The Neo4j Graph database was lacking a declarative query language. We wanted to add a humane query language which is easy to read and understand. It borrows on other languages like SQL and SPARQL but brings it it's own flavor. Cypher uses ASCII ART to describe graph patterns that you're looking for. We used Scala's parser combinator library in combination with functional approaches and lazy evaluation to develop the Cypher query language. The talk describes the internals of the Cypher implementation.

2. Geekout
Tallin
Estonia
Got a Graph Database?
Need a Query
Language!
(Neo4j) -[:LOVES]-> (Cypher)

3. (Michael) -[:WORKS_ON]-> (Neo4j)
console
Cypher
community
graph
Community
ME
Server
Spring Cloud

4. YOU
?
SQL
NOSQL
Graph Database

5. Graphs are
everywhere

7. (Neo4j) -[:IS_A]-> (Graph Database)
Lucene
Sharding 1 M/s
Master/
Index
Slave
LS
TRAVERSA
HIG S
H_A TE
VA RA
IL.
TEG
IN
PROVIDES
Server ACID TX
RUN LI
S_A CE
S
NS
ED
O _L
Ruby IK
ES_T
RU
JS E
MySQL
S
NS
_A
_O
SC AL
Clojure
NS
.net
N
RU
Mongo
embedded 34bn Heroku
Java Nodes

8. Homework
- so Pay Attention
• Go To http://bit.ly/geekout-cypher
• With your Cypher knowledge
• Add yourself to the graph
• Determine a path from you to me
• Share the console on Twitter by
Monday
• Win this AR-Drone

9. Query a Graph
with
___ _____ _____ _____ _____ ___ ______ _____
/ _ / ___/ __ _ _|_ _| / _ | ___ _ _|
/ /_ `--.| / / | | | | / /_ |_/ / | |
| _ |`--. | | | | | | _ | / | |
| | | /__/ / __/_| |_ _| |_ | | | | | | |
_| |_|____/ ____/___/ ___/ _| |_|_| _| _/
Whirlwind Tour

10. A Graph
Can‘t see the Patterns for the Trees
http://maxdemarzi.com/2012/02/13/visualizing-a-network-with-cypher/

11. Patterns?
Where?

12. Patterns?
Describe them!

13. Patterns?
Find them on bound nodes

14. Patterns?
There are more !

15. Patterns?
Make them visible!

16. How ?
A Graph Query Language
with
Pattern Matching

17. SparQL, SQL ?
No,
Cypher
developed by and for
Neo4j

18. Can I draw Patterns?
A
B C

19. Sure, with ASCII ART!
A
B C

20. Sure, with ASCII ART!
A
B C
A --> B --> C, A --> C

21. Sure, with ASCII ART!
A
B C
A --> B --> C, A --> C
A --> B --> C <-- A

22. Example
Directed Relationship
A B

23. Example
Directed Relationship
A B
(A) --> (B)

24. Example
Labeled Directed Relationship
LOVES
A B

25. Example
Labeled Directed Relationship
LOVES
A B
(A) -[:LOVES]-> (B)

26. Example
Transitive Relationship Path
A B C

27. Example
Transitive Relationship Path
A B C
(A)-->(B)-->(C)

28. Example
Variable Length Path
A B
A B
A B
...

29. Example
Variable Length Path
A B
A B
A B
...
(A) -[*]-> (B)

30. Real World
Examples

31. Give me: JOINS
SELECT skills.*, user_skill.*
FROM users
JOIN user_skill ON users.id = user_skill.user_id
JOIN skills ON user_skill.skill_id = skill.id
WHERE users.id = 1
START user = node(1)
MATCH user -[r:USER_SKILL]-> skill
RETURN skill, r

32. Give me:
Old, Influential Friends
START me = node(...)
MATCH (me) - [f:FRIEND] - (old_friend)
- [:FRIEND ] - (fof)
WHERE ({today}-f.begin) > 365*10
WITH old_friend, collect(fof.name) as names
WHERE length(names) > 100
RETURN old_friend, names
ORDER BY old_friend.name ASC
f:FRIEND :FRIEND
me friend fof

33. Give me:
Simple Recommendation
START me = node(...)
MATCH (me) -[r1:RATED ]->(thing)
<-[r2:RATED ]- (someone)
-[r3:RATED ]->(cool_thing)
WHERE ABS(r1.stars-r2.stars) <= 2
AND r3.stars > 3
RETURN cool_thing, count(*) AS cnt
ORDER BY cnt DESC LIMIT 10
r1:RATED thing r2:RATED
me so
r3:RATED
cool
thing

34. Results ?
• Tables:
for Human Brainz & Tools
• Graphs:
to highlight, visualize,
export & refining queries

35. One Step Back
• Goals / Intent
• Origins
• Decisions
• Implementation
• Future

36. What is Cypher?
• Graph Query Language for Neo4j
• Querying for Humans

37. Goals, Origins & Design
Picking good ideas and having some of our own

38. Cypher: Some Goals
ASCII-art
Pattern
patterns
Matching
Declarative
External
DSL
Closures
SQL
Familiarity

39. Something new?
• Existing Neo4j query mechanisms were not
simple enough
• Too verbose (Java API)
• Too prescriptive (Gremlin)
• Other query languages

40. Java API?
• Object oriented
• Node, Relationship, Path objects
• Imperative
• Verbose
• Traversers, for-loops, Index
• mostly lazy-eval

41. Gremlin?
• DSL for pipes
• imperative
• a single path expression
• loop constructs
• side-effects
• lots of closures

42. SQL?
• Well known and understood
• Unable to express paths
• these are crucial for graph-based
reasoning
• Cumbersome Mutation
• Neo4j is schema/table free

43. SPARQL?
• SPARQL designed for a different data model
• namespaces / URI‘s
• reified properties as nodes
• SPARQL/RDF mostly in academia
• developers don‘t get it
• Pattern matching is cool

44. Cypher: Some Goals
ASCII-art
Pattern
patterns
Matching
Declarative
External
DSL
Closures
SQL
Familiarity

45. Cypher: Some Goals
ASCII-art
Pattern
patterns
Matching
Declarative
External
DSL
Closures
SQL
Familiarity

46. Design Decisions
Closures / Quantifiers
START london = node(1), moscow = node(2)
MATCH path = london -[*]-> moscow
WHERE all(city in nodes(path) where city.capital)
RETURN path

47. Cypher: Some Goals
ASCII-art
Pattern
patterns
Matching
Declarative
External
DSL
Closures
SQL
Familiarity

48. Design Decisions
Parsed, not an internal DSL
Execution Semantics Serialisation
Type System Portability

49. Cypher: Some Goals
ASCII-art
Pattern
patterns
Matching
Declarative
External
DSL
Closures
SQL
Familiarity

50. Design Decisions
Familiar for SQL users
select start
from match
where where
group by return
order by order by
skip
limit

51. Design Decisions
Database vs Application
Design Goal: single user
interaction expressible as
single query
Queries have enough logic to
find required data, not
enough to process it

52. Implementation
Docs from
Execution Tests
Java Bridge
Plan
Pattern
Matching
Parsing
Scala

53. CODE

54. Scala
• Parser Combinator
• Query Object Creation
• Combining Pipes
• Pattern Matching
• Lazy Evaluation

55. Example: Pipes
START n=node(0)
MATCH n-[*]->m
RETURN n, count(*)
ORDER BY n.name
Resulting Pipes:
Parameters()
Nodes(n) // Identifier n
PatternMatch(n-[*]->m) // Identifier m
Extract([n,n.name])
Sort(n.name ASC)
EagerAggregation( keys: [n], aggregates: [count(*)])
ColumnFilter([n.name,n,count(*)])

56. Why Scala?
• Fun Language
• Parser Combinators
• Functional Concepts
• Collection library
• Immutable Values
• Scala Pattern Matching

57. Problems with Scala?
• Scala versions
• Slow compilation
• Low Collection Write Throughput
• Code gets easily complicated
• Hard to ramp up for other devs
• Big separate library

58. Implementation
Query Parts
• START
• MATCH
• Pattern Matching
• WHERE
• Expressions, Predicates
• RETURN

59. START
WHERE
Aggregation
BINDS
FILTERS
DELETE RESTRUCTURES
Identifier
REMOVE_FROM Result
BOUND_TO USED_IN PAGINATES
BUILDS_UP
SKIP/LIMIT
Graph FOUND_IN Pattern
DESCRIBES
ADD_TO CREATES
MATCH
FIX COMPLETES
CREATE
RELATE

60. START
START dev=node:user(name=“Andres“)
• lazy source of identifiers bound to nodes
and relationships
• each identifier is an iterable
• spawns execution per value
• cross product between multiple
• index lookup or direct lookup

61. MATCH
MATCH dev-[:WORKED_ON]->project
• describe patterns with ASCII art
• declare identifiers
• paths
• var. length paths
• graph algorithms
• optional relationships
• expands results with each found subgraph

62. Core Acitvity:
Pattern Matching
• Match clause with ASCII-art
• derive pattern description
• bound Nodes and Relationships
• finds patterns attached to bound entities
• each found Pattern spawns Subgraph Result
• recursive PM with Backtracking

63. Pattern Matching
• Scala
• incremental search with backtracking
• allows:
• Variable length paths
• Filter during matching
• optional relationships
• powerful but slower

64. Pattern Matching
• Java
• existing graph matching libray
• fast but less capable
• integrated with lazy Scala API
• NEW Traversal Framework approach
• for one or two bound nodes
• use Neo4j traversal framework
• pattern description & complexity
determine Pattern Matcher selection

65. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

66. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

67. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

68. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

69. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

70. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

71. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

72. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

73. Implementation
• Recursive matching with backtracking
START x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

74. WHERE
WHERE project.name = „Cypher“
• filters results
• single big boolean expression
• needs existing identifiers to work with
• much like SQL

75. Expressions
• expressions compute values
• composite Specification Pattern
• input is ExecutionContext Map
• have name
• declare symbolic dependencies
• self & composite
• directly derived from parser
• probably rewritten in between

76. Predicates
• WHERE clause filters results with a single
composed predicate
• boolean expressions
• composable boolean algebra
• Patterns as predicates
• Quantifiers (ALL, NONE, SINGLE)
• Collections as predicates

77. RETURN
RETURN project, collect(idea),count(fun)
• determines what to return
• column names or aliases with AS
• automatic aggregation
• when aggregation function exists
• all non-aggregated values are grouping
• lazyness is killed with aggregation

78. SKIP LIMIT ORDER BY
SKIP 5 LIMIT 10
ORDER by length(ideas) DESC
• the usual suspects
• lazyness is killed with ordering

79. Mutation
Transaction
WITH Idem-
(Separation) potence
DRY
Mass Data
Handling
Lazyness

80. Mutation
• need transactions
• must not disturb reads / traversals
• separation of read and write query parts
• explicit context change and scope: WITH
• implicit split for simple queries
• granularity of mutation aligned with # of
executions
• work with parameters

81. Mutation - Impl.
• Query Builder create UpdateCommands
• UpdatePipe ??
• Transaction Scope for whole Query
• collect statistics
• create new entities
• add identifiers to the ExecutionContext
• need to iterate through to get all updates
executed (no lazyness), special Pipe
• have to track already deleted entities

82. Implementation
Mutating Query Parts
• WITH
• CREATE
• RELATE
• SET, DELETE
• FOREACH

83. START
WHERE
Aggregation
BINDS
FILTERS
DELETE RESTRUCTURES
Identifier
REMOVE_FROM Result
BOUND_TO USED_IN PAGINATES
BUILDS_UP
SKIP/LIMIT
Graph FOUND_IN Pattern
DESCRIBES
ADD_TO CREATES
MATCH
FIX COMPLETES
CREATE
RELATE

84. WITH
WITH me, count(friend) as friends
• syntax like RETURN
• separate query parts like a pipe
• declares a new scope with new identifiers
• all other ids will be gone
• useful for HAVING
• spawns a new ExecutionContext
• continue with read or write part

85. CREATE
CREATE me = {name: „Michael“}
• create new nodes or relationships
• can work with map params (or Iterables
thereof)
• assign new identifiers
• can create full paths

86. RELATE
RELATE posts-[:POST]->(p {title: „..“}
• FIX the graph
• construct missing relationships and nodes
• need at least one bound node
• match given properties
• try to advance (from multiple sides) and
create missing stuff then iterate

87. DELETE
DELETE n, rel, m.prop
• as expected
• idempotent deletion
• can only delete unconnected nodes
• so delete relationships first
START n=node(*)
MATCH n-[r?]-()
DELETE n,r

88. SET
SET n.name = „Father of “+m.name
• can work with arbitrary expressions
• use coalesce for idempotent defaults
• can mass assign with map-parameters

89. FOREACH
FOREACH ( f in new_friends :
RELATE me-[:FRIEND]->f)
• iterable loop for mutating operations
• saves a lot of repetetive code
• wraps current execution context in a
temporary proxy

90. Self Documenting Tests
• Provides
• Title & Description
• Define Sample Graph
• Declare & Execute Query
• Results

91. Self Documenting Tests
• Asserts
• No Syntax Errors
• Multiple Cypher Versions
• No Execution Errors
• Results
• Resulting graph

92. Self Documenting Tests
• Generates
• Graph Rendering Graph-Viz
• Tabular Results
• Ascii-doc for Documentation
• Live-Console Integration

93. Self Documenting Tests
• Good enough for
• Fun to write
• Integration Tests
• Manual
• Blog-ready Cookbook Examples

94. Self Documenting
Tests

95. Cypher Console(s)
• Integrated in Neo4j-Shell
• Integrated in Webadmin (Shell & Search)
• REPL & GIST - console.neo4j.org
• Interactive learning GCLI
• Scala spray-can & cypher = bansky
• py2neo, bash
• Google Document

96. Live Console
• In Memory GDBs in Web Session
• Set up with mutating Cypher (or Geoff)
• Executes Cypher (also mutating)
• Visualizes Graph & Query Results (d3)
• Multiple Cypher Versions
• Share: short link, tweet, yUML
• Embeddable <iframe>
• Live Console for Docs

97. The Rabbithole
http://console.neo4j.org
This Graph: http://tinyurl.com/7cnvmlq

98. Homework
- Apply your Wits
• Go To http://bit.ly/geekout-cypher
• With your new Cypher knowledge
• Add yourself to the graph
• Determine a path from you to me
• Share the console on Twitter by
Monday
• Win this AR-Drone

99. Graph your World
• Neo4j.org
• Google Group
• Grab your own ego-Graph

100. Thanks for Listening!
Questions?
michael.hunger@neotechnology.com
@mesirii