Neo4j GraphTalks Oslo Kees Vegter, Neo4j

1. Next Generation Solutions built on Neo4j
Kees Vegter, Neo4j
Graphtalk Oslo, May 9 2018

2. OSLO, MAY 9, 2018!
Agenda!
● Solutions using Neo4j!
● Recommendations!
● GDPR!
● Conclusions!

3. OSLO, MAY 9, 2018!
Solutions: new mindset required?!

4. OSLO, MAY 9, 2018!
Solutions: new mindset!
Yesterday:!
- Static Applications!
- Designed to fulfill current
requirements!
- Performance Constraints!
- Domain experts versus IT
experts!
Tomorrow:!
- Flexible Applications!
- Designed to fulfill tomorrows
requirements!
- Performance is not limiting !
- Domain experts work hand in
hand with IT experts!

5. OSLO, MAY 9, 2018!
Look at this data…

6. OSLO, MAY 9, 2018!
Swap glasses…

7. OSLO, MAY 9, 2018!
… now look at it again, this time as a graph

8. Speed: Real time query
enabled
Graph Based Solutions!
Enables Up-Sell / Cross-sell
Key Features Added Value
360 degree view on data
Using data Connections as a value
Intuitive: Supports Business Needs
Flexible: enabled for
additional requirements
Finding patterns within the data
Detect anomalies
Prevent rather than detect
Enables conversation across Functions
Comply to regulations
What-if Analysis
Telco
OSS
GDPR
Fraud
Telco BSS
Recomm
endations
MDM
Resource efficient

9. OSLO, MAY 9, 2018!
Evolution using Neo4j!
!
Neo4j Platform!
!
Graph Transactions Graph Analytics
Data Integration
Development &
Admin Analytics Tooling
Drivers & APIs Discovery & Visualization
Developers
Admins
Applications Business Users
Data Analysts
Data Scientists
!
3rd Party Tools!
!
!
“The Graph Advantage”!
!
!
Domain know-how!
!
!
Professional Services!
!
!
PS Packages!
!
!
Graph Based Solution!
!

10. OSLO, MAY 9, 2018!
Evolution using Neo4j!
!
Neo4j Platform!
! Graph Transactions Graph Analytics
Data Integration
Development & Admin Analytics Tooling
Drivers & APIs Discovery & Visualization
Developers
Admins
Applications Business Users
Data Analysts
Data Scientists
!
3rd Party Tools!
!
!
“The Graph Advantage”!
!
!
Domain know-how!
!
!
Professional Services!
!
!
PS Packages!
!
!
Graph Based Solution!
!
Neo4j enables Graph Based
Solutions with a need for:!
- Agility!
- Intuitiveness !
- High Performance to support
connected data scenarios!
- Scalable on traversing through
connected data!

11. OSLO, MAY 9, 2018!
Recommendation Engines
Building Powerful Recommendation Engines With Neo4j!

12. OSLO, MAY 9, 2018!
“If you liked this, you might like that…”
Powerful, real-time, recommendations and
personalization engines have become
fundamental for creating superior user
experience and commercial success in retail

13. OSLO, MAY 9, 2018!
Creating Relevance in an
Ocean of Possibilities

14. OSLO, MAY 9, 2018!
How Graph Based Recommendations Transformed
the Consumer Web!
People Graph
“People you may know”
Disruptor: Facebook!
Industry: Media Ad-business!
Disruptor: Amazon!
Industry: Retail!
People & Products
“Other people also bought”
People & Content
“You might also like”
Disruptor: Netflix!
Industry: Broadcasting Media!

15. OSLO, MAY 9, 2018!
Today Recommendation Engines are At the
Core of Digitization in Retail!
Product
Recommendations
Effective product recommendation
algorithms has become the new
standard in online retail — directly
affecting revenue streams and the
shopping experience.!
Logistics/Delivery
Routing recommendations allows
companies to save money on routing
and delivery, and provide better and
faster service.!
Promotion
recommendations
Building powerful personalized
promotion engines is another area
within retail that requires input from
multiple data sources, and real-time,
session based queries, which is an
ideal task to solve with Neo4j.!

16. OSLO, MAY 9, 2018!
... and Recommendation Engines are at
the core of:
Content
Recommendations
Content recommendation algorithms
are the basis to use portals providing
value added content — directly
affecting the behaviour of the users
and have them stay on the web page!
Fraud
Taking timely action based on
patterns / recommendations you find
inside connected data . May require
input from multiple data sources,
and real-time, session based
queries, which is an ideal task to
solve with Neo4j.!
!
Social Networks
Building powerful personalized
engines to recommend new contacts,
friends, based on patterns,
preferences, status „friends-of-friends“
taking advantage of the value of
connected data!

17. OSLO, MAY 9, 2018!
Why Graph Based
Recommendation Engines?!
• Increase revenue!
• Create Higher Engagement!
• Mitigate Risk!Value
• Real-Time capabilities!
• Ability to use the most recent transaction data!
• Flexibility to incorporate new data sources!Performance

18. OSLO, MAY 9, 2018!
The Impact of Bad Recommendations!
Characteristic! Impact for Recommendations! Examples!
• “Batch Oriented
Recommendation”!
• Unable to react on real-time changes!
• Unable to fulfill real-time needs!
• Recommending “out-of-stock” products!
• Content recommendation, eg news: the latest
news are the most important ones!
• Lack of
Performance!
• Recommendation slow down the user
interaction!
• Recommendation alternatives limited!
• Delayed response time lead to customer
dissatisfaction!
• Recommend just the obvious (“similarities”) and
inability to recommend more complex scenarios
(Account specific and product specific and
buying history and …)!
• Limited by Data
Connections!
• Recommendations are limited by
number of hops!
• Inability to recommend more complex correlations
(eg product hierarchies and dependencies)!
• No complex recommendation algorithms
supported!
• Missing Feedback
Loop!
• Inability to react on Feedback! • Customer never picks Top 3 recommendations!
• Recommendations are getting meaningless!
• No Graph
Algorithm support!
• Limitations on Machine Learning
approaches!
• “Centrality” for Products to be recommend can be
essential!

19. OSLO, MAY 9, 2018!
Case Studies

20. OSLO, MAY 9, 2018!
Case study!Solving real-time recommendations for
the World’s largest retailer.!
Challenge
• In its drive to provide the best web experience for
its customers, Walmart wanted to optimize its
online recommendations.!
• Walmart recognized the challenge it faced in
delivering recommendations with traditional
relational database technology.!
• Walmart uses Neo4j to quickly query customers’
past purchases, as well as instantly capture any
new interests shown in the customers’ current
online visit – essential for making real-time
recommendations.!
Use of Neo4j
“As the current market leader in
graph databases, and with
enterprise features for scalability
and availability, Neo4j is the
right choice to meet our
demands”.
- Marcos Vada, Walmart
• With Neo4j, Walmart could substitute a heavy
batch process with a simple and real-time graph
database.!
Result/Outcome

21. OSLO, MAY 9, 2018!
Case study!eBay Tackles eCommerce Delivery Service Routing
with Neo4j!
Challenge
• The queries used to select the best courier for
eBays routing system were simply taking too long
and they needed a solution to maintain a
competitive service.!
• The MySQL joins being used created a code base
too slow and complex to maintain.!
• eBay is now using Neo4j’s graph database
platform to redefine e-commerce, by making
delivery of online and mobile orders quick and
convenient.!
Use of Neo4j
• With Neo4j eBay managed to eliminate the
biggest roadblock between retailers and online
shoppers: the option to have your item delivered
the same day.!
• The schema-flexible nature of the database
allowed easy extensibility, speeding up
development.!
• Neo4j solution was more than 1000x faster than
the prior MySQL Soltution.!
Our Neo4j solution is literally
thousands of times faster than the
prior MySQL solution, with queries
that require 10-100 times less code.
Result/Outcome
– Volker Pacher, eBay

22. OSLO, MAY 9, 2018!
Example Recommendation
Solution Architecture

23. Neo4j Database Cluster!
Neo4j APOC!
Recommen!
dation!
Algorithms!
(Scheduled)!
Management
Dashboard!
Neo4j Bolt Driver!
Data Ingest!
Mgmt.!
…!
Customer Data Sources / Systems / Applications!
Legend:!
Neo4j Provided Components!
Custom built Neo4j/Customer!
Customer/SI!
Batch!
Data Buffering
(Queue)!
Real-Time!
Admin UI!
System Specific Adapters / Scripts / Connecters!
Admin / Superuser!
Apps! Websites!
Affiliate!
Programs!
Points of sale!
User Interface!
Retail Web Shop functionality / !
Shipment /!
etc.!

24. OSLO, MAY 9, 2018!
Why Graph is Superior for Recommendation Engines!
Recommendation Requirement Traditional Approaches Neo4j Approach
Usage of connected data over unlimited
amount of „hops“
Complex queries with hundreds of join
tables
Simple single query traverses all
enterprise systems
Real-time 360 degree view on data
within your System
Performance limitations with increasing
number of connections / hops
Traversing over connections in near
real-time provided
Effort required to add additional data
sources to support reco
Days to weeks to rewrite schema and
queries
Minutes to draw new data connections
Time to deployment Months to years Weeks to months
Response time to Recommendations Minutes to hours per query Milliseconds per query
Machine Learning Enablement Static Database scheme leads to static
processes
ML algorithms can use Graph algorithms and
take advantage of connected data
Bottom line Long, ineffective and expensive Easy, fast and affordable

25. OSLO, MAY 9, 2018!
Why Graph is Superior for Recommendation Engines!
Recommendation Requirement Traditional Approaches Neo4j Approach
Usage of connected data over unlimited
amount of „hops“
Complex queries with hundreds of join
tables
Simple single query traverses all
enterprise systems
Real-time 360 degree view on data
within your System
Performance limitations with increasing
number of connections / hops
Traversing over connections in near
real-time provided
Effort required to add additional data
sources to support reco
Days to weeks to rewrite schema and
queries
Minutes to draw new data connections
Time to deployment Months to years Weeks to months
Response time to Recommendations Minutes to hours per query Milliseconds per query
Machine Learning Enablement Static Database scheme leads to static
processes
ML algorithms can use Graph algorithms and
take advantage of connected data
Bottom line Long, ineffective and expensive Easy, fast and affordable
A Fortune 500 customer brought in Neo4j to improve content
recommendations quality... and will decommission 48 ‘wide column
store’ servers (half a million USD in list EC2 hosting costs) in favor
of a *3-machine* Neo4j cluster which handles the same load.

26. OSLO, MAY 9, 2018!
How Neo4j Differentiates from other Databases!
Visualization
Queries
Processing
Storage
Non-Native Graph DB
Native Graph DB
RDBMS
Optimized for graph workloads

27. OSLO, MAY 9, 2018!
What about Machine Learning?!
Neo4j is an enabler technology:!
• Detecting Recommendation patterns via Cypher
queries!
• Recommendation Algorithms based on scores!
• Feedback loop !
• Learn from feedback (eg never used “friends
recommendation”) and change scoring!
• Algorithm to automatically add relevant connections!
• ….!

28. OSLO, MAY 9, 2018!
Neo4j powered Recommendation
Engine!Characteristic! Benefit for Recommendation Solution!
• Agility! • Constant learning of recommendations given feedback enabled!
• Enabled for Future Requirements!
• Solution can be built iteratively!
• Fast implementation cycles!
• Schema free DB supports “connect anything”!
• Intuitiveness! • Enable Business Analysts to use technology!
• All channels and data sources can be easily connected!
• Speed! • Unlimited number of traversals to detect potential recommendations !
• Response time enables fraud prevention!
• Leverage Data Connections! • 360 degree customer view enabled / provided!
• Scalability! • Hardware efficiency with real-time patterns!
• TCO/ROI! • Adding on top of existing infrastructure protects investments!

29. GDPR Compliance

30. OSLO, MAY 9, 2018!
GDPR Summary!
• GDPR = General Data Protection Regulation!
• Adopted by the EU Parliament on 24th May 2016!
• Will apply from 25th May 2018!
• Applies to both Controllers and Processors!
• Applies to organisations operating within the EU, as well as organisations outside
the EU that offer goods or services to individuals in the EU.!
• Covers a broad definition of personal data!
• Defines lawful basis for processing personal data, which include consent and
contract!
• Defines significant fines for non-compliance!

31. OSLO, MAY 9, 2018!
Individual Rights Under GDPR!
Right to be
informed!
Right of
access!
Right to
rectification!
Right to
erasure!
Right to
restriction of
processing!
Right to data
portability!
Right to object!
Rights
regarding
automated
decision
making!

32. OSLO, MAY 9, 2018!
Key GDPR Requirements !
Organizations that embrace the new GDPR regulations and provide the right levels of transparency and traceability
for personal information have a big opportunity to win the hearts, minds and business of consumers.
What data do you
have? Is it accurate?
Where is the data
stored?
How and when did
you obtain the data?
Why do you have the
data?
Who has access to the
data?
Do you have
permission to use the
data? For what
purpose?
Is the data secure?
How does the data
travel through your
systems?
Does the data ever
cross international
borders?

33. OSLO, MAY 9, 2018!
GDPR: Risk Mitigation vs. Competitive Advantage!
Be a leader and have a solution
ready on time
Improve
Brand!
Reduce Risk
Leverage connected data to drive
analytics for threat detection &
business forecasts
Competitive
Advantage
Spend is strategic
Increase ROI!
Reduce Risk
Become a trusted enterprise,
delight customers and DPA
Increase
CSAT!
Become
Trusted!
Improve
Brand
Strategic solution ensures data
governance and solution
maintenance
Reduce Risk!
Reduce Cost
Stay on the sidelines to see what
others are doing
Increased
Risk
Look to get by with bare minimum
solution
Increased
Risk
Spend is sunk investment to just
mitigate risk
Low to No
ROI!
Unknown
Risk
Mitigation
Solution results in less than happy
subjects, DPO and DPA
Lower CSAT!
Minimal Risk
Reduction
Focus on data governance and
solution maintenance is low
Increased
Risk!
Increased
Cost

34. OSLO, MAY 9, 2018!
Why Graphs?!

35. OSLO, MAY 9, 2018!
GDPR needs: Connected Data & Visualization!
Graph database is the perfect solution to this vast amount of connected data; traditional
approaches with an RDBMS or other NoSQL databases just cannot cut it

36. OSLO, MAY 9, 2018!
Graph Database is the Right GDPR Foundation !
Neo4j includes powerful visualization tools that enable you to model and
track the movement of sensitive data through your systems

37. OSLO, MAY 9, 2018!
Data Modeling and Definition
1
Data Transformation2
Consent Management
3
Entitlement 4
GRAPHS IN
METADATA
MANAGEMENT

38. OSLO, MAY 9, 2018!
#1 Data Modeling and Definition

39. OSLO, MAY 9, 2018!
Party
CUST_SCHEMA
Party
First
Name
HAS_LOGICAL_ATT
CUST_SCHEMA
_PARTY.FIRST
_NM
CUST_SCHE
MA.PARTY
SHEMA_HAS_TABLE
Party
Last
Name
HAS_LOGICAL_ATT
SHEMA_HAS_TABLE
GENERATES
CONCEPT_FOR
CONCEPT_FOR
CONCEPT_FOR
HAS_NAME
CUSTOMER
NAMECUSTOMER
HAS_PHONE
TABLE_HAS_COL
CUST_SCHEMA
_PARTY.LAST_
NM
TABLE_HAS_COL
GENERATES
Enterprise!
Ontology!
Application
Logical
Model!
Physical !
Schema!
CUST_SCHE
MA.ROLE

40. OSLO, MAY 9, 2018!
#2 Data Transformation

41. OSLO, MAY 9, 2018!
ETL_PR
OC_1
SALES_SCH
EMA
Normali
ze_Date
SLS_SCHE
MA.PROD
UCT
SLS_SCHEM
A.SALES.DA
TE
SLS_SCH
EMA.SAL
ES
#2 Data Transformation
Channel_
Normaliz
ation
SLS_SCHEM
A.SALES.CH
ANNEL
BusinessView!
Integration Middleware!Operational Systems!
HAS_INPUT
HAS_INPUT
Time.time_
key
Time.day_
of_week
Enterprise DWH!
HAS_OUTPUT
HAS_OUTPUT
HAS_COL
HAS_COL
TechnicalView!
Billing
System
EDWH
CDE:
Transact
ion_Dat
e
SENDS RECEIVES
CONCEPTUAL_ELEMENT_FOR
Star_Sche
ma
Star_Sc
hema.Ti
me

42. OSLO, MAY 9, 2018!
#3 Consent management

43. OSLO, MAY 9, 2018!
#3 Consent Management + MDM
Amsterdam"
NL
K.Vegter
+316239004
…
kees@neo4j
.com
EMAIL_FOR
ADDRESS_FOR
kees@gmai
l.com
EMAIL_FOR
{ contrib: ‘XYZ’,
permittedFor: [UC1,UC4],
consentUntil : 31-12-19 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
EMAIL_FOR
{ contrib: ‘LMN’,
permittedFor: [UC2,UC6],
consentUntil : 31-12-20 }

44. OSLO, MAY 9, 2018!
#4 Entitlement

45. OSLO, MAY 9, 2018!
#4 Entitlement
User 1
User 3
User 2
Exclusio
n List G1
Resource
1
Group 1
Resource
2
Group 3
D
CRUD
MEMBER
MEMBER

46. OSLO, MAY 9, 2018!
#4 Entitlement
#3 Consent
Management
#2 Data
Transformation
#1 Data Modelling
and Definition
Graphs in Metadata Management and Data Governance!
# …

47. OSLO, MAY 9, 2018!
Why Graph is Superior for GDPR !
GDPR Task Traditional Approaches Modern Neo4j Approach
Trace data through enterprise systems Complex queries with hundreds of join
tables
Simple single query traverses all
enterprise systems
Preserve the integrity of data lineage Broken data paths and lineage,
especially with NoSQL databases
Continuous, unbroken data paths at all
times
Effort required to add new data and
systems
Days to weeks to rewrite schema and
queries
Minutes to draw new data connections
Time to deployment Months to years Weeks to months
Response time to GDPR requests Minutes to hours per query Milliseconds per query
Form of GDPR responses Text reports that are not visual and
prove very little
Visuals of personal data and the path it
follows through your systems
Bottom line Long, ineffective and expensive Easy, fast and affordable

48. OSLO, MAY 9, 2018!
Dashboards & Visual Reports!

49. Personal Data Map
Role Based Dashboards - Subject View!

50. Personal Data Map!

51. Role Based Dashboards - Management View!

52. Consents per Subject

53. Data Lineage Report for ‘John Doe’!

54. John
Doe

55. Example Architecture!

56. OSLO, MAY 9, 2018!
Graph Database is the Right GDPR Foundation !
Marketing! CRM!
Customer!
Service!
Online!
Store!
Logistics!
Extract GDPR Events/Data!
Financials!

57. OSLO, MAY 9, 2018!
Graph Database is the Right GDPR Foundation !
Extract GDPR Events/Data!
Marketing! CRM!
Customer!
Service!
Online!
Store!
Logistics! Financials!

58. OSLO, MAY 9, 2018!
Conclusion!
(graphs)-[:ARE]-> (everywhere)
and!
(Solutions)-[:NEED]-> (graphs)

59. OSLO, MAY 9, 2018!
Who can help?!
!
Neo4j Platform!
! Graph Transactions Graph Analytics
Data Integration
Development & Admin Analytics Tooling
Drivers & APIs Discovery & Visualization
Developers
Admins
Applications Business Users
Data Analysts
Data Scientists
!
3rd Party Tools!
!
!
“The Graph Advantage”!
!
!
Domain know-how!
!
!
Professional Services!
!
!
PS Packages!
!
!
Graph Based Solution!
!
Professional Services:!
- Extend and leverage Domain Expertise!
- Best Practices!
- Using Building Blocks!
- Don’t “re-invent the wheel”!
- Speed up development and deployment!
- Access to Neo4j infrastructure
(Development, Support, Product
management)!

60. Q & A

61. Extra: Demo