Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Next Generation Solutions built on Neo4j
Kees Vegter, Pre-Sales Engineer
Graphtalk Amsterdam, Oct 18 2018

AMSTERDAM, OCT 18, 2018
Agenda
● Solutions using Neo4j
● Recommendations
● AI ML
● GDPR
● Conclusions

Solutions: new mindset required?

Solutions: new mindset
Yesterday:
- Static Applications
- Designed to fulfill current
requirements
- Performance Constraints
- Domain experts versus IT
experts
Tomorrow:
- Flexible Applications
- Designed to fulfill tomorrows
requirements
- Performance is not limiting
- Domain experts work hand in
hand with IT experts

New Mindset
Store Data in a Different Way

Look at this data…

Swap glasses…

… now look at it again, this time as a graph

Node
Relationship

Speed: Real time query
enabled
Graph Based Solutions
Enables Up-Sell / Cross-sell
Key Features Added Value
360 degree view on data
Using data Connections as a value
Intuitive: Supports Business Needs
Flexible: enabled for
additional requirements
Finding patterns within the data
Detect anomalies
Prevent rather than detect
Enables conversation across Functions
Comply to regulations
What-if Analysis
Telco
OSS
GDPR
Fraud
Telco BSS
Recomm
endations
MDM
Resource efficient

Evolution using Neo4j
Neo4j Platform
Graph Transactions Graph Analytics
Data Integration
Development &
Admin
Analytics Tooling
Drivers & APIs Discovery & Visualization
Developers
Admins
Applications Business Users
Data Analysts
Data Scientists
3rd Party Tools
“The Graph Advantage”
Domain know-how Professional Services PS Packages
Graph Based Solution

Evolution using Neo4j
Neo4j Platform
Data Integration
Development &
Admin
Analytics Tooling
Developers
Admins
Data Analysts
Data Scientists
3rd Party Tools
Neo4j enables Graph Based
Solutions with a need for:
- Agility
- Intuitiveness
- High Performance to support
connected data scenarios
- Scalable on traversing through
connected data

OSLO, MAY 9, 2018
Recommendation
EnginesBuilding Powerful Recommendation Engines With Neo4j

“If you liked this, you might like that…”
Powerful, real-time, recommendations and
personalization engines have become
fundamental for creating superior user
experience and commercial success in retail

Creating Relevance in
an Ocean of
Possibilities

How Graph Based Recommendations
Transformed the Consumer Web
People Graph
“People you may know”
Disruptor: Facebook
Industry: Media Ad-business
Disruptor: Amazon
Industry: Retail
People & Products
“Other people also bought”
People & Content
“You might also like”
Disruptor: Netflix
Industry: Broadcasting Media

Today Recommendation Engines are At
the Core of Digitization in Retail
Product
Recommendations
Effective product recommendation
algorithms has become the new
standard in online retail — directly
affecting revenue streams and the
shopping experience.
Logistics/Delivery
Routing recommendations allows
companies to save money on routing
and delivery, and provide better and
faster service.
Promotion
recommendations
Building powerful personalized
promotion engines is another area
within retail that requires input from
multiple data sources, and real-time,
session based queries, which is an
ideal task to solve with Neo4j.

... and Recommendation Engines are at
the core of:
Content
Recommendations
Content recommendation algorithms
are the basis to use portals providing
value added content — directly
affecting the behaviour of the users
and have them stay on the web page
Fraud
Taking timely action based on
patterns / recommendations you find
inside connected data . May require
input from multiple data sources, and
real-time, session based queries,
which is an ideal task to solve with
Neo4j.
Social Networks
Building powerful personalized engines
to recommend new contacts, friends,
based on patterns, preferences, status
„friends-of-friends“ taking advantage of
the value of connected data

Why Graph Based
Recommendation Engines?
• Increase revenue
• Create Higher Engagement
• Mitigate RiskValue
• Real-Time capabilities
• Ability to use the most recent transaction data
• Flexibility to incorporate new data sourcesPerformance

The Impact of Bad Recommendations
Characteristic Impact for Recommendations Examples
• “Batch Oriented
Recommendation”
• Unable to react on real-time changes
• Unable to fulfill real-time needs
• Recommending “out-of-stock” products
• Content recommendation, eg news: the latest
news are the most important ones
• Lack of
Performance
• Recommendation slow down the user
interaction
• Recommendation alternatives limited
• Delayed response time lead to customer
dissatisfaction
• Recommend just the obvious (“similarities”) and
inability to recommend more complex scenarios
(Account specific and product specific and buying
history and …)
• Limited by Data
Connections
• Recommendations are limited by
number of hops
• Inability to recommend more complex correlations
(eg product hierarchies and dependencies)
• No complex recommendation algorithms supported
• Missing Feedback
Loop
• Inability to react on Feedback • Customer never picks Top 3 recommendations
• Recommendations are getting meaningless
• No Graph
Algorithm support
• Limitations on Machine Learning
approaches
• “Centrality” for Products to be recommend can be
essential

Case Studies

Case studySolving real-time recommendations for
the World’s largest retailer.
Challenge
• In its drive to provide the best web experience for
its customers, Walmart wanted to optimize its online
recommendations.
• Walmart recognized the challenge it faced in
delivering recommendations with traditional
relational database technology.
• Walmart uses Neo4j to quickly query customers’
past purchases, as well as instantly capture any
new interests shown in the customers’ current
online visit – essential for making real-time
recommendations.
Use of Neo4j
“As the current market leader in
graph databases, and with
enterprise features for scalability
and availability, Neo4j is the
right choice to meet our
demands”.
- Marcos Vada, Walmart
• With Neo4j, Walmart could substitute a heavy
batch process with a simple and real-time graph
database.
Result/Outcome

Case studyeBay Tackles eCommerce Delivery Service Routing
with Neo4j
Challenge
• The queries used to select the best courier for
eBays routing system were simply taking too long
and they needed a solution to maintain a
competitive service.
• The MySQL joins being used created a code base
too slow and complex to maintain.
• eBay is now using Neo4j’s graph database platform
to redefine e-commerce, by making delivery of
online and mobile orders quick and convenient.
Use of Neo4j
• With Neo4j eBay managed to eliminate the biggest
roadblock between retailers and online shoppers:
the option to have your item delivered the same
day.
• The schema-flexible nature of the database
allowed easy extensibility, speeding up
development.
• Neo4j solution was more than 1000x faster than
the prior MySQL Soltution.
Our Neo4j solution is literally
thousands of times faster than the
prior MySQL solution, with queries
that require 10-100 times less code.
Result/Outcome
– Volker Pacher, eBay

Example Recommendation
Solution Architecture

Neo4j Database Cluster
Neo4j APOC
Recommen
dation
Algorithms
(Scheduled)
Management
Dashboard
Neo4j Bolt Driver
Data Ingest
Mgmt.
…
Customer Data Sources / Systems / Applications
Legend:
Neo4j Provided Components
Custom built Neo4j/Customer
Customer/SI
Batch
Data
Buffering
(Queue)
Real-Time
Admin UI
System Specific Adapters / Scripts / Connecters
Admin / Superuser
Apps Websites
Affiliate
Programs
Points of sale
User Interface
Retail Web Shop functionality /
Shipment /
etc.

Why Graph is Superior for Recommendation Engines
Recommendation Requirement Traditional Approaches Neo4j Approach
Usage of connected data over unlimited
amount of „hops“
Complex queries with hundreds of join
tables
Simple single query traverses all
enterprise systems
Real-time 360 degree view on data
within your System
Performance limitations with increasing
number of connections / hops
Traversing over connections in near
real-time provided
Effort required to add additional data
sources to support reco
Days to weeks to rewrite schema and
queries
Minutes to draw new data connections
Time to deployment Months to years Weeks to months
Response time to Recommendations Minutes to hours per query Milliseconds per query
Machine Learning Enablement Static Database scheme leads to static
processes
ML algorithms can use Graph algorithms and
take advantage of connected data
Bottom line Long, ineffective and expensive Easy, fast and affordable

Why Graph is Superior for Recommendation Engines
Recommendation Requirement Traditional Approaches Neo4j Approach
Usage of connected data over unlimited
amount of „hops“
Complex queries with hundreds of join
tables
enterprise systems
Real-time 360 degree view on data
within your System
Performance limitations with increasing
number of connections / hops
Traversing over connections in near
real-time provided
Effort required to add additional data
sources to support reco
queries
Response time to Recommendations Minutes to hours per query Milliseconds per query
Machine Learning Enablement Static Database scheme leads to static
processes
ML algorithms can use Graph algorithms and
take advantage of connected data
A Fortune 500 customer brought in Neo4j to improve
content recommendations quality... and will decommission
48 ‘wide column store’ servers (half a million USD in list
EC2 hosting costs) in favor of a *3-machine* Neo4j cluster
which handles the same load.

How Neo4j Differentiates from other Databases
Visualization
Queries
Processing
Storage
Non-Native Graph DBNative Graph DB RDBMS
Optimized for graph workloads

Neo4j powered Recommendation Engine
Characteristic Benefit for Recommendation Solution
• Agility • Constant learning of recommendations given feedback enabled
• Enabled for Future Requirements
• Solution can be built iteratively
• Fast implementation cycles
• Schema free DB supports “connect anything”
• Intuitiveness • Enable Business Analysts to use technology
• All channels and data sources can be easily connected
• Speed • Unlimited number of traversals to detect potential recommendations
• Response time enables fraud prevention
• Leverage Data Connections • 360 degree customer view enabled / provided
• Scalability • Hardware efficiency with real-time patterns
• TCO/ROI • Adding on top of existing infrastructure protects investments

Graphs Provide Connections
& Context for ML and AI

Knowledge Graphs
GraphConnect speakers 2015-2017

What Your AI & ML Looks Like Today

“Increasingly we're learning that you can make
better predictions about people by getting all the
information from their friends and their friends’
friends than you can from the information you
have about the person themselves”
— Dr. James Fowler

Connected Feature Extraction

Four Pillars of Graph-Enhanced AI/ML
1. Knowledge
Graphs
Context for Decisions
2. Connected Feature
Extraction
Context for Credibility
4. AI Explainability3. Graph
Accelerated AI
Context for Efficiency
Context for Accuracy

Relational WorldGraph World
Morpheus Generated
SQL Graph View
HDFS .
Hive
Metadata
SQL Views
SQL
Database
Morpheus: Graph-Relational, Spark-Based Workbench
Neo4j
This is coming,
not available yet!

GDPR Summary
• GDPR = General Data Protection Regulation
• Adopted by the EU Parliament on 24th May 2016
• Will apply from 25th May 2018
• Applies to both Controllers and Processors
• Applies to organisations operating within the EU, as well as organisations outside
the EU that offer goods or services to individuals in the EU.
• Covers a broad definition of personal data
• Defines lawful basis for processing personal data, which include consent and
contract
• Defines significant fines for non-compliance

Individual Rights Under GDPR
Right to be
informed
Right of
access
Right to
rectification
Right to
erasure
Right to
restriction of
processing
Right to data
portability
Right to
object
Rights
regarding
automated
decision
making

Key GDPR Requirements
Organizations that embrace the new GDPR regulations and provide the right levels of transparency and traceability
for personal information have a big opportunity to win the hearts, minds and business of consumers.
What data do you
have? Is it accurate?
Where is the data
stored?
How and when did
you obtain the data?
Why do you have the
data?
Who has access to the
data?
Do you have
permission to use the
data? For what
purpose?
Is the data secure?
How does the data
travel through your
systems?
Does the data ever
cross international
borders?

GDPR: Risk Mitigation vs. Competitive Advantage
Be a leader and have a solution
ready on time
Improve
Brand
Reduce Risk
Leverage connected data to drive
analytics for threat detection &
business forecasts
Competitive
Advantage
Spend is strategic
Increase ROI
Reduce Risk
Become a trusted enterprise,
delight customers and DPA
Increase
CSAT
Become
Trusted
Improve
Brand
Strategic solution ensures data
governance and solution
maintenance
Reduce Risk
Reduce Cost
Stay on the sidelines to see what
others are doing
Increased
Risk
Look to get by with bare minimum
solution
Increased
Risk
Spend is sunk investment to just
mitigate risk
Low to No
ROI
Unknown
Risk
Mitigation
Solution results in less than happy
subjects, DPO and DPA
Lower CSAT
Minimal Risk
Reduction
Focus on data governance and
solution maintenance is low
Increased
Risk
Increased
Cost

GDPR needs: Connected Data & Visualization
Graph database is the perfect solution to this vast amount of connected data; traditional
approaches with an RDBMS or other NoSQL databases just cannot cut it

Graph Database is the Right GDPR Foundation
Neo4j includes powerful visualization tools that enable you to model and
track the movement of sensitive data through your systems

OSLO, MAY 9, 2018
Data Modeling and Definition
1
Data Transformation2
Consent Management 3
Entitlement 4
GRAPHS IN
METADATA
MANAGEME
NT

OSLO, MAY 9, 2018
#1 Data Modeling and Definition

Party
CUST_SCHE
MA
Party
First
Name
CUST_SCH
EMA_PART
Y.FIRST_NM
CUST_SC
HEMA.PA
RTY
Party
Last
Name
CUSTOMER
NAMECUSTOMER
CUST_SCH
EMA_PART
Y.LAST_NM
Enterprise
Ontology
Application
Logical Model
Physical
Schema
CUST_SC
HEMA.RO
LE

OSLO, MAY 9, 2018
#2 Data Transformation

ETL_P
ROC_1
SALES_S
CHEMA
Normal
ize_Da
teSLS_SC
HEMA.P
RODUC
T
SLS_SCH
EMA.SAL
ES.DATE
SLS_S
CHEMA
.SALES
#2 Data Transformation
Channel
_Normal
ization
SLS_SCH
EMA.SAL
ES.CHAN
NEL
Integration MiddlewareOperational Systems
Time.time
_key
Time.day
_of_week
Enterprise DWH
Billing
Syste
m
EDW
H
CDE:
Transa
ction_D
ate
Star_Sch
ema
Star_S
chema
.Time

OSLO, MAY 9, 2018
#3 Consent managemen

OSLO, MAY 9, 2018
#3 Consent Management + MDM
Amsterdam
NL
K.Vegter
+31623900
4…
kees@neo
4j.com
kees@gm
ail.com
{ contrib: ‘XYZ’,
permittedFor: [UC1,UC4],
consentUntil : 31-12-19 }
{ contrib: ‘internal’,
permittedFor: [UC3],
{ contrib: ‘LMN’,
permittedFor: [UC2,UC6],

OSLO, MAY 9, 2018
#4 Entitlement

OSLO, MAY 9, 2018
#4 Entitlement
User 1 User 3User 2
Exclusi
on List
G1
Resourc
e 1
Group
1
Resourc
e 2
Group
3

#4 Entitlement#3 Consent
Management
#2 Data
Transformation
#1 Data Modelling
and Definition
Graphs in Metadata Management and Data Governance
# …

Why Graph is Superior for GDPR
GDPR Task Traditional Approaches Modern Neo4j Approach
Trace data through enterprise systems Complex queries with hundreds of join
tables
enterprise systems
Preserve the integrity of data lineage Broken data paths and lineage,
especially with NoSQL databases
Continuous, unbroken data paths at all
times
Effort required to add new data and
systems
queries
Response time to GDPR requests Minutes to hours per query Milliseconds per query
Form of GDPR responses Text reports that are not visual and
prove very little
Visuals of personal data and the path it
follows through your systems

OSLO, MAY 9, 2018
Dashboards & Visual Reports

Personal Data Map
Role Based Dashboards - Subject View

Role Based Dashboards - Management View

Data Lineage Report for ‘John Doe’

Graph Database is the Right GDPR Foundation
Extract GDPR Events/Data
Marketing CRM
Customer
Service
Online
Store
Logistics Financials

Conclusion
(graphs)-[:ARE]-> (everywhere)
and
(Solutions)-[:NEED]-> (graphs)

Who can help?
Neo4j Platform
Data Integration
Development &
Admin
Analytics Tooling
Developers
Admins
Data Analysts
Data Scientists
3rd Party Tools
Professional Services:
- Extend and leverage Domain Expertise
- Best Practices
- Using Building Blocks
- Don’t “re-invent the wheel”
- Speed up development and deployment
- Access to Neo4j infrastructure
(Development, Support, Product
management)

Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Similar a Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j (20)

Más de Neo4j

Más de Neo4j (20)

Último

Último (20)

Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j