This technical tutorial is designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. The session starts with a quick introduction to Redis and the capabilities Redis provides. It will cover the basic data types provided by Redis and the module system. Using an ad serving use case, Griffith will look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. You will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark and then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will also be discussed, focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these features. By the end of the session, you should feel confident building a prototype/proof-of-concept application using Redis and Spark. You’ll understand how Redis complements Spark, and how to use Redis to serve complex, ML-models with high performance.

1. Tague Griffith – Head of Developer Advocacy
Redis Labs
Getting Ready to Use
Redis with Apache Spark

2. Agenda
• Introductions
• Why Redis and Spark
• What is Redis
• Redis-ML module

3. Redis Labs – Home of Redis
• Founded in 2011
• HQ in Mountain View CA, R&D center in Tel-Aviv IL
The commercial company behind Open Source Redis
Provider of the Redis Enterprise (Redise)
technology, platform and products

4. Who I am
• Head of Developer Advocacy
• Developer and architect turned Evangelist
• Infrastructure and Distributed Systems
• Large Scale Redis Systems
• Former: Apple, Netscape, Yahoo/Flickr, GoPro

5. ML Models Challenges
• Models are becoming bigger and more complex
• Can be challenging to deploy
• Do not scale well, speed and size
• Reliable services are hard to do
• Can be very expensive

6. What is Redis

7. • NoSQL Database
• Open Source
• In-Memory
• Most operations performed < 1ms

8. Why Use Redis
Simplicity ExtensibilityPerformance
ListsSorted Sets
Hashes Hyperloglog
Geospatial
Indexes
Bitmaps
SetsStrings
Bit field
NoSQL Benchmark Redis Data Structures Redis Modules

9. Redis Modules
• Any C/C++ program can now run on Redis
• Use existing or add new data-structures
• Enjoy simplicity, infinite scalability and high availability while keeping the native
speed of Redis
• Can be created by anyone
New Capabilities
New Commands
New Data Types

10. Redis - ML
• Define data structures for models
• Store training output as “hot model”
• Perform evaluation directly in Redis
• Easily integrate existing C/C++ ML libs
Spark Training
Data loaded into Spark Model is saved in
Redis-ML
Redis-ML
Serving Client
Client
App
Client
App
Client
App
Any Training
Platform

11. Redis ML

12. Redis ML Module
Redis Module
Tree Ensembles
Linear Regression
Logistic Regression
Matrix + Vector Operations
More to come...

13. Random Forest Model
• A collection of decision trees
• Supports classification & regression
• Splitter Node can be:
◦ Categorical (e.g. day == “Sunday”)
◦ Numerical (e.g. age < 43)
• Decision is taken by the majority of decision trees

14. Perform classification/regression of a feature vector:
Forest Data Type API
Add nodes to a tree in a forest:
ML.FOREST.ADD <forestId> <treeId> <path>
[ [NUMERIC|CATEGORIC] <splitterAttr> <splitterVal> ] |
[LEAF] <predVal>
ML.FOREST.RUN <forestId> <features>
[CLASSIFICATION|REGRESSION]

15. Forest Data Type Example
> MODULE LOAD "./redis-ml.so"
OK

16. Forest Data Type Example
> MODULE LOAD "./redis-ml.so"
OK
> ML.FOREST.ADD myforest 0 . CATEGORIC sex “male” .L
LEAF 1 .R LEAF 0
OK

17. Forest Data Type Example
> MODULE LOAD "./redis-ml.so"
OK
> ML.FOREST.ADD myforest 0 . CATEGORIC sex “male” .L
LEAF 1 .R LEAF 0
OK
> ML.FOREST.RUN myforest sex:male
"1"

18. Forest Data Type Example
> MODULE LOAD "./redis-ml.so"
OK
> ML.FOREST.ADD myforest 0 . CATEGORIC sex “male” .L
LEAF 1 .R LEAF 0
OK
> ML.FOREST.RUN myforest sex:male
"1"
> ML.FOREST.RUN myforest sex:something_else
"0"

19. Redis Spark-ML Module
Spark Redis-ML

20. Resources
• Redis-ML: https://github.com/RedisLabsModules/redis-ml
• Spark-Redis-ML: https://github.com/RedisLabs/spark-redis-ml
• Online Demo: http://bit.ly/sparkredisml

21. tague@redislabs.com
@tague on Twitter
Thank You.