1.
Vegas
The Missing Matplotlib for
Scala/Spark
DB Tsai
Roger Menezes

2.
Homepage Kids Page Downloads Page
Netflix Recommendations
Every aspect
of the
Experience is
Machine
Learned

3.
3
2017
> 100M members
> 190 countries

4.
Multiple Devices

5.
Genres: 23 rows/page average
Sims: 10 rows/page average

6.
My List:
Continue Watching:
Popular on Netflix:
Trending Now:
Watch It Again:
Top Picks:
Because You Watched:
Genres:
New Releases:
Recently Added:
Originals RowBillboard:

7.
Machine Learning at Netflix
● Optimize the Experimentation usecase vs Productionization
● Experimentation
○ Opportunity sizing, Data Exploration
○ Feature Identification and Selection
○ Tweaks to ML algos
○ Model Evaluation

8.
Experimenter’s loop
Problem
Explore
Data
Identify
Features
Produce
Model
Evaluate
Model
Share
Findings

9.
Notebooks
● Optimal for Experimentation
● Sharing reproducible research
○ Facilitates feedback loop with Product Managers
● End to end ML experiment.
○ Interactivity drives productivity

10.
Python Notebooks

11.
Python Notebooks
● Seamless Experience - ML experimentation
● Well known Scientific computing libraries
● Huge catalog of Visualization plotting libraries
○ Matplotlib, Seaborn, Bokeh, BQPlot, Lightning, etc.

12.
Scala Notebooks
● Zeppelin, Jupyter, Databricks, Spark-Notebooks, ...
● Computing library gap filling up
● Lack of Visualization Libraries
○ Main friction point in adoption
○ End to End ML use case not convincing

13.
Introducing Vegas
● Visualization Library in Scala
● Mainly built for the notebook use case
● Scala wrapper around Vega-Lite
○ Missing MatPlotLib for the Scala/Spark world.

14.
DECLARATIVE
STATISTICAL
VISUALIZATION
GRAMMAR
IN SCALA
You tell it WHAT should be done with the data, and it knows
HOW to do it!
Operations such as filtering, aggregation, faceting are built
into the visualization, rather than putting the burden on the
user to massage the data into shape.
Complex visualizations can be built with a few high level
abstractions:
DATA
TRANS-
FORMS
SCALES
GUIDES MARKS
cf : Altair Talk by Brian Granger in PyData 2016 https://youtu.be/v5mrwq7yJc4

15.
Added Bonus of Declarative
Visualizations:
INTERACTIVITY!
D3JS
VEGAS
VEGAS CODE EXPANDS OUT TO D3JS CODE!

16.
Anatomy of a plot: Channels
X/Y channel
Shape Channel
Size Channel
Color Channel

17.
Features…

18.
1. Supports most plot types

19.
2. Trellis plots

20.
3. Layers
Layer 1.
Layer 2.
Layer 3.

21.
4. Notebook and Consoles

22.
5. Built-in spark support
Vegas
.withDataFrame(myDataFrame)
.encodeX(“population”)
.encodeY(“age”)
Mapped Columns
Pass In DF.

23.
6. Visual statistics
● Advanced Binning
● Sorting
● Scaling
● Custom Transforms
● Time Series
● Aggregation
● Filtering
● Math functions (log, etc)
● Descriptive Statistics

24.
How It Works !

25.
1. Specify in Scala
2. Embed HTML
(iFrame)
3. Render within
iFrame using JS

26.
VEGA
D3JS
VEGA-LITE*
VEGAS
MOREABSTRACTION SCALA DSL EMITS TYPE-CHECKED
VEGA-LITE JSON
VEGA-LITE CONVERTS INTERNALLY
TO VEGA JSON SPEC
VEGA TRANSLATES JSON TO D3JS
CODE THAT CAN BE VERY VERBOSE
A SCALA DSL FOR VEGA-LITE
* Vega-Lite

27.
What’s coming

28.
1. Interactive selections

29.
2. Selections transforms

30.
Contributors
Aish DB Roger
Sudeep Jeremy

31.
Thank you.

32.
@NetflixResearch
@rogermenezes @dbtsai
The missing MatPlotLib
for Scala/Spark
http://vegas-viz.org