DataOps expands DevOps philosophy to include data-heavy roles (data engineering & data science). DataOps uses better cross-functional collaboration for flexibility, fast time to value and an agile workflow for data-intensive applications including machine learning pipelines. (Strata Data San Jose March 2018)

1. © 2018 MapR Technologies 1
DataOps: An Agile Method for
Data-driven Organizations
Ellen Friedman, PhD
Principal Technologist
7 March 2018 #StrataData

2. © 2018 MapR Technologies 2
Contact Information
Ellen Friedman, PhD
Principal Technologist, MapR Technologies
Committer Apache Drill & Apache Mahout projects
O’Reilly author
Email efriedman@mapr.com ellenf@apache.org
Twitter @Ellen_Friedman #StrataData

3. © 2018 MapR Technologies 3
Big Data Applications Used Widely in Production
Quoted from:
New Vantage Partner Big Data Executive Surveys for 2016 & 2017
http://newvantage.com/wp-content/uploads/2016/01/Big-Data-Executive-Survey-2016-Findings-FINAL.pdf
http://newvantage.com/wp-content/uploads/2017/01/Big-Data-Executive-Survey-2017-Executive-Summary.pdf

4. © 2018 MapR Technologies 4
How do you measure Earth’s oceans?
By NASA Goddard Space Flight Center from Greenbelt, MD, USA (Full Disk Image of Earth Captured August 24, 2011)
[CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

5. © 2018 MapR Technologies 5

6. © 2018 MapR Technologies 6
Changing How People Work with Data
A 19th century big data story
Matthew Fountain Maury
extracted data from ship’s logs
to build amazing charts for
navigation

7. © 2018 MapR Technologies 7
Big data project: Maury’s Wind and Currents charts

8. © 2018 MapR Technologies 8
Big data project: Maury’s Wind and Currents charts
At first, nobody was
interested in them…

9. © 2018 MapR Technologies 9
Big data project: Maury’s Wind and Currents charts
At first, nobody was
interested in them…
…until Captain Jackson
shaved a month off the run
from Baltimore to
Rio de Janeiro
Then everybody
wanted one!

10. © 2018 MapR Technologies 10
© 2014 Ellen Friedman
People with “vision” think with their eyes closed

11. © 2018 MapR Technologies 11
Aadhaar Project: Largest Biometric DB in the World
• Unique 12 – digit number for each person in India
• Proof of identity, authenticated anytime, anywhere
• Runs on NoSQL database MapR-DB since 2014
Revolution: Changing a Society
Photo credit PANOS, with permission
1.3 B
people

12. © 2018 MapR Technologies 12
Changing Rhythm to How We Work with Data
Utility providers using
smart meters
Collect data every 15 min
instead of once a month

13. © 2018 MapR Technologies 13
Image © E Friedman
Self-driving cars:
Huge volume of
sensor data
Time value of data
Analysis at the Speed of Life

14. © 2018 MapR Technologies 14
Changing Rhythm to How We Work with Data
Apache Drill SQL query engine with schema discovery for data exploration
May shorten prep time when running a new query from days/ weeks to hours
Follow community on Twitter: @ApacheDrill

15. © 2018 MapR Technologies 15
We
need
a
better
fit
to
the
way
business
happens

16. © 2018 MapR Technologies 16
A Better Fit
• The way business happens
• A dataflow (datafabric) that matches the shape of business
• Technologies with capabilities to support flexibility and timely
response across data centers
• Organization at the human level matches as well

17. © 2018 MapR Technologies 17
Build a Global Data Fabric
Flexibility & agility to respond as life changes

18. © 2018 MapR Technologies 18
Global Data Fabric
• Comprehensive view of data
• Breaks silos
• Works with multi-tenancy
• Computation (and data) where you want them
• Fine-grained control over who has (and does not have) access

19. © 2018 MapR Technologies 19
A
DataOps
approach
improves
a
project’s
ability
to
stay
on
target
&
on
time

20. © 2018 MapR Technologies 20
DataOps: Brings Flexibility & Focus
Platform&network
Operations
Softwareengineering
Architecture&planning
Dataengineering
Datascience
Productmanagement
DataOps Team B
DataOps Team A
Cross functional DataOps teams
• Expands DevOps to include data-heavy roles
• Organized around data-related goals
• Better collaboration and communication between roles
From Chap 2 of Machine Learning Logistics, by Ted Dunning & Ellen Friedman © 2018 (O’Reilly Media)

21. © 2018 MapR Technologies 21
DataOps Principles
“DataOps teams seek to orchestrate data, tools, code and environments from
beginning to end.”
They “…measure performance of data analytics by the insights they deliver.”
Thor Olavsrud interview with Ted Dunning & Ellen Friedman for CIO
https://www.cio.com/article/3237694/analytics/what-is-dataops-data-operations-analytics.html

22. © 2018 MapR Technologies 22
Advantages of a DataOps Approach
• Able to pivot & respond to real-world events as they happen
• Improved efficiency and better use of people’s time
• Faster time-to-value
• A good fit to working with a global data fabric

23. © 2018 MapR Technologies 23
How
do
you
keep
people
from
feeling
threatened
by
change?

24. © 2018 MapR Technologies 24
Don’t
be
threatening!

25. © 2018 MapR Technologies 25
Why Stream?
Munich surfing wave Image © 2017 Ellen Friedman

26. © 2018 MapR Technologies 26
Stream
transport
supports
microservices

27. © 2018 MapR Technologies 27
Stream Transport that Decouples Producers & Consumers
P
P
P
C
C
C
Transport Processing
Kafka /
MapR Streams
Good stream transport is persistent, performant & pervasive!

28. © 2018 MapR Technologies 28
More on Streaming Microservices
• Chapter 3 of Streaming Architecture by Ted Dunning & Ellen Friedman
© 2016 (O’Reilly Media)
http://shop.oreilly.com/product/0636920049463.do
• “Streaming Microservices” chapter by Ted Dunning & Ellen Friedman in
Encyclopedia of Big Data Technologies, Sherif Sakr and Albert
Zomaya, editors. In press 2018 (Springer International Publishing)
• Chapter 4 in A Practical Guide to Microservices & Containers by
James A. Scott © 2017 (MapR)
https://mapr.com/ebooks/microservices-and-containers/title.html

29. © 2018 MapR Technologies 29
Legacy Applications
How Does MapR Solve This?
Big Data 1.0 Applications Next-Gen Applications
MapR Converged Data Platform
High Availability Real Time Unified Security Multi-tenancy Disaster Recovery Global Namespace
Real-Time NoQL Database Stream TransportWeb-Scale Storage

30. © 2018 MapR Technologies 30
With MapR, Geo-Distributed Data Appears Local
stream
Data
source
Consumer

31. © 2018 MapR Technologies 31
With MapR, Geo-Distributed Data Appears Local
stream
stream
Data
source
Consumer

32. © 2018 MapR Technologies 32
With MapR, Geo-distributed Data Appears Local
stream
stream
Data
source
ConsumerGlobal Data Center
Regional Data Center

33. © 2018 MapR Technologies 33
90% of the effort in successful
machine learning isn’t the
algorithm or the model…
It’s the logistics

34. © 2018 MapR Technologies 34
Why?
• Just getting the training data is hard
• ! The myth of the unitary model
• Model-to-model evaluation
• Respond as the world changes: Agile roll out & roll back when
deploy to production

35. © 2018 MapR Technologies 35
Metrics
Metrics
ResultsRendezvous
Enter Rendezvous Architecture
Scores
ArchiveDecoy
m1
m2
m3
Features /
profiles
InputRaw
Rendezvous Architecture described in:
- Machine Learning Logistics book by Ted Dunning & Ellen Friedman © 2018 (O’Reilly)
- “Rendezvous Architecture” chapter in Encyclopedia of Big Data Technologies. Sherif Sakr and Albert
Zomaya, editors. Springer International Publishing, in press 2018.

36. © 2018 MapR Technologies 36
Best thing about Rendezvous: Agile deployment
• Many “good” models ready and waiting
– Already running
– Ready to deploy into production
• To deploy a new model: just stop ignoring it

37. © 2018 MapR Technologies 37
Rendezvous to the Rescue: Better ML Logistics
• Stream-1st architecture is a powerful approach with
surprisingly widespread advantages
– Innovative technologies emerging for streaming data
• Microservices approach provides flexibility
– Streaming supports microservices (if done right)
• Containers remove surprises
– Predictable environment for running models

38. © 2018 MapR Technologies 38
Preparation of Input (and Training) Data
Model 1
Model 2
Model 3
request
Raw
Add
external
data
Input
Database
The world
Raw data may contain features you’ll want in future

39. © 2018 MapR Technologies 39
Raw data is gold!

40. © 2018 MapR Technologies 40
Decoy Model in the Rendezvous Architecture
Input
Scores
Decoy
Model 2
Model 3
Archive
• Looks like a model, but it just archives inputs
• Safe in a good streaming environment, less safe without good isolation

41. © 2018 MapR Technologies 41
Why do you need new models?
Conditions may (will) change

42. © 2018 MapR Technologies 42
Advantages of Rendezvous Architecture
Real
model
Result
Canary
Decoy
Archive
Input

43. © 2018 MapR Technologies 43
Rendezvous: Mainly for Decisioning Type Systems
• Decisioning style machine learning
– Looking for a “right answer”
– Simpler than interactive machine learning (such as in self-driving car)
• Examples include:
– Fraud detection
– Predictive analytics / market prediction
– Churn prediction (as in telecommunications)
– Yield optimization
– Deep learning in form of speech or image recognition, in some cases

44. © 2018 MapR Technologies 44
Described in new book on ML management:
Download free pdf via @MapR:
https://mapr.com/ebook/machine-learning-logistics/
Includes a discussion of DataOps

45. © 2018 MapR Technologies 45
Example: Tensor Chicken
Label
training
data
Run the
model
Deploy
model
Gather
training
data
Labeled
image files
Train
model
Update
model
Deep learning project by
software engineer Ian Downard
(see blog + @tensorchicken)

46. © 2018 MapR Technologies 46
DataOps: A Good Way to Adapt to Emerging Data Practices
• Faster time-to-value & better ability to pivot
– Better collaboration/communication across skill groups
– Focused around data-related goals
– More efficient use of team members’ time
• A good fit to working with a data fabric
• A good fit for a streaming microservices style

47. © 2018 MapR Technologies 47
Please support women in tech – help build
girls’ dreams of what they can accomplish
© Ellen Friedman 2015#womenintech #datawomen

48. © 2018 MapR Technologies 48
Related events at Strata
• “Better Machine Learning Logistics with Rendezvous
Architecture” talk by Ted Dunning Wed at 5:10pm
• “Rendezvous Architecture” booth talk at MapR booth
Thur at 11:30 am
• Chat with us in the MapR booth

49. © 2018 MapR Technologies 49
Thank You !

50. © 2018 MapR Technologies 50
Additional Resources: Available Now
O’Reilly report by Ted Dunning & Ellen Friedman © March 2017
Read free courtesy of MapR:
https://mapr.com/geo-distribution-big-data-and-analytics/
O’Reilly book by Ted Dunning & Ellen Friedman
© March 2016
Read free courtesy of MapR:
https://mapr.com/streaming-architecture-using-
apache-kafka-mapr-streams/

51. © 2018 MapR Technologies 51
Book signings at MapR booth
• Wed afternoon break 3:35 pm – 4:15 pm
• Thur morning break 10:30 am – 11:10 am
Get a free copy of the book & meet the authors
Ted Dunning & Ellen Friedman
Or download free pdf via @MapR:
https://mapr.com/ebook/machine-learning-logistics/

52. © 2018 MapR Technologies 52
Please tell me how DataOps works out for you.
Ellen Friedman
Twitter @Ellen_Friedman
Email @efriedman@mapr.com ellenf@apache.org