There are exciting and boring elements in any project, both of which must be addressed. This session will cover how to overcome the difficult but necessary problems that are essential to success. Presented by Aaron Cordova CTO, co-founder at Koverse Inc.

1. #datapopupseattle
AARON CORDOVA
CTO and Co-Founder, Koverse
aaroncordova
Making Big Data Projects Successful
koverse

2. #datapopupseattle
UNSTRUCTURED
Data Science POP-UP in Seattle
www.dominodatalab.com
D
Produced by Domino Data Lab
Domino’s enterprise data science platform is used
by leading analytical organizations to increase
productivity, enable collaboration, and publish
models into production faster.

3. Keys to making successful
big data projects repeatable

4. ©Koverse | Company Confiden<al 2
Intro
Aaron Cordova
CTO, co-founder at Koverse Inc.
Built successful big data systems for DOD, Intelligence,
Finance

5. ©Koverse | Company Confiden<al 3
Big Data Projects
How it tends to be
How it should be

6. ©Koverse | Company Confiden<al 4
Big Data Projects
Interes<ng part

7. ©Koverse | Company Confiden<al 5
Big Data Projects
Interes<ng part

8. ©Koverse | Company Confiden<al 6
Big Data Projects
Interes<ng part
More propellant
Support
Infrastructure
Propellant
Launch plaSorm
U<li<es

9. ©Koverse | Company Confiden<al 7
Step 1: Import
Bring the data to the data scien<st
From where?

10. ©Koverse | Company Confiden<al 8
Step 1: Security
Sensi<ve data requires access controls
Using more than 1 dataset require fine-grained access controls

11. ©Koverse | Company Confiden<al 9
Step 1: Security

12. ©Koverse | Company Confiden<al 10
Step 2: Data Assump<ons
Need to find out
1. Structure of the data (field names, types)
2. Data seman<cs (is CustomerID in dataset A equal to CID from dataset B?)
Ini<al assump<ons are almost certainly wrong.
Need to see actual data samples.
Go back, get more datasets; normalize, clean up data

13. ©Koverse | Company Confiden<al 11
Step 2: Data Assump<ons
If primary analy<cal system can’t handle discovery, need another system for
sampling, viewing, cleaning up, normalizing data

14. ©Koverse | Company Confiden<al 12
Step 3: Interes<ng Part!
Run analy<cs!
Need some sort of system for running analy<cs:
R
Python
Spark MLLib
MapReduce
SAS

15. ©Koverse | Company Confiden<al 13
Step 4: Delivering Results
Reports are rela<vely easy to deliver – run once a day .. small output
Some results are large, need to stay in the system
Indexing makes results searchable for a large number of consumers
Results can be embedded in interac<ve decision-making apps with an API

16. ©Koverse | Company Confiden<al 14
Step 4: Delivering Results
Find some system for indexing analy<cal results – possibly copying data,
address consistency issues
Apply some solu<on for making results available via an API so they can be
embedded in applica<ons …
Then build applica<ons

17. ©Koverse | Company Confiden<al 15
Scalability
Even if original data sets are small, mul<ple data sets need to be co-located
Original data is transformed into deriva<ves
Indexed data requires more space
Scalability becomes a problem eventually

18. ©Koverse | Company Confiden<al 16
Scalability
Migrate original solu<on to a scalable system.
Rewrite analy<cs, data flow for the scalable system.

19. ©Koverse | Company Confiden<al 17
Repeatability
System works! Now what?
As new data arrives, the whole process needs to be re-run, or run on all the
available data
If any assump<ons or structure of the data change, need to be able to re-
process data
Live updates need to be scheduled, resource demands need to be balanced
Oh yeah, and go back and address security … if possible

20. ©Koverse | Company Confiden<al 18
Working backwards

21. ©Koverse | Company Confiden<al 19
Working backwards
Want to provide value from data but first have to:
Address data discovery, security, scalability, repeatability …

22. ©Koverse | Company Confiden<al 20
Yak Shaving
Avoid

23. ©Koverse | Company Confiden<al 21
Recommended approach
1. Start with scalable technologies
2. Build in security from the start
3. Admit that data is messy, make it possible to address data quality issues
within the system
4. Integrate with whatever analy<cal tools data scien<sts want to use
5. Integrate indexing and search into the system, avoid copying data
6. Allow for prototyping new data flows, analy<cs, apps in produc<on system.
Going live a mamer of configura<on .. not a rewrite

24. ©Koverse | Company Confiden<al 22
Recommended approach
Go from 2-3 successful projects per year to 20-30