Half humoristic presentation featuring the success, paradox and technologies behind big data today

1. The Paradox of
Big Data

2. 2001 Programming Languages
2004 Natural Language Processing
2006 Social Recommendation
2008 Distributed Computing
2011 Social Gaming
2012 Advertising
2013 Dataiku
2009 Web Mining
Type Spent
Coding
2010
100%
100%
80%
50%
20%
0%
10%
50%
20%
Favorite
Language
C
Exascript
Exascript
Exascript
Python
Powerpoint
Python
Java
None
Largest
Dataset
100GB
100GB
10GB
10TB
100TB
100kB
500GB
100TB
10TB
I’m Florian and I like data

3. www.dataiku.com
Dataiku in short
Software
editor
behind
Data
Science
Studio,
the
«
Photoshop
for
Data
Science
»
COMMUNITY
EDITION
http://www.dataiku.com/dss/trynow/

4. Goals For Today
• Big Data with the bias of what I know of it
(Analytics …)
• Big Data: History and Feelings
• What are the key technologies to watch ?
• Some practical use cases ?
• How to get started ?

5. Dataiku
Motivation
1/8/14
4
First Hard Drive: 3,75 Megabytes
Access Time: 1 second

6. IN 2008
man
invented
big data
Volume Variety Velocity

7. WHAT IF THE MARKETING GUY
HAD CHOSEN ANOTHER LETTER?
Capacity Complexity Celerity

8. OR SIMPLER
Size Serendipity Speed

9. OR AFTER A DRINK
Big Blur Blazing

10. Or Combine
C… B.. S….

11. Or Combine
Complete Bull Sh..

12. SOOO
WHAT IS
BIG DATA ?

13. PARADOX #1
SIMPLEXITY

14. SUBTLE
PATTERNS

15. "MORE
BUSINESS"
BUTTONS

16. PARADOX #2
SELF-AWARE

17. DATA SCIENTIST
AT NIGHT

18. DATA CLEANER
THE DAY

19. DATA PLUMBERER
THE WEEK-END

20. WAIT COMPUTATION
BETWEEN
COFFEES

21. PARADOX #3
WHERE TO STORE DATA?

22. MY DATA
IS WORTH
MILLIONS

23. I SEND IT
TO THE
MARKETING
CLOUD
AND BACKUP
IT TO GOOGLE

24. PARADOX #4
IS IT BIG OR NOT ?

25. WE ALL LIVE
IN A BIG DATA
LAKE

26. ALL MY DATA
MAY FITS
IN HERE

27. PARADOX #5 (at last)
HUMAN OR NOT ?

28. TECHCRUNCH
SAYS THAT
MACHINE
LEARNING
WILL SAVE
US ALL

29. I JUST WANT
MORE REPORTS

30. BIG DATA
TECH TRENDS

31. ELEPHANT MAKE BABIES

32. WELCOME TO
TECHNOSLAVIA
Hadoop
Ceph
Sphere
Cassandra
Kafka Flume
Spark
Scikit-Learn GraphLAB
prediction.io jubatus
Mahout
WEKA
MLBase LibSVM
RapidMiner
Panda
Kibana
InfiniDB Drill
Spark SQL
Hive
Impala
…
Elastic Search
SOLR
MongoDB
Riak
Membase
Pig
Cascading
Talend
Machine Learning
Mystery Land
Scalability Central
SQL Colunnar Republic
Vizualization County Data Clean Wasteland
Statistician Old
House
R
Real-time island
Storm
NOSQL Nihiland

33. DRIVER
1:
BACK
TO
THE
BASICS
RAM
-­‐
CPU
-­‐
DISK

34. 2000 2013
1000$
/
GB
6$
/
GB
$10
/
GB
$0.06
/
GB
memory
divided
by
150
disk
cost
divided
by
250
MAP
REDUCE
times
HACK
REDUCE
times
A
PERSISTENT
MEMORY
PROBLEM

35. DATA
IS
BIGGER

36. IS
USEFUL
DATA
BIGGER
?
WHOLE
DATA
REFINED
DATA

37. GOLD
NEEDLE
IN
HAYSTACK
?

38. OILD
REFINE
BEFORE
USE

39. HOW
BIG
IS
BIG
DATA
?
Web
Site
– $1Billion
revenue
per
year
– 10
Millions
Unique
Visitor
per
month
– 100.Millions
orders
/
actions
/
per
day
10TB
RAW
DATA
1TB
REFINED
DATA

40. 1
TERABYTE
FITS
IN
MEMORY
1TB

41. DRIVER
2
:
ECOSYSTEM
GROWS
• GOOGLE
• 1
Circle
OPEN
SOURCE
– YAHOO
–
IBM
–
LINKEDIN
-­‐
FACEBOOK
• 2
Circle
– STANDFORD
BERKELEY
– STARTUPS

42. STARTUPS
64m$
6.75m$
14m$
2m$
40m$
20m$
20.5m$
19m$
4m$
100m$
1.8m$
17m$
11m$
7.75m$
1.7m$
2013
2012
2011
2010
2009
$1B
per
year
Invested
in
Big
Data
TECH
223m$
301m$

43. ALL
>
SPARK
Real-­‐Time
Resilient
Distributed
Memory
Framework
• Abstraction
with
any
DAG
operation
on
data:
-­‐ Filter
-­‐ Map
-­‐ Reduce
-­‐ Cache

44. SPARK
AND
ITS
ECOSYSTEM
SHARK
MLBASE
STREAMING
Real-­‐Time
Queries
Real-­‐Time
Updates
In-­‐Memory
Learning
SPARK

45. SooOOo
WHAT IS IT IN
PRACTICE?

46. www.dataiku.com
Turn Device Logs
Into Next Years' Business
Parking
ticket
machine
data
OpenStreetMap
data
Cleaning
and
enrichment
of
data
Crossing
data
Data Science Studio
Creation
of
a
predictive
algorithm
Availability
of
the
predictions
Each
street
is
segmented
into
small
pieces
that
are
enriched
with
geospatial
information.
The
parking
ticket
history
is
joined
with
the
points
of
interest
from
OpenStreetMap.
The
availability
of
parking
lots
is
predicted
by
street
segments
from
the
joined
data.
The
algorithm
is
finally
integrated
in
the
iPhone
app
«
Find
me
a
space
».
by

47. www.dataiku.com
Optimizing Last Mile with
Data Science Studio
Data Science Studio
Historical delivery
and retrieval data
Modeling of a score
for each delivery
Cleaning and temporal
enrichment of data
Data aggregation by
geographic location
Incorporation of new deliveries
to the existing model
by

48. • Reformulation de la
recherche
• Pas de réponse
• Clic sur un pro
• Top recherche
• Clic de navigation ou filtre
COMMENT AMÉLIORER LA PERTINENCE DE NOS RÉPONSES
VIA L’ANALYSE DU COMPORTEMENT UTILISATEUR ?
20 M
Analyse &
corrections
automatisation
>10
occurrences1,4M
requêtes
>200M
recherches
✗ ✓
0,5M requêtes
priorisées

49. SOLUTION
Machine
Gestion Exploration
pagesjaunes.fr
Annuaire
hadoop PIG+Hive
Exportindexation
Moteur
d’interprétation
crawl
Autres
référentiels
Sickit-learn

50. www.dataiku.com
Analyst
Panels
1970 : Birth of Computer Analytics
Computer
Expensive Software
Marketing
Studies

51. www.dataiku.com
Multiple
Data
Sources
Analyst Team
Many
Models
CRM
Logs
2015 : BUILD YOUR FACTORY
Server Cluster
Light Software
Personalised
Experience Model
Acquisition
Cost Opportunity
Model
Stock Optimisation
Model
Optimize
Delivery

52. www.dataiku.com
Churn
Volume Forecast
RecommenderSegmentation LifetimeValue
Risk Score Hot Location
Pricing Ranking FraudEvent Paths
A MODEL
An automated way to make a computer
take a decision from raw (historical) data
The model can be used to take immediate (real-time)
actions through an API

53. www.dataiku.com
Churn
Volume Forecast
RecommenderSegmentation LifetimeValue
Risk Score Hot Location
Pricing Ranking FraudEvent Paths

54. SooOOo
How To I ENTER
WONDERLAND ?

55. STEP 1 : LEARN
• PYTHON + PANDAS +
SCIKIT
• R
• SCALA
http://scikit-learn.org/
https://www.coursera.org/course/rprog

56. STEP 2 : PRACTICE
• Try to enter in a Contest on kaggle.com or
• or datascience.net
• Join a meetup

57. www.dataiku.com
http://www.dataiku.com/dss/trynow/
Dataiku HQ
2 rue Jean Lantier
75001 Paris France
Dataiku West
2423A Durant Avenue
Berkeley, CA 94704
Florian
florian.douetteau@dataiku.com
You have ideas
“My data is too dirty. I don’t even know where to start ”
“We could probably better understand ours users. But how ?
“There’s a trend here, but our full historical data is just too big”
You have data
You need a tool