1. Big Data & APIs
A recon tour on how to successfully do Big Data

4. More events, users
Facebook user post 4.5 billion
items a day (as of Sep 2013)
Facebook MAU 1.2 billion
(as of Sep 2013)

5. More messages, transactions
WhatsApp
From 0 to 31 billion
messages sent daily
(as of Aug 2013)

7. for {
x <- post.stream
user <- getUser(x)
message <- getData(x)
friend <- getFriends(user)
} {
yield notifyFriend(friend,user,message.id)
}
1 billion posts a day!
Example: Notify all my friends

9. Pleasingly parallel problems
●
●
●
●
●
●

10. News filtering
This is a tougher problem.
You cannot read
all that stuff !!!

11. News filtering:
“a machine feeds you what to read”

12. for {
x <- post.stream
user <- getUser(x)
message <- getData(x)
friend <- getFriends(user)
hustle <- getFriendNonsense(friend)
weather <- getWeather(user)
mood <- getMood(user),
vibe <- getMood(friend),
topics <- getTrendingTopics(friends)
market <- getChart(‘gold, ‘bigmac)
interesting <- hal9000(hustle,weather,mood,vibe,topics,market)
if interesting
}{
yield notifyFriend(friend,user,message.id)
}
1 billion posts a day!
Notify only those who care.
The context is
much bigger now.

13. Dealing with context

14. Machine learning to the rescue
●
●
●
The problem
Constraints

15. Data science: random forests
from bigml.com
Solve a
classification
problem

16. Million of features.
Million of users and
preferences.
Very large
sparse matrix !

17. Data science: Time series prediction
Extract features.
Correlate time
series
Very large sparse
matrix !

18. RAM: 100 Tera Byte, DISK: 100 Peta Byte, CPU: 100 Tera Flops

19. Bummer.

20. Why?

22. Nature went that way too.
Ain’t that funny?
“Evolving to multi cellular
organisms”
More resiliant
cells die: organism lives on
Complex tasks:
cannot be handled by a single cell

24. Distributed parallel problems

25. A few distributed computing paradigms
MPI, supercomputing, layered memory arch. , locking, acid
homogeneous, simpler model
heterogeneous, actor model, state-machine

26. The Map Reduce computing

27. The Map Reduce computing

28. Map-Reduce: How well are we doing?

30. CAP theorem: 12 years later
The CAP theorem is largely misunderstood.

31. High Availability
A system can be up, but not available
(think of a network outage: your system is in P mode)
How to improve it .
Replication / Redundancy:
3, 5 replicas are common in highly available systems
Dynamic Commission - Decommission:
re-balance the cluster for dead/new nodes

32. Tuning CAP: understand your use cases

34. Hadoop Distributed FS
Haddop Distribute Run-Time (Map-Reduce)
Hive (DB) Python R
Cassandra (distributed low-latency datastore)
Akka (web server, in-memory runtime)
A proven stack today: Functional

35. Hadoop Distributed FS
Haddop Distribute Run-Time (Map-Reduce)
Hive (DB) Python R
Cassandra (distributed low-latency datastore)
Akka (web server, in-memory runtime)
A proven stack today: Monitoring-Logging
Atmos
DataStax
OpsCenter
Hue
Ambari
Ganglia
Elastic
Search
Logstash
KibanaMarvel

36. Everything Distributed

37. Latency tradeoffs

38. Hmm, thats a complex system.
How to manage?

39. Hmm, thats a complex system.
How to manage?
lazy evaluated
scheduled

40. APIs are everywhere.

41. Thanks