Discover how the world of big data is evolving and becoming faster, more reliable and better organized-- powering many of the cooler new features that you see in the client today!

1. TheEvolution
ofBigDataat
Spotify
Josh Baer (jbx@spotify.com)

2. WhoAm I?
‣ Technical Product Owner at
Spotify, responsible for
Hadoop
@l_phant

3. Overview
• Creating Music Charts in three parts:
• Playing Music
• Collecting Data
• Processing Data
• The Future

4. Building
MusicCharts

6. Building Music Charts
Play Music Collect Data Process

7. PlayingMusic

8. What is Spotify?
• Music Streaming Service
• Browse and Discover Millions of
Songs,Artists andAlbums
• Bythe end of 2014
• 60 Million Monthly Users
• 15 Million Paid Subscribers

9. What is Spotify?
• Data Infrastructure
• 1300 Hadoop Nodes
• 42 PB Storage
• 30TB data ingested via Kafka/day
• 400TB generated by Hadoop/day

10. Powered by Data
• RunningApp
• Matches music to running tempo
• Personalized running playlists in
multiple tempos for millions of
active users
http://www.theverge.com/2015/6/1/8696659/spotify-running-is-great-for-discovery

11. Powered by Data
• Now Page
• Shows, podcasts and playlists
based on day-parts
• Personalized layout so you always
have the right music forthe right
moment

13. Building Music Charts
10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/
aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/
aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/
courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser
HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/1291726"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/
instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/
dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81
Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/
courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
• Raw data is complicated
• Often dirty
• Evolving structure
• Duplication all over
• Getting data to a central
processing point is HARD

14. Collecting
Data

15. “It’s simple, we just
throw the data into
Hadoop”
A naive data engineer

16. LogArchiver
• Original method to transport logs fromAPs to HDFS
• Lasted from 2009 - 2013
• Relies on rsynch/scp to move files around
• Regularly scheduled via cron

18. LogArchiver Fails
• Worked well with small number ofAPs
• Issues with scale
• Manual Processes of adding new hosts
• Frequent dying of hosts or network issues caused massive congestion
• Manual process of overrides

19. Apache Kafka to the rescue!
• Apublish-subscribe messaging system open sourced by LinkedIn in 2011
• High level overview:
• Topic: Feeds of messages
• Producer:Amessage publisher
• Consumer :Asubscriber oftopics

20. Apache Kafka to the rescue!
• Log -> HDFS latency reduced from hours to seconds!
• Benefits:
• Community supported
• Division of responsibilities
• Allowed for enhanced streaming use-cases

22. Processing
Data

23. Workflow Management Fail!
0
*
*
*
*
spotify-­‐core
hadoop
jar
merge_hourly.jar
15
*
*
*
*
spotify-­‐core
hadoop
jar
aggregate_song_plays.jar
30
*
*
*
*
spotify-­‐analytics
hadoop
jar
merge_artist_song.jar
*
1
*
*
*
spotify-­‐core
hadoop
jar
daily_aggregate.jar
*
2
*
*
*
spotify-­‐core
hadoop
jar
calculate_royalties.jar
*/2
22
*
*
*
spotify-­‐radio
hadoop
jar
generate_radio.jar

24. Handles the ‘plumbing’ for Hadoop jobs
https://github.com/spotify/luigi
Luigi - Python Workflow Manager
Easy to get started, no xml like Oozie

25. Hadoop Availability
• In 2013:
• Hadoop expanded to 200 nodes
• It was business critical
• It was not very reliable :-(
• Created a ‘squad’ with two missions:
• Migrate to a new distribution withYarn
• Make Hadoop reliable

26. How did we do?
HadoopUptime
90%
92%
94%
96%
98%
100%
Q3-2012 Q4-2012 Q1-2013 Q2-2013 Q3-2013 Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015
Hadoop ownerless Dedicated
squad launches
Upgrade
instability
Continually improving

27. Going from Python to Crunch
• Most of our jobs were Hadoop (python) streaming
• Lots of failures, slow performance
• Had to find a betterway

28. Moving from Python to Crunch
• Investigated several frameworks*
• Selected Crunch:
• Real types - compile time error detection, bettertestability
• Higher levelAPI - let the framework optimize foryou
• Better performance #JVM_FTW
*thewit.ch/scalding_crunchy_pig

30. Play Music Collect Data Process
Data driven features that
allows for new ways to
play and discover music
Lower latency and
enhanced reliability for
passing data from Access
Points to HDFS via Kafka
Increased Hadoop
reliability, Luigi scheduling
and better performance
with Crunch
Improving Charts!

31. TheFuture

32. Growth%
0
500
1000
1500
2000
2500
3000
3500
2012 2013 2014 2015
Hadoop Usage Spotify Users
Growth of Hadoop vs. Spotify Users

33. Explosive Growth
• Increased Spotify Users
• More users listening to more music -> more data -> longer running jobs
• Increased Use Cases
• Beyond simple analytics into Machine Learning, advanced processing
• Increased Engineers
• In 2014, growth of data and machine learning engineers grew rapidly

34. Scaling Machines: Easy
Scaling People: Hard

35. User Feedback:
Automate it!

36. Inviso
Developed by Netflix: https://github.com/Netflix/inviso

37. Hadoop Report Card
• Contains Statistics
• Guidelines and Best
Practices
• Sent Quarterly

38. RealTime Use Cases
• Expanding our use of Storm for:
• TargetingAds based on genres
• Visualizing Data
• Quicker recommendations
• More information:
• https://labs.spotify.com/2015/01/05/how-spotify-scales-apache-storm/

39. Two takeaways
• Getting data into Hadoop is halfthe challenge.Think early
and often about scale.
• Increasing infrastructure reliability and performance leads to
expanded use.This adds challenges but it’s a good problem
to have.

40. Join The Band!
Engineers needed in NYC, Stockholm
http://spotify.com/jobs