Presentation from Gluecon 2013 on building and running the Netflix API.

1. Netflix API Crash Course
Building & Running the API in 30 minutes
Ben Schmaus, Netflix
May 2013, Gluecon
bschmaus@netflix.com
@schmaus

2. Streaming TV Shows & Movies Globally

3. > 1000 Devices

4. 1/3 of
Internet
at peak

5. Programmer not Distributor

6. More than 36 million
subscribers in over
40 countries

7. How does the API fit
into the picture?

8. Personalization
Engine
User Info
Movie
Metadata
Ratings
Similar
Movies
Instant
Queue
A/B Test
Engine
API

9. Personalization
Engine
User Info
Movie
Metadata
Ratings
Similar
Movies
Instant
Queue
A/B Test
Engine
API
Enable UX
Innovation
Insulate from
Failure

10. > 2 Billion Requests per Day

11. Growth Over Time

23. Automation
Visibility
Operational awareness
Balance speed
& quality

24. How's the API
put together?

25. ELB
Routing
Cluster
Mid-tier
Services
Backend
App
Cluster
Backend
App
Cluster
+
API Layer

26. ELB
Routing
Cluster
Mid-tier
Services
Backend
App
Cluster
Backend
App
Cluster
+
API Layer

27. Inside an
API
App Server
RxJava
Hystrix
Service Client 1 Service Client 2 Service Client N

28. HystrixRx+Java
Service
Layer
Service Client
(provided JAR)
Application
Service
/device/endpoint
(provided script)
Service
UI
Teams
Mid-tier
Service
Teams
API
Team

29. Continually changing UI scripts and
mid-tier services
Functionality, resiliency and
performance drifts over time

30. Deployment & Ops

31. REMOVE MANUAL WORK pushing
code to multiple AWS regions/clusters
ENABLE RAPID DEPLOYMENT of
code despite limited visibility into how it's
changed
KEEP TEAM INFORMED about what's
happening in prod
MITIGATE RISK of systemic failure

32. Tools

33. End-to-end Traceability Using
Python/Java Glue

34. Code Flow

35. Run 1% of your traffic on the new
code and see how it does

36. API ami-123 API ami-456
2xx
4xx
5xx
latency
busy threads
load
...

37. Manually looking at graphs and SSH-
ing into servers and grep-ing logs
doesn't scale
(although we used to do that)

38. Confidence score for each AMI based on
comparison of 1000+ metrics

39. Scannable visualization of metric space
More
important
Less
important

40. Cross-reference Jira, Link to code diffs

41. Track lib changes

42. Easy to access report artifacts for each AMI

43. Your basic red/black push

48. Doing red/black by hand for multiple
clusters across multiple regions is
not fun

49. Automate multi-cluster/region
pushes

50. Automate multi-cluster/region
pushes
Don't forget to
automate
rollbacks, too!

51. $Who, $What, $Where, $When
e.g., "bschmaus, ami-123, Sandbox Canary, 2013-05-06 19:05"
Latest prod change in chat topic

52. Quickly see status of all clusters in a
region

53. What the #%*! just happened!?

54. Historical & realtime metrics, sort
realtime by error/request rate

55. Distributed grep + tail
2013-05-09.20:38:54 MX 200 us-east-1c i-1824cb73 i-1c61b77f prod
NFPS3-001-8G50FJCX... 288404769389848058 90ms api-global.
netflix.com GET /tvui/release/470/plus/pathEvaluator -
amazon.ami-id: ami-502eb039
amazon.availability-zone: us-east-1c
amazon.instance-id: i-1824cb73
amazon.instance-type: m2.2xlarge
amazon.local-ipv4: 10.6.213.112
amazon.public-hostname: ec2-54-243-4-69.compute-1.amazonaws.
com
amazon.public-ipv4: 54.243.4.69
cookie_esn: NFPS3-001-8G50FJCX...
country: MX
currentTime: 1368131934468
duration-millis: 90
esn: NFPS3-001-8G50FJCX...
geo.city: CIUDADOBREGON
...
$ ./simple_stream.py -f -q 'e["country"]=="MX" && e["esn"]
==~/NFPS3.*/' -r us

56. Go for haystack handing you
the needle

57. Or at least be able to make smaller
haystacks

58. Continuously experiment to make
hard things easier

59. Even with the best tools, building
software is hard work.
Great engineers build great
software.

60. Want to help us build the API?
bschmaus@netflix.com
@schmaus