SlideShare a Scribd company logo
1 of 20
Download to read offline
Netflix Cloud Architecture
and Open Source
Andrew Spyker (@aspyker)
Engineering Manager, Netflix
About Netflix
● 109M+ members
● 2000+ employees (1400 tech)
● 80+ countries
● > 100M hours watch per day
● > ⅓ NA internet download traffic
● 500+ Microservices
● 100,000+ VM’s
● 3 regions across the world
Netflix Github -
http://netflix.github.io
Why does Netflix open source?
Improve Engineering
● Great feedback from wider community
● Collaborate through open code
Recruit new and retain engineering talent
● Hard problems are openly worked on
Industry Alignment
● Content and cloud technologies
Elastic, Web and Hyper Scale
Doing this
Not doing that
…
…
…
…
…
…
•
•
•
•
•
•
•
•
…Geo DNS
A truly global service
● Replicate data across
regions
● Be able to redirect traffic
from region to region
● Be able to migrate
regional traffic to other
regions
● Have automated control
across regions Flux Demo
Testing is only way to prove HA
● Chaos Monkey
○ Kill instances in production - runs regularly
● Chaos Kong
○ Kill entire region and shift traffic globally
○ Run frequently but with prior scheduling
… …
Spinnaker
Works at
Netflix
scale
Views of
global
pipelines
From simple
deployment to
advanced CI/CD
pipelines
……
…
…
Security
Dynamic
Security
Done in new ways
NOT
Dynamic, Web Scale & Simpler Security
Security Monkey
● Monitors security policies, tracks changes, alerts on situations
Scumblr
● Searches internet for security “nuggets” (credentials, hacking discussions)
Sketchy
● A safe way to collect text and screenshots from websites
Sleepy Puppy
● Delayed cross site scripting propagation testing framework
Lemur
● x.509 certificate orchestration framework
Stethoscope
● Automated security monitoring and alert for employees
What did we not cover?
Big Data, Data Persistence and UI Engineering
● Big Data tools used well beyond Netflix
● Ephemeral, semi and fully persistent data systems
● Recent addition of UI OSS and Falcor
Other parts of the Netflix business
● Personalization and Algorithms
● Content Encoding and Delivery
● Studio and Partners facing applications
Questions
?

More Related Content

What's hot

Season 7 Episode 1 - Tools for Data Scientists
Season 7 Episode 1 - Tools for Data ScientistsSeason 7 Episode 1 - Tools for Data Scientists
Season 7 Episode 1 - Tools for Data Scientists
aspyker
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
Ruslan Meshenberg
 

What's hot (20)

Velocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ NetflixVelocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ Netflix
 
Netflix OSS Meetup Season 5 Episode 1
Netflix OSS Meetup Season 5 Episode 1Netflix OSS Meetup Season 5 Episode 1
Netflix OSS Meetup Season 5 Episode 1
 
Container World 2018
Container World 2018Container World 2018
Container World 2018
 
Herding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes PublicHerding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes Public
 
The new Netflix API
The new Netflix APIThe new Netflix API
The new Netflix API
 
Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015
 
Netflix Cloud Platform and Open Source
Netflix Cloud Platform and Open SourceNetflix Cloud Platform and Open Source
Netflix Cloud Platform and Open Source
 
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
 
CS80A Foothill College Open Source Talk
CS80A Foothill College Open Source TalkCS80A Foothill College Open Source Talk
CS80A Foothill College Open Source Talk
 
Re:invent 2016 Container Scheduling, Execution and AWS Integration
Re:invent 2016 Container Scheduling, Execution and AWS IntegrationRe:invent 2016 Container Scheduling, Execution and AWS Integration
Re:invent 2016 Container Scheduling, Execution and AWS Integration
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
Season 7 Episode 1 - Tools for Data Scientists
Season 7 Episode 1 - Tools for Data ScientistsSeason 7 Episode 1 - Tools for Data Scientists
Season 7 Episode 1 - Tools for Data Scientists
 
Monitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloudMonitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloud
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
 
Dev309 from asgard to zuul - netflix oss-final
Dev309  from asgard to zuul - netflix oss-finalDev309  from asgard to zuul - netflix oss-final
Dev309 from asgard to zuul - netflix oss-final
 
Arc305 how netflix leverages multiple regions to increase availability an i...
Arc305 how netflix leverages multiple regions to increase availability   an i...Arc305 how netflix leverages multiple regions to increase availability   an i...
Arc305 how netflix leverages multiple regions to increase availability an i...
 
NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1
 
The Art of Decomposing Monoliths - Kfir Bloch, Wix
The Art of Decomposing Monoliths - Kfir Bloch, WixThe Art of Decomposing Monoliths - Kfir Bloch, Wix
The Art of Decomposing Monoliths - Kfir Bloch, Wix
 
Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015
Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015
Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015
 
Netflix viewing data architecture evolution - EBJUG Nov 2014
Netflix viewing data architecture evolution - EBJUG Nov 2014Netflix viewing data architecture evolution - EBJUG Nov 2014
Netflix viewing data architecture evolution - EBJUG Nov 2014
 

Similar to Netflix Cloud Architecture and Open Source

Netflix’s Success through Technology and Culture - Andicom 2014
Netflix’s Success through Technology and Culture - Andicom 2014Netflix’s Success through Technology and Culture - Andicom 2014
Netflix’s Success through Technology and Culture - Andicom 2014
Mikey Cohen - Hiring Amazing Engineers
 

Similar to Netflix Cloud Architecture and Open Source (20)

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthUSENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
 
Netflix Architecture and Open Source
Netflix Architecture and Open SourceNetflix Architecture and Open Source
Netflix Architecture and Open Source
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
 
Engineering Leader opportunity @ Netflix - Playback Data Systems
Engineering Leader opportunity @ Netflix - Playback Data SystemsEngineering Leader opportunity @ Netflix - Playback Data Systems
Engineering Leader opportunity @ Netflix - Playback Data Systems
 
How we scale up our architecture and organization at Dailymotion
How we scale up our architecture and organization at DailymotionHow we scale up our architecture and organization at Dailymotion
How we scale up our architecture and organization at Dailymotion
 
Stream Processing in Uber
Stream Processing in UberStream Processing in Uber
Stream Processing in Uber
 
Distributed architecture in a cloud native microservices ecosystem
Distributed architecture in a cloud native microservices ecosystemDistributed architecture in a cloud native microservices ecosystem
Distributed architecture in a cloud native microservices ecosystem
 
Netflix’s Success through Technology and Culture - Andicom 2014
Netflix’s Success through Technology and Culture - Andicom 2014Netflix’s Success through Technology and Culture - Andicom 2014
Netflix’s Success through Technology and Culture - Andicom 2014
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
 
OpenKilda: Stream Processing Meets Openflow
OpenKilda: Stream Processing Meets OpenflowOpenKilda: Stream Processing Meets Openflow
OpenKilda: Stream Processing Meets Openflow
 
Closing Keynote
Closing KeynoteClosing Keynote
Closing Keynote
 
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
 
Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)
 
Sobanski odl summit_2015
Sobanski odl summit_2015Sobanski odl summit_2015
Sobanski odl summit_2015
 
Bringing DevOps to Routing with evolved XR: an overview
Bringing DevOps to Routing with evolved XR: an overviewBringing DevOps to Routing with evolved XR: an overview
Bringing DevOps to Routing with evolved XR: an overview
 
Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016
 
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
 
Druid meetup @ Netflix (11/14/2018 )
Druid meetup @ Netflix  (11/14/2018 )Druid meetup @ Netflix  (11/14/2018 )
Druid meetup @ Netflix (11/14/2018 )
 
Managing python at scale without breaking the bank
Managing python at scale without breaking the bankManaging python at scale without breaking the bank
Managing python at scale without breaking the bank
 
Netty training
Netty trainingNetty training
Netty training
 

More from aspyker

Ibm cloud nativenetflixossfinal
Ibm cloud nativenetflixossfinalIbm cloud nativenetflixossfinal
Ibm cloud nativenetflixossfinal
aspyker
 
Netflix s2e1lightningtalk
Netflix s2e1lightningtalkNetflix s2e1lightningtalk
Netflix s2e1lightningtalk
aspyker
 

More from aspyker (11)

SRECon Lightning Talk
SRECon Lightning TalkSRECon Lightning Talk
SRECon Lightning Talk
 
Netflix OSS Meetup Season 4 Episode 4
Netflix OSS Meetup Season 4 Episode 4Netflix OSS Meetup Season 4 Episode 4
Netflix OSS Meetup Season 4 Episode 4
 
Netflix Open Source Meetup Season 4 Episode 3
Netflix Open Source Meetup Season 4 Episode 3Netflix Open Source Meetup Season 4 Episode 3
Netflix Open Source Meetup Season 4 Episode 3
 
Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016
 
Netflix Container Runtime - Titus - for Container Camp 2016
Netflix Container Runtime - Titus - for Container Camp 2016Netflix Container Runtime - Titus - for Container Camp 2016
Netflix Container Runtime - Titus - for Container Camp 2016
 
Netflix Open Source Meetup Season 4 Episode 1
Netflix Open Source Meetup Season 4 Episode 1Netflix Open Source Meetup Season 4 Episode 1
Netflix Open Source Meetup Season 4 Episode 1
 
Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2
 
Ibm cloud nativenetflixossfinal
Ibm cloud nativenetflixossfinalIbm cloud nativenetflixossfinal
Ibm cloud nativenetflixossfinal
 
Docker Demo IBM Impact 2014
Docker Demo IBM Impact 2014Docker Demo IBM Impact 2014
Docker Demo IBM Impact 2014
 
Netflix s2e1lightningtalk
Netflix s2e1lightningtalkNetflix s2e1lightningtalk
Netflix s2e1lightningtalk
 
Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse
Going Cloud Native with IBM Cloud and NetflixOSS for Dev@PulseGoing Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse
Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse
 

Recently uploaded

Recently uploaded (20)

A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 

Netflix Cloud Architecture and Open Source

  • 1. Netflix Cloud Architecture and Open Source Andrew Spyker (@aspyker) Engineering Manager, Netflix
  • 2. About Netflix ● 109M+ members ● 2000+ employees (1400 tech) ● 80+ countries ● > 100M hours watch per day ● > ⅓ NA internet download traffic ● 500+ Microservices ● 100,000+ VM’s ● 3 regions across the world
  • 4. Why does Netflix open source? Improve Engineering ● Great feedback from wider community ● Collaborate through open code Recruit new and retain engineering talent ● Hard problems are openly worked on Industry Alignment ● Content and cloud technologies
  • 5. Elastic, Web and Hyper Scale Doing this Not doing that
  • 7.
  • 10. A truly global service ● Replicate data across regions ● Be able to redirect traffic from region to region ● Be able to migrate regional traffic to other regions ● Have automated control across regions Flux Demo
  • 11. Testing is only way to prove HA ● Chaos Monkey ○ Kill instances in production - runs regularly ● Chaos Kong ○ Kill entire region and shift traffic globally ○ Run frequently but with prior scheduling
  • 12.
  • 14. Spinnaker Works at Netflix scale Views of global pipelines From simple deployment to advanced CI/CD pipelines
  • 15.
  • 18. Dynamic, Web Scale & Simpler Security Security Monkey ● Monitors security policies, tracks changes, alerts on situations Scumblr ● Searches internet for security “nuggets” (credentials, hacking discussions) Sketchy ● A safe way to collect text and screenshots from websites Sleepy Puppy ● Delayed cross site scripting propagation testing framework Lemur ● x.509 certificate orchestration framework Stethoscope ● Automated security monitoring and alert for employees
  • 19. What did we not cover? Big Data, Data Persistence and UI Engineering ● Big Data tools used well beyond Netflix ● Ephemeral, semi and fully persistent data systems ● Recent addition of UI OSS and Falcor Other parts of the Netflix business ● Personalization and Algorithms ● Content Encoding and Delivery ● Studio and Partners facing applications