In this talk, we will describe globo.com's live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .
Nginx is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.
this was presented by leandro moreira and juarez bochi at nginx.conf in 09/23/15
2. an amazing teamwork
Leandro
Moreira
other teamsDaniel
Martins
Flávio
Ribeiro
Juarez
Bochi
Hugo
Roque
Robson
Soares
Rafael
Borsani
Bernardo
Camilo
Lucas
Mundim
Juan
Almeida
more people
5. Agenda (4+ years of work)
2010 Architecture for FIFA World Cup 10
2012 Mobile support
2013 HLS adoption
2014 DVR
2014 Waiting room
2014 FIFA World Cup results
2015 Recap & Next steps
22. {2013} instrumentation
Graphite
● status code
● request t.
● cache hit
● cpu usage
● memory
● buffer rate
● bitrate
time-series data
graph api
http
enqueue
dequeuemetric x t
snmp
metric x t
agents
26. {2014} OS fine tuning
19Gpbs maxconn: 70k, cpu affinity
15Gpbs maxconn: 40k, cpu affinity
12Gpbs maxconn: 25k, irqbalance
packet loss
4Gbps, no fine tuning
*each machine has 2 NIC bonded (10GB each one)
27. {2014} what we’ve got here is
Scalabilityport:80 UX Instrumentation
2G
4G
Wifi
1080p
720p
320p
Caching
45. ● > 500k simult. users (peak 1 game)
● ~= 125.000 video req/s (peak 1 game)
● > 640 Gbps (peak 1 game)
FIFA 2014 World Cup Brazil data
46. FIFA 2014 World Cup Brazil data
● ~= 1600 watched years (sum all)
● > 40M video views (sum all)
● ~= avg 6 Gbps, max 19 Gbps
47. ● ~= cpu usage max 10% (per node, FE)
● ~= avg bitrate 480 Kbps (crowded)
● ~= avg bitrate 1864 Kbps (less crowded)
FIFA 2014 World Cup Brazil data