More Related Content Similar to Inside the ABC's new Media Transcoding system, Metro (20) Inside the ABC's new Media Transcoding system, Metro2. metro - Media Transcoder
Daphne Chong
Nick McHardy
@daphnechong
@nickmchardy
4. metro - Media Transcoder
Project Background
- What is Transcoding?
- Examples
- FFmpeg
- Why did we build our own?
5. metro - Media Transcoder
Architecture
- Architecture overview
- Technology choices
- Transcoder autoscaling policies
- Costs
6. metro - Media Transcoder
Future Roadmap
- Cost optimisations
- Validation and Introspection
- Features
7. metro - Media Transcoder
Project Background:
What is
Transcoding?
9. metro - Media Transcoder
Converting video
to different formats
14. metro - Media Transcoder
Project Background:
Transcoding
Example
23. metro - Media Transcoder
other rendition sizes
1000k
650k
500k
220k
audio only
28. metro - Media Transcoder
GXF, ABC1
Profile:
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
30. metro - Media Transcoder
GXF, ABC1
MOV, Arts
Profile:
- Adjust ratio
- ProRes MOV to MP4
34. metro - Media Transcoder
"hi+": {
"transcoder": "ffmpeg",
"transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,
scale=800x450,setdar=16/9[main], [1:v]scale=42:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-37:
main_h-overlay_h-34" -crf 22 -maxrate 904k -bufsize 1000k -pix_fmt yuv420p -profile:v high -level
4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2,
dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]"
-movflags +faststart -threads 0 -y",
"bitrate": "1000k",
"outputfilemimetype": "video/mp4"
},
"hi++": {
"transcoder": "ffmpeg",
"transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,
scale=1024x576,setdar=16/9[main], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:
main_h-overlay_h-44" -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -
level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1]
amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0
-map "[aout]" -movflags +faststart -threads 0 -y",
"bitrate": "1500k",
"outputfilemimetype": "video/mp4"
}
FFmpeg filters are fun
35. metro - Media Transcoder
af_sidhaiompress f_senmd vfropdett vf_kerndeint vf_signalstats af_adelay af_sileedett f_zmq vfurves
vf_lenorrtion vf_smartblur af_aho af_sileeremove fifo vf_tdnoiz vf_libopev vf_spp af_aemphasis
af_sofalizer filtfmts vf_deband vf_lut vf_ssim af_afade af_stereotools formats vf_dimate vf_lut3d vf_stk
af_aformat af_stereowiden framepool vf_dejudder vf_maskedmerge vf_stereo3d af_agate af_tremolo
framesy vf_delogo vf_deint vf_subtitles af_alimiter af_vibrato generate_wave_table vf_deshake
vf_mergeplanes vf_super2xsai af_amerge af_volume graphdump vf_deteline vf_mpdimate vf_swapuv
af_amix af_volumedett graphparser vf_disple vf_neighbor vf_teline af_anequalizer allfilters lavfutils
vf_drawbox vf_noise vf_thumbnail af_anull asink_anullsink log2_tab vf_drawtext vf_null vf_tile af_apad
as_anoises lswsutils vf_edgedett vf_r vf_tinterle af_aphaser as_anulls opel_allkernels vf_elbg vf_overlay
vf_transpose af_apulsator as_flite pthread vf_eq vf_owdenoise vf_unsharp af_aresample as_sine setpts
vf_extrtplanes vf_pad vf_uspp af_asetnsamples audio settb vf_fade vf_palettegen vf_vtorope af_asetrate
avf_aphasemeter split vf_fftfilt vf_paletteuse vf_vflip af_ashowinfo avf_avtorope s_movie vf_field
vf_persptive vf_vidstabdett af_astats avfoat transform vf_fieldmah vf_phase vf_vidstabtransform af_asyts
avf_shoqt trim vf_fieldorder vf_pixdetest vf_vignette af_atempo avf_showfreqs unsharp_opel vf_find_rt
vf_pp vf_w3fdif af_biquads avf_showsptrum vf_alphamerge vf_format vf_pp7 vf_waveform af_bs2b
avf_showvolume vf_aspt vf_fps vf_psnr vf_xbr afhannelmap avf_showwaves vf_atadenoise vf_framepk
vf_pullup vf_yadif afhannelsplit avfilter vf_bbox vf_framerate vf_qp vf_zoompan afhorus avfiltergraph
vf_blkdett vf_framestep vf_random vf_zale afompand bbox vf_blkframe vf_frei0r vf_removegrain video
afompensationdelay buffersink vf_blend vf_fspp vf_removelogo vidstabutils af_shift buffers vf_boxblur
vf_geq vf_repeatfields vsink_nullsink af_dynaudnorm deshake_opel vfhromakey vf_gradfun vf_rotate
vsellauto af_earwax drawutils vfodview vf_hflip vf_sab vs_life af_extrastereo dualinput vfolorbalae
vf_histeq vf_ale vs_mandelbrot af_flanger f_drawgraph vfolohannelmixer vf_histogram vf_seltivolor
vs_mptests af_join f_ebur128 vfolorkey vf_hqdn3d vf_separatefields vs_tests af_ladspa f_interleave
vfolorlevels vf_hqx vf_setfield window_fu af_pan f_perms vfolormatrix vf_hue vf_showinfo af_replaygain
f_realtime vfopy vf_idet vf_showpalette af_resample f_reverse vfover_rt vf_il vf_shuffleframes
af_rubberband f_selt vfrop vf_interle vf_shuffleplanes
other FFmpeg filters
41. metro - Media Transcoder
GXF, ABC1
Profile:
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
43. metro - Media Transcoder
Project Background:
Why did we
build Metro?
72. metro - Media Transcoder
Original
length
Time to produce rendition
60 min
7 min
42 min
Lowest
(audio)
Highest
(1500k)
75. metro - Media Transcoder
Node + Go
- AWS: EC2, SQS,
- S3, Dynamo/RDS
78. metro - Media Transcoder
Node.js
- Existing team experience
- Rapid development
- Solid choice for APIs
- Reservations for long lived processes
- Untyped. Sadface.
81. metro - Media Transcoder
Golang
- Typed and compiled
- Cross-platform
- Easy deployment
- Concurrency (we don’t use this heavily)
82. metro - Media Transcoder
Golang
- Reasonable learning curve
- Magic interfaces
- Different techniques for testing
- Dependencies are challenging
83. metro - Media Transcoder
AWS - EC2
Elastic Compute
Load Balancing
Autoscaling
87. metro - Media Transcoder
SQS
- First in first out (FIFO)
- Single consumer per message
- Message has “visibility timeout”
- Adjustable visibility timeout
90. metro - Media Transcoder
S3 - File Storage
- Store originals + output files
- Purge content after 7 days
95. metro - Media Transcoder
DynamoDB
- NoSQL
- Key-value pair storage
- Managed hosting
- Pay per record retrieval throughput
96. metro - Media Transcoder
DynamoDB
- Our use case was not optimal
- No aggregations = no stats
- Higher cost than expected
- Doesn’t scale automatically
97. metro - Media Transcoder
MySQL
- Switched to MySQL on RDS
- Managed hosting
- Aggregations!
- Data structure ported as-is
98. metro - Media Transcoder
Node + Go
- AWS: EC2, SQS,
- S3, Dynamo/RDS
99. metro - Media Transcoder
System Overview:
Autoscaling
Transcoders
102. metro - Media Transcoder
Scaling Considerations
- Hourly billing
- Instance size: Cost vs Speed
- Content priority
- Ad-hoc requests
- Varied content length
103. metro - Media Transcoder
AWS limitations
- AWS scaling metrics can be slow
- No automatic off switch
- Can’t choose instances to scale down
- (Instance protection now available)
108. metro - Media Transcoder
Time vs Cost
for 1 hour of content
Time Cost
?
110. metro - Media Transcoder
Up: step scaling
Down: stop everything
123. metro - Media Transcoder
EC2 Instances: $34.31
EC2 (other): $ 4.24
RDS: $ 2.66
S3: $ 1.65
SQS: $ 0.00
Total: $42.86
124. metro - Media Transcoder
Next 24 hours
Approx 70% of the volume
136. metro - Media Transcoder
On demand: 29c / hour
Spot: 4c / hour
c4.xlarge
139. metro - Media Transcoder
Future Roadmap:
Introspection &
Validation
140. metro - Media Transcoder
GXF, ABC1
Profile:
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
154. metro - Media Transcoder
We had opportunity
for a new system
161. metro - Media Transcoder
www.iridetheharlemline.com/2013/05/13/a-commuters-rainbow
162. metro - Media Transcoder
http://developers.digital.abc.net.au/
we’re hiring :-)