These are the slides from my Demuxed 2017 talk. VOD will be available in a few weeks. Original description: Think you know all there is to know about different video formats, codecs, and containers? Your users are here to prove you wrong! Let's take a quick dive the insanity of petabytes of user uploads on a top video website.

1. Things Developers Believe About Video Files
(Proven Wrong by User Uploads)
derekb@vimeo.com
@daemon404
Derek Buitenhuis
5 October 2017
San Francisco, USA

2. /whois derekb
15 October 2017
• Senior Video Encoding Engineer at Vimeo
• Distributed chunk transcoder
• On-the-fly image service (before it was cool)
• HLS/DASH on-the-fly packager
• Open Source Multimedia Troll^WDeveloper
• FFmpeg, L-SMASH, FFMS2, x264, etc.
• Daemon404 @ FreeNode
• VideoLAN Non-Profit Association Member

3. Everything’s Great Until Users Show Up
25 October 2017

4. Everything’s Great Until Users Show Up
25 October 2017
• Most users are well behaved (H.264 + AAC in MOV/MP4)

5. Everything’s Great Until Users Show Up
25 October 2017
• Most users are well behaved (H.264 + AAC in MOV/MP4)
• Others…
• In the last 90 or so days:
• 119 Video Codecs
• 73 Audio Codecs
• 39 Container
• N Combinations of the above

6. Everything’s Great Until Users Show Up
25 October 2017
• Most users are well behaved (H.264 + AAC in MOV/MP4)
• Others…
• In the last 90 or so days:
• 119 Video Codecs
• 73 Audio Codecs
• 39 Container
• N Combinations of the above
• Observation: Users with the worst files tend to complain the loudest

7. Everything’s Great Until Users Show Up
25 October 2017
• Most users are well behaved (H.264 + AAC in MOV/MP4)
• Others…
• In the last 90 or so days:
• 119 Video Codecs
• 73 Audio Codecs
• 39 Container
• N Combinations of the above
• Observation: Users with the worst files tend to complain the loudest
• Fixing the worst files has lead to a much more robust all around product

8. The “Obvious” Ones
35 October 2017

9. The “Obvious” Ones
35 October 2017

10. The “Obvious” Ones
35 October 2017

11. The “Obvious” Ones
35 October 2017
• That properties cannot change between frames

12. Things Change…
3.55 October 2017
• Pixel packing format
• Color family
• Color primaries
• Transfer function
• Color matrix
• Resolution
• Frame rate (more on this later!)
• Color range
• Codec
• Sampling rate
• Aspect ratio
• Bit depth
• Whether it’s interlaced or not
• Whether it’s telecined or not
• Field order
• Chroma location
• Number of audio channels
• Channel layout
• Audio sample format
• About 10 billion more!

13. The “Obvious” Ones
35 October 2017
• That properties cannot change between frames

14. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)

15. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)
• That non-constant timestamps mean you don’t have a framerate
• Oscillating timestamps due to use of 90kHz time base
• Insane android “constant” frame rate based off camera timings (?)

16. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)
• That non-constant timestamps mean you don’t have a framerate
• Oscillating timestamps due to use of 90kHz time base
• Insane android “constant” frame rate based off camera timings (?)
• That even when you do have a constant frame rate, that you play at that rate
• mvhd rate and edts media_rate in ISOBMFF / MOV

17. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)
• That non-constant timestamps mean you don’t have a framerate
• Oscillating timestamps due to use of 90kHz time base
• Insane android “constant” frame rate based off camera timings (?)
• That even when you do have a constant frame rate, that you play at that rate
• mvhd rate and edts media_rate in ISOBMFF / MOV
• That you don’t need to do rate conversion
• Example: sub-1fps slideshow videos → bad seeking, bad subtitles

18. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)
• That non-constant timestamps mean you don’t have a framerate
• Oscillating timestamps due to use of 90kHz time base
• Insane android “constant” frame rate based off camera timings (?)
• That even when you do have a constant frame rate, that you play at that rate
• mvhd rate and edts media_rate in ISOBMFF / MOV
• That you don’t need to do rate conversion
• Example: sub-1fps slideshow videos → bad seeking, bad subtitles
• That every frame will have a timestamp

19. The “Obvious” Ones: Mostly Timestamps
35 October 2017
• That properties cannot change between frames
• That you’ll have a frame rate at all (variable framerate)
• That non-constant timestamps mean you don’t have a framerate
• Oscillating timestamps due to use of 90kHz time base
• Insane android “constant” frame rate based off camera timings (?)
• That even when you do have a constant frame rate, that you play at that rate
• mvhd rate and edts media_rate in ISOBMFF / MOV
• That you don’t need to do rate conversion
• Example: sub-1fps slideshow videos → bad seeking, bad subtitles
• That every frame will have a timestamp
• That discontinuous timestamps are invalid
• OGG, MPEG-TS

20. 45 October 2017
• That you shouldn’t display negative timestamped samples
• AllF entry in udta atom
Yes, Timestamps Are Pain, Have Some More

21. 45 October 2017
• That you shouldn’t display negative timestamped samples
• AllF entry in udta atom
• That you don’t need sample durations
• Duration of the last frame + sub-1fps slideshows
Yes, Timestamps Are Pain, Have Some More

22. 45 October 2017
• That you shouldn’t display negative timestamped samples
• AllF entry in udta atom
• That you don’t need sample durations
• Duration of the last frame + sub-1fps slideshows
• That you’ll have sample durations
• Looking at you, Matroska
Yes, Timestamps Are Pain, Have Some More

23. 45 October 2017
• That you shouldn’t display negative timestamped samples
• AllF entry in udta atom
• That you don’t need sample durations
• Duration of the last frame + sub-1fps slideshows
• That you’ll have sample durations
• Looking at you, Matroska
• That these will match the timestamp deltas
Yes, Timestamps Are Pain, Have Some More

24. Less Obvious: That Was a Fun Day of Debugging
55 October 2017

25. Less Obvious: That Was a Fun Day of Debugging
55 October 2017
• That you can seek in the input without pre-indexing every packet
• IPPPPPPP
• Periodic Intra Refresh

26. Less Obvious: That Was a Fun Day of Debugging
55 October 2017
• That you can seek in the input without pre-indexing every packet
• IPPPPPPP
• Periodic Intra Refresh
• That coded samples match what should be displayed
• Edit lists, ordered chapters

27. Less Obvious: That Was a Fun Day of Debugging
55 October 2017
• That you can seek in the input without pre-indexing every packet
• IPPPPPPP
• Periodic Intra Refresh
• That coded samples match what should be displayed
• Edit lists, ordered chapters
• That you only need to display what’s in the file
• Audio silence gap-filling, post-stream black frames

28. Less Obvious: That Was a Fun Day of Debugging
55 October 2017
• That you can seek in the input without pre-indexing every packet
• IPPPPPPP
• Periodic Intra Refresh
• That coded samples match what should be displayed
• Edit lists, ordered chapters
• That you only need to display what’s in the file
• Audio silence gap-filling, post-stream black frames
• That you should display every packet/sample
• VP8, VP9, AV1

29. Less Obvious: That Was a Fun Day of Debugging
55 October 2017
• That you can seek in the input without pre-indexing every packet
• IPPPPPPP
• Periodic Intra Refresh
• That coded samples match what should be displayed
• Edit lists, ordered chapters
• That you only need to display what’s in the file
• Audio silence gap-filling, post-stream black frames
• That you should display every packet/sample
• VP8, VP9, AV1
• That you only need to play one track of a kind at once
• Multiple default-on MOV tracks

30. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved

31. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved
• That packets will even be interleaved at all

32. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved
• That packets will even be interleaved at all
• That packets will even be vaguely in-order

33. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved
• That packets will even be interleaved at all
• That packets will even be vaguely in-order
• That you should use the bitstream color properties
• That you should use the container color properties

34. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved
• That packets will even be interleaved at all
• That packets will even be vaguely in-order
• That you should use the bitstream color properties
• That you should use the container color properties
• That you shouldn’t use both

35. Less Obvious: That Was a Fun Week of Debugging
65 October 2017
• That packets will be well interleaved
• That packets will even be interleaved at all
• That packets will even be vaguely in-order
• That you should use the bitstream color properties
• That you should use the container color properties
• That you shouldn’t use both
• That the color properties will even map to some existing standard
• MOV’s embedded ICC profile in the colr atom

36. “multimedia is basically neverending pain”
75 October 2017
— wm4, mpv-player lead

37. 75 October 2017
• That file actually has an end
• Segment linking or dref atom loop (A → B → A → B → …)
• LOOP entry in udta atom
“multimedia is basically neverending pain”
— wm4, mpv-player lead

38. 75 October 2017
• That file actually has an end
• Segment linking or dref atom loop (A → B → A → B → …)
• LOOP entry in udta atom
• That the video track is even a video “codec”
• PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV
“multimedia is basically neverending pain”
— wm4, mpv-player lead

39. 75 October 2017
• That file actually has an end
• Segment linking or dref atom loop (A → B → A → B → …)
• LOOP entry in udta atom
• That the video track is even a video “codec”
• PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV
• That you can easily downmix or play a stereo track on a mono speaker
• Phase cancellation is great fun
“multimedia is basically neverending pain”
— wm4, mpv-player lead

40. 85 October 2017
• Non-bit-exact IDCT in MPEG-2 and MPEG-4
“multimedia is basically neverending pain”
— wm4, mpv-player lead

41. 85 October 2017
• Everything is MP3:
“multimedia is basically neverending pain”
— wm4, mpv-player lead

42. Tools, Sources, etc.
85 October 2017
• ffprobe: http://git.videolan.org/?p=ffmpeg.git
• boxdumper: https://github.com/l-smash/l-smash
• bxd: https://github.com/dwbuiten/bxd
• Beyond Compare is great as well
• ffms2: https://github.com/FFMS/ffms2/
• dvbsnoop: http://dvbsnoop.sourceforge.net/
• JPEG/MP3 sample epic bug report:
• https://github.com/mpv-player/mpv/issues/3973
• Source of IDCT Images:
• https://guru.multimedia.cx/the-mpeg124-and-h26123-idct

43. 95 October 2017
Thanks!
Questions (or 🍅)?