The document discusses video coding techniques for compression and transmission. It covers traditional hybrid video coding standards using motion compensation (H.261, H.263, MPEG), as well as newer techniques like wavelet video coding, error resilient transmission, rate-scalable coding, and distributed video coding without layers. These newer techniques can provide better rate-distortion performance than standard codecs or more graceful quality degradation over lossy networks.

1. Video Coding For Compression . . . and Beyond Bernd Girod I nformation Systems Laboratory Department of Electrical Engineering Stanford University Compression

2. Bit Consumption of US Households Bit equivalent, assuming state-of-the-art compression, year 2000 [Source: UC Berkeley: How much Information] 0.0003% Internet 0.6% Video games 3.3% Home video 0.0002% Magazines 0.0002% Books 0.0003% Newspaper 0.4% Recorded Music 1.7% Radio 94% Television ~230 Exabyte/year Total for 70M households

3. Desirable Compression Ratios DSL ~200 kbps ~ 1,000 : 1 Dial-up modem, wireless link ~ 20 kbps ~ 10,000 : 1 ITU-R 601 166 Mbps SDTV broadcasting ~2 Mbps ~ 100 : 1 CIF QCIF

6. “ It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “ In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

7. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -

8. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in ¼-pixel accuracy Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -

9. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Adaptive block sizes . . .

10. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Multiple Past Reference Frames

11. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Generalized B-Frames Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -

13. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~15%

14. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference >25%

15. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~40%

17. Surprising Success of ITU-T Rec. H.263 ?? What H.263 was developed for . . . Analog videophone . . . and what is was used for. Internet video streaming

21. Lifting P U Even Frames Synthesis: Odd Frames Low Band High Band P U Even Frames Analysis: Odd Frames Low Band High Band Motion Compensation [Secker & Taubman, 2001] [Popescu & Bottreau, 2001]

23. Wavelet Synthesis with Lossy Motion Vector MC Wavelet Transform Motion Estimator Embedded Encoding Embedded Encoding Decoder Decoder Inverse Wavelet Transform Video in Video out [Taubman & Secker, ICIP03] Minimize J=D+  R Minimize J=D+  R

24. R-D Performance with Lossy Motion Vector [Taubman & Secker, VCIP 2003] courtesy D. Taubman Bit - Rate (kbps) Video PSNR (dB) 0 200 400 600 800 1000 1200 24 26 28 30 32 34 36 38 40 Embedded wavelet coefficients Lossless motion Non-embedded single-rate Embedded wavelet coefficients Lossy motion CIF Foreman

26. Priority Encoding Transmission (PET) information symbols block of packets [Albanese, Blömer, Edmonds, Luby, Sudan, 19 96] [Davis & Danskin, 1996] [Horn, Stuhlmuller, Link, Girod, 1999] [Puri, Ramchandran, 1999] [Mohr, Riskin, Ladner, 2000] [Stankovic, Hamzaoui, Xiong, 2002] [Chou, Wang, Padmanabhan, 2003] . . . and many more . . . … redundancy symbols enhancement layer base layer Reed-Solomon codeword K N-K packet network

27. Packet Delay Jitter and Loss delay     pdf loss lead-time loss probability lead-time loss probability

28. Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001] Request stream Rate-distortion preamble Packet Schedule Video packets Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule

31. Markov Decision Tree for One Packet ... N transmission opportunities before deadline send: 1 ack: 1 0 0 0 send: 1 0 send: 1 0 ack: 1 0 1 0 1 0 0 1 1 1 0 0 0 0 t current t current +  t t current +2  t Action Observation “ Policy“ minimizing J = D +  R

36. Wyner-Ziv MPEG Codec [Rane, Aaron, Girod, VCIP 2004] Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S * MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder

37. Graceful Degradation with Forward Error Protection Main Stream @ 1.092 Mbps FEC (n,k) = (40,36) FEC bitrate = 120 Kbps Total = 1.2 Mbps WZ Stream @ 270 Kbps FEP (n,k) = (52,36) WZ bitrate = 120 Kbps Total = 1.2 Mbps

38. Visual Comparison of Degradation at Same PSNR With FEC 1 Mbps + 120 kbps (38.32 db) Foreman 50 CIF frames @ symbol error rate = 4 x 10 -4 With FEP 1 Mbps + 120 kbps (38.78 db)

39. Superior Robustness of FEP With FEC 1 Mbps + 120 kbps (33.03 db) Foreman 50 CIF frames @ symbol error rate = 10 -3 With FEP 1 Mbps + 120 kbps (38.40 db)

40. Lossy Compression with Side Information Source Encoder Decoder Source Encoder Decoder [Wyner, Ziv, 1976] For mse distortion and Gaussian statistics, rate-distortion functions of the two systems are the same .

41. Ultra-Low-Complexity Video Coding Interframe Decoder Intraframe Encoder K’ Interpolation/ Extrapolation Key frames K Conventional Intraframe coding Conventional Intraframe decoding X’ Scalar Quantizer Turbo Encoder Buffer WZ frames X Turbo Decoder Request bits Slepian-Wolf Codec Reconstruction Y [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003]

43. Ultra-Low-Complexity Video Coder H263+ Intraframe Coding 330 kbps, 32.9 dB Wyner-Ziv Codec 274 kbps, 39.0 dB

44. Ultra-Low-Complexity Video Coder H263+ I-B-I-B 276 kbps, 41.8 dB Wyner-Ziv Codec 274 kbps, 39.0 dB

45. Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

46. Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

47. Light Field Compression Rate: 0.11 bpp PSNR 39.9 dB Rate: 0.11 bpp PSNR 37.4 dB Wyner-Ziv, Pixel-Domain JPEG-2000

49. Acknowledgments Anne M. Aaron Jacob Chakareski Philip A. Chou J=D+  R Markus Flierl Sang-eun Han Mark Kalman Marc Levoy Yi Liang Shantanu Rane David Rebollo-Monedero Andrew Secker David Taubman Thomas Wiegand Xiaoqing Zhu Rui Zhang

50. Progress is a wonderful thing, if only it would stop . . . Robert Musil