Distinct concepts seen in Communication Engineering / Multimedia services classes

1. Multimedia Services:
Image
Sep-2015
Dani Gutiérrez Porset
Associate Professor
Communications Engineering
Eman ta zabal zazu

2. 2
2Multimedia Services:
Image
Thanks, Licences and Tools
● Thanks to people and organizations who took or take part
in free software and free knowledge projects, specially
Wikimedia Foundation and KDE
●
This presentation is licensed as CC BY-SA 3.0 ES
http://creativecommons.org/licenses/by-sa/3.0/es/
● This presentation has been made with KDE, LibreOffice,
Inkscape, Gimp, Chromium, Firefox

3. 3
3Multimedia Services:
Image
Sources and References
● Images from Wikimedia Foundation, if not referenced other
source. Logos and trademarks belong to respective organizations
● Texts:
– Wikipedia pages and referenced articles and material
– “Guide to Voice and Video over IP” - Sun, Mkwawa, Jammeh,
Ifeachor
– “Video over IP” - Wes Simpson
– “Computer Networking, a top-down approach” - Kurose, Ross

4. 4
4Multimedia Services:
Image
Index
● Introduction
● Color
● Raster images
● Compression
● Image files and formats

5. 5
5Multimedia Services:
Image
Visual perception in humans
● Images and colors are sensations in our brain
● Photoreceptors in eye:
– Rods, for brightness
– LMS cones for color:
● S (short wavelength = blue)
● M (medium wavelength = green)
● L (long wavelength = red)
Introduction

6. 6
6Multimedia Services:
Image
Sensitivity of human eye/brain
● More sensible to brightness than color
● Less sensitive to the higher spatial frequency components
● Less sensitive to quantizing distortion at high luminance
levels
● Not all theoretical colors can be perceived (“imaginary
colors“)
● More sensible to green. Actually, color sensitivity depends
on bright
Introduction

7. 7
7Multimedia Services:
Image
Image Devices and
Corrections (1/2)
● Devices:
– Input: camera, eye,...
– Output: monitor, projector, printer,...
● Gamut: which colors can be perceived or generated by a
device
● Corrections:
– For Brightness: Gamma: Mathematical function to consider human
perception of brightness. Relates Luma and Luminance
– For Color: ICC profile: color space that characterizes colors in input
or output device, and uses a Profile Connection Space
Introduction

8. 8
8Multimedia Services:
Image
Image Devices and
Corrections: Workflow (2/2)
● In capturing devices, e.g. camera, when saving an
image:
– If raw format is used no correction applied
– If non-raw format (e.g. JPEG), image is saved with
gamma encoded or compressed
● Each display device has a gamma value to decode
gamma compressed images
Introduction

9. 9
9Multimedia Services:
Image
Types of Digital Still Images
● Raster or Bitmap: represented by a matrix of pixels1
● Vector: represented by vectorial elements: points, vectors (strokes or
paths: bézier curves), text,...
– No quality loss if size or rotation altered.
– for 2D and 3D. E.g.
● 2D: SVG, MathML, POV-Ray
● 3D: Blender .blend, CAD .dwg,...
● Possible to mix raster and vectorial in some formats, e.g. SVG, WMF, EMF,
PDF, EPS, Postscript, SWF (Shockwave Flash, includes animations,...)
1: All of the next slides in this presentation refer to raster images
Introduction

10. 10
10Multimedia Services:
Image
Applications of Raster Images
● Icons: PNG, BMP, SVG
● Web: JPG, PNG, GIF. Streamable using
progressive (interlaced) rendering
● Cameras: RAW, EXIF
● Graphic Edition: formats with lossless
compression, (proprietary formats)
● Print Publishing: TIFF, PDF
Introduction

11. 11
11Multimedia Services:
Image
Pixel Resolution
● For Surface processing devices (e.g. camera,
display)
● Concept: pixel dimensions or Width x height pixels
● Units: pixels (or megapixels,...)
● For an image file it's the real size in pixels
Introduction

12. 12
12Multimedia Services:
Image
Standards for
Pixel Resolution
● CIF (and variations)
● SDTV (Standard Definition TV)
● HDTV (High Definition TV): 1080p, 1080i, 720p
● 4K family
● 8K and Ultra HD TV
● VGA, XGA,...
http://en.wikipedia.org/wiki/Graphics_display_resolution
https://en.wikipedia.org/wiki/Common_Intermediate_Format
https://en.wikipedia.org/wiki/List_of_common_resolutions
Introduction

13. 13
13Multimedia Services:
Image
Aspect ratio
● Ratio of width to height pixels expressed as A:B
● Some frequently used e.g. 4:3, 16:9 (HDTV), 21:9
● Types:
– SAR: Storage Aspect Ratio
– DAR: Aspect Ratio for the Display
– PAR: Pixel Aspect Ratio. PAR = DAR / SAR
● Scaling, cropping and vertical/horizontal bands are
used to adapt SAR to DAR
Introduction

14. 14
14Multimedia Services:
Image
Spatial Resolution
● For line processing devices (e.g. scanner and
printer)
● Concept: granularity in a line
● Units: ppi (pixels per inch) or dpi (dots per inch)
● For an image file the number of dpi or the real size
in metric units are just metadata
Introduction

15. 15
15Multimedia Services:
Image
Introduction to Color
● Color spectrum (increasing wavelengths):
● Cases:
– Device independent models
– Device/eye dependent models
Color

16. 16
16Multimedia Services:
Image
Color models and Color spaces
● Color model: mathematical model that uses tuples to reference colors
– e.g. RGB, CMYK, YUV, HSV, LAB,...
– Conversions between models with matrices
● Color space: organization of colors to represent colors precisely in
terms of perception, coming:
– From a model transformation for a color model, e.g.
RGB: sRGB, Adobe RGB,...
– From a list of colors, e.g. Pantone System
● Profile Connection Space: reference color space used for conversions :
CIELAB and CIEXYZ
● Terms “Model” and “Space” sometimes mixed
Color

17. 17
17Multimedia Services:
Image
RGB Color model
● Red, Green, Blue. RGBA: 4th component A = Alpha
● Additive model
● Device dependent
● Uses: electronic displays
● Cube
● Problems:
– Not efficient with “real-world” images
– Not efficient for some image processing, e.g. modify intensity
Color

18. 18
18Multimedia Services:
Image
CMY, CMYK Color model
● Cyan, Magenta, Yellow
● Subtractive model
● Device dependent
● Derived from old RYB
● Uses: printing processes, with BlacK
Color

19. 19
19Multimedia Services:
Image
Y'UV, YUV, YCbCr, YPbPr,.
Color models
● Components:
– Non-color info (black, gray, white info). Terms:
● Luminance: measured. Symbol: Y
● Luma: perceived. Symbol: Y'
– Chroma or Chrominance. Symbol: C. Similar concepts:
● U, V: color information
● Pb, Pr: Chroma for blue (B − Y) and for red (R − Y). For analog encoding (analog component video)
● Cb, Cr: Idem for digital encodings
● I, Q: In-phase, Quadrature, from rotation of U, V
– Sometimes YUVA: 4th component A = Alpha
● Uses: video systems (Y'UV in PAL, SECAM, YIQ in NTSC)
● Advantages:
– Compatible with black and white analog TV
– Some Chroma info can be discarded (better perception than RGB with higher compression)
Color

20. 20
20Multimedia Services:
Image
HSV, HSL, HSB, HSI
Color models
● Components:
– Hue: distinguishes wich colour is
– Saturation: intensity or colorfulness
– 3rd component: Value, Lightness
(from black to white), Brightness
and/or Intensity
● Cylindrical coordinates. Same RGB
color model represented in other way
● Uses: by digital artists. More intuitive representation for
attributes recognized by human vision
Color

21. 21
21Multimedia Services:
Image
LAB Color model
● LAB or La*b*:
– L: Lightness
– A, B: Chromacity, in color-opponent dimensions
● Device independent
● Designed to approximate human vision
● Wider gamut than RGB and CMYK and human eye. More data per
pixel required
● Uses:
– Conversions from RGB to CMYK
– Interchange, device-independent format
Color

22. 22
22Multimedia Services:
Image
Color depth
● Number of bits to indicate the color:
– For a single pixel: Bits per pixel
– For each color component: “Bits per channel”, “Bits per color” (RGB,
YCbCr, CMYK) or “Bits per sample”. Alpha channel if transparency is
used
● Representations:
– Indexed or palette
– Direct color:
● Number of bits depending on color space: 8, 15, 16, “True” color = 24, “Deep”
color (30, 36, 48) bits. e.g. RGBA 32 bits
● Type e.g. floating values for HDR
Color

23. 23
23Multimedia Services:
Image
Indexed Colors
● Better for images with low number of colors or large
areas with solid colors, e.g. cartoons
● Types of palette:
– Adaptive: optimized for each image, contains the most
frequent colors in image
– Master: contains a miniaturized version of full RGB colors
– ...
● Dithering algorithms are usually needed to transform
from Direct (e.g. RGB) to Palette schema
Color

24. 24
24Multimedia Services:
Image
Lossless compression methods
Compression
RLE or Run-length
encoding
Adaptive dictionary
Deflation = combination
of Lz77 and Huffman
Chain codes
TIFF, BMP,...
LZW GIF, TIFF
PNG, TIFF
Monochrome images

25. 25
25Multimedia Services:
Image
Lossy Image compression
and Human Perception
http://en.wikipedia.org/wiki/Image_compression
To relative differences between
darker tones than lighter tones
To brightness information
than color information
To green than blue and red
Gamma correction
to convert luminance in luma
Chroma subsampling
Luma with more G info than R, B:
Y, Cb, Cr
To spatial low frequency
transitions than high freq
Quantization matrix
for DCT
Human eye is more sensitive... Method to apply:
Compression

26. 26
26Multimedia Services:
Image
YCbCr and Chroma sub-sampling
● Lossy compression method based on Y'CbCr that takes less chroma
samples than luma samples (horizontally and/or vertically), based on
human visual perception
● Taking a region of w x 2 pixels, the notation is w:h:v
– w: pixel width of the region. Generally w=4
– h: number of Cb or number of Cr samples in the 1st row of the region
– v: number of chroma changes samples between 1st and 2nd row of the region
– Exception: not valid for 4:1:0 (region of 4 pixels height)
● Internal packaging in a file or stream: distinct dispositions, e.g. in
Y'UV420p (p=“Planar”) first one component for all pixels, second other
component for all pixels,...
http://en.wikipedia.org/wiki/Chroma_subsampling
http://en.wikipedia.org/wiki/YUV#Y.27UV420p_.28and_Y.27V12_or_YV12.29_to_RGB888_conversion
Compression

27. 27
27Multimedia Services:
Image
Examples of
YCbCr and Chroma sub-sampling
Compression

28. 28
28Multimedia Services:
Image
Other Lossy compression methods
● Palette and color quantization for reduced color
space (GIF, PNG)
● Transform coding, ej. DCT (JPEG), Wavelet (JPEG
2000, DjVu)
● Fractal compression
Compression

29. 29
29Multimedia Services:
Image
Structure of Image files
● Structure:
– File metadata, e.g. Pixel resolution. If chroma subsampling there
maybe distinct pixel blocks
– Possible file chunks and chunk metadata
– Info for each pixel, packed in pixel by pixel schema or in channel
(planes) schema:
● Color: finite or ~ infinite number of colors. Special cases: BW, gray
● Alpha channel: Transparency (0%) - Opacity (100%)
● Image File size = f (chunks, resolution, color, compression)
http://en.wikipedia.org/wiki/Image_file_format
http://en.wikipedia.org/wiki/Comparison_of_image_file_formats
Image files and formats

30. 30
30Multimedia Services:
Image
Image Formats for Internet
● Supported natively in browsers:
– Mandatory: GIF, JPEG, PNG
– Some browsers: SVG, PDF,...
● Interlacing: incremental decoding in browsers, so they render
progressively images showing first a degraded version that
changes to final image:
– Better for human perception. Targeted mainly to slower communications
– Formats:
● GIF or PNG: images saved with "interlaced" option
● JPEG: images saved with "progressive" option
Image files and formats

31. 31
31Multimedia Services:
Image
JPEG compression and file format
● Theorically available compression types:
– Lossy compression in three phases (see appendix)
– Lossless coding, but not supported in most products
● Modes:
– Baseline
– Progressive: Data is compressed in multiple passes of progressively
higher detail. Progressive rendering in browsers
● Can embed an ICC color profile
● Does not support transparency nor animation
● Variations: JPEG 2000 (wavelets compression), JPEG XR,...
Image files and formats

32. 32
32Multimedia Services:
Image
JPEG Encoding flow (1/3):
Chroma subsampling
● 1.a Convert from RGB to Y'CbCr
● 1.b Apply chroma subsampling, from 4:4:4 to
typically 4:2:2 or 4:2:0 [1st compression, lossy]
Image files and formats

33. 33
33Multimedia Services:
Image
JPEG Encoding flow (2/3):
Discard high-frequency variations
● 2.a Split the result image in blocks of 8x8 pixels for luma. MCU
(Minimum Coded Unit) or macroblocks = blocks for luma and
chroma: 8x8 (if 4:4:4), 16x8 (if 4:2:2) or 16x16 (if 4:2:0)
● 2.b Transform to frequency domain with DCT applied to each
macroblock
● 2.c Quantize frequency components. One matrix for luma and
one for chrominance
● 2.d Discard high-frequency coefficients due to human
perception. This gives the bigger or smaller compression [2nd
compression, lossy]
Image files and formats

34. 34
34Multimedia Services:
Image
JPEG Encoding flow (3/3):
Final entropy encoding
● 3. Compress the resulting 8 x8 blocks in zig-zag with
entropy encoding [3rd compression, lossless]
The matrix from 2.d is like: Path for compression in 3:
(Logo in FFmpeg library: )
Image files and formats

35. 35
35Multimedia Services:
Image
PNG file format
● Lossless compression: filter + Deflate algorithm
● 3 Type of images: Palette-based, Grayscale, Direct color RGB[A]
● Color depth: bits/channel x no. of channels (Gray, R,G,B,A)
● Bits/channel: 1, 2, 4, 8, 16 bits/channel (if alpha 64 bits/pixel), e.g.
PNG8: indexed with 8 bits, PNG24: RGB 8 bits/channel, PNG32: PNG24+Alpha
● RGB space color
● Chunks:
– Criticals: Header (width, height, color type, bit depth), Palette, Data
– Others: ICC profile, gamma, text,...
● Interlacing: 2-dimensional, 7-pass
● Does not support animation
Image files and formats

36. 36
36Multimedia Services:
Image
TIFF file format
● Can be seen as a file format, and as a container for lossy and
lossless data
● Much more powerful and complicated than PNG or JPEG
● Applications:
– Base for lot of raw proprietary formats in cameras
– Professional image editing
● Allowed compressions:
– Lossless: CCITT Group 3 and Group 4 bi-level, LZW
– Lossy: JPEG
● Color spaces of images: RGB, CMYK, YCbCr,...
Image files and formats

37. 37
37Multimedia Services:
Image
Formats for image metadata
● Metadata: date and time, geolocation, data about
the image (e.g. orientation),...
● Formats that specify how metadata is inserted into
standard formats (e.g. JPEG, TIFF,...):
– EXIF: Used in digital cameras and scanners
– XMP
– IPTC IIM
Image files and formats