1. Eng: Mohammed Hussein1
Republic of Yemen
THAMAR UNIVERSITY
Faculty of Computer Science&
Information System
Lecturer, and Researcher atThamar University
By Eng: Mohammed Hussein
mohammedhbi@thuniv.net
2. Outline
Introduction
Audio and its digitization
Video and its digitization
Bandwidth requirements
Multimedia transmission requirements
SAS Factors for Audio andVideo
Network Performance parameters
Multimedia over Internet &Wireless
Problems with Internet & solutions
2 Eng: Mohammed Hussein
3. Introduction
The continuous media are played during some
well-defined time interval with user interaction.
The most demanding are audio and video
Audio: what we hear through our ears
Video : what we see through our eyes
3 Eng: Mohammed Hussein
5. What is Streaming???
Streaming media refers to media types with time constraints
and a continuous Video and audio transmission
Playback of streaming media starts while data is being
received, i.e. it is not necessary to download the entire file
before playback starts
Advantage: doesn‘t require to download complete
content first (thus it avoids long delays)
5 Eng: Mohammed Hussein
8. MATLAB SIMULINK
To start the Simulink software, you must first start the MATLAB®
technical computing environment.You can then start the Simulink
software in two ways:
On the toolbar, click the Simulink icon.
Enter the simulink command at the MATLAB prompt.
8 Eng: Mohammed Hussein
9. How we can digitize audio signal?
Basic steps:
1. Conversion to electronic form using microphone (analog signal )
2. Sampling the analog signal based on PAM or PCM.
3. Quantization using Analog to Digital converter
9 Eng: Mohammed Hussein
12. Why we use these signals?
Based on these Signals we can create complex Signals.
Sinusoid, unit step and exponential are used to approximate
basically more complex signals.
They help us to predict and analysis as we take the signals as input
to the system such as filters, high pass, low pass, band pass, etc
filters and see what the output is.
12 Eng: Mohammed Hussein
15. Discrete-time and continuous-time signals
A discrete-time signal is quantities that defined only on a discrete
set of times.A simple source for a discrete time signal is
the sampling of a continuous signal, approximating the signal by a
sequence of its values at particular time instants.
A continuous-time real signal is any real-valued (or complex-
valued) function which is defined at every time t in an interval, most
commonly an infinite interval.
15 Eng: Mohammed Hussein
16. Analog and digital signals
The figure shows a digital signal that results from approximating an
analog signal by its values at particular time instants.
Digital signals are discrete and quantized, while analog signals
possess neither property.
16 Eng: Mohammed Hussein
20. Video
Best way to understand electronic video signal is understand how
we get a picture from a digital camera.
Electronic sensors Camera Gain Calibration (CCD) convert
different levels of light to electronic signals of different amplitudes.
It is useful to know this conversion for evaluating the performance of
the CCD camera.
The light is passed through a RGB filter
20 Eng: Mohammed Hussein
21. Video
Eng: Mohammed Hussein21
The video consist of a sequence of frames
The frames are displayed fast enough to get the impression of
motion.
To get flicker free display, the frames are repeated 50 times per
second. Because the property of human eye at the rate of 50 frames
per second doesn’t identify the changes in the video
Frame 1 Frame 2
22. Digitizing video
Each frame is divided in small grids, called pixels.
For black-and-whiteTV, grey level of each pixel is represented by 8
bits.
In case of color, each pixel is represented by 24 bit, 8-bit for each
primary colors (R G B ).
Assuming that a frame is made of 640*480 pixels, the bandwidth
requirement is 2*25*640*480*24 =368.64 Mbps
The 25 frames repeated 2 times with 24 bits color
This is too high for transmission without compression through the
internet.
22 Eng: Mohammed Hussein
23. Resolution
How many Megapixel of your camera?
Example of resolution a 3.1 Megapixel digital camera
provides the following resolution options:
The highest resolution 2048* 1536
2048* 1536
1600*1200
1024*768
640*480
23 Eng: Mohammed Hussein
25. Bandwidth Requirements : Video
Now the requirements for applications such as : HDTV,TV,…The CIF
and QCIF used for conferencing applications.
The number of pixels is horizontal (HR) 1920 and vertically (VR) 1080,
with each pixel 24 bits (CR) and 60 frames per second (FR) will
Required 2986 Mbps. But by using compression, it reduced.
25
Eng: Mohammed Hussein
26. Multimedia Transmission Requirements
Eng: Mohammed Hussein26
Qualitative requirements are:
1. Response of the human ear (20 Hz to 20 KHz, sensitive to changes
to signal levels rather than absolute values, for animal can be more
then human ).
2. Response of the Human Eye (retains for few msec before decaying)
3. Tolerance to Error ( Higher error rate tolerance for uncompressed
signals), also as we communicate with network there became an
error after compressed.
4. Tolerance to delay and variation in delay (small for live applications)
5. Lip synchronization (most critical aspect) to avoid overlap.
27. Difference with classic applications
Eng: Mohammed Hussein27
Highly delay-sensitive
Packets are useless if they arrive too late
Loss-tolerant (for the most part)
Packet loss can be concealed
28. Performance Parameters
Synchronization Accuracy Specification (SAS) factors used to
specify the goodness of sync which are:
1. Delay:Acceptable time gap between transmission and
reception
2. Delay jitter: the variation between the desired presentation
times and actual presentation times of streamed multimedia
objects.
3. Delay skew:Average difference between the desired and
actual presentation times.
4. Error rate: Level of error is specified by bit-error rate (BER)
28 Eng: Mohammed Hussein
29. SAS Factors for Audio
Delay : for conversation,one-way delay should be in 100-500
msec range, which requires echo cancellation.
Delay Jitter: 10 times better than delay
Lip synchronization : between audio and video should be
better than 80 ms
BER: required less than 0.01 for telephones.
Less than 0.001 for uncompressed CD quality audio
less than 0.0001 for compressed CD quality audio
29 Eng: Mohammed Hussein
30. SAS Factors for Video
Delay and jitter
Required less than 50 ms for HDTV
Less than 100 ms for broadcast qualityTV
Less than 500 ms for video conference
Error rate
Required less than 0.000001 for HDTV
Less than 0.00001 for broadcastTV
Less than 0.0001 for video conference
30 Eng: Mohammed Hussein
31. Traffic Characterization Parameters
Variability of bit rates are :
1. Constant bit rate (CBR) applications:
Example: uncompressed digitized voice
transmission required CBR.
2. Variable bit rate (VBR) applications:
Example: video transmission using compression
Most Multimedia applications generateVBR traffic
31 Eng: Mohammed Hussein
32. Network Performance Parameters
1. Throughput: Effective rate of transmission of information
bits.
2. Delay:Time required for a bit to traverse through a network
(end-to-end) (250 msec max).
3. Delay variance:Variation of delay as a packet traverses
though a network (10 msec max).
4. Error rate: Specified by bit-error rate (BER), Packet error
rate (PER), Packet Loss Rate (PLR) and Cell Loss Rate(CLR).
The values of these NPPs, as compared to the bandwidth and SAS
factors of an application determines the QoS of a network.
32 Eng: Mohammed Hussein
33. Multimedia over Wireless and Internet
The Internet Protocol (IP)-
based Internet and
wireless systems are
converging into a
ubiquitous all-IP
information transport
infrastructure.
Thus, will allow mobile and
stationary users to access
the wireless Internet for
multimedia services
anywhere, and anytime.
33 Eng: Mohammed Hussein
34. Multimedia over LAN
The packet may suffer of a larger number of collisions. So some
LAN type are not suitable for multimedia traffic.
34 Eng: Mohammed Hussein
35. Why Multimedia over Wireless?
WLAN technologies are being increasingly used for
multimedia transmissions.
IP multimedia is partly real-time, which needs low delay.
There will be multiple of simultaneous services, with different
QoS requirements.
All of these should be supported by cost-efficient manner by
using network resources efficiently.
35 Eng: Mohammed Hussein
36. QoS in Broadband wireless networks
New services based on multimedia applications
VoIP
Video conferencing
Video on demand (VoD)
-They demand strict network guarantees such as reserved
bandwidth or bounded delays in BroadbandWireless Network
(BWN)
-The challenge for BWN is to provide QoS simultaneously to
services with different characteristics
36 Eng: Mohammed Hussein
37. Wireless Vs. Wired
Multimedia supported in wireless networks is much
more difficult than wired networks.
Wireless links are moving and unpredictable.
Wired links are fixed position.
-That is why the QoS in wireless networks is managed at
a MAC layer (medium access control)
37 Eng: Mohammed Hussein
39. Multimedia over Internet
The internet was not designed to carry multimedia
traffic.
The existing protocols are:
1. TCP: Includes connection establishment, error
control, flow control and hence unsuitable for real-
time multimedia traffic.
2. UDP: connectionless protocol at the transport layer,
can deliver real-time data if error can be tolerated, as
with uncompressed audio and video.
39 Eng: Mohammed Hussein
40. Internet for Multimedia Traffic
To make the internet support multimedia we need some
enhancements:
Multicasting:Used in audio and video conferencing (one-to-
many). Original IP is a “best-effort” is unicast approach.
IP multicast: an extension to original IP protocol supports
Dynamic and distributed group membership.
Multiple group membership.
Multiple send/receive nodes.
Multicast Backbone (Mbone): Real-world implementation of
IP Multicast
40 Eng: Mohammed Hussein
41. Multicast Backbone
It can be considered as internet radio orTV.
User call up for movie and Mbone net view uncompressed
movies.
It implement virtual overlay network on top of internet
It consists of multicast islands connected by tunnels
41 Eng: Mohammed Hussein
Mbone
tunnel
42. Problems with Internet & solutions :
TCP/UDP/IP suite provides best-effort, no guarantees on
expectation or variance of packet delay
Performance deteriorate if links are congested (transoceanic)
Most router implementations use only First-Come-First-Serve
(FCFS) packet processing and transmission scheduling
42 Eng: Mohammed Hussein
43. Problems and solutions
Limited bandwidth
Solution: Compression
Packet Jitter
Solution: Fixed/adaptive playout delay for Audio (example:
phone over IP)
Packet loss
Solution: FEC, Interleaving
43 Eng: Mohammed Hussein
44. Forward Error Control (FEC)
Eng: Mohammed Hussein44
FEC is a technique used for controlling
errors in data transmission over unreliable
or noisy communication channels.
The central idea is the sender encodes their
message in a redundant way by using
an error-correcting code (ECC).
Packet-level Forward Error Control (FEC)
for video streaming over a wireless
network.
packet-level interleaving when combined
with FEC presents a remedy to time-
correlated error bursts, though it can
further increase delay.
45. Problem: Limited bandwidth Intro:
Digitalization
Audio
x samples every second (x=frequency)
The value of each sample is rounded to a finite
number of values (for example 256).This is called
quantization.
Video
Each pixel has a color
Each color has a value
45 Eng: Mohammed Hussein
46. Problem: Limited bandwidth
Need for compression
Audio
CD quality: 44100 samples per seconds with 16 bits per
sample, stereo sound
44100*16*2 = 1.411 Mbps
For a 3-minute song: 1.441 * 180 = 254 Mb = 31.75 MB
Video
For 320*240 images with 24-bit colors
320*240*24 = 1843200/8000= 230KB/image
15 frames/sec: 15*230KB = 3.456MB
3 minutes of video: 3.456*180 = 622MB
46 Eng: Mohammed Hussein
254/8=31.75
47. Audio compression
Several techniques
GSM (13 kbps), G.729(8 kbps), G723.3(6.4 and 5.3kbps)
MPEG 1 layer 3 (also known as MP3)
Typical compress rates 96kbps, 128kbps, 160kbps
Very little sound degradation
If file is broken up, each piece is still playable
Complex (psychoacoustic masking, redundancy reduction, and bit
reservoir buffering)
3-minute song (128kbps) : 2.8MB
47 Eng: Mohammed Hussein
48. Discrete Cosine Transform (DCT)
Eng: Mohammed Hussein48
DCT brought blocks of 8x8 pixels information into frequency
domain.The low frequency means smoother image.
Some data appeared more important to others. Because of some
properties of the human eye it isn't necessary give color
information for every pixel. So, to reduce image size by using
important data only. If the image does not change too rapidly in the
vertical direction, than apply DCT to the collection of eight values.
يالحظ االنسان نظر
اختالفات
low frequency
من اكثر
High frequency
49. Image compression: JPEG
Divide digitized image in 8x8 pixel blocks
Pixel blocks are transformed into frequency blocks using DCT
(Discrete CosineTransform).This is similar to FFT (Fast
FourierTransform).
The quantization phase limits the precision of the frequency
coefficient.
The encoding phase packs this information in a dense fashion
49 Eng: Mohammed Hussein
DCT
50. Square waves to Sine Waves
Eng: Mohammed Hussein
50
Square sampled by 5 different coefficient frequency
High Frequency
Low Frequency
51. Square waves to Sine Waves
Eng: Mohammed Hussein51
If we take out number of high frequency to reduce amount of data
(Compression).
54. Video compression
Popular techniques
MPEG 1 for CD-ROM quality video (1.5Mbps)
MPEG 2 for high quality DVD video (3-6 Mbps)
MPEG 4 for object-oriented video compression
54 Eng: Mohammed Hussein
55. Video Compression: MPEG
MPEG uses inter-frame encoding
Exploits the similarity between consecutive frames
Three frame types
I frame: independent encoding of the frame (JPEG) :Coded without reference to
other frames.
P frame: encodes difference relative to I-frame (predicted) : Coded with reference to
a previous reference frame (either I or P).
B frame: encodes difference relative to interpolated frame: Coded with reference to
both previous and future reference frames (either I or P).
Note that frames will have different sizes
Complex encoding, e.g. motion of pixel blocks, scene changes, …
Decoding is easier then encoding
MPEG often uses fixed-rate encoding
I PB B BB B BP
55
Eng: Mohammed Hussein
Transmit Order
IBBPBBPBB
57. MPEG System Streams
Eng: Mohammed Hussein57
Combine MPEG video and audio streams in a single
synchronized stream.
Consists of a hierarchy with meta data at every level
describing the data:
System level contains synchronization information
Video level is organized as a stream of pictures group
Pictures are organized in slices
The motion vector isn't valid for the whole frame. Instead of this the
frame is divided into macro blocks of 16x16 pixels.
60. Problem: Packet Jitter
Jitter:Variation in delay
Example
1
3
5 4 3 2Sender
No jitter
125 46
6
Receiver
Jitter
pkt 6
pkt 5
60 Eng: Mohammed Hussein
61. Dealing with packet jitter
How does Phone over IP applications limit the effect of jitter?
A sequence number is added to each packet
A timestamp is added to each packet
Playout is delayed
Fixed playout delay
61 Eng: Mohammed Hussein
62. Dealing with packet jitter
Adaptive playout delay
Objective is to use a value for p-r that tracks the network delay
performance as it varies during a transfer.
The following formulas are used:
di = (1-u)di-1 + u(ri – ti) u=0.01 for example
i = (1-u)i-1 + u|ri-ti-di|
Where
ti is the timestamp of the ith packet (the time pkt i is sent)
ri is the time packet i is received
pi is the time packet i is played
di is an estimate of the average network delay
i is an estimate of the average deviation of the delay from the estimated average delay
62 Eng: Mohammed Hussein
63. Problem: Packet loss
Eng: Mohammed Hussein63
Loss is in a broader sense: packet never arrives or arrives later than
its scheduled playout time
Since retransmission is inappropriate for RealTime applications,
FEC or Interleaving are used to reduce loss impact.
64. Recovering from packet loss
Forward Error Correction
Send redundant encoded chunk every n chunks (XOR original
n chunks)
If 1 packet in this group lost, can reconstruct
If >1 packets lost, cannot recover
Disadvantages
The smaller the group size, the larger the overhead
Playout delay increased
64 Eng: Mohammed Hussein
65. Recovering from packet loss Piggybacking
Lo-fi stream
With one redundant low quality chunk per chunk, scheme can recover
from single packet losses
65 Eng: Mohammed Hussein
66. Recovering from packet loss Interleaving
Divide 20 msec of audio data into smaller units of 5 msec each and
interleave.
Upon loss, have a set of partially filled chunks
66 Eng: Mohammed Hussein
استرجاع
الباكتات
الخسرانه
67. Recovering from packet loss
Receiver-based Repair
The simplest form: Packet repetition
Replaces lost packets with copies of the packets that arrived
immediately before the loss.
A more computationally intensive form: Interpolation
UsesAudio before and after the loss to interpolate a suitable
packet to cover the loss .
67 Eng: Mohammed Hussein
68. Internet for Multimedia Traffic
Another enhancements required.
UDP is more suitable thanTCP for interactive traffic. But, it requires the
services of RTP.
Real-timeTransport protocol (RTP):
Application layer protocol designed to handle real time traffic
RTP provides and-to-end transport services to real-time audio, video
and simulation data. So RTP used for data communication and RCTP
used for control signal communication.
However, It does not provide QoS guarantees. RTP encapsulation is
only seen at the end systems: it is not seen by intermediate routers.
68 Eng: Mohammed Hussein
69. RTP runs on top of UDP
• RTP does not provide any mechanism to ensure timely delivery of data or
provide other quality of service guarantees.
• RTP encapsulation is only seen at the end systems: it is not seen by
intermediate routers.
• Routers providing best-effort service do not make any special effort to ensure that
RTP packets arrive at the destination in a timely matter.
69
Eng: Mohammed Hussein
RTP extend UDP and uses a even numbered
UDP port and the next number is used by a
companion protocol RealTimeTransport
Control protocol (RCTP).
• port numbers, IP addresses
• payload type identification
• packet sequence numbering
• time-stamping
70. RTP packet format
PayloadType: 7 bits, providing 128 possible different types
of encoding; eg PCM, MPEG2 video, etc.
Sequence Number: 16 bits; used to detect packet loss
Timestamp: 32 bytes; gives the sampling instant of the first
audio/video byte in the packet; used to remove jitter
introduced by the network
Synchronization Source identifier (SSRC): 32 bits; an id
for the source of a stream; assigned randomly by the source
70 Eng: Mohammed Hussein
71. Timestamp vs. Sequence No
Timestamps relates packets to real time
Timestamp value sampled from a media specific clock
Sequence number relates packets to other packets
Audio silence example :
Consider audio data type
What do you want to send during silence?
Not sending anything
Why might this cause problems?
Other side needs to distinguish between loss and silence
Receiver usesTimestamps and sequence No. to figure out what
happened .
71 Eng: Mohammed Hussein
72. RTP Control Protocol (RTCP)
Used in conjunction with RTP. Used to exchange control information
between the sender and the receiver.
Three reports are defined: Receiver reception, Sender, and Source
description.
Reports contain statistics such as the number of packets sent, number
of packets lost, inter-arrival jitter
Typically, limit the
RTCP bandwidth to 5%.
Approximately one
sender report for three
receiver reports
72 Eng: Mohammed Hussein
73. Feedback During Media Streaming
Feedback can be used to control performance
Sender may modify its transmissions based on feedback
73 Eng: Mohammed Hussein
74. Streaming Stored Multimedia Example
Audio/Video file is segmented and sent over eitherTCP or
UDP, public segmentation protocol: Real-Time Protocol
(RTP)
User interactive control is provided, e.g. the public protocol
RealTime Streaming Protocol (RTSP)
74 Eng: Mohammed Hussein
75. Streaming Stored Multimedia Example
Helper Application: displays content, which is typically
requested via aWeb browser; e.g. RealPlayer; typical functions:
Decompression
Jitter removal
Error correction: use redundant packets to be used for reconstruction
of original stream
GUI for user control
75 Eng: Mohammed Hussein
76. Streaming from a Web Server (cont)
Alternative: set up connection between server and player, then
download.
Web browser requests and receives a Meta File.
(a file describing the object) instead of receiving the file itself;
Browser launches the appropriate Player and passes it the Meta
File;
Player sets up aTCP connection with a streaming server and
downloads the file.
76 Eng: Mohammed Hussein
77. Options when using a streaming server
Use UDP, and Server sends at a rate (Compression andTransmission) appropriate for
client; to reduce jitter, Player buffers initially for 2-5 seconds, then starts display.
UseTCP, and sender sends at maximum possible rate underTCP; retransmit when error
is encountered; Player uses a much large buffer to smooth delivery rate ofTCP
77 Eng: Mohammed Hussein
78. Real Time Streaming Protocol (RTSP)
For user to control display: rewind, fast forward, pause, resume, etc…
Out-of-band protocol (uses two connections, one for control messages
(Port 554) and one for media stream)
RFC 2326 permits use of eitherTCP or UDP for the control messages
connection, sometimes called the RTSP Channel
As before, meta file is communicated to web browser which then
launches the Player; Player sets up an RTSP connection for control
messages in addition to the connection for the streaming media
78 Eng: Mohammed Hussein
81. References
Streaming in Mobile Networks
Jun-Zhao Sun, Douglas Howie,Antti Koivisto and Jaakko Sauvola,“A
Hierarchical Framework Model of Mobile Security,” MediaTeam Oulu, infotech
Oulu, Finland, 2001, pp.A56-A59
Claudio Cicconetti, Luciano Lenzini and Enzo Mingozzi,“Quality of Service
Support inWireless Networks,” University of Pisa, Nokia Research Center,
Italy,April 2006, pp. 50-55
http://www.ece.umd.edu/~minwu/public_paper/Jnl/0603MuserVideo_IEEEfinal_
NetMag.pdf
http://research.microsoft.com/china/papers/Scalable_Video_Coding_Transport.pdf
http://islab.cis.nctu.edu.tw/seminar/doc/1.pdf
http://www.tml.tkk.fi/Opinnot/T-
110.5120/2005/slides/06a.Streaming%20Multimedia%20Architecture,%20Codecs%
20and%20QoS2.pdf
http://www.hindawi.com/journals/am/2009/982867/
81 Eng: Mohammed Hussein