Adaptive Streaming for Immersive Communication

1. Adaptive Streaming for
Immersive Communication
Silvia Rossi and Laura Toni
UCL - University College London
57th FITCE Congress, UK 2018 - Delivery and Consumption of Digital Media
MediaCityUK, Salford

2. • 360º video streaming: an overview
• Main Challenges
• Possible solutions
• Conclusion and future works
Outline
2

3. Virtual Reality (VR)
3
A virtual - rather than physical - world in which
any user can be fully immersed

4. VR content: 360º video
4
360º or spherical video is a new multimedia format that
provides the user with a full immersion sensation
Displayed viewport

5. 360º video streaming chain
5

6. 360º video streaming chain
5

7. 360º video streaming chain
5

8. 360º video streaming: main challenges
• New spherical geometry
• Large volume of data to store, deliver and display
• Ultra-low-delay constraints over bandwidth-limited
resources
• Uncertainty on the portion of content
that will be displayed by the user
6

9. • New spherical geometry
• Large volume of data to store, deliver and display
• Ultra-low-delay constraints over bandwidth-limited
resources
• Uncertainty on the portion of content
that will be displayed by the user
360º video streaming: main challenges
7
VR KEY QUESTIONS
• Which QoE to capture the new spherical geometry?
• Which content to store based on user position?
• Which content to send based on user position
➡ user centric era

10. A novel QoE for a new content geometry?
8
Map projection problem
• Omnidirectional video (ODV) is projected on a planar frame
• Equirectangular (ERP) projection is the most popular and
simplest projection

11. • Omnidirectional video (ODV) is projected on a planar frame
• Equirectangular (ERP) projection is the most popular and
simplest projection
A novel QoE for a new content geometry?
8
Map projection problem

12. A novel QoE for a new content geometry?
8
Spherical domain
Planar domain
From the sphere to planar with ERP projection. User VP is in red.
• Omnidirectional video (ODV) is projected on a planar frame
• Equirectangular (ERP) projection is the most popular and
simplest projection
• Projection introduces artifacts affecting the user quality of
experience (QoE)
Need for a geometry-aware distortion evaluation

13. • 360º video streaming: an overview
• Main Challenges
• Possible solutions
• Conclusion and future works
Outline
13

14. Which content to store?
• High cost of storage and coding
• Limited storage space
9
Need to limit what to store

15. Which content to store?
• High cost of storage and coding
• Limited storage space
• Non-homogenous user requests and content characteristics
Need to create diversity in what we store
(to accomodate any users)
9
Need to limit what to store

16. Which content to store?
• High cost of storage and coding
• Limited storage space
• Non-homogenous user requests and content characteristics
Need for a popular-representation set to store
9
Need to limit what to store
• Need to limit what to store
• Need to create diversity in what we store
➡ Need to find the best trade-off between
storage cost and user versatility

17. • 360º video streaming: an overview
• Main Challenges
• Possible solutions
• Conclusion and future works
Outline

18. Which content to send?
• Prefetching requested viewport prevents freezing of the video
• Delivering non-homogenous quality content saves bandwidth
• Resolution of content improves user QoE
Need for navigation-aware adaptive logic
10

19. VR360 video streaming solutions
11
Interactive Users
Time
Quality
Content Provider
QualityQEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
Time
Entire video at
uniform quality

20. VR360 video streaming solutions
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Entire video at
uniform quality

21. VR360 video streaming solutions
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Entire video at
non-uniform
quality

22. VR360 video streaming solutions
Content Provider
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Entire video at
non-uniform
quality

23. VR360 video streaming solutions
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Tile-based
system

24. VR360 video streaming solutions
Content Provider
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Tile-based
system

25. VR360 video streaming solutions
Content Provider
11
Interactive Users
Content Provider
QEC1QEC2
HTTP request
HTTP segment
Interactive Users
Time
QualityQuality
Time
Time
Tile-based
system

26. • Tile-based adaptive streaming system
• Optimized transmission strategy, taking into account:
-users' navigation paths
-geometry-based MSE as quality metric
To efficiently transmit a 360˚ videos, we propose:
Proposed Navigation-aware adaptive streaming
12
Silvia, Rossi, and Laura Toni. “Navigation-aware adaptive streaming strategies for omnidirectional video”
Multimedia Signal Processing (MMSP), 2017 IEEE 19th International Workshop on. IEEE, 2017.

27. Proposed Navigation-aware adaptive streaming
12

28. • New spherical geometry
• Large volume of data to store, deliver and display
• Ultra-low-delay constraints over bandwidth-limited
resources
• Uncertainty on the portion of content
that will be displayed by the user
360º video streaming: main challenges
VR KEY QUESTIONS
• Which QoE to capture the new spherical geometry?
• Which content to store based on user position?
• Which content to send based on user position
➡ user centric era
13

29. User navigation prediction
Where will the user be looking in the future frames?
STEP 1:
Users profiling
• Analysis on the sphere taking into
account both physical structure
and users’ features
• Users clustered based on their
position on the sphere during the
navigation
Need for users analysis on the
spherical domain 14
15 Clusters
Frames
Frames
5 Clusters

30. User navigation prediction
Understanding where the user is most likely going to look at
is the key to improve the delivery of VR applications
STEP 2:
Viewport Prediction
• Users in the same class have
similar characteristics
• The user attention is strongly
related to content features
• Popular navigation paths
can give a global behaviour
Need for head movement
prediction algorithm
Where will the user be looking in the future frames?
14

31. Conclusion
To optimize the entire pipeline of immersive communication
leading the user to the best QoE:
✓ Need for a spherical distortion evaluation
• Need for a popular-representation set to store at the
server
✓ Need for navigation-aware adaptive logic
• Need for user movements analysis on the spherical domain
➡ User navigation prediction algorithm
15

32. Thank you for your attention!
➡ Questions ?
(At the Panel…)