Addressing and Annotating Multimedia Fragments

Addressing and annotating
multimedia fragments
Raphaël Troncy <raphael.troncy@eurecom.fr>

Use Case

10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2

Use Case

 Aidem received on her Facebook
wall a status message containing
a Media Fragment URI
 Use a ‘#’ !
 Highlight a
video
sequence
 Highlight a
region
to pay
attention to


What are Media Fragments?

0 20 temporal media fragment 35 t

spatial media fragment

track media fragment


Temporal Media Fragments

Original resource
length

Fragment beginning Playback progress Fragment end


Spatial Media Fragments

highlighted
fragment
semi-opaque
overlay


Media Fragments Use Cases

 Bookmark / Share parts (fragments) of
audio/video content
 Annotate media fragments
 Mash-ups
 Conserve bandwidth

http://www.w3.org/TR/media-frags-reqs/


Requirements
 r01: Temporal fragments:
 a clipping along the time dimension from a start to an end time that
are within the duration of the media resource

 r02: Spatial fragments:
 a clipping of an image region, only consider rectangular regions

 r03: Track fragments:
 a track as exposed by a container format of the media resource

 r04: Named fragments:
 A temporal media fragment that has been given a name through
some sort of annotation mechanism


Media URIs

 Using URI query part:
http://www.example.org/video.ogv?t=60,100

 Using URI fragment part:
http://www.example.org/video.ogv#t=60,100

 Mixing both:
http://www.example.org/video.ogv?t=60,100
#t=10,15


URI Fragments vs. URI Queries
#t=20,30 ?t=20,30
secondary resource, primary resource,
notion of context no notion of context
extraction needs to be no adaptation restrictions
expressible in byte ranges
no provisions for key-value pairs are sent to
communi-cating the server
fragments to the server
potentially cacheable not cacheable
 The media fragment URI syntax can be used
for URI queries
 We will focus on URI fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10

Media Fragments Resolution

 For the URI query part:
 The media file is only processed on server side
 The UA receives a new video file

 For the URI fragment part:
 Smart UA may strip out the fragment definition and
encode it into custom http headers (Range header)
 (Media) Servers will handle the request, slice the media
content and serve just the fragment (corresponding byte
ranges)
… while old ones will serve the whole resource


Influence of Media Formats

 Fragment extraction needs to be expressible in
terms of byte ranges
 Requirements for the different axes
 temporal: presence of intra-coded frames
(i.e., random access points)
 spatial: presence of independently coded spatial regions
 track: need to be identifiable by a name

 Conclusion: temporal and track axes are
realistic, spatial fragments can hardly be
expressed in terms of byte ranges


Can I use Media Fragments now?

<video id='v'
src='AudioAPI.webm#t=50,100'
onloadedmetadata='update()'
onpause='update()'
onplay='update()'
onseeked='update()'
controls></video>

 Firefox 9+ (or WebKit)
 No bandwidth saving



 Sysnote over YouTube
 No bandwidth saving



 Ninsuna
 Bandwidth
saving


Examining HTTP Traffic

 HTTP request:
 retrieving URI
 parsing key=values pairs from fragment part
 setting Range header

 HTTP response:
 checking Content-Type
and Content-Range-Mapping headers values
 attaching custom playback controls to page
 creating spatial dimension overlay (if specified)


Example: Requesting a Time Fragment

 A web developer specifies a video source with
a temporal fragment URI:
http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15

 key=value pair is analyzed,
fragment begin and end time are matched
t=5,15

 Media Fragments Extension analyses the
fragment part, retrieves beginning and end time
and sets proper Range header value:
Range: t:npt=5-15


 The NinSuna server responds with the 206
Partial Content response and Content-Range-
Mapping header showing the mapped time
ranges and media fragment in the message
payload:
HTTP/1.1 206 Partial Content

Content-Type: video/ogg
Accept-Ranges: bytes, t, track, id
Content-Range: bytes 629578-1690588/4055466
Content-Range-Mapping:
{t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466}



 ... and it won’t work!

 Because the
player does not
have the media
file header



 Player needs to be initialized, thus bare
fragment is not playable
 To fix this, we add ;include-setup to the Range
header value:
Range: t:npt=5-15;include-setup



 The response from the server is slightly different:

Content-Type: multipart/byteranges;boundary=End
Content-Range-Mapping:
{t:npt 4.8-14.8/0-38.33;include-setup}
={bytes 0-5998,629578-1690588/4055466}
...

--End

{binary data}
--End

{binary data}
--End
--End--



 This is why we need to attach a Stream Listener
to the HTTP channel:

Content-Type:
multipart/byteranges;boundary=End
...
--End
Content-Type: video/ogg Content-Type: video/ogg
Content-Range: bytes 0-5997/4055466 ...

{binary data} {binary data}
--End {binary data}
Content-Range: bytes 629578-
1690588/4055466

{binary data}
--End
--End--


Spatial Fragments

 Without additional information send to server
 Overlay is created by appending four additional
DIVs to the webpage
 They are styled to create the impression of
semi-opaque layer over the video element


NinSuna: Overview

 Fully integrated media adaptation and delivery
platform
 media adaptation and media packaging core is
independent of media formats
based on model-driven content adaptation & delivery technique
 only high-level adaptation operations such as scene
selection and frame rate scaling
no transcoding is applied
ideal candidate for MF implementation
 more information: http://ninsuna.elis.ugent.be


MF-specific Features of NinSuna

 Support for Media Fragment URI queries
 both HTTP and RTSP implementation
 try some URIs at
http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test
http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t
railer_2.mp4?track=5;6

 Support for Media Fragment URI fragments
 i.e., support for the MF-specific HTTP headers

 Combining Media Fragment URI queries and
fragments
 e.g., http://foo.com/media.mp4?t=10,40#t=5,10


Impact on the Existing Web Infrastructure

 Updating existing infrastructure is not trivial
 web servers need to be extended with
media extractors
MF-specific HTTP communication
 existing web caches are only effective with HTTP byte
ranges
specialized media caches need to be developed in the future to
cache the other cases
 user agents need to be extended with
a Media Fragment URI parser and interpreter
MF-specific HTTP communication
MF-specific visualization


Media Fragments Proxy

 Goal: make existing media resources, served
by generic HTTP Web servers, available as
Media Fragments
 Biggest problem for content providers
willing to support Media Fragments
 media extractor (dependent on underlying media
formats)
 Solution: an approach that
 works with existing HTTP Web servers (apache, IIS)
 works with existing Web caches
 works with not so smart user agents


Media Fragments Proxy http://foo.com/video.ogv#t=11,19

GET /?url=http://foo.com/video.ogv HTTP/1.1
Host: MFProxy.com
HTTP Accept: video/*
server Range: t:npt=11-19
Accept-Range-Redirect: bytes

HTTP/1.1 307 Temporary Redirect
Location: http://foo.com/video.ogv
-get (only) the header info
Accept-Ranges: bytes, t, track
Content-Length:a fragment-to-byte mapping
-try to find 0
-construct the redirect response
Content-Range-Mapping: t:npt 10-20/0-50
Range-Redirect: 24000-32000
Vary: Accept-Range-Redirect

GET /video.ogv HTTP/1.1
MF Proxy Host: foo.com
MF User Accept: video/*
Agent Range: bytes=24000-32000

http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI>


Annotating Media Fragments yuma.min.js


Annotating Media Fragments

Benchmarking: Sphinx, HTK,
Julius

 NER + full text index with the
transcription
 Interlinking with the Linked Data
Cloud to enable semantic search


http://www.slideshare.net/troncy


Addressing and Annotating Multimedia Fragments

Recomendados

Recomendados

Más contenido relacionado

Más de Raphael Troncy

Más de Raphael Troncy (20)

Último

Último (20)

Addressing and Annotating Multimedia Fragments