These are the slides of my talk WebKit and GStreamer of the GStreamer Conference on 2013, cohosted with LinuxCon.
The HTML5 version with its effects can be found at http://people.igalia.com/xrcalvar/talks/20131022-GstConf-WebKit
3. Who is this dude?
· Born in the Basque Country
· Grown and living in Galicia
- One of the seven Celtic Nations, like Scotland
· Studied Computer Science at Universidade da Coruña
· Partner and developer at Igalia
- Working in Multimedia and WebKit
7. WebKit
· Web content engine
· Processes and renders web content
· Started as a fork of KTHML and KJS
· Open source since 2005
· Goals: open source, compatibility, compliance, stability,
performance, security, portability, usability, hackability
· Non-goals: being a full web browser, being a science project,
having reusable components, unlimited scope
· Forked by Google as Blink
· Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows
7/36
9. WebKitGTK+
· Each WebKit port is composed of
- Platform interfaces
- API layer
· WebKitGTK+ platform layer:
- libsoup for networking
- cairo for rasterization
- OpenGL for making the scene graph and WebGL
- GStreamer for media
- Various GTK+ APIs for talking with the system
· API layer is a GtkWidget and a set of GObject APIs
· WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp
9/36
10. WebKit2
· Give the web rendering parts of WebKit their own process
· Page crashes don't crash the browser
· Can put vulnerable data into a separate address space
· Sandbox web rendering
· Prevent pages from accessing the disk and operating system
interface
10/36
14. WebKit Multimedia
· Mac port: QuickTime, QtKit, AVFoundations
· Apple Win port: QuickTime
· GTK+ and EFL ports: GStreamer 1.0 backend
· Blackberry port: Bridges over to its own Media playback framework
· Chromium: bridges over to WebKit/Chromium backend which
relays to FFMPeg implementations
· Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on
Windows
14/36
16. GStreamer MediaPlayer - Overview
· Mainly developed and tested on Linux platform
· Successfully ported to MacOS/Win platfoms (not fully upstream
yet though)
· Playbin-based: the best GStreamer element for playback usecases
- Playbin2 with GStreamer 0.10
· MediaPlayer registers and uses two WebKit-specific GStreamer
elements: webkitwebsrc and a video-sink
16/36
17. GStreamer MediaPlayer - Main Features
· On-disk-buffering
· Frame-accurate seeking
· Basic metrics reporting
· Trick-modes
· WebAudio
· Hardware decoding support with VA-API (gst 1.2.x)
· Video accelerated compositing
· Codec installer support
17/36
19. GStreamer MediaPlayer - HTTP source element
· a GStreamer bin implementing the URI handler interface for
http(s):// and blob:// uris.
· A single source pad provides raw data access to downstream
elements.
· appsrc is used internally to interface with the WebCore
SubResourceLoader object
· Element is registered with high rank in the GStreamer registry so
playbin is more prone to pick it up to read http:// URIs.
19/36
20. GStreamer MediaPlayer - Video rendering
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input
· Pass those to the MediaPlayerPrivateGStreamer via a signal
20/36
21. GStreamer MediaPlayer - Audio rendering
· Use autoaudiosink
· Check for pulseaudio for playback request messages
· Pitch preservation
· Preserve audio passthrough when possible
· Volume managed by the sink
21/36
22. GStreamer MediaPlayer - Next steps
· Improvements and bug fixes in the player:
- Store on-disk buffered media in WebCore's cache directory
- Make the webkitwebsrc element private to WebKit
· MediaStream (getUserMedia) and WebRTC spec implementation
· MediaSource adaptive streaming API spec implementation
· Integration of WebAudio with media elements and mediastream
stack
· Finish porting to 1.2
· Remove GStreamer 0.10 code path
22/36
24. WebAudio
· A JavaScript API allowing to create a new generation of Web
games and applications
· Effects, convolution engine for room simulation, etc
· Realtime analysis, playback and visualization
· W3C spec driven and implemented mainly by Chromium and
Firefox
· In WebKit, currently implemented for:
- Mac
- Chromium
- GStreamer based Ports: GTK+, EFL, Qt
24/36
25. WebAudio GStreamer - Reading Audio
· Spec allows two ways to read audio data:
- from memory: we can use giostreamsrc GStreamer element
- from a file path: we can use the filesrc GStremer element
· Pipeline: src ! decodebin2 ! audioconvert ! audioresample !
capsfilter ! deinterleave ! queue ! appsink
· deinterleave separates multi-channel audio to n audio mono
channels (Pipeline above can contain multiple queue ! appsink
chains)
· appsink used to collect mono audio buffers, store them in a list and
convert data to floating point values
· Data is stored in a WebCore::AudioBus, containing one or multiple
channel Float arrays
25/36
26. WebAudio GStreamer - Playing Audio
· The WebAudio stack processed or generated some audio data to be
sent to the sound card
· Similarly to the "Reading Data" step, a WebCore::AudioBus is used
to store the data
· Custom GStreamer element to convert an AudioBus to an
interleaved audio data flow: webkitwebaudiosrc
· webkitwebaudiosrc: a bin with n internal pads pushing to an
interleave element. Source pad outputs a single, interleave, wavencoded data flow
· Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert !
autoaudiosink
26/36
27. WebAudio GStreamer - Work in Progress
· Some layout tests don't pass yet
· Not yet enabled by default in release builds
· Support for stereo+ streams in the reader side. Playback side
already handles those.
· Integration with <video> and <audio> elements: WebAudio
MediaElementSourceNode
27/36
28. Hardware Accelerated Video
· We use GStreamer VA-API. (1.2.x)
· That's it! Hardcore comes when compositing that.
28/36
30. Accelerated Video Compositing
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input.
· Video frame is transferred to the MediaPlayerPrivateGStreamer via
a GLib signal.
· MediaPlayerPrivateGStreamer uploads it to a GL texture and
renders it asynchronously in the WebKit context.
30/36
32. Accelerated Video Compositing - Rendering paths
· Software rendering path
- It uses Cairo
· Accelerated compositing paths
- TextureMapperLayer:
- Light-weight implementation of a scene-graph
- The TextureMapperLayer tree is equivalent to the
GraphicsLayer tree.
- TextureMapperGL
- Provides a GPU accelerated implementation of the
drawing primitives, using shaders compatible with GL/ES
2.0
32/36
33. Accelerated Video Compositing - Next steps
· More video decoders with GstVideoGLTextureUploadMeta support
· Support EGLImage in WebKit as video frames handlers
- And video decoders that deliver this kind of capabilities.
· Handle the vertical blank interval in WebKit
- GTK+ 3.8 seems to fix this issue.
33/36
