Architecture diagrams and difference between webrtc implementation in broswers and stacks.

1. Browsers and stack
implementation differences
Dr. Alex Gouaillard

2. Plan
• global architectures of browsers and stacks
• -- chrome / firefox / webkit
• -- webrtc stand alone, webrtc in chrome,
webrtc in firefox
• - how to trace the different browsers without
recompiling

3. WebRTC Arch

4. LibWEBRTC Variation

5. Chrome Variation

6. Chrome Variation
• Wait .. What? Chrome does not use libwebrtc?
Why?

7. Chrome Layers

8. Chrome Layers

9. Chrome Variation
• Wait .. What? Chrome does not use libwebrtc?
Why?
– Chrome layering
– Sandbox and per tab security
– Echo canceling
– Special capturers (screen/window/tab)
– Capturer access security
– Pretty much the same applies to all browsers

10. Chrome Variations
• OK, Chrome does not use libwebrtc? Why
should I Care?
• If you use the webrtc lib in your project, and
expect the same results as chrome, be ready
for big surprises!
– No Screen/windows/tab sharing
– Obsolete mac capturer
– No HTTP PROXY support

11. LibWEBRTC src Files layout

12. Libwebrtc src file layout
• base/
• common_audio/
• common_video/
• libjingle/
• modules/
• p2p/
• sound/
• system_wrappers/
• tools/
• video/
• video_engine/
• voice_engine/
• Too many to list
• Basic signal processing (FFT, window, transforms, …)
• Frame buffer, I420, libyuv, …
• Session management
• See next slide
• ICE Transports: TURN, STUN, DTLS, ….
• Platform specific sound system wrappers
• Platform specific low level wrappers (threads, timer, …)
• Poach and weaponize!!
• Video encoder/decoder/rec and send stream
• Capturer, Channel, Encoder, Sync, REMB, Send
• Channel, DTMF, …

13. Libwebrtc src file layout
• audio_coding/
• audio_conference_mixer/
• audio_device/
• audio_processing/
• bitrate_controller/
• desktop_capture/
• media_file/
• pacing/
• remote_bitrate_estimator/
• rtp_rtcp/
• utility/
• video_capture/
• video_coding/
• video_processing/
• video_render/
• ACM2, NACK, NETQ, DTMF tone gen., ….
• mixer
• Platform specific
• Noise, typing, gain detection, NS, Echo Cancellation, …
• Bandwidth estimation and bitrate controller
• Desktop capture (abstract only)
• Audio files reading/writing
• Paced sender
• Obvious
• RTP, RTCP, Payloads, FEC, …
• More Audio files utilities, rtp_dump, recorder, player
• Generic and platform dependent capturers (OLD)
• Codecs, encoders and decoders, send and rec, jitter, …
• Video processing and image enhancement
• Generic and platform dependent renderers (OLD)

14. Libwebrtc src file layout
• Why do I care?
– 1. you might need to go in the code
– 2. you need to know the layout for debugging:
chrome.exe --enable-logging --
vmodule=*/webrtc/*=2,*/libjingle/*=2,*=-2 --no-
sandbox

15. LibWEBRTC Class Arch

16. LibWEBRTC Arch

17. LibWEBRTC DEPS
• binutils
• Boringssl
• Colorama
• Drmemory
• Expat
• Icu
• instrumented_libraries
• Jsoncpp
• Libjpeg
• libjpeg_turbo
• Libsrtp
• libudev
• libvpx
• Libyuv
• llvm-build
• Nss
• Ocmock
• openmax_dl
• Opus
• Protobuf
• Sqlite
• Syzygy
• Usrsctp
• Yasm
• zlib

18. Webrtc.org

19. Chrome Specific Webrtc Bits
• //src/content/renderer/media/
• //src/third_party/WebKit/public/platform/
• //src/third_party/WebKit/Source/modules/mediastream
• //src/content/browser/media/
• //src/content/browser/media/capture/
• //src/media/capture/video/
• //src/content/browser/renderer_host/media/
• .
• .
• .
• .
• .
• .
• .

20. Chrome Specific Webrtc Bits
// MediaStreamManager is used to open/enumerate media capture devices (video
// supported now). Call flow:
// 1. GenerateStream is called when a render process wants to use a capture
// device.
// 2. MediaStreamManager will ask MediaStreamUIController for permission to
// use devices and for which device to use.
// 3. MediaStreamManager will request the corresponding media device manager(s)
// to enumerate available devices. The result will be given to
// MediaStreamUIController.
// 4. MediaStreamUIController will, by posting the request to UI, let the
// users to select which devices to use and send callback to
// MediaStreamManager with the result.
// 5. MediaStreamManager will call the proper media device manager to open the
// device and let the MediaStreamRequester know it has been done.

21. Chrome GUM example
capturers / frames / Encoding / ….

22. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)GUM JS API
(tab/sandbox)
1. Send request

23. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)GUM JS API
(tab/sandbox)
2. Check if MST is already available

24. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)GUM JS API
(tab/sandbox)
Security Manager
(source, origin)
3. Check rights
2. Check if MST is already available - NO

25. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
GUM JS API
(tab/sandbox)
4. Ask Corresponding capturer
type to start capturing
3. Check rights - OK

26. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
A
GUM JS API
(tab/sandbox)
4. Ask Corresponding capturer
type to create one - OK
V
5. Store the MST

27. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
A
GUM JS API
(tab/sandbox)
V
6. Trigger callback
Keep feeding frames

28. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
A
GUM JS API
(tab/sandbox)
V
NOTE 1: second call for same device with same constraints will directly return the
MST, that allows to share streams across tabs without blocking

29. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
A
GUM JS API
(tab/sandbox)
V
NOTE 2: Recently, a second call for the same device but with different constraints
(think simulcast) will indeed return a different resolution. Before it would return the
first resolution asked.

30. Chrome Specific Webrtc Bits
Media Stream Manager
(singleton@browser)
Audio
Capturer
Video
Capturer
Security Manager
(source, origin)
A
GUM JS API
(tab/sandbox)
V
NOTE 3: Not only this allow to share cams across processes, it allows for global echo
cancellation (yes, including the key strokes). Before tabs could cross feed.

31. Chrome Screensharing 2 steps (1)
Media Stream Manager
(singleton@browser)
Screen/Windows/Tab
Capturer
Security Manager
(source, origin)
Screensharing
(extension)
1
2
3
4
S
5

32. Chrome Screensharing 2 steps (2)
Media Stream Manager
(singleton@browser)
Screen/Windows/Tab
Capturer
S
GUM JS API
(tab/sandbox)
With ID

33. Firefox Variation

34. Firefox Variation – Desktop 1process

35. Firefox Variation – B2G

36. FireFox Arch
• PeerConnection
– PeerConnection.js -- shim translation layer to let us do API adaptation to the C++
– PeerConnectionImpl -- C++ implementation of the PeerConnection interface.
– SIPCC -- handles SDP and media negotiation. Provided by Cisco but not a downstream.
• Media
– Webrtc.org/GIPS -- handles media encoding and decoding. Downstream from Google.
– MediaConduit -- Generic wrapper around Webrtc.org
– MediaPipeline -- Wrapper to hold the MediaConduit, mtransport subsystem, and the SRTP
contexts, as well as interface with MediaStreams.
• Transport
– mtransport -- generic transport subsystem with implementations for ICE, DTLS, etc.
– NSS -- new DTLS stack. Mentioned because we need to land the new version of NSS
– nICEr -- ICE stack; downstream from reSIProcate project
– nrappkit --portable runtime, utility library; downstream from nrappkit.sourceforge.net
• DataChannel
– DataChannel implementation in the DOM
– libsctp -- SCTP implementation; downstream from the BSD SCTP guys

37. Webkit (safari ?)
Current work:
- Implement datachannel for webkit (yin liu)
- Implement webrtc.org as a back end for
webkit (adam tiamou)
- Implement windows support for OWR

38. Webkit Arch

39. Webkit Arch

40. Webkit Webrtc Arch

41. Webkit Arch – Data Channel

42. Webkit Arch – Data Channel