This describe the limitation of usual single page, single browser, single OS approach, illustrate current IMTC sponsored effort to improve interoperability testing, and describe a roadmap to improve the quality of app testing.

1. WebRTC Testing SOA
Dr Alex Gouaillard
Citrix,
IMTC, W3C, IETF

2. Manual Single page tests
web-platform, adapter.js, whatever
50
Dev Nightly
50 50 49 49 - 47 0 4
https://github.com/w3c/web-platform-tests
https://github.com/webrtc/adapter
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

3. Automated Single page tests
web-platform, adapter.js, whatever
?
Dev Nightly
50 49 ?? ? ? ? ?
https://github.com/webrtc/adapter/tree/master/test
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

4. Automated Single page tests
how easy would it be to extend the automation?
?
Dev Nightly
50 49 ?? ? ? ? ?
Easy Easy
Med.
Hard Medium
Easy
Med.
Hard
Just add the browser binary on the (virtual) machine
Need a new web-driver, and the browser binary.
Need everything, and it’s on a mobile OS !
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

5. Automated Single page tests
WIP (AFAIK)
?
Dev Nightly
50 49 ?? ? ? ? ?
Nils IMTC Alex
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

6. Automated Single page tests
web-platform, adapter.js, whatever
?
Dev Nightly
50 49 ?? ? ? ? ?
Wait, automated …
on Debian ONLY !!
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

7. NightlyDev
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
Manual Single page tests
web-platform, adapter.js, whatever
50 50 50 49 49 - - - -
? ? ? ? ? - 47 - -
? ? ? ? ? - - 0 4
? ? ? ? ? -
? ? ? ? ? - -
? ? ?
- -

8. Automated Single page tests
WIP (AFAIK)
Dev Nightly
50 49 - - - -
? ? ? - - -
? ? ? ? ? - -
? ? ? ? ? -
? ? ? ? ? - -
? ? ?
- -
Alex
IMTC
Nils??
IMTC IMTC
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

9. Intermediate Conclusion (1)
• With webrtc, single-page show their limit
– 2 PC objects in one page
• One browser testing is not enough
– 2 PC objects in two tabs
• Next step is to test interoperability between two separate browsers
– Possibly with proxy, NAT, in-between
• Original, single-machine, interop code part of adapter.js test suite is
a good start, but need support for more browsers.
• IMTC sponsors the work to add edge browser support, and full app
testing.
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

10. Interop Tests (Debian)
appRTC Hosted + raw local
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

11. Interop Tests (Debian)
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

12. Interop Tests (Debian)
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

13. Interop Tests (Win10)
IMTC
IMTC
IMTC
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

14. Interop Tests (MacOSX)
AlexAlex
Alex
Alex
Alex
Alex Alex
Alex Alex
Alex
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

15. Interop Tests (Android and iOS)
Dr.AlexGouaillard@CitrixSystems,2016.DocumentprovidedunderCCBY-NC4.0

16. Intermediate Conclusion (2)
• Powerpoint is not the good tool for
multidimensional data visualization 
• The real interesting cases are not covered today:
– Desktop to mobile
– Cross OS: Chrome on Mac against Edge on Windows
• How to quantify what is done, and what is left
TBD?
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

17. WhatweCOULDtesttoday
Thanks to IMTC
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

18. WhatweACTUALLYtesttoday
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

19. Intermediate Conclusion (3)
• State of the (open source) Art is two browsers
running the same tests on the same OS.
• How to quantify what is done, and what is left
TBD?
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

20. WhatweWANTtotest.
496
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

21. Whatwecouldtesttoday 115
23%
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
381
77%
1 OS
2 Oses
(Not tested)

22. © Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
6
5%
WhatweACTUALLYtesttoday
21
18%
88
77%
Tested
WIP
Not tested

23. 2016 RoadMap (not in order)
• Improve W3C tests
• Additional browser support in Adapter.js
• new web drivers to the interop test suite.
– New desktop Browsers, and mobile browsers.
• Improve webdrivers
– security prompt support
– Safari / GTK+ / webkit support
• Add a Conductor for cross-browser interop tests
• Add appRTC standalone support (GAE SDK)
• Add a SIP interop app to the suite
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

24. Want to have some of the fun?
• Add improving the tests by joining one of the
numerous projects. PR welcome.
• Join IMTC SuperOp 2016 in Miami to run your app
and your infra against many others
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
http://www.imtc.org/event/imtc-superop-2016/