HKG15-411: Browser Testing Framework for LHG --------------------------------------------------- Speaker: Trevor Woerner, Will Chen Date: February 12, 2015 --------------------------------------------------- ★ Session Summary ★ The purpose of this talk is to provide the audience with an introduction to the testing framework used in Web browser performance testing as implemented by LHG. The browser test suite is used to compare browser performance and compliance by using a series of benchmarks in key test categories. Sample browser results for both Android and RDK will be presented. -------------------------------------------------- ★ Resources ★ Pathable: https://hkg15.pathable.com/meetings/250839 Video: https://www.youtube.com/watch?v=TtAMRnzB2nw Etherpad: http://pad.linaro.org/p/hkg15-411 --------------------------------------------------- ★ Event Details ★ Linaro Connect Hong Kong 2015 - #HKG15 February 9-13th, 2015 Regal Airport Hotel Hong Kong Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org

1. Presented by
Date
Browser Testing
Framework for LHG
Trevor Woerner, Will Chen
February, 2015

2. Outline
● Overview of the test suite
● Test category
● Run the test
○ On Linux
○ On Android
○ Verified platforms
● Test output
○ Output file
○ Browser comparison

3. Overview of the test suite

4. Overview of the test suite
● Linaro Web Browser Test Suite is a
benchmark tool to evaluate the
performance of web browsing.
● It consists of Bash script, Python script,
html, and Javascript files.

5. Overview of the test suite
● Test suite contain six categories (Sites,
Layout, Image, Rendering, Javascript,
HTML5).
● The testing result is reported in
milliseconds, which is a measurement of
the test suite processing time.

6. Overview of the test suite
● Features:
○ Automatic
○ Can be used on Linux and Android platform
○ Easy to add test cases for future extension
○ Browser options can be configured
○ Test case/category/iteration number can be
configured

7. Overview of the test suite
● Compared to other test suite:
○ High testing coverage - Combine most of the
existing browser test suite on the internet.
○ Easy to compare - One score to represent the
performance.
○ Has recent popular test cases - Add web album,
4k image, and web game test.

8. Test category

9. Test category
● Sites: This benchmark is an offline copy
of the front page of some real and
commonly used web sites.

10. Test category
● Layout: this benchmark performs mostly
static HTML/CSS or javascript controlled
dynamic HTML layout of objects on the
page. This is supposed to benchmark
sites that are mainly static and with little
computation, few images and little user
interaction.

11. Test category
● Image: this benchmark performs mainly
image rendering and layout. This is
supposed to benchmark sites with lots
of images. For instance, it should
demonstrate the laying out of the
images in an online album.

12. Test category
● Rendering: this benchmark mainly
consists of rendering animated vector
graphics into a canvas. It is controlled
by Javascript, but most of the time is
spent in graphics libraries. This is
supposed to benchmark sites that are
graphic intensive.

13. Test category
● Javascript: this benchmark consists of
Javascript tests, mainly focusing on the
Javascript engine performance. It is
supposed to reflect performance of rich
internet application (RIA) sites. For
instance it should demonstrate the
usage of an online mail agent.

14. Test category
● HTML5: this benchmark performs
image/canvas drawing, graphic
manipulation, and 2D shape rendering
specific for HTML5. For instance it
should demonstrate the usage of the
web game.

15. Run the test
● On Linux

16. Run the test on Linux
● We need a browser, bash script, and
Python to run the test suite on Linux.
● Put test suite in the storage device and
issue command, then the test suite will
finish all the tests and generate testing
result automatically.

17. Run the test on Linux

18. Run the test
● On Android

19. Run the test on Android
● We need to establish ADB connection
between testing device and host
computer to run the test.
● The test suite can install itself in the
testing device automatically. The
installed path is configured in script file.

20. Run the test on Android
Host computer
Target device
ADB connection

21. Run the test
● Verified platforms

22. Run the test on Android

23. Test output
● Output file

24. Test output
suite_scores.out: Record
Real Time and Process Time
of each testing category.
Also record the summing
time of all the categories
except html5. The total
Process Time is the final
benchmarking score (the
unit is ms).

25. Test output
all_scores.out: Similar with
suite_scores.out, but it will
append each testing result in
the file. So, we can check
the previous result in this file.

26. Test output
run.”category”.log: The log
file when run the test for
specific “category”.

27. Test output
run.”category”.out: Record
the Suite Score of each test
case in the specific
“category”. This file will
record each case’s score
and sum of the score. It will
also contain the real time,
user time, and system time
while running this category
(the unit is second).

28. Test output
test_result_”
category”.html:
Similar with run.”
category”.out file, but
it is more easy to
read. It shows the
name and Suite
Score of each case
by table with html
format.

29. Test output
● Browser comparison

30. Test output
Comparison between chromium and firefox browser: (Platform is Intel i5
CPU + Ubuntu on Laptop)

31. Test output