Resource: LCA13 Name: Android Infrastructure Automation Improvements Date: 05-03-2013 Speaker: Tyler Baker

1. ASIA 2013 (LCA13)
Tyler Baker – March 2013
LAVA Workshop @ LCA13
A LAVA Success Story

2. ASIA 2013 (LCA13)
www.linaro.org
Introduction
Tyler Baker – Former Android Platform Engineer at Intermec
Started working with Android early 2008
Board bring ups on custom hardware designs
Boot loader porting
Development of BSP and device drivers
1D/2D Imaging solutions
System services and application development
RIL integration and debugging
System administrator for Gerrit, Jenkins, and LAVA
Android system internals course instructor at Marakana
Studied Computer Science and Applied Mathematics at Western Washington University
In my spare time I enjoy designing autonomous aerial drones, snowboarding, and fishing.
Located in Bellingham, WA

3. ASIA 2013 (LCA13)
www.linaro.org
Perspective
Working for an OEM designing and developing enterprise
products has given me unique perspective
The intent of this presentation is to highlight the success I
have had utilizing LAVA
Takeaway Points
Continuous Integration Philosophy
What are the pain points associated with existing test frameworks
Why LAVA was adopted
How LAVA can be easily extended
How LAVA can be integrated into CI environments
Why everyone here should consider LAVA

4. ASIA 2013 (LCA13)
www.linaro.org
Continuous Integration Philosophy
Imagine CI as the LIM(x->0) of 1/x
There is a vertical asymptote at x = 0
The function increases without bound
Think of true CI being the asymptote
Select a number close to zero, and I will select one that is closer

5. ASIA 2013 (LCA13)
www.linaro.org
Continuous Integration Philosophy Continued
Achieving true CI is very difficult and some say impossible
There is always something else that can be automated
CI practices and tools are constantly evolving
New hardware platforms constantly drive CI software changes
Early stages of hardware and software development can be unstable
Boot loader CI?
What is the return on investment?
In my opinion, the best approach to CI is to evolve your strategy as the
technologies changes

6. ASIA 2013 (LCA13)
www.linaro.org
Pain Points
Seemingly many companies build their own automated test
framework and would like you to purchase it
Intellectual property
Licensing
Closed Source
Paid support
Costly integration efforts
Technical debt incurred to modify the system
Integration with corporate IT infrastructure I.E (LDAP)
Scalability?
Extensibility?
Multi platform?
Reporting?
Dashboards?
KPI?
There is no perfect solution yet

7. ASIA 2013 (LCA13)
www.linaro.org
Linaro Automated Validation Architecture
I was first introduced to LAVA at ELC 2011
Began to add support for our rugged Android OMAP3 based
handset shortly after
Why LAVA was adopted
Flexible architecture
Compartmentalized
Easy to extend to meet the any need
Automated recovery of bricked devices
Well supported by LAVA team
Tests already integrated
Open Source
LAVA itself is an enabler
Contributing changes upstream reduces technical debt

8. ASIA 2013 (LCA13)
www.linaro.org
LAVA Extensions
CLI tools are wrapped, exposed as actions to the dispatcher
Very simple to extend the existing software once you
understand entry points
Developers who write services or applications should also be
able to take advantage of LAVA
The issues that needed to be solved
Facilitate downloading, installing and uninstalling applications/system
services
Download and deploy files to the file system
Run commands on the client side over ADB
Run unit or functional tests and get the the results

9. ASIA 2013 (LCA13)
www.linaro.org
LAVA Extensions Continued
Extended lava-android-test
Download and push files to a specific location on the file system
Install, uninstall, and reinstall applications
Run client side shell commands
insmod, chmod, reboot
Download, install, and run any Gtest or Junit binary with correct parse
Can be referenced in a job file
Was able to successfully test Android OTA updates

10. ASIA 2013 (LCA13)
www.linaro.org
LAVA and LDAP
To manage existing user accounts an extension was needed
to allow LDAP authentication
Gerrit and Jenkins both support LDAP
LAVA server extension written in a single day
Used the OpenID authentication implementation as reference
Straight forward
LAVA can adapt to almost any need

11. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration
Having Jenkins submit a job to LAVA after a build is trivial
It would be nice to have a plug-in, but not necessary
LAVA CLI tools work just fine
The issues
When you submit a job in LAVA it is asynchronous operation
LAVA dashboard holds the test results
The developer must seek out the test results for his/her build
This violates the “rapid feedback loop” CI principle
How will LAVA be instructed to download the correct binaries from Jenkins
in a dynamic fashion when the job file is static?

12. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
Extended lava-tool
Created a function called “testjob-run”
It schedules the job
It blocks like a synchronous operation
It monitors the job progress
When the job completes, the results are fed to STDOUT
There is also a option to write the results out into xUnit format
Now the results can be visualized right from Jenkins!
Not only just graphs, but the same data you might find in a results bundle
This data can be pushed into an existing test results database

13. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued

14. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued

15. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
How are the job files being dynamically created?
The job files used are “templates”
sed is used to replace “variables” in the LAVA job template
This allows the dynamic creation of Jenkins links in the job template submitted to
LAVA
A bit of a hack, but so far been very robust
When the upstream build finishes, the modified job template(s) are
downloaded to the downstream LAVA test job
These modified job templates are scheduled and results pushed up into
Gerrit

16. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration
Jenkins can listen for Gerrit events to trigger builds
Once triggered Jenkins will download the patch set, applied it to the
source tree and build
This will allow a developer build and test their code before ever merging
Since the test results are now being reported to Jenkins
It is possible to use the Gerrit SSH API to “code review” the change set that got built and
tested
A thin bash wrapper parses the STDOUT from the lava-tool extension
The failed results get put into the code review message and -1 is applied to the change set
If all the results pass a +1 is applied to the change set
If the is job status is returned as “incomplete” then 0 is applied to the change set
In every case a link to the LAVA test bundle is included in the message
The feedback from the build and the test(s) will be reported all in one place
This is what CI is all about, giving the developer timely feedback about his/her changes

17. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration Continued

18. ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration Continued

19. ASIA 2013 (LCA13)
www.linaro.org
CI Architecture Overview

20. ASIA 2013 (LCA13)
www.linaro.org
LAVA Canary Testing
Build machines can build boot loaders, kernel, modules, and
Android file system in less than two minutes
Change set applied
Dirty build, no CCACHE
18 minutes to build all components from a clean 4.2.1 tree
Quad Xeon processors with 48GB of RAM per processor
Building on a ramdisk
All units can recover from a bad kernel or file system
Once builds are smoked tested on the canary units, they are
pushed out to larger populations as OTA updates
Commit, build, and receipt of test results takes just under 8
minutes
This includes full OS build, downloading the build to LAVA server, flashing
full OS using fastboot, booting, running a quick smoke testing which
includes monkey

21. ASIA 2013 (LCA13)
www.linaro.org
LAVA Canary Testing

22. ASIA 2013 (LCA13)
www.linaro.org
CI Build Machines

23. ASIA 2013 (LCA13)
www.linaro.org
LAVA Multi Platform Support
Built a extension in a matter of days to support the Windows
Mobile platform
Flashing of the OS
Running tests
Reporting results
IOS support is being implemented

24. ASIA 2013 (LCA13)
www.linaro.org
Conclusion
Planning to upstream all the extensions mentioned in this
presentation for the benefit of the community
LAVA may not do everything you want but can be easily
extended
LAVA is easy to use
LAVA has a great team of developers
LAVA can be tightly integrated into a CI environment
I would personally like to see LAVA emerge as the gold
standard for automated testing frameworks

25. ASIA 2013 (LCA13)
www.linaro.org
Thank you!
You can reach me at tyler.baker@linaro.org or on IRC as
tyler-baker

26. LAVA Hands On – Wednesday 4pm - 6pm
• Requirements
• Laptop
At least 4GB Ram
Virtualisation support (i.e. Intel i3-i7)
• Software
VirtualBox 4.1 or later (Linux, Windows and OS X
supportedhttps://www.virtualbox.org/
LAVA VM Image
http://deb.li/lavavm
• Limited space, but some spaces available
• E-mail lca13-lava@linaro.org and turn up

27. More about Linaro Connect: www.linaro.org/connect/
More about Linaro: www.linaro.org/about/
More about Linaro engineering: www.linaro.org/engineering/
ASIA 2013 (LCA13)