My final talk on the Appium mobile grid and getting started.
- Benefits of parallelization.
- The pros and cons of both cloud services and local setup.
- Getting connected devices information.
- Setting up Grid and Appium servers.
- Getting critical metadata to insert into Allure.
- The benefits of distributed tests vs parallel and when best to use them.
- Setup examples for Android and iOS.
- Leveraging cloud services by sending only selected tests to the cloud.
- Reduce service costs.
- Expose your app to a greater amount of devices and os's.
- A demo of Wunderlist's local grid.
3. And each one of these cases?
• Validate every locale / languages your app supports.
• Validate every OS version your app supports.
• Validate different resolutions.
• Validate different screen sizes (phones, tablets, phablets etc..).
• Validate different device manufacturers.
4. You could try the single test
approach!
Not recommend…
12. Test On The Cloud
• The Pros:
• It’s the present and the future of automated testing.
• Ease of setup and integration.
• Save on costs of buying mobile devices or machines.
• Some test services provide you with excellent analytics and reporting.
• Some test services integrate well with cloud CI services. e.g. Travis CI.
• You expose your app to a much larger range of devices and OS’s.
• The Cons:
• It can be difficult sometimes to narrow down issues.
• Network problems.
• Latency issues.
• Outages can occur.
• Bugs in their software.
• Acquired by another company and then shutdown.
• Costs can become significant when running CI on every commit.
13. Cloud Test Services
• Sauce Labs
• Test Object
• Amazon Test Cloud (was Appthwack)
• Google Test Lab (was Appurify)
• Perfecto Mobile
• Testdroid
• TouchTest
• Keynote (was Device Anywhere)
Just to name a few…
14. The Mobile Grid
• The Pros:
• You can run tests an unlimited amount without worrying about costs.
• You’re not handcuffed to a cloud test services 24/7 availability.
• You have all the critical test data local for debugging.
• Have full access to every connected device via adb or instruments.
• No latency issues.
• You don't expose your app or sensitive data to the cloud.
• The Cons:
• Can be tough finding documentation or good examples specific to your bindings.
• Takes a bit of time to setup and smooth things out.
• Require some degree of ops knowledge.
• You might not be able to expose your app to a large range of devices and os’s.
• Build and maintain the entire setup (software & hardware).
• Devices can be a real pain! (I’ll cover this later.)
15. What I’ll demo…
• Getting all of the connected devices programmatically.
• Launching Grid hub & Appium nodes programmatically.
• Capturing screenshots, logs, video and logcat for reporting.
• Running single threaded.
• Running tests distributed.
• Running tests in parallel.
• Leveraging cloud test services such as Sauce Labs.
• Reduce your runtime costs by only executing selected scenarios.
• Generating an Allure report with all above metadata attached.
• The grid setup at Wunderlist.
• and lets briefly look at the core code…
16. Get Connected Device Data
• We retrieve connected device data and insert into an environment variable.
• The “thread” value will then match our “TEST_NUMBER” env var for
parallelization. TEST_NUMBER is created from the parallel gem.
17. • Capture device data for
every connected device.
• At a minimum, you need the
UDID to connect appium to
a grid node.
• However, it is important to
know the OS, and SDK
versions too.
18. Node Config Method
• Programmatically create your node based on the connected devices.
20. Appium Server Setup
• Method to setup appium server and pass in any arguments.
• The wonderful Appium developers baked a grid node into the
appium server.
21. Launch Hub & Appium Server Methods
• We set a “THREAD” variable. This will tell our parallel library how
many processes it should run.
• Launch the nodes in parallel so all start at the same time.
22. ADB Methods (Video and Logcat)
• Video records up to 180 seconds for real devices.
• Capture logcat data.
23. Spec Helper Methods
• Parse the DEVICES env variable and match it’s thread to
the running process, then initialize appium.
24. Test Setup and Teardown
• Set the UDID environment
variable for your tests.
• Before each test, start video
and logcat.
• After each test, stop video
and logcat.
• Only attach output to report
on failures!
25. Test Example
• Here is where the UDID environment variable is used. This
will let us know which device ran a specific test.
29. Single Thread Test Example
• As you can see, this example was pretty slow. 48 seconds
in total.
30.
31. • Distributed means we spread each
test to an available node. e.g.
device.
• This example only has two tests
(specs). So in theory we only need
two devices, with each running a
test.
• In this example we have one
emulator (left) and one real device
(right).
• Our runtime goes from 48 seconds
single-threaded to 27 seconds
multi-threaded.
Distributed Test
Example
33. Parallel Test Example
• Parallel in this case means we
run same tests on every
available node. e.g. device.
• This example has two tests
(specs). All tests will run on all
devices/emulators.
• We have five emulators, each
running a different SDK version,
and one real device (bottom
right).
• By testing on all SDK versions,
this gives us a warm fuzzy
feeling our app works correctly
on all versions.
• As you saw, emulator-5554
(GRID5) crashed. Lets look at
the report!
35. Run On The Cloud
• In the example tests theres only one spec tagged with “sauce”.
If you we had more, we could run in parallel for faster test
execution in Sauce Labs. e.g rake android[dist,sauce].
36. Allure Report Sauce Labs
• If needed, you can also download the test assets and attach
to the allure. Example: https://github.com/isonic1/appium-
mobile-grid/blob/master/common/dowload_sauce_assets.rb
37.
38. iOS Setup
• Enable Developer Mode on your devices.
• Pair your devices to your machine.
• Enable UI Automation in the Developer menu.
• Ideally, connect a device running each os version your app
supports. iOS is generally pretty good at backwards
compatibility of os versions.
Real devices…
42. • Mac Pro - 3.5 GHz 6-Core - 32 GB Ram.
• OSX Server with performance mode enabled.
• Two Anker 9 Port USB Hubs.
• Energenie programmable power strip. (Reboots USB
hubs, router, Mac)
• Blink(1) programmable usb light to display CI status.
• Jenkins - builds and runs smoke tests on every commit.
CI Machine Specs
46. Challenges
• Adb disconnects.
• I programmatically rebooting USB hubs before every test run. This reconnects adb.
• WiFi issues.
• I toggle airplane mode on/off prior to every test run.
• Other Possible Solutions:
• Reverse USB tethering. (requires rooting.)
• Using multiple routers on different broadcasting channels.
• Alternatives to real devices:
• Android Studio emulators.
• Cannot video record. *I will open a source a solution to this.
• A big plus IMO is that you have access your apps files (database) without rooting.
• Can use up a significant amount of memory depending on specs and the number running.
• Using Genymotion emulators. It’s the closest vm software to a real device.
• Has video recording but no API or CLI to start the recording. :/
• Some devices just behave erratically. (random reboots, adb connection issues, wonky text output,
battery issues).
47. Things to be excited about!
• A replacement for Apple's xcodebuild that makes it easier to build and test iOS or OSX apps.
• https://github.com/facebook/xctool
• WebDriverAgent is a WebDriver server for iOS that runs inside the Simulator and is written
entirely in Objective-C. Can run multiple iOS simulators in parallel.
• https://github.com/facebook/webdriveragent
• Run iOS tests on multiple simulators in parallel at the same time.
• https://github.com/plu/parallel_ios_tests
• Appium iOS driver, backed by Apple XCUITest.
• https://github.com/appium/appium-xcuitest-driver
• Remote control all your Smartphone devices From the comfort of your browser.
• http://openstf.io/
48.
49. Lastly, some thank you’s
• Thank you to the Selenium open source developers and the
community whom make WebDriver happen.
• Also Dan Cuellar, the creator of Appium, whom is here at the
conference with us.
• If you see Dan, please give him a big bearhug! He’ll love it!
• Without Dan, people like him, and all the open source developers
we would never have these amazing tools to help do our jobs
more efficiently, and for free.