1. Improve Your Automated Web
Application Testing
Scott McMaster
Google
Kirkland, Washington USA

2. About Me
• Software Design Engineer @ Google.
– Building high-traffic web frontends and services in Java.
– AdWords, Google Code
• Ph.D. in Computer Science, U. of Maryland.
• Formerly of Amazon.com (2 years), Lockheed Martin (2 years), Microsoft
(7 years), and some small startups.
• Frequent adjunct professor @ Seattle University, teaching software
design, architecture, and OO programming.
• Author of technical blog at http://www.scottmcmaster365.com.

3. Testing and Me
• Doing automated testing since 1995.
• Ph.D. work in test coverage and test suite maintenance.
• Champion of numerous unit, system, and performance
testing tools and techniques.
• Co-founder of WebTestingExplorer open-source automated
web testing framework (www.webtestingexplorer.org).

4. Agenda
• Web Testing Challenges
• The Present: Selenium/WebDriver Automation.
– Best Practices
– Things to Avoid
• The Future? Automated Exploratory Testing with
WebTestingExplorer

5. A Few Words About Testing @
Google
• At Google, test engineers are developers. They develop:
– Test code.
– Test frameworks and productivity tools that are used by software design engineers.
• Test code quality == development code quality.
– Test code must be maintainable, robust, and meet other general software engineering
guidelines.
• For more on “How Google Tests Software”, see the series of blog posts by James
Whittaker (starting at http://googletesting.blogspot.com/2011/01/how-google-
tests-software.html).

6. Assumptions in This Talk
• You have at least some background in:
– Web application development and/or testing.
– Basic Selenium and/or WebDriver usage.
• And you would like to improve your testing
through the user interface!

7. How Would You Test This?

8. Some Things on That Page
• Dynamic content and hundreds of HTML elements.
• User content (reviews, likes).
• Multiple dropdown menus.
• Search.
• Security (login).
• State (shopping cart).
• Embedded video.

9. It’s Even Worse
• After we write the tests, customers /
designers will want changes.
– Our tests will probably break.
– Managers don’t want to wait for testing to fix
their broken tests before releasing the software.

10. !!!

11. Calm Down. What Can We Do?
• Write more robust, more maintainable tests!

12. Let’s Use a Slightly Simpler Example

13. Use Cases
• Enter the name of a country and load cities.
– Error message if not found.
• Select a city and load the current weather.
• The weather appears in a table.
• Signed-in user gets weather for their “favorite”
city.

14. Architecture
Rich Browser UI (GWT)
Frontend Server (GWT RPC
servlet)
Remote Web
Service (XML-over- User Database
HTTP)

15. What Testing Should We Automate?
• Unit?
– Yes!
• System?
– Yes!
• End-to-end?
– Yes! This will be the focus of this talk.

16. Web UI System Test Automation
• What tools do we use?
– Selenium WebDriver
– http://seleniumhq.org/projects/webdriver/

17. Selenium WebDriver
“Driving a browser natively as a user would either locally or on a remote machine…”
• Selenium IDE
• Browser plugin for play/record.
• Mostly for non-programmers – don’t use this.
• Selenium RC / Selenium 1
• Deprecated – do not start using this!
• Selenium 2 (WebDriver)
• More concise API.
• Uses browser’s native automation support, works in all
major browsers (but Firefox tends to work “best”).
• Specially designed for rich (AJAX) web UIs.
• (For example, performs style calculations to help
you only work with displayed elements.)

18. Selenium WebDriver Language
Bindings
• Java
• Python
• C#
• Ruby
• PHP
• Perl
• Javascript
My examples will use Java.

19. “Simple” WebDriver Test: Spec
• Load the homepage.
• Find the “country” text box.
• Type in the name of a country that does not exist.
• Find the “load cities” button.
• Click it.
• Wait (important!)
• Verify that the “cities” UI is not displayed.
• Verify that the “country not found” error is displayed.

20. “Simple” WebDriver Test: Code
@Test
public void testUnknownCountryName() {
FirefoxProfile firefoxProfile = new FirefoxProfile();
WebDriver driver = new FirefoxDriver(firefoxProfile);
WebElement countryElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.COUNTRY_NAME_ID));
countryElement.sendKeys("Does not exist");
WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG +
DebugIds.LOAD_CITIES_ID));
loadCitiesButton.click();
WebDriverWait wait = new WebDriverWait(driver, 5000);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG +
DebugIds.NO_CITIES_ID)));
WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.NO_CITIES_ID));
assertTrue(noCitiesElement.isDisplayed());
WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.CITY_ROW_ID));
assertFalse(cityRowElement.isDisplayed());
}

21. Bad News / Good News
• This code is verbose and hard to maintain.
• But we can make it better!

22. Tip#1: Organize Your Locators
• Shortens the code and gives you one easy-to-find
place to change when the UI changes.
private static interface WeatherHomeLocators {
By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID);
By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID);
By noCities = By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID);
By cityRow = By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID);
By city = By.id(GWT_DEBUG + DebugIds.CITY_ID);
By loadWeather = By.id(GWT_DEBUG + DebugIds.LOAD_WEATHER_ID);
By weatherCity = By.id(GWT_DEBUG + DebugIds.WEATHER_CITY);
By weatherTemp = By.id(GWT_DEBUG + DebugIds.WEATHER_TEMP);
By userName = By.id(GWT_DEBUG + DebugIds.USER_NAME);
By login = By.id(GWT_DEBUG + DebugIds.LOGIN);
By weather = By.id(GWT_DEBUG + DebugIds.WEATHER);
By unknownUser = By.id(GWT_DEBUG + DebugIds.UNKNOWN_USER);
}
…
WebElement countryElement = driver.findElement(WeatherHomeLocators.
countryName);

23. Bonus Tip #1a: How to Locate
• In priority order, locate elements by:
1. id (ask developers to add these, make them stable, and
available in all builds)
2. name
3. css class
4. link text (but watch for dynamic content and localization)
5. Xpath expression

24. Tip #2: Use a Base Class for Tests
• Some things you can do in a base class:
1. Create the browser profile and WebDriver.
2. Safely close the WebDriver.
3. Improve on the WebDriver API.

25. Base Class Example
public class AbstractWebDriverTest {
protected static final String GWT_DEBUG = "gwt-debug-";
protected WebDriver driver;
@Before
public void setUp() throws IOException {
FirefoxProfile firefoxProfile = new FirefoxProfile();
driver = new FirefoxDriver(firefoxProfile);
}
@After
public void tearDown() {
driver.close();
}
protected WebElement findElementOrFail(By by) {
WebElement element = driver.findElement(by);
if (element == null) {
org.junit.Assert.fail("Element " + by + " not found");
}
return element;
}
}

26. Other Good Base Class Methods
• Action methods:
protected void click(By by) {
WebElement element = findElementOrFail(by);
element.click();
}
API improvements:
protected void setText(By by, String text) {
WebElement element = findElementOrFail(by);
element.clear();
element.sendKeys(text);
}
Rich assertions:
protected void setText(By by, String text) {
WebElement element = findElementOrFail(by);
assertTrue(element.isDisplayed());
}

27. Remember Our “Simple” Test?
@Test
public void testUnknownCountryName() {
FirefoxProfile firefoxProfile = new FirefoxProfile();
WebDriver driver = new FirefoxDriver(firefoxProfile);
WebElement countryElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.COUNTRY_NAME_ID));
countryElement.sendKeys("Does not exist");
WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG +
DebugIds.LOAD_CITIES_ID));
loadCitiesButton.click();
WebDriverWait wait = new WebDriverWait(driver, 5000);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG +
DebugIds.NO_CITIES_ID)));
WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.NO_CITIES_ID));
assertTrue(noCitiesElement.isDisplayed());
WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG +
DebugIds.CITY_ROW_ID));
assertFalse(cityRowElement.isDisplayed());
}

28. Now It Looks Like…
@Test
public void testUnknownCountryName() {
setText(WeatherHomeLocators.countryName, "Does not exist");
click(WeatherHomeLocators.loadCities);
WebDriverWait wait = createDefaultWait();
wait.until(ExpectedConditions.visibilityOfElementLocated(WeatherHomeLocators.noCities));
assertIsDisplayed(WeatherHomeLocators.noCities);
assertIsNotDisplayed(WeatherHomeLocators.cityRow);
}
Much Better!!!

29. Any More Problems?
• Today you select a country by:
– Typing a country name.
– Clicking the “Load Cities” button.
• What if we change it to:
– Select the country from a dropdown list.
• Our tests all break!

30. Tip #3: Use Page Objects
• Create higher-level action abstractions on a
per-page basis.
• Write tests that use those abstractions.
• Then, if the underlying UI changes, we can
update a single Page Object instead of many
individual tests.

31. Page Object Design Considerations
• Put your Locator definitions there.
• Give it access to your WebDriver API helpers.
• Put your page-specific action and (more
optionally) assertion methods there.

32. Page Object Example
public class WeatherHomePageObject {
public static interface WeatherHomeLocators {
public static final String GWT_DEBUG = "gwt-debug-";
By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID);
By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID);
// etc.
}
private WebDriverWrapper driverWrapper;
public WeatherHomePageObject(WebDriverWrapper driverWrapper) {
this.driverWrapper = driverWrapper;
}
public void loadCitiesForCountry(String countryName) {
driverWrapper.setText(WeatherHomeLocators. countryName, "Does not exist");
driverWrapper.click(WeatherHomeLocators. loadCities);
}
public void assertCitiesNotLoadedDisplay() {
driverWrapper.assertIsDisplayed(WeatherHomeLocators.noCities);
driverWrapper.assertIsNotDisplayed(WeatherHomeLocators.cityRow);
}
}

33. Now It Looks Like…
@Test
public void testUnknownCountryName() {
pageObject.loadCitiesForCountry("Does not exist");
WebDriverWait wait = createDefaultWait();
wait.until(ExpectedConditions.visibilityOfElementLocated(
WeatherHomeLocators.noCities));
pageObject.assertCitiesNotLoadedDisplay();
}
This change is subtle…but important!
It makes your tests maintainable and
robust to change in the UI!
We are pretty happy, but…

34. That Seems Like a Lot of Work!
• It is! Selenium WebDriver tests are:
– Time-consuming to develop.
– Somewhat susceptible to being “flaky” (example: use of “wait”s).
– Highly susceptible to breaking when the UI changes.
– Sometimes hard to maintain.
• Consequently, we often focus more on unit testing, less on
end-to-end tests.
• Usually this is pretty successful, but can we do better?

35. Big Idea
What if we could generate and regenerate interesting tests automatically,
and find bugs in the process?

36. WebTestingExplorer
• Supports Automated Exploratory Testing.
– Automated: Easy to replay (and regenerate) interesting
test cases.
– Exploratory: Systematically discovers new application
states.
– Testing: Includes test case generation, oracle generation,
real-time bug detection, and test case replay.

37. Advantages
• Efficiently test through the user interface.
• Test like a real user.
• Don't miss integration issues and things that
are hard to observe.
– Behind the scenes in the browser.
– Or on other application tiers.

38. WebTestingExplorer Design
• Assumes a testable application.
– Example: We can ask developers to add id attributes.
• Built on top of WebDriver.
– Easy to integrate with existing WebDriver code.
• Highly configurable and extensible.
• Easy to skip/ignore uninteresting and/or unstable regions of
the web page.
– Examples: Ads, “Like” buttons, common headers/footers, etc.

39. WebTestingExplorer Workflow
•Two Complementary Phases:
1. Exploratory
– Incrementally searches for new application states.
– Writes out new test cases and oracles.
– Can search for bugs.
– Tester iterates on the configuration to guide the tool into new areas of the application
(the feedback-driven, exploratory approach to testing).
2. Replay
– Executes tests generated by the Exploratory phase.
– For regression testing.
– A/B, comparison of actual and expected states.

40. State
• An abstraction of the current state of the web application, customizable by
the tester.
• Looking at “state” is key to how WebTestingExplorer generates test cases.
• State can be any combination of:
– Web browser state (HTML, DOM, cookies, ...)
– Web server state (session data, variables, ...)
– Database state (persistent application data)
– User-defined (but some state implementations are included and ready-to-
use!)

41. Exploratory Workflow
• Iteratively and systematically moves through the web
application via the user interface.
• Monitors for different types of bugs.
• Generates increasingly-long and sophisticated test cases
(and oracles) for later replay.
– Driven by feedback derived from state.
• Tester iterates on the configuration to guide the tool into interesting new
areas of the application and away from uninteresting ones (the feedback-
driven, exploratory approach to testing).

42. Automated Exploration
• Changes in observed state between actions move the exploration forward into
longer test cases.
• High-level algorithm:
– Pick a candidate test case (a series of actions – click, set-text, select, etc.).
– Take a state snapshot.
– Do something.
– Take another state snapshot.
– Compare the “before” and “after” states.
– Different? Then:
• Generate a new test case.
• Enqueue new, longer candidate test cases to execute.
– Repeat until out of time, or all test cases of a specified length are executed.

43. Automated Error Detection (Oracles)
• HttpResponseOracle: Checks for allowed and disallowed
HTTP response codes.
• JSErrorCollectorOracle: Looks for Javascript runtime errors in
Firefox.
• FinalStateCheckOracle: Uses expected and observed final
state for regression testing.
• Testers can plug in custom oracle implementations.
– Example: If your application displays an error bar when an error
occurs, you can write an oracle that looks for the error bar!

44. Regression Testing
• Exploration creates a test case.
– Test case includes saved final state for the configured state checkers.
• Replayer is configured with the FinalStateCheckOracle.
• Replayer replays the test case.
• At the end, a state snapshot is taken and compared to the
final state saved in the test case.
• If they are different, a failure is recorded.

45. Configuration
• Selectors for Actionable and Stateful web elements.
• Pre-packaged and custom Oracles.
• Action sequence filters to eliminate redundant or uninteresting
exploration paths.
– Example: MaxRepeatedActionSequenceFilter eliminates action sequences
that repeat the same action more than a set number of times.
• Action generators to create custom actions on specific elements.
• Action sequence prioritizer for test case prioritization during exploration.
• Custom wait conditions to increase stability while executing an action
sequence.

46. Let’s Look at Some Code!

47. Custom Action Generator
• If you have links to external web sites (e.g. www.google.com) , you
probably don’t want to write tests that navigate there.
new TagActionGeneratorConfig("a") {
@Override
public Set<Action> generateActions(WebElementWithIdentifier elementWithId) {
Set<Action> actions = Sets.newHashSet();
String href = elementWithId.getElement().getAttribute("href");
if (href != null && href.startsWith("javascript:")) {
actions.add(new ClickAction(elementWithId.getIdentifier()));
}
return actions;
}
}
• This will disallow test cases for all <a>’s that do NOT execute Javascript.
• This is sufficiently useful that it is included as a library class:
• JavascriptAnchorActionGeneratorConfig

48. Another Example
• If you have a textbox that takes login names, you want to try both valid
and invalid logins.
new IdentifierActionGeneratorConfig(
new IdWebElementIdentifier(GWT_DEBUG + DebugIds.USER_NAME)) {
@Override
public Set<Action> generateActions(WebElementWithIdentifier elementWithId) {
Set<Action> actions = Sets.newHashSet();
actions.add(new SetTextAction(elementWithId.getIdentifier(), "scott@example.com"));
actions.add(new SetTextAction(elementWithId.getIdentifier(), "not a real user"));
return actions;
}
})
This will generate test cases with combinations of valid and invalid login names.

49. A True Story
• I built the WeatherApp just for this conference.
• The first time I ran WebTestingExplorer against it, I
found a bug!
– An empty country name tries to load all the cities in the
world that have weather data!
– Not what I intended, not a good idea.

50. Test Case (XML Format)
<?xml version="1.0" ?>
<TestCase>
<url>http://127.0.0.1:8888/WeatherAppV1.html</url>
<actionSequence>
<actions>
<SetTextAction>
<keysToSend>Not a country</keysToSend>
<identifier class= "IdWebElementIdentifier">
<id>gwt-debug-CountryName</id>
</identifier>
<initial>false</initial>
</SetTextAction>
<ClickAction>
<identifier class= "IdWebElementIdentifier">
<id>gwt-debug-LoadCities</id>
</identifier>
<initial>false</initial>
</ClickAction>
</actions>
</actionSequence>
<finalState>
<VisibleElementsState>
<elementProperties>
<!-- Lots more XML, properties and values of page elements -->

51. Test Case (Pretty Format)
1. Click the element with id gwt-debug-LoadCities
2. Click the element with id gwt-debug-LoadCities
3. Set the text in the element with id gwt-debug-CountryName
to 'China'
4. Set the text in the element with id gwt-debug-UserName to
'scott@example.com'
5. Refresh the page
PASSED

52. WebTestingExplorer is Open Source
Software
• http://www.webtestingexplorer.org
• http://code.google.com/p/webtestingexplorer/
• Contributors welcome!

53. Planned Features
• Parallel exploration.
• Code coverage integration.
• Different state abstractions.
– And advice and data on which ones work best.

54. Conclusions
• End-to-end testing of web applications is hard. But we can do
it!
• We can organize our Selenium WebDriver code to support
rapid development of concise, robust, maintainable tests.
– Organize Your Locators.
– Use a Base Class for Tests
– Use Page Objects
• And for automated exploratory testing:
– Try WebTestingExplorer.

55. Questions?
Scott McMaster
Google
Kirkland, Washington USA

56. 2012-12-20