3. About me
• 29 years coding
• Agile Practices (inc. TDD) since 2003
• Certified ScrumMaster, Agile Coach,
Trainer, Speaker
• 4+ years with Atlassian (JIRA
Development Team Lead)
• Co-founder of Spartez
14. XStream
OSGi REST Velocity
Jersey
Pico
Active OfBiz EntityEngine Guava
Objects LESS
Java OpenSocial
OS Workflow
Quartz
Jackson
Mixture of Lucene
technologies JQuery
Spring
XML Underscore Soy
Seraph
OAuth Maven2
JSP Webwork JDBC
Javamail Backbone.js SpringDM
63. You commit at 3 PM
You get “Unit Test Green” email at 4PM
64. You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
65. You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
You get flood of “Red Test X” emails at 4 - 9PM
66. You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the
other side of the globe
67. You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the
You
other side of the globe
68. “Slow CI loop and non-
deterministic tests are
strong inhibitor of change
instead of the catalyst”
by W. Seliga
69. “We probably spend more
time dealing with the JIRA
test codebase than the
production codebase”
72. Outcomes
• Development slows down
• Devs are afraid of change
• Software difficult to release
• Significant amount of time spent on
analysing test failures
• Morale goes down
74. Feedback Loop Speed
• Tiniest change triggers test avalanche
• Lack of responsibility syndrome
• Devs do not run tests locally (speed)
• Before you get the results you are at
home
75. Quality
• Non-deterministic tests (races,
timeouts)
• Catching up with UI changes
• 1 red test hides new failures
• Ignoring always red tests in
dangerous ...
80. Decisions which do not
scale
• All unit tests in one maven module
• All functional tests in one maven module
81. Decisions which do not
scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one
module
82. Decisions which do not
scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one
module
• Every commit triggers rebuild and re-test of
everything
83. Decisions which do not
scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one
module
• Every commit triggers rebuild and re-test of
everything
• Monolithic test framework / utils
84. Decisions which do not
scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one
module
• Every commit triggers rebuild and re-test of
everything
• Monolithic test framework / utils
• Opaque fixtures
90. Page Objects Pattern
• Page Objects model UI elements (pages,
components, dialogs, areas) your tests
interact with
• Page Objects shield tests from changing
internal structure of the page
• Page Objects generally do not make
assertions
• Designed for chaining
91. Page Objects Example
public class AddUserPage extends AbstractJiraPage @Override
{ public TimedCondition isAt()
{
private static final String URI = return and(username.timed().isPresent(),
"/secure/admin/user/AddUser!default.jspa"; password.timed().isPresent(), fullName.timed().isPresent());
}
@ElementBy(name = "username")
private PageElement username; public AddUserPage addUser(final String username,
final String password, final String fullName, final
@ElementBy(name = "password") String email, final boolean receiveEmail)
private PageElement password; {
this.username.type(username);
@ElementBy(name = "confirm") this.password.type(password);
private PageElement passwordConfirmation; this.passwordConfirmation.type(password);
this.fullName.type(fullName);
@ElementBy(name = "fullname") this.email.type(email);
private PageElement fullName; if(receiveEmail) {
this.sendEmail.select();
@ElementBy(name = "email") }
private PageElement email; return this;
}
@ElementBy(name = "sendemail")
private PageElement sendEmail; public ViewUserPage createUser()
{
@ElementBy(id = "user-create-submit") return createUser(ViewUserPage.class);
private PageElement submit; }
@ElementBy (id = "user-create-cancel")
private PageElement cancelButton; public <T extends Page> T createUser(Class<T> nextPage,
Object...args)
@Override {
public String getUrl() submit.click();
{ return pageBinder.bind(nextPage, args);
return URI; }
}
...
92. Using Page Objects
@Test
public void testServerError()
{
jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class)
.addUser("username", "mypassword", "My Name",
"sample@email.com", false)
.createUser();
// assertions here
}
93. Using Page Objects
@Test
public void testImportSampleProject() {
final PivotalImporterSetupPage setupPage = getSetupPage();
Assert.assertEquals("1. Connect", setupPage.getActiveTabText());
final PivotalProjectsMappingsPage projectMappingPage = setupPage.next();
Assert.assertEquals("2. Project Mapping", setupPage.getActiveTabText());
Assert.assertTrue("Expecting all project to be selected by default",
projectMappingPage.areAllProjectsSelected());
projectMappingPage.setImportAllProjects(false);
projectMappingPage.setProjectImported(sampleProject, true);
projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE");
final ImporterFinishedPage importerLogsPage =
projectMappingPage.beginImport().waitUntilFinished();
Assert.assertTrue(importerLogsPage.isSuccess());
Assert.assertEquals(0, importerLogsPage.getGlobalErrors().size());
Assert.assertEquals("1", importerLogsPage.getProjectsImported());
// more assertions here
}
94. More on Page Objects
• Design for reusability
• Design for sharing - libraries of Page
Objects
• Good support by WebDriver/Selenium 2
• Atlassian Selenium 2.0
120. Splitting Codebase
• Tests closer to tested code
• Less to test
• Testing less frequently
• Increased team responsibility
• Restructuring CI hierarchy - more
complicated picture
133. Execution Time:
Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
134. Execution Time:
Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
135. Execution Time:
Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
136. Execution Time:
Test Level
Speed
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
137. Execution Time:
Test Level
Speed
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
138. Execution Time:
Test Level
Speed Confidence
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
139. Execution time - Cont.
• Batching
• Several tests per single set-up
(violation of test isolation)
• REST-based assertions
• Remove / merge overlapping tests
140. Execution time - Cont.
• IDs over CSS/JQuery Selectors over XPath
• JUnit tests running in the container
• In-process testing
• In-memory DBs
• Mocking web servers
• Reducing framework initialization time
• Test Optimization (Clover)
141. Waiting time
• More build agents
• Shorter and smaller tests
• No sleep()
• Avoiding long fixture setup (hot
container, fragmented setup)
142. Preparation Time
• SCM performance
• Container set-up
• Compilation time (GWT...)
• Maven...
• Artifacts passing
143. So how about the
goals?
Is 10 seconds build realistic?
146. Realistic Goals (for us*)
Time Type
for unit tests for 95% of the
2 min
commits
for base smoke functional tests
5 min
for 95% of the commits
for ALL tests for 95% of the
15 min
commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams
147. Realistic Goals (for us*)
Time Type
for unit tests for 95% of the
2 min
commits
for base smoke functional tests
5 min
for 95% of the commits
for ALL tests for 95% of the
15 min
commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams
148. Realistic Goals (for us*)
Time Type
for unit tests for 95% of the
2 min
commits
for base smoke functional tests
5 min
for 95% of the commits
for ALL tests for 95% of the
15 min
commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams
149. Realistic Goals (for us*)
Time Type
for unit tests for 95% of the
2 min
commits
for base smoke functional tests
5 min
for 95% of the commits
for ALL tests for 95% of the
15 min
commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams
150. Realistic Goals (for us*)
Time Type
for unit tests for 95% of the
2 min
commits
for base smoke functional tests
5 min
for 95% of the commits
for ALL tests for 95% of the
15 min
commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams
153. Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams
154. Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams
155. Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams
156. Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams
157. Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams
158. Our Possible Future
• Further splitting the code-base, incubation
and maturity
• Finer-grained team responsibilities
• Merciless quarantine and purging of flakey
tests
• More page objects, less old-school Selenium
• Refactoring/removal of slow tests
• More REST-driven test fixtures and assertions
185. There are no universal
rules - silver bullets
We are expected to find
optimum balance for our
specific case
186. There are no universal
rules - silver bullets
We are expected to find
optimum balance for our
specific case
Definition of “optimum”
constantly changes