4. Goals
Goals of Test Automation
Improve quality.
Understand the system under test.
Reduce and not introduce risk.
Leo Arias (Canonical) Defect Localization September 24, 2013 4 / 26
5. Goals
Improve Quality
Tests enable us to specify the behavior in various scenarios
captured in a form that we can then execute.
Prevent bugs from being introduced.
Defect localization.
Leo Arias (Canonical) Defect Localization September 24, 2013 5 / 26
6. Goals
Defect Localization
Customer tests tell us that some behavior isn’t working as expected,
the Unit tests tell us why.
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails but no unit tests fail, it indicates a missing unit
test.
Leo Arias (Canonical) Defect Localization September 24, 2013 6 / 26
7. Goals
Defect Localization
Customer tests tell us that some behavior isn’t working as expected,
the Unit tests tell us why.
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails but no unit tests fail, it indicates a missing unit
test.
Leo Arias (Canonical) Defect Localization September 24, 2013 6 / 26
9. Isolation
Isolation
We should strive to isolate the system under test as much as possible
the parts we choose not to test.
Failure to isolate lead to context sensitivity because we end up testing
too much software all at once.
Leo Arias (Canonical) Defect Localization September 24, 2013 8 / 26
10. Isolation
Isolation
We should strive to isolate the system under test as much as possible
the parts we choose not to test.
Failure to isolate lead to context sensitivity because we end up testing
too much software all at once.
Leo Arias (Canonical) Defect Localization September 24, 2013 8 / 26
22. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26
23. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26
24. Defect Localization problems
Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).
Leo Arias (Canonical) Defect Localization September 24, 2013 20 / 26
26. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26
27. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26
28. Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will affect.
Leo Arias (Canonical) Defect Localization September 24, 2013 22 / 26
29. Proposal
Proposal
We should focus on defect localization and isolation while adding test
coverage.
Leo Arias (Canonical) Defect Localization September 24, 2013 23 / 26
30. Proposal
Short-term actions
When an automated test fails because of an error or a behavior change
on another project:
File a bug describing the problem and the projects affected.
<QA> Subscribe to the bug and follow the process to its
resolution.
Make sure all the people affected are notified.
<developers> Add automated customer tests with every fix to
prevent regressions.
<QA> Make sure no fix comes without a customer test. Check that
they are added on the right project.
Leo Arias (Canonical) Defect Localization September 24, 2013 24 / 26
31. Proposal
After the release...
<QA> Analyze the current API and GUI coverage.
<developers> Add isolated tests to fill the holes.
<developers> Make sure we only rely on APIs and behaviors with
automated customer tests.
<QA, developers> Write a test suite specific to test the
integration of all the projects on the different Ubuntu images and
devices.
<design, QA, developers> Start adding user tests since the
expected user experience is defined.
Establish an effective way for us to communicate between teams.
Leo Arias (Canonical) Defect Localization September 24, 2013 25 / 26