Everyone knows you need testing, but what are the different types of testing, how will each type benefit you and what libraries are available to ease the pain? This talk will run through an explanation of each type of testing (unit, integration, functional, acceptance, fuzz, mutation...) explaining upon each level of an Android app, the testing involved, how this will benefit you and how it will benefit your users. It will also explain the architecture of a well tested app. Finally ending with some examples and libraries that ease your accessibility into testing and help with faster more descriptive feedback.
3. - Define different types of testing
- Clarify testing & naming
- Benefits of each type of testing
- Codez available to help you
- Some architecture hints & tips
Abstract
6. - Test a single block of code / function
- Test independently from collaborators
- Only test against interface definitions
- New behaviour : new unit test
Unit Tests - What
7. @Test
public void testGetLastPathSegmentReturnsFilename() {
SecureUrl secureUrl = new SecureUrl(
"http://test.com/some/random/url/filename.mp4");
String lastPathSegment = secureUrl.getLastPathSegment();
assertThat(lastPathSegment).isEqualTo("filename.mp4");
}
Unit Tests - Example
8. - Daily development work
- Allow for continuous refactoring
- Test Driven Development
- Documentation for legacy developers
Unit Tests - When
9. Unit Tests - Where
- All building blocks
- All levels of an application architecture
- Help pinpoint a place of failure
11. - Highlight UI issues
- Guarantee combined system integrity
- Prove integration of components
Unit Tests - Won’t
12. - Interaction between two or more classes
- Use real objects
- Can use threads
- Can access database
Integration Tests - What
13. @Test(expected = IllegalStateException.class)
public void testFindFragmentWithUnknownTagResourceIdThrowsError() {
Activity activity = Robolectric.buildActivity(Activity.class).get();
FragmentManager fM = activity.getFragmentManager();
Resources resources = activity.getResources();
TaskFragmentFinder finder = new TaskFragmentFinder(resources);
finder.findTaskFragment(fM, UNKNOWN_TASK_ID);
}
Integration Tests - Example
14. - Ensure parity of collaboration between objects
- Testing expected changes - db schema
- Testing environment integration
Integration Tests - When
15. - All building blocks
- Interactions between layers
- Help pinpoint integration issues
Integration Tests - Where
17. - Highlight UI issues
- Show specific code block of failure
- Give instant feedback
- Hard to diagnose some failures
- Full system confidence
Integration Tests - Won’t
18. - Whole system, nearly end to end
- Don’t care about intermediary steps
- Slower to run
- Mostly controller driven
Functional Tests - What
20. - Evaluate expectation of sum of the parts
- Confirming feature completion
- TDD Keep you focused
- Confidence in features of system
Functional Tests - When
21. - Span whole architecture
- Touching live systems
- Above integrated components
Functional Tests - Where
23. - Show specific broken units of code
- Lack of knowledge of the details
- Give quick feedback for specific problems
Functional Tests - Won’t
24. - Black box tests
- From user perspective
- Specialised form of functional tests
- Model the final complete system
Acceptance Tests - What
25. public void testRotationInBrowseScreenMaintainsActionBar() {
swipeToBrowseScreen();
Activity activity = rotate(this);
assertActionBarOpen(activity);
}
private void assertActionBarOpen(Activity activity) {
assertTrue("Expected ActionBar open but was closed.",
activity.getActionBar().isShowing());
}
Acceptance Tests - Example
26. - User interface is integrated
- TDD feedback loop
- Capturing differences across devices
- Screenshots for greater feedback
Acceptance Tests - When
27. - Span whole architecture
- Touching live systems
- Above integrated components
Acceptance Tests - Where
29. - Run very fast
- No feedback upon cause (just symptoms)
- Help day-to-day incremental improvements
Acceptance Tests - Won’t
30. - Mutate the state of your code
- Insert faults in your software
- Unit tests are ran
- Tests fail “mutant is killed” thumbs up
- Tests pass “mutant survived” thumbs down
Mutation Tests - What
32. Mutation Tests - Example
if (a == b) {
// do something
}
will be mutated to
if (a != b) {
// do something
}
Different
Mutators:
if (a == b) {
// do something
}
will be mutated to
if (true) {
// do something
}
public int method(int i) {
i++;
return i;
}
will be mutated to
public int method(int i) {
i--;
return i;
}
34. - Find holes in your test suite
- Depends on % code coverage
- Confidence in your unit tests
- Complexity of your problem is high
Mutation Tests - When
35. - Outside of your code
- Around your test suite
- Not written yourself but configured
- Configuration is key
Mutation Tests - Where
37. - Give any confidence in working software
- Show collaboration between objects
- Prove if the application actually works
- Help at all without unit tests
Mutation Tests - Won’t
38. - can be unit, integration, functional
- learn how 3rd
party libraries work
- confirming understanding
- safety net for outside changes
Vendor Tests - What
39. @Test
public void setAndGetActiveSessionAreTrustworthy() {
Session.setActiveSession(mockFacebookSession);
FacebookSession session = Session.getActiveSession();
assertThat(mockFacebookSession).isEqualTo(session);
}
Vendor Tests - Example
40. - Incorporating 3rd party library
- Not beneficial to add retrospectively
- Replacing libraries with confidence
- Wanting to ‘in house’ a library feature
Vendor Tests - When
41. - Around your 3rd party libraries
- One off test
- Suite inside unit tests
Vendor Tests - Where
43. - Confirm your domain code
- Replace unit, integration or acceptance tests
- Adds overhead to development
Vendor Tests - Won’t
44. - Feeding your software random data
- Wait to see what breaks
- It is not logical
- Android = Application Exerciser Monkey
- Combinatory with other tools
Fuzz Tests - What
45. adb shell monkey -p com.your.package.name -v 50000
Fuzz Tests - Example
// Allowing start of Intent { act=mubi.intent.action.ON_BOARD cmp=com.mubi/.onboard.OnboardActivity } in package com.
mubi
:Sending Touch (ACTION_DOWN): 0:(1091.0,659.0)
:Sending Touch (ACTION_UP): 0:(1085.1356,667.17145)
:Sending Touch (ACTION_DOWN): 0:(467.0,404.0)
:Sending Touch (ACTION_UP): 0:(472.5769,398.45746)
// CRASH: com.mubi (pid 754)
// Short Msg: java.lang.IllegalStateException
// Long Msg: java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager
// Build Label: google/nakasi/grouper:4.1.1/JRO03H/405518:user/release-keys
// Build Changelist: 405518
// Build Time: 1364293068000
// java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager
// at android.app.FragmentManagerImpl.saveFragmentInstanceState(FragmentManager.java:586)
// at android.support.v13.app.FragmentStatePagerAdapter.destroyItem(FragmentStatePagerAdapter.java:140)
// at android.support.v4.view.ViewPager.setAdapter(ViewPager.java:413)
// at com.mubi.onboard.OnboardActivity.updateViewPager(OnboardActivity.java:139)
// at com.mubi.onboard.OnboardActivity.onRetrieved(OnboardActivity.java:132)
// at com.mubi.onboard.OnboardTaskFragment$3.run(OnboardTaskFragment.java:77)
// at android.os.Handler.handleCallback(Handler.java:725)
// at android.os.Handler.dispatchMessage(Handler.java:92)
// at android.os.Looper.loop(Looper.java:137)
// at android.app.ActivityThread.main(ActivityThread.java:5195)
// at java.lang.reflect.Method.invokeNative(Native Method)
// at java.lang.reflect.Method.invoke(Method.java:511)
// at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:795)
// at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:562)
// at com.android.internal.os.ZygoteInit.main(Native Method)
// at dalvik.system.NativeStart.main(Native Method)
//
** Monkey aborted due to error.
Events injected: 4191
46. - From the beginning
- Thin slice (UI)
- Continuous integration, faster feedback
- Identifying differences across Android versions
- Highlight refactoring effort
Fuzz Tests - When
47. - Black box testing
- On top of your application (Android)
- Custom fuzz testing - input level
- Continuous integration
Fuzz Tests - Where
49. - Prove application is running correctly
- Only reports crashes
- Highlights quality rather than bugs
- Not a replacement for unit, integration etc
Fuzz Tests - Won’t
50. Helpers / Libraries
- Brief overview
- What test types it benefits
- Any positives
- Any negatives
51. Mockito
//mock creation
List mockedList = mock(List.class);
//using mock object - doesn’t throw exceptions
mockedList.add(“one”);
//selective & explicit verification
verify(mockList).add(“one”);
Mockito is a mocking framework that tastes really good. It lets you write beautiful tests with clean
& simple API. Mockito doesn't give you a hangover because the tests are very readable and they
produce clean verification errors.
- Unit Tests : mocking dependencies
- Integration Tests : questionable use
52. Fest
FEST is a collection of libraries, whose mission is to simplify software testing.
No more confusion about the order of the "expected" and "actual" values. Our assertions are very
readable as well: they read very close to plain English, making it easier for non-technical people
to read test code.
Regular JUnit:
assertEquals(View.GONE, view.getVisibility());
Regular FEST:
assertThat(view.getVisibility()).isEqualTo(View.GONE);
FEST Android:
assertThat(view).isGone(); - Unit Tests
- Integration Tests
- Functional Tests
- Acceptance Tests : anywhere you do assertions
improving readability
53. “Robolectric is a unit test framework that de-fangs the Android SDK jar so you can test-drive the
development of your Android app. Tests run inside the JVM on your workstation in seconds. “
Robolectric
@RunWith(RobolectricTestRunner.class)
public class ViewingQueryTaskTest {
@Test
public void testAlwaysClosesCursor() {
viewingQueryTask.run();
verify(mockCursor).close();
}
}
- Unit tests : for day to day activity including TDD
- Integration tests : to handle Android
- Positive : speed vs on device tests
- Negative : the rabbit hole
54. “Robotium is an Android test automation framework. Robotium makes it easy to write powerful
and robust automatic black-box UI tests. With the support of Robotium, test case developers can
write functional and user acceptance test scenarios, spanning multiple Android activities.“
Robotium
- Functional tests : check when you click a button the
shared preferences are updated
- Acceptance tests : ensure features have been
completed and take screenshots for feedback
public class EditorTest extends ActivityInstrumentationTestCase2<EditorActivity> {
private Solo solo;
public void setUp() throws Exception {
solo = new Solo(getInstrumentation(), getActivity());
}
public void testFileExtensionIsInMenu() {
solo.sendKey(Solo.MENU);
solo.clickOnText("More");
solo.clickOnText("Preferences");
solo.clickOnText("Edit File Extensions");
Assert.assertTrue(solo.searchText("rtf"));
}
}
55. Use Espresso to write concise, beautiful, and reliable Android UI tests
Espresso tests state expectations, interactions, and assertions clearly without the distraction of boilerplate
content, custom infrastructure, or messy implementation details getting in the way.
Espresso
public void testSayHello() {
onView(withId(R.id.name_field).perform(typeText("Dave"));
onView(withId(R.id.greet_button)).perform(click());
onView(withText("Hello Dave!")).check(matches(isDisplayed()));
}
- Functional tests : check when you click a button the shared
preferences are updated
- Acceptance tests : ensure features have been completed, doesn’t
support screenshots as an API but you can still do it
http://tiny.cc/disableAnim
http://tiny.cc/espressoScreenshot
56. The Monkey is a program that runs on your emulator or device and generates pseudo-random
streams of user events such as clicks, touches, or gestures, as well as a number of system-level
events. You can use the Monkey to stress-test applications that you are developing, in a random
yet repeatable manner.
UI Exerciser Monkey
adb shell monkey -p com.your.package.name -v
50000
- Fuzz tests : as already discussed allows you to test input events
Don’t get the Application Exerciser Monkey & the Monkey Runner mixed
up.
App Ex Monkey – Fuzz Testing
Monkey Runner – Android device control using Python scripts
57. The uiautomator testing framework lets you test your user interface (UI) efficiently by
creating automated functional UI testcases that can be run against your app on one or more
devices.
UI Automator
extend UIAutomatorTestCase
UIObject & UISelector
Min SDK 16 Jelly Bean
adb shell uiautomator runtest MyTests.jar -c com.example.MyApp
Tests are in the jar, and the package is the package of the app under test.
- Acceptance tests : acts just like a user.