This document discusses testing strategies for the iOS Sync Engine. It provides overall test stats, including that unit tests make up 80% of tests and integration tests make up 20%. The main testing strategy is to test against a fake backend called MockTransportSession, which provides advantages like speed, flexibility and independence over using stubs. MockTransportSession simulates the real backend by making fake requests, managing an internal database, and responding to requests. Future plans include splitting tests into smaller frameworks to reduce runtime, running tests on pull requests, and using record/replay to keep MockTransportSession in sync with the real backend.

1. Testing Sync Engine

2. Testing (iOS) Sync Engine

3. Agenda
• Overall tests stats
• Our test strategies
• MockTransportSession
• Workflow
• Problems
• Future plans

4. Overall tests stats

5. • 3356 tests on OSX
Overall tests stats

6. • 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
Overall tests stats

7. • 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
• ~ 7.5 minutes to build and run all the tests on
one platform (build ~ 40s, tests ~ 7 min)
Overall tests stats

8. • 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
• ~ 7.5 minutes to build and run all the tests on
one platform (build ~ 40s, tests ~ 7 min)
Overall tests stats

9. • Unit tests ~ 80%
Overall tests stats

10. • Unit tests ~ 80%
• Integration tests ~ 20%
Overall tests stats

11. • Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
Overall tests stats

12. • Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
• Tests of tests
Overall tests stats

13. • Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
• Tests of tests
• Sorry, no UI tests
Overall tests stats

14. Our test strategies

15. What we DON’T do:
Our test strategies

16. What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Our test strategies

17. What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Why?
• Stubs are not that flexible.
Our test strategies

18. What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Why?
• Stubs are not that flexible.
• It’s still slow.
Our test strategies

19. What we really do:
• Test against fake backend
Our test strategies

20. What we really do:
• Test against fake backend
Advantages
• Fast
• Flexible
• Independent
• Testable
Our test strategies

21. What we really do:
• Test against fake backend
Advantages
• Fast
• Flexible
• Independent
• Testable
Disadvantages
• Need to write (and
test!) code
• Need to be in sync
with real backend
(can be improved)
Our test strategies

22. MockTransportSession

23. WTF is THAT?
MockTransportSession

24. WTF is THAT?
ZMTransportSession
ZMTransportSessi
onErrorCode
ZMBackgroundAc
tivity
ZMURLSessionCa
ncelTimer
ZMURLSessionSw
itch
ZMURLSession
ZMTransportCodec
ZMAccessToken
ZMUserAgent
ZMReachability
ZMTransportData
ZMTransportResp
onse
ZMTaskIdentifierM
ap
ZMExponentialBa
ckoff
ZMAccessTokenH
andler
ZMTransportRequ
estScheduler
ZMNetworkSocket
ZMDataBuffer
ZMWebSocketFra
me
ZMWebSocketHan
dshake
ZMWebSocket
ZMPushChannelC
onnection
ZMTransportPush
Channel
ZMStreamPairThre
ad
MockTransportSession

25. WTF is THAT?
ZMTransportSession
MockTransportSession

26. WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
MockTransportSession

27. WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
MockTransportSession

28. WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
• Handles authentication
MockTransportSession

29. WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
• Handles authentication
• Opens/closes web socket connection
MockTransportSession

30. WTF is THAT?
MockTransportSession - is a mock of
ZMTransportSession
MockTransportSession

31. WTF is THAT?
MockTransportSession - is a mock of
ZMTransportSession
MockTransportSession - acts as a fake backend
MockTransportSession

32. WTF is THAT?
• Make fake requests
MockTransportSession

33. WTF is THAT?
• Make fake requests
• Records requests and responds to them
MockTransportSession

34. WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
MockTransportSession

35. WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
• Can simulate remote changes.
MockTransportSession

36. WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
• Can simulate remote changes.
• Can send push channel (web socket) events
MockTransportSession

37. • Before each test we insert objects in internal
database (in memory)
MockTransportSession

38. • Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
MockTransportSession

39. • Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
• It records and process incoming request (filtering by
path and lots of ifs)
MockTransportSession

40. • Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
• It records and process incoming request (filtering by
path and lots of ifs)
• Performs some CRUD action on database
depending on request (add message, create
conversation, fetch conversation)
MockTransportSession

41. • Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
• It records and process incoming request (filtering by
path and lots of ifs)
• Performs some CRUD action on database
depending on request (add message, create
conversation, fetch conversation)
• Creates and sends response. Optionally can
delegate it to other object (i.e. test case).
MockTransportSession

42. Workflow

43. • Finished feature - run all the tests
Workflow

44. • Finished feature - run all the tests
• Create pull request
Workflow

45. • Finished feature - run all the tests
• Create pull request
• Review, merge
Workflow

46. • Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
Workflow

47. • Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pushes to remote branches (master/develop)
Workflow

48. • Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pushes to remote branches (master/develop)
• Xcode bots make integration on each push to master/
develop
Workflow

49. • Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pushes to remote branches (master/develop)
• Xcode bots make integration on each push to master/
develop
• Each bot for each platform runs all the tests
Workflow

50. • Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pushes to remote branches (master/develop)
• Xcode bots make integration on each push to master/
develop
• Each bot for each platform runs all the tests
• Bots archive and upload binary to the cloud
Workflow

51. Problems

52. Solved:
• Tests isolation (mostly database state)
Problems

53. Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Problems

54. Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
Problems

55. Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time
U don’t like
me? =(
Problems

56. Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time
• Sync with real backend
Problems

57. Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time
• Sync with real backend
• Flaky tests
Problems

58. Future plans

59. • Split to smaller frameworks - to improve run time
Future plans

60. • Split to smaller frameworks - to improve run time
• Setup Buildasaur - to run tests on Github pull
requests
Future plans

61. • Split to smaller frameworks - to improve run time
• Setup Buildasaur - to run tests on Github pull
requests
• Record/replay tests for MockTransportSession -
to keep it in sync
Future plans

62. That’s it!