In this webinar presentation, Brian Jordan gives a tour of Code.org’s continuous, automated cross-browser testing suite, newly integrated with CircleCI. Brian discusses how Code.org’s small engineering team approaches testing throughout the product development cycle, including a suite of cross-browser tests using Sauce Labs that now run for every single commit an engineer pushes to their feature branch.

1. Tests for Every Branch
Using CircleCI & Sauce Labs to
Continuously Test CS Curriculum at Code.org
@bcjordan
Brian Jordan, software engineer at Code.org

2. Code.org

3. non-profit

4. expanding
participation in CS

5. Hour of Code

7. full curricula,
district partnerships,
professional
development,
policy change

8. K-12 CS Curriculum

9. 43,000 K-12
teachers trained

10. courses:
400k teachers
12mm students

12. how did Code.org
start automated testing?

13. development
in 2013-2014

14. why test?
what is Code.org's
testing context?

15. the context

16. the context

17. the context

18. the context

19. the context
RTL LTR

20. the context

21. the context

22. so how do we
test all that?

23. how we started
Selenium testing

24. how our tests are
organized and run
selenium-webdriver
cucumber

27. takeaways

28. Build a library of
steps

29. Build a library of
test levels

30. Build a library of
annotations
@no_mobile
@no_ie9
@skip
@db_access@as_student
@eyes

31. who writes tests?

32. who/what
runs tests?

33. 2015: one pipeline

34. 2016: every commit!

35. what challenges
did we end up facing?

36. challenge:
multiple browsers

37. solution:
selenium-webdriver
Sauce Labs

40. challenge:
testing new changes

41. locally:
chromedriver
Sauce Connect

42. challenge:
big buckets of changes

43. pull request tests in
❤

44. pull request tests in
circle.yml
github.com/code-dot-org/code-dot-org

45. tips and tricks
github.com/code-dot-org/code-dot-org
⚡

46. tips and tricks
github.com/code-dot-org/code-dot-org
⏰

47. tips
adding more [commit tags]

48. tips
use Sauce Connect for Circle <=> Sauce Labs

49. tips
use Sauce Connect for Circle <=> Sauce Labs

50. tips
use Debug via SSH and Manual Sessions for debugging

51. tips
debugging failures—APIs to the rescue!

52. tips
debugging failures—APIs to the rescue!

53. challenge:
visual, responsive
changes

54. hackathon solution?

55. not easy...

56. actual thing!

57. using eyes

61. what do tests look
like when running?

62. Speed Run!

63. (not) wasting time

64. speeding up
test runs

65. 2014:
60-90 min of tests

66. parallelization
at browser level

67. 2015:
60-90 min of browser tests
to
20-30 min

68. parallelization
by containers

69. parallelization
by containers

70. justifying
tests

71. the bug collection

72. missing button

73. button height change

74. extra margin

75. overflow: hidden added

76. duplicate button

77. stray pixel

78. Firefox getBBox()

79. Chrome 50
(deprecated SVGElement.offsetWidth)

80. my favorite
bug ever

85. who investigates
failures?

86. dev of the day

87. dev of the day
😭

88. dev of the day
+

89. what's next?

90. further flakiness
investigations
(less flakiness: more
parallelization with less pain)
💪

91. Applitools Eyes test results as
GitHub PR comments

92. takeaways

93. start small

94. everybody
tests

95. periodically
invest in
speed-ups

96. In memory of Laurel Fan
Thanks to the Code.org team who spread the
burden of fleshing out, monitoring and
improving our test infrastructure, especially
Laurel Fan, Brendan Reville, and Brad Buchanan.
Thanks to the kind folks at Sauce Labs, CircleCI
and Applitools who helped us get set up, and to
Dave Haeffner @tourdedave for sharing his
advice on testing.

97. code.org/help
github.com/code-dot-org
Thanks!
@bcjordan
Brian Jordan, software engineer at Code.org
Q&A