1. Is It Handmade Code If You Use Power Tools?
Photo by matsbeads on Etsy
Laura Beth Denker (@elblinkin)

2. Is It Handmade Code If You Use Power Tools?
In the beginning...
•Deployments happened weekly
•Development happened on branches
•Release Manager role rotations
efo re
B

3. Is It Handmade Code If You Use Power Tools?
Deployinator
•One button deployment
•Eliminated painful weekly deploys
• Deployments went up to a handful a day
•Eliminated release manager role rotations
• If you committed code, it was your responsibility to push the button
efo re
B

4. Is It Handmade Code If You Use Power Tools?
Voila! Continuous Deployment
•Buy-in
•One-button deployment process (NO fear!*)
•Monitoring and graphs
* NO fear in the deploy process, but still some healthy fear in the code changes

5. Is It Handmade Code If You Use Power Tools?
“Confidence is contagious; so is lack of
confidence.”
Vince Lombardi

6. Is It Handmade Code If You Use Power Tools?
Buildbot
•Developer written and maintained CI System
• Resulted in Test Infrastructure being convoluted with site architecture
• Test infrastructure was untested
•Tests written in Python
• But we’re a PHP shop
•Slow and resource intensive
• Could not run on a developer VM
•Non-deterministic failures
• Flaky tests
• Assertions in helpers
• EC2 connectivity issues

7. Is It Handmade Code If You Use Power Tools?
(Hudson) Jenkins CI
•Open source
•High community involvement
•Highly configurable
•Numerous plugins
•Eliminated the pitfalls of developer written and
maintained CI
21
J u ne

8. Is It Handmade Code If You Use Power Tools?
PyUnit to PHPUnit
•~1,500 tests moved to our lingua franca
•~3 minutes to run new PHPUnit tests
•Eliminated context-switching between
languages
ly 19
Ju

9. Is It Handmade Code If You Use Power Tools?
Commit Mutex
1. Commit code to trunk (maybe merge a development branch into trunk)
2. Wait for Buildbot (and Jenkins) to tests
a. If tests PASS, continue to Deployinator and begin deployment
b. If tests FAIL, kick Buildbot (and Jenkins) because it might be a flake,
then do any of the following
i. Repeat step (b) until it passes
ii. Debug failures and begin fixing trunk
iii. Rollback
Note: While there are committed changes in trunk that have not been
deployed, DO NOT COMMIT

10. Is It Handmade Code If You Use Power Tools?
Why is there a Commit Mutex?
1.Branch development
2.Tests cannot run in a timely fashion on the
developer VMs

11. Is It Handmade Code If You Use Power Tools?
Try Server v.1
•Jenkins parameterized build + shell script
•Parameters
• username
• patch file
•Shell script
• Create a patch file
• Create the JSON
• Send a request to Jenkins via cURL
•Branches NOT SUPPORTED
•Eliminated the Commit Mutex

12. Is It Handmade Code If You Use Power Tools?
g us t 19
Au

13. Is It Handmade Code If You Use Power Tools?
Try Server v.1.1
•Jenkins parameterized build + PHP script
•Parameters
• username
• patch file
•PHP script
• Support for creating git-svn patches
• Support for dry runs
• Support for uploading manually created patches

14. Is It Handmade Code If You Use Power Tools?
Test Classification
•Unit
•Integration
•Functional (System)

15. Is It Handmade Code If You Use Power Tools?
Test Classification
•Functional (System)
• Provides confidence that a particular use case works
• Unit
• Integration

16. Is It Handmade Code If You Use Power Tools?
Legacy CI System is Dead!
er 15
pte mb
Se

17. Is It Handmade Code If You Use Power Tools?
Test Classification
• Functional (System)
•Integration
• Tests interaction between a discrete number of components
• A set of interacting classes
• Interaction with an external service
• Unit

18. Is It Handmade Code If You Use Power Tools?
Test Classification
• Functional (System)
• Integration
•Unit
• Fast
• Deterministic
• NO database
• NO file system
• NO network
• NO threading, forking processes
• NO sleep
• NO time dependencies
• Tests a single object
• Best way to obtain line coverage
• Reduces debug time

19. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
•@group to declare external dependencies
•PHPUnit XML Configuration
•Jenkins Project for each configuration
er 21
Oc to b

20. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group cache
for tests that use memcache or another cache
er 21
Oc to b

21. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group database
for tests that use a database like postgres or mysql
er 21
Oc to b

22. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group network
for tests that talk to external services (in-house or 3rd party)
er 21
Oc to b

23. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group sleep
for tests that call a sleep() function
er 21
Oc to b

24. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group time
for tests that depend on the time of day or day of the year, etc.
er 21
Oc to b

25. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group smoke
for tests that execute cURL and regex based tests against an instance of
the web service under test
er 21
Oc to b

26. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
@group flaky
for tests that fail without code changes
er 21
Oc to b

27. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
There is NO @group for unit tests.
er 21
Oc to b

28. Is It Handmade Code If You Use Power Tools?
PHPUnit @group
•@group to declare external dependencies
•PHPUnit XML Configuration
•Jenkins Project for each configuration
er 21
Oc to b

29. Is It Handmade Code If You Use Power Tools?
40 Deploys in a Day
•November 18th, 2010
• <5 months after first dedicated Test & Automation engineer
• <3 months after releasing Try Server
• 1 week before Thanksgiving freeze
•34 deploys the day after
ber
No ve m !
!!
18

30. Is It Handmade Code If You Use Power Tools?
How do we keep the momentum?
•The tests ported to PHPUnit are flaky
• Re-building intermittent failures is wasting time in the push queue
•We are hiring more engineers
• We need to deploy as fast, if not faster
•We are writing more tests
• More tests can take more time, so we need to write faster tests
• Started at 1,500 tests, now running >7,000 tests
ber !
Cy y !
Mo n da

31. Is It Handmade Code If You Use Power Tools?
Random Data BAD
•Fixtures Framework
•Written and maintained by developers
•Original intent
• Generate data to run test instance of the website
•Unexpected usage
• Flaky and hard to debug integration tests posing as unit tests

32. Is It Handmade Code If You Use Power Tools?
DBUnit for Multiple Databases
•Shard-ed database architecture
•Our ORM talks to multiple databases
• At least the index and a shard
•PHPUnit_Extensions_Database_MultipleDatab
ases
•Increased clarity in old tests ported to PHPUnit
• Data explicitly specified in YAML, not at random
• Developer needs to know what databases 8test will touch
ra
em be
De c

33. Is It Handmade Code If You Use Power Tools?
Worth Mentioning: Try Server v.2
•Master try project in Jenkins
• PHP Script executed in Jenkins shell builder
• Started all sub-projects over cURL
• Pinged all sub-projects for status
• Used Jenkins Text Finder plugin to toggle project state
• Sent a unified result e-mail
•Eliminated problems finding test results
be r8
De cem

34. Is It Handmade Code If You Use Power Tools?
Worth Mentioning: Try Server v.2
be r8
De cem

35. Is It Handmade Code If You Use Power Tools?
Why to mock?
unit tests
test the functionality of a single method in a single object with the
expectations of the collaborating objects
test case
as a suitcase contains suits, a test case contains tests

36. Is It Handmade Code If You Use Power Tools?
PHPUnit Mock Helpers
Making it easier to create mocks,
should support writing more unit tests
• Tweaking prod code
• Writing short, tested helpers for internal types
• PHPUnit_Extensions_MockObject_Stub_ReturnMapping
• UPDATE: Now in PHPUnit as $this->returnValueMap()
• Still more to do...
ry 31
Ja nua

37. Is It Handmade Code If You Use Power Tools?
PHPUnit Extensions
Open-sourced on Github
ry 27
Fe b r ua

38. Is It Handmade Code If You Use Power Tools?
Try Server v.3
Jenkins Master Project Plugin
Try’d out on Try Server
ch 24
M ar

39. Is It Handmade Code If You Use Power Tools?
Pake
Now test locally, just like Jenkins!
Ap ril 5

40. Is It Handmade Code If You Use Power Tools?
Deployinator Info in Jenkins!
Jenkins Master Project Plugin
Just like Try
Jenkins Deployinator Plugin
Special CLI Command
ONE Jenkins job per deployment stage
Previously, ~15 Jobs that were difficult to correlate with a push
pr il 9
A

41. Is It Handmade Code If You Use Power Tools?
On the Big Monitors!

42. Is It Handmade Code If You Use Power Tools?
Still More to Come...
•Flaky test resolution
•More focused DBUnit Tests
•Make pure unit testing even easier
•Try to solve whatever comes next...
on d !!
Bey

43. Is It Handmade Code If You Use Power Tools?
References
Code As Craft
http://codeascraft.etsy.com/
Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
Divide and Concur
http://codeascraft.etsy.com/2011/04/20/divide-and-concur/
Etsy PHPUnit Extensions
https://github.com/etsy/phpunit-extensions
Jenkins CI
http://jenkins-ci.org/
PHPUnit
https://github.com/sebastianbergmann/phpunit/

44. Is It Handmade Code If You Use Power Tools?
Wardrobe Furnished By
Spring Daisies Short Patchwork Dress
http://www.etsy.com/shop/pineapplepop of Nashville, TN
Asymmetric Flower and Beads Necklace (Silver)
Created by Michelle D’Netto, Software Engineer @ Etsy
Knitted DROPS Bolero with Lace Pattern
http://www.garnstudio.com/lang/en/visoppskrift.php?d_nr=127&d_id=24
Design by DROPS Design Kni t
bu t te
Knitted by Laura Beth Denker a l as d at t h
, e
f o r i t wa s C o n f,
the t
dre s o o d a r k
s :-(

45. Is It Handmade Code If You Use Power Tools?
Our software is artisanally crafted using the
ancient techniques from the 1960s. All releases
are aged in a handmade filled cave for at least
20 minutes before decanting to the Web.
--Noah Sussman