NYC Metro Selenium Users Meetup 03/15/12
http://www.meetup.com/NYCSelenium/events/52638682/
Dev + Ops = DevOps --> QA + Ops = QaOps
Chris Hoey, Senior IT Manager at Wireless Generation, will share some of the surprising ways Selenium has brought value to his company outside of the testing function.
Chris began his career at Wireless Generation as a Lead QA engineer responsible for writing many of the Selenium tests. When he transitioned over to Operations, he decided to carry the tests over to see new life not previously thought of.
This included periodically benchmarking applications from tests, helping customer care debug browser specific issues seen on their site and playing back the tests into a jmeter proxy to extend the tests into load testing scripts. It also spawned a new name (coined by Chris) to describe the joining of these two groups - QaOps.
More than Just Lines on a Map: Best Practices for U.S Bike Routes
QaOps - NYC Selenium Users Meetup - 03/15/12
1. Christopher Hoey
■ Senior IT Manager - Wireless Generation
■ Started in QA
■ Switched over to Operations
■ Selenium abuser since 2006
■ Got to see the Selenium vs Webdriver Cage Fight live
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
2. Thanks!
Ashley Wilson
Lindsey Baron
Sauce Labs
Conductor
NYC Metro Selenium Users
Selenium Developers
Jmeter Developers
Founded 12/14/06 Closed 07/16/09
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
3. Show of hands
Numbers collected and updated before sharing slides
About 35 attended and participated in informal poll
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
4. Show of hands - primary job title
Quality Assurance ?
12
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
5. Show of hands - primary job title
Developer ?
10
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
6. Show of hands - primary job title
Operations ?
5
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
7. Show of hands - experience with Selenium
Have not used it?
5
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
8. Show of hands - experience with Selenium
Tried Selenium
IDE?
8
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
9. Show of hands - experience with Selenium
Mastered Selenium
IDE?
1
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
10. Show of hands - experience with Selenium
Tried Selenium
RC/Webdriver?
13
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
11. Show of hands - experience with Selenium
Mastered Selenium
RC/Webdriver?
6
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
12. Show of hands - experience with Selenium
Selenium tied to CI
build system?
10
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
13. Show of hands - experience with Selenium
Running a
Selenium Grid?
7
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
14. Show of hands - use of Selenium
Functional Testing
New Features?
9
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
15. Show of hands - use of Selenium
Regression
Testing?
10
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
16. Show of hands - use of Selenium
Load/Stress
Testing? really?
1
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
17. Show of hands - use of Selenium
Benchmarking
Performance?
1
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
18. Show of hands - use of Selenium
Automating Data
Entry?
1
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
19. Show of hands - use of Selenium
Monitoring for
Operations?
1
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
20. Show of hands - use of Selenium
Something Strange?
2
Filling out forms to renew trial licenses when expired
Running build jobs for a build tool that does not have a cli
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
21. Dev + Ops = DevOps --> QA + Ops = QAOps
Disclaimers
Not a talk about best practices
Forcing Selenium to do things not originally intended
A lot of talk about Selenium IDE
QAOps term kind of meant as a joke but might have meaning?
Times have changed, tools improved and storms have passed
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
22. About Wireless Generation
● Create tools that make teacher's jobs easier
● Primarily focused around assessment and reporting apps
● Mix of mature code/platforms and new development projects
● Experiencing rapid growth year over year
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
23. Going MacGyver in early days of Ops
Faith in monitors pretty low
Manual configuration management
Serious bugs in third party tools
Large increases in user traffic
Lots of new projects to juggle while
keeping existing work online
Sysadmins manually checking site =(
Reuse anything that helps!
Selenium to the rescue!
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
24. Suites of Smoke Tests for Operations
● Use Selenium IDE so anyone in Operations with Firefox can use
● Suites are happy path smoke tests that are not brittle
● Focus on touching all moving parts of infrastructure
● All suites can run with any username
● All suites can run on any environment
● Sysadmins running Selenium tests -- still done manually =/
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
25. Suites for Operations
DEMO
TIME
Removing brittleness from Selenium
IDE tests against a site with difficult
locators
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
26. Suites for Operations - DEMO TIME
All we really want to know is if we threw any errors while loading pages that touch on all parts of our
platform.
Two simple regex searches for every page gets us the desired minimal coverage
Yes that looks way too simple but it did the job very well.
These tests started finding issues before our customers did.
Confidence increased dramatically!
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
27. Suites for Operations - DEMO TIME
https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do?
ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstI
d=33645685&ppkSchoolGroupId...
We know the starting URL pattern.
The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium
friendly. We could use XPath but that has proven to be really brittle for us.
Solution - Store the html source and use storeEval with a regex to find the correct link every time
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
28. Automation - judges.py
■ Headless Selenium servers
■ Suites of tests for each production webapp using
smoke tests
■ Run through suites serially in a never ending loop
■ Any failure in suites marks the status as failed for
that server
■ Look at each servers total result time running the
suite based on thresholds
■ Running helps see what server is currently being
tested
■ Results projected on wall in Ops area
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
29. We have a fundamental problem
Sysadmins Customers
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
30. Solution - Any browser any platform
Bromine
Grid
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
31. Loadtesting Selenium vs JMeter
● Lots of start up overhead ● Loads quickly
● ~30 concurrent sessions before ● ~300 concurrent sessions before
strange things happen strange things happen
● Lots of requirements / coordination ● Java and a jar, headless is easy
headless is a hack
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
32. Loadtesting with JMeter
Queries to find
interesting users
Selenium Scripts IN
Scripts to replace variables
for Selenium suites
Playback Selenium scripts
through JMeter proxy
JMeter Scripts OUT
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
33. Loadtesting with JMeter
DEMO
TIME
Setting up jmeter as a proxy
recorder for Selenium
playback
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
34. Loadtesting with JMeter - DEMO TIME
Start JMeter and create a new basic WorkBench
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
35. Loadtesting with JMeter - DEMO TIME
Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy Server
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
36. Loadtesting with JMeter - DEMO TIME
Set your port, any record filters of what to include or exclude and click start to fire up the proxy
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
37. Loadtesting with JMeter - DEMO TIME
Set Firefox network settings to use localhost and the port you configured JMeter proxy to use
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
38. Loadtesting with JMeter - DEMO TIME
Playback your tests using Selenium and watch as JMeter records the hits
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
39. Loadtesting with JMeter - DEMO TIME
Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread Group
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
40. Loadtesting with JMeter - DEMO TIME
Name your thread group, pick how many threads you want, ramp up time and how many loops
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
41. Loadtesting with JMeter - DEMO TIME
Run your tests and set servers ablaze. Run -> Start
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
42. Finding hard bugs
Uh oh! Just throwing more memory at the problem will not solve this!
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
43. Finding hard bugs
Did we see this in the environments below Production? Not very noticeable unless looking hard for it.
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
44. Finding hard bugs
Ramping up JMeter threads of a specific user in an endless loop reproduces issue in every environment
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
45. Dashboard Setup?
Use Selenium to login and navigate to your various dashboards for TV's and sharing with teams.
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
46. QAOps?
A lot of quick wins can
be made here.
Getting most of the focus these days.
Certainly best of all worlds coming
together. Takes a while to be a good Selenium IDE makes it trivial
practitioner based on tech debt | servers | for Ops to start running and
timeboxes | complexity | culture developing tests
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
47. Questions?
We are hiring!
http://www.wirelessgeneration.com
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
48. Picture Credits
http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg
http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg
http://www.russpainter.com/shop/images/clip.jpg
http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg
http://www.agilegeoscience.com/storage/post-images/python-logo-small.png?
__SQUARESPACE_CACHEVERSION=1316752792052
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
49. Picture Credits
http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands-
together-powerpoint-backgrounds.jpg
http://seleniumhq.org/images/selenium-ide-logo.png
http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg
http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif
http://gigaom2.files.wordpress.com/2011/01/firefox-ie.png
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
50. Picture Credits
http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg
http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg
http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg
http://stylreesta.files.wordpress.com/2011/07/caution_big.png
http://en.wikipedia.org/wiki/File:Devops.png
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012
51. Picture Credits
http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png
http://www.wisedatasecurity.com/imagenes/ie-8.jpg
http://seleniumhq.org/images/selenium-grid-logo.png
http://www.studiosysadmins.com/site_media/attachments.
gallery/2009/05/12/ServerRoomFireWeb.jpg
http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.png
Dev + Ops = DevOps --> QA + Ops = QAOps
NYC Metro Selenium Users Meetup
March 15, 2012