San Francisco Selenium Meetup, May 2013

1. Quality Assurance
Continuous
…using Selenium WebDriver
San Francisco Selenium Meetup Group - May 16, 2013

2. Fabrizio
BrancaLead System Developer at

3. San Francisco, CA
Janine
Fiona
that’s me
It’s a boy!

4. E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, FLOW,…
Mobile Searchperience:
SOLR
120 people
in 7 offices
world-wide
High
Performance
/Scale
Global
Enterprise
Projects

5. Continuous
Integration
Deployment
Development
Delivery
Improvement

6. Quality Assurance
Continuous
Everything sounds better
with “continuous”

7. “Continuous”
==“Doingstuffoverandoveragain”
Builds
Deployment Tests Infrastructure
Setup
Devbox
Setup
== “Automation”

8. timespent
task size
does it
manually
does it
manually
gets annoyed
writes script to
automate
runs script
wins
loses
Geeks vs. Non-Geeks
Makes fun of geek’s
complicated method

9. — Ovidiu Platon
We are not shipping your machine!
if it works on your machine!
I don’t care

10. Picture
The BIG

11. Vagrant
Boxes

12. Awesome
stuff
Every single
line potentially
can break
production!
Bratwurst
(Germans)
Donut
(Americans)
Pizza
(Italians)

13. Version Control
GIT, SVN,…
PHP lint

14. Version Control Code Reviews
“Gerrit”

15. Code Reviews Build
Files
DB
Installer
Settings
Triggered by
commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“virtual package”
(revision)

16. Multiple Release
Branches?*
Multiple Build
Pipelines!
…
…
…
*http://nvie.com/posts/a-successful-git-branching-model/

17. Version Control Code Reviews Build
Files
DB
Installer
Settings

18. Build
Files
B
ler
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend

19. Static Code
Analysis
Install on
“Integration”
First system where
code from all
developers/teams
meets each other
unstable system.
Might be broken.
Will be rebuild
without warning

20. Install on
“Integration”
Unit Tests
PHPUnit
Code
coverage

21. Unit Tests Integration Tests
AOE PMD
(for Magento)
broken links,
HTTP headers,
W3C validity,…
Test features that rely on other
parts (e.g. framework, external
services,…)
External components
(SSL, Redis, Varnish,…)
Basically this is also
PHPUnit, but
addressing other stuff

22. Integration Tests Acceptance Tests
Here we go:
Selenium!

23. Acceptance Tests
Install on
“Staging”
Behat
(Cucumber)
Same setup as production,
but different machines
Client QA /
approval

24. Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC2
instances in the
cloud to produce
traffic*

25. Again:
multiple servers or
autoscaling setup
Stress
Tests
Install on
“Production”
A/B
Deployments
on different
scales
Plan B:
rollback
Zero Downtime

26. Varnish ArrayEC2
Frontend ArrayBackend ArrayEC2 EC2
EC2
Worker ArrayEC2EC2
Deployment x
ELB
Load Balancer
Route 53
DNS-Service
shop.angrybirds.com
Deployment

27. Varnish Array
Frontend Array Worker ArrayEC2
Deployment x
ELB
Load Balancer
VaEC2
FroBackend ArrayEC2 EC2
EC2
EC2
Deployment x+1
Release Y
ELB
Load Balancer
Route 53
DNS-Service
shop.angrybirds.com
Deployment

28. Stress
Tests
Install on
“Production”

29. Deployment !=
Feature Release

30. Install on
“Production”
Enable Features

31. Version
Control
Code
Reviews
Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
Tests
Install on
“Staging”
Install on
“Production”
Enable
Features

32. Pipeline Visualization
#284
#283
#282
#281
#280
Build
Static Code
Analysis
Unit
Tests
Install on
“Integration”
Integration
Tests
Acceptance
Tests
#285
for team dashboards

33. Acceptance Tests

34. test automation
is a developer’s task
we need to provide
them some good tools…
us

35. Selenium1vs.
Selenium2

37. What’s out there?
Selenium for PHP
PHPUnit_Selenium
(Sebastian Bergmann)
WebDriver-PHP
(Chibimagic)
Php-webdriver-bindings
(Lukasz Kolczynski)
php-webdriver
PHPUnit_Selenium
(Giorgio Sironi)

38. php-webdriver
Selenium 2
08/2011 Facebook/php-webdriver (Justin Bishop)
Element-34/php-webdriver (Adam Goucher)
instaclick/php-webdriver (Anthon Pang)

39. php-webdriver
“A very thin wrapper
of WebDriver”

40. So what?
A thin layer is good!
But not enough…
no test logic
no higher level
components
dealing with
Selenium specific tasks
not compatible to
legacy test cases
No abstraction /
structure

41. Architecture
HTTP
AUT
No direct access
(DB, classes,…)
REST*
(over HTTP)
*http://code.google.com/p/selenium/wiki/JsonWireProtocol

43. Project test cases
PHPUnit
Project Components
Components Library
Menta
php-webdriver
reusable
project
specific

44. Menta
Selenium 2 Testing Framework for PHP
Who is
“Menta”?
What is
“Menta”?

45. Why
“Menta”?

47. Features
MentaThe framework.
Not the cat…
Components Session Management
?Bonus Features :)Event / ObserverConfiguration Management
PHPUnit integration

48. Why?
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Data Providers
✔
✔
✔
✔
✔
✔
✔
✔
✔
Dependent Tests
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Test Infrastructure

49. Menta for PHPUnit
Abstract test class
PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase
Very thin layer on
top of PHPUnit
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout
the “magic” happens
in the components

50. Legacy Selenium 1 tests
PHPUnit_Framework_TestCasePHPUnit_Extensions_SeleniumTestCase
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout

51. Legacy Selenium 1 tests
PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout
Menta_PHPUnit_Testcase_Selenium1
Selenium1Facade
Compatible with
PHPUnit_Extensions_SeleniumTestCase

52. HTML ReportScreenshots
(on demand
and on failure)
test description
from phpdoc
additional
messages
error details

53. HTML Report“Duration
Heatmap”
shows long
running tests

54. Menta for PHPUnit
Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.
Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml
> SUITE:
> SUITE:
> SUITE: Acceptance_Tests_Account_Register
> TEST: Acceptance_Tests_Account_Register::register
SUCCESS. (Duration: 8.18 sec)
> TEST: Acceptance_Tests_Account_Register::login
SUCCESS. (Duration: 6.5 sec)
> TEST: Acceptance_Tests_Account_Register::testRegistrationMail
FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22
< Duration: 36 sec
< Duration: 36 sec
> SUITE:
> SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest
> TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart
FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Fail
< Duration: 9.87 sec
> SUITE: Acceptance_Tests_Checkout_CheckoutTest
> TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView
SUCCESS. (Duration: 1.79 sec)
> TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart
FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec)
SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)

55. Components?
Components!

56. Manager
Component
Menta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart

57. Rewrites
Component
PageObject_CartProjectZ_PageObject_Cart
extends
change
locators
add
functionality
change
behavior
overwrite
labels

58. Manager
Component
Menta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_Cart
Menta_ComponentManager::addRewrite(
‘PageObject_Cart’,
‘ProjectZ_PageObject_Cart’
);

59. Components
Common Assert Wait Screenshot Selenium1Facade
Imap WebService W3C Validator Curl …
Some might not even
require a Selenium session

60. Gallery
Screenshot

61. Components!
PageObjects?
PageObjectsComponents
{ }}⊇{ PageObjects are
components

62. PageObjects?
represents
services offered by
a page or feature
is the only thing with
a deep knowledge of
the HTML
clickCheckoutButton() isInCart($product) getTotals()
✔
✔
✔

63. Component
Libraries
Menta
Magento
(your library)
Project-
specific
Magento
(community-provided)
General
(your library)

64. Session
Management
Let Menta manage
your sessions
auto-close
auto-start
incl. on exceptions
and ctrl+c
Reusing sessions
Cleaning sessions

65. Configuration
ManagementFallback mechanism
for global settings
Create multiple
configurations for
different browsers,
application instances
or Selenium Servers
Compatible to
PHPUnit (xml)

66. Event / Observer
Menta_Events::dispatchEvent('after_session_create');
Menta_Events::addObserver('after_session_create', function($session) {
$session->window('main')->postSize(array('width' => 1024, 'height' => 768));
}

67. Bonus Features

68. Sauce Labs Support
*configuration: http://username:accesskey@ondemand.saucelabs.com:80/wd/hub
works fine*
update test
result (REST)

69. Sauce Labs Support
Speaking name Test results

70. Perceptual diffs
Build X
Build X-1
Perceptual
Diff

71. github.com / AOEmedia / Menta
github.com / AOEmedia / Menta_SampleProject
It’s composer based and
you’ can get started
in a few minutes!

72. Thank you!
Any questions?
http://www.aoemedia.com
http://www.fabrizio-branca.de
@fbrnc
Follow me on twitter!
My blog