Open Source Test Workshop for QA Testers, Developers, IT Managers

Surface and Mitigate Performance Bottlenecks

Open Source Test Workshop
For QA Testers, Developers, Operations

Frank Cohen, +01 (408) 871-0122, fcohen@pushtotest.com

June 9, 2011

Thursday, June 9, 2011

Open Source Test Automation

Slides on Twitter
‣ http://twitter.com/fcohen
‣ Please retweet
‣ Please follow me
‣ #ost

2



About PushToTest

3



About PushToTest
‣ Our Mission: We Test, For A More Reliable World

3



About PushToTest
‣ Make It Easier To Move From Manual To Automated Testing

3



About PushToTest
‣ Support Advanced Testing, including Agile, Load, Integration

3



About PushToTest
‣ Support Advanced Testing, including Agile, Load, Integration
‣ Open Source Testing (OST) For Everyone

3



Manual Testing Process

Manual
Testing

4




1

Explore
Manual
Testing

4



Results

4 1

Recommend Changes
Explore
Manual
Testing

Tes
3 t
2 Plan

4 Run The Test Steps
Create Test Plans



Side Effects

5



Side Effects
‣ Exhaustion
‣ Fatigue
‣ Blurry Vision
‣ Night Sweats
‣ Insanity
‣ Hair Loss
‣ Acne
‣ Divorce
5



Proprietary Testing Falling Behind
‣ Adobe Releases Testing APIs for Flex/Flash in 2009
‣ JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010
‣ Appcelerator Titanium Makes 3 Releases in 2010
‣ Microsoft Draws 8,000 to Mix for StarLight in 2010
‣ StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC,
LinuxWorld Feature Keynote Talks On Ajax

‣ “HP Refundedcustomer in theFor The Ajax industry. Handler”,
a PushToTest
Our Payment
automotive
Protocol

6



Open Source Testing Emerges

Lots Of Options

7




W

tness
Lots Of Options
Selenium

HtmlUnit
soapUI

DBUnit
Sahi

Watir
FlexMonkey

FIT

Windmill Hudson TestMaker jMeter

7




W

tness
Lots Of Options
Selenium

HtmlUnit
soapUI

DBUnit
Sahi

Watir
FlexMonkey

FIT

Windmill Hudson TestMaker jMeter

‣ Signs of Maturity:Logger, Recovery Scenario Manager, Object
Iteration Engine,
Record/Playback, Data Generation/Importer,

7 Recognizer



OST Architecture
Application Under Test
Web, RIA, BPM, SOA

8



OST Architecture
Web, RIA, BPM, SOA

Functional
Test Test Authoring
SoapUI (Pro)
Sahi (Pro)
JUnit Class Framework
Load Test Eclipse Java IDE
Fitness
Languages: Java, Python
Wiki Collaboration

Production
Monitor

Continuous Test
Repository Bug Tracker Monitoring
Integration Management

8



OST Architecture
Web, RIA, BPM, SOA

Functional
Test Test Authoring
SoapUI (Pro)
Sahi (Pro)
Fitness
Test Operation Console Languages: Java, Python
Wiki Collaboration

Production Designer Grid and Cloud Repository Results Analysis
Record/Playback Test Deployment MySQL RDBMS Business Intelligence Tool
Monitor

Continuous Test

8



OST Architecture
Application Under Test PTTMonitor
Web, RIA, BPM, SOA CPU, Net, Memory

Functional
Test Injector Test Authoring
Test
Bundles ScriptRunners Data Production Libraries SoapUI (Pro)
MySQL, HtmlUnit Sahi, Selenium, SoapUI, Java csv files, rdbms queries Sahi (Pro)
Fitness
Test Operation Console Languages: Java, Python
Wiki Collaboration

Production Designer Grid and Cloud Repository Results Analysis
Monitor

Continuous Test

8



TestMaker Architecture

9



TestMaker Architecture

TestNode
Bundles ScriptRunners Data Production Libraries
MySQL, HtmlUnit Sahi, Selenium, SoapUI, Java csv files, rdbms queries

TestMaker Console
Designer Grid and Cloud Repository Results Analysis

9



Test Script
Selenium: Search Component

1 Load Home Page

2 Enter Value
3 Validate Results

10



Test Script
Selenium: Search Component Selenium Script Runner
Data Driven, Step Time,

1 Load Home Page
HTMLUnit

2 Enter Value
3 Validate Results

10



Test Script
Selenium: Search Component Selenium Script Runner
Data Driven, Step Time,

1 Load Home Page
HTMLUnit

2 Enter Value TestMaker Transformer: Java Unit Test

3 Validate Results
public class CheckEmail {
public void setUp() {}
...
public void runTest {
selenium.setBaseUrl( "http://myurl" );
selenium.open("/login.srf”);
selenium.type("search", "ctest");
selenium.click("//Item[6]");
selenium.assertTextExists("mytext");
}
10
}



Browser Compatibility Tests

TestNode
Selenium RC SeleniumHtmlUnit

Selenese Test

TestMaker Console

11



ScriptRunners
ScriptRunner Type Name (langtype)

Designer Script IE, Htmlunit designerscript
Selenium HtmlUnit Browser selenium
Selenium RC IE, Firefox, Chrome seleniumrc
Sahi IE, HtmlUnit sahi
HTTP Archive Page Optimization HAR
Selenium RC IE, Firefox, Chrome seleniumrc
SoapUI SOAP and REST soapui
TestGen4Web Web 1.0 Tests testgen4web
Mozmill Mozilla XUL Tests mozmill
Unit Tests jUnit TestCase Java, Jython, JRuby,
PHP, Perl, Groovy

DotNet .NET Tests dotnet
Visual Basic VB tests vb
12
Command Line Desktop and utils commandline



Your cloud or mine? Or both!
Application or Service
HTTP REST AMF AJAX

TestNode TestNode TestNode TestNode
test test test test

PTTMonitor PTTMonitor PTTMonitor PTTMonitor

SOAP SOAP SOAP SOAP
Test RunTime

TestMaker Console

13



Lots of Possibilities TestMaker

Test Sent

Seattle, Datacenter
TestNode Generating Load

Dallas, Texas
Datacenter

14



Lots of Possibilities TestMaker

Test Sent

Seattle, Datacenter
TestNode Generating Load

Dallas, Texas
Datacenter

You’re Such A Geek!
14



TestMaker Demonstration
‣ Fun And Easy Test Script Creation
‣ Selenium and TestMaker
‣ Repurposing A Single Test
‣ Functional Test, Load and Performance Test, Production Monitor
‣ Results Analysis
‣ Chart Engine

15



Root Cause Analysis

‣ PTTMonitor - Agent Based Monitor Gateway
‣ Glassbox - Performance Monitoring for Java Applications
‣ dynaTrace - Analysis In Application and DB Tiers
16



PTTMonitor
‣ Correlate Test Operation To Resource Usage
‣ Deploy as Web Application (WAR) and Service
‣ Observes CPU, Network, Memory Utilization As Test Runs
‣ Gateway to Glassbox, dynaTrace, Windows PerfMon

17



Results Analysis Engine

18
And 350 Additional Charts


Glassbox
‣ Correlate Test Operation To Application Tier Issues
‣ Memory Leaks
‣ Thread Deadlocks
‣ Slow DB Connections
‣ No Instrumentation of Applications
‣ Live Performance Alert View
‣ Run In Production

19



Glassbox Troubleshooter

20



dynaTrace PurePaths
‣ PurePath from TestMaker Virtual Users To Application & DB Tier
‣ Answers: What Are The Transactions and Why Are They Slow?
dynaTrace monitors
applications for
functional issues
dynaTrace
Database Agent
and performance
bottlenecks.

dynaTrace
Application Server Agent

PurePath Token

TestNode TestNode TestNode
test test test
PTTMonitor

Test RunTime

21
TestMaker Console



Pinpoint The Problem
‣ Down to the Line of Source Code

22



.NET Monitoring
‣ Correlate PerfMon Statistics with Test Operation
‣ PTTMonitor Gateways Data from PerfMon or MS SQL Server
.NET Application Tier PerfMon MS SQL Server

SOAP HTTP REST

TestNode TestNode TestNode
PTTMonitor
test test test

Test RunTime

TestMaker Console
23



Dual License

‣ Functional Testing with TestMaker Object Designer ‣ TestMaker Enterprise expands on the success of

‣
TestMaker Community
Repurposing Selenium, soapUI Sahi and unit tests into
data-driven functional tests, load and performance ‣ Load Testing to Millions of Virtual Users

‣
tests, and business service monitors
Cloud and Grid Testing
‣ Load Testing To 50 Virtual Users
‣ Data-driven tests using Data Production Libraries for
‣ Data-driven tests using CSV files. CSV files, RDBMS, Data Generating Objects

‣ PTTMonitor Correlates CPU, Network, Memory ‣ Root cause analysis and mitigation using Glassbox

‣
Utilization As Tests Operate.
Mock services for agile service development
‣ 25 Charts and Reports
‣ Flex AMF Protocol Handler for Flex/Flash testing
‣ Community Support and access to all documentation
‣ 1000+ Charts and Reports

‣ Business service monitors, including email notification

‣ Support, QuickStart, Training, Consulting
24



Where To Go From Here
‣ Download TestMaker and Read The Tutorials
‣ Please Complete The Evaluation Form
‣ And We Will Send You The Slides
‣ Watch the Screencasts
‣ http://www.pushtotest.com/screencastcentral
‣ Ask For A Proposal on Licenses, Training, Support, Consulting
‣ Call +01 408 871 0122, sales@pushtotest.com
25


Open Source Test Workshop for QA Testers, Developers, IT Managers

Recomendados

Recomendados

Más contenido relacionado

Similar a Open Source Test Workshop for QA Testers, Developers, IT Managers

Similar a Open Source Test Workshop for QA Testers, Developers, IT Managers (20)

Más de Clever Moe

Más de Clever Moe (20)

Último

Último (20)

Open Source Test Workshop for QA Testers, Developers, IT Managers