PushToTest TestMaker 6.5 Open Source Test Design Document

Collaborative Results, Resource Repository,
UI Enhancements

TestMaker 6.5 Plan
Frank Cohen, CEO
(408) 871-0122
fcohen@pushtotest.com
November 8, 2011
Intellectual Property of PushToTest. (c) 2011 All rights reserved.
Note: All trademarks mentioned in this document are the trademarks of their respective holders.

Open Source Test Automation


Product Roadmap
TestMaker 6.0 TestMaker 6.1 TestMaker 6.5 TestMaker 7
2011 2011 2011 2012

• Improved Test • Tabular Results Charts • Easier Test Controller • TestCards Phase 2
Operation Speed
• Flex 4 In-Browser • Reports • Web Reports
• Jasper Business Recording
• Collaborative Charts • Web Controller
Intelligence Reporting
• TestMaker JAT for • Live Charts for Drill • Test Management
• Test Object Designer Oracle Applications Downs
• Rally
• Record/Playback • Citrix Applications • Test Object Repository • TestLink
• Firefox, IE, Safari, • TestCards Phase 1 • Share Test
Chrome, Opera • HP QC
• Rally Integration Resources
• Open Source • Zephyr
• Web Reports • Search
Alternative to • QMetry
HP QTP • Web Controller • Access Control
• Adds Sahi, Flex • Bug Fixes • LDAP/Rally
• TestCards 1
• Data and Object
Enabled
• HtmlUnit Viewer

3 Open Source Test Automation


TestMaker 6.5
‣ Resource Storage in the Repository
‣ Access Control for users and LDAP integration
‣ Search
‣ Reports
‣ Live Reports
‣ Collaborative Reports
‣ Easier Test Controller
‣ Streaming Results using Snapshot Protocol



Needs Research for 6.5
‣ TestMaker to restore Windows and IE proxy on quit
‣ Console Connects To TestNodes (Cloud) Through Proxy
‣ URL to Web-based TestMaker Test Results Automatically Saved To
Rally Test Case

‣ L7 and HPOpenView reports gathered automatically



Not In TestMaker 6.5
‣ Full Web Interface for TestMaker
‣ Test ManagementFailed Previous Test Operations
‣ Rerun Tests of Zephyr, Quality Center Integration
‣ TestLink, QMetry,Unique DPL v2)
‣ Colossus“Push” Bundle Deployment, Use Resources Instead
DPL (aka
‣ DynamicTestNode Sahi Proxy SSL Certificate Registration in
‣ Remote
Community
‣ Open Designer,view “How Are We Doing”
see Editor too
‣ No dashboard has a custom status display (icon and text) settable
‣ Each Controller step context
by API from a test
‣ Graphical view to edit and manage RDBMS for DPL content



Resource Repository
‣ File System or Repository
‣ Editor, TestMaker Object Designer, Console
‣ Open, Save, Save As
Open Test Resource Save Test Resource

Repositories
Repositories
Local Repository
Local Repository
QA Unit 1 Repository

/ Last Modiﬁed Creator

/ Last Modiﬁed Creator Project X June 12, 2011 6:05 pm mcohen
3rd Party Connector Project June 12, 2011 6:12 pm mcohen
Project X June 12, 2011 6:05 pm mcohen
TestScenarios June 12, 2011 12:05 pm mcohen
Load Test 103 Scenario June 12, 2011 1:05 pm mcohen
Project Y June 12, 2011 3:05 pm mcohen
Project Z June 12, 2011 5:05 pm mcohen
Project Z June 12, 2011 5:05 pm mcohen Load Test 133 Scenario

Cancel Open Show Versions Cancel Save
Show Versions



Resource Repository Search
‣ Search File Names
‣ Search TestMaker Object Designer Meta Data
‣ Name, Creator, Links, Version, Types, Operations, Goals
‣ Object Repository objects: Story, Form, Object
‣ Object name, Type, Accessor, Comment
‣ Search Editor Meta Data
‣ Author Name, Creator, Links, Versions, Categories



Access Control
‣ TestCard Viewable by Anonymous, Role, User Name
‣ TestScenarioRole, User Name Stop, Pause, Create, Delete by
Anonymous,
View, Change, Run,

‣ Repository Read, Change, Create DB, Create Contents, Delete by
Anonymous, Role, User Name

‣ Role Create, Change, Delete by Anonymous, Role, User Name
‣ Reports Create, Change, Delete by Anonymous, Role, User Name



SSO Option with Rally
‣ Optionally Uses Rally LDAP for User Authentication
PushToTest TestCards - Best Buy

help

Welcome To TestMaker TestCards

TestMaker Enterprise 6.0.1
User ID fcohen
Licensed for 50 virtual users
and 10 service monitors
Password •••••••••

Sign-in

Options

Sign-in Using Rally



Enhanced Controller
‣ Web (for TCS 1) and SWING
‣ Reports Button -> New Reports
Ajax Load Test

‣ Tabs Expand To New Window 513
40 Seconds Remaining
Executing iteration: 1 Users * 1
513 Success
12.67 % Failure 65 Failed

Instant Results TestNode Status Local TestNode Log Local TestNode Error Log

Real Time Scalability Index Chart

6

5

4

3

2

1 2.5 TPS 4.6 TPS
1 TPS
0
1 Users * 1 2 Users * 1 4 Users * 1
213 Success 475 Success 513 Success
5 Failed 35 Failed 65 Failed
.02% Failure 7.36% Failure 12.67% Failure

Reports Conﬁguration Logs



Next let’s talk about

Reports Design

‣ Resource Plan
‣ User Interface Design
‣ Required Changes to TestMaker 6.0
11



Reports Summary
‣ User Interface Design (Tomas)
‣ Reports - A report is a collection of charts
‣ Charts - Athe Transaction of the test results repository. For
example,
graphical view
Distribution chart

‣ Live Charts - A user interface to see custom step logging data
‣ ScriptRunner (EOP) Enhancements (Luis Carlos)
‣ DesignerScript -commandtoinstore screenshots, DOM after
executing each
Options
a .ds file

‣ soapUI - Options to save each TestCase steps request & response
‣ Java - Options to save request, response, stack trace



Required Changes
‣ PTTStepListener and Repository Schema
‣ StepRequest (String, may contain CR&LF characters)
‣ StepResponse (Blob, may contain CR&LF characters)
‣ StepStackTrace (String, up to 10K characters)
‣ StepExceptionName (String)
‣ Screenshot (Blob)
‣ TestScenario Schema Changes
‣ Define Report, Charts, Drill Downs



Optional Enhancements
‣ Caching of Jasper generated charts to avoid reprocessing time



Run A Report
PushToTest TestMaker 6 - Best Buy

help
TestMaker Results Repository

Run Report Edit Report Delete Report New Report
Reports
s Weekly Scalability Report
img TSH Working Team 2

s Load Test for Mods To 3rd Party Gov
img Performance after changes

s Functional Test for Geek Squad Reg
img Testing new registration process



Report Appears In Window
PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011

Weekly Scalability Report
Created on September 10, 2011

Transaction Distribution Chart



New/Edit A Report

help

TestMaker Enterprise 6.0.1 New Report
and 10 service monitors Report Name:

Cancel Next



Add Charts To The Report

help


Report Settings
Licensed for 50 virtual users Report Name: Weekly Scalability Report
Header: Weekly Scalability Report

Footer: M Cohen - mcohen@bestbuy.com x3817

New Chart Edit Chart Delete Chart

Charts

Transaction Distribution Chart !

Scalability Index

!
Transactions by Avg Duration

Report Details Cancel Save



Chart Types
‣ Scalability Chart
‣ Bar Chart
‣ Grouped Data Points
‣ Transaction Chart
‣ Line and Data Point Chart
‣ Shows Individual Points of Data
‣ Resource Chart
‣ Bar Chart
‣ Resource (CPU, Net, Memory) Usage Percentage
‣ Data Chart
‣ Grid Chart
‣ Transaction and Step Data



Scalability Chart Settings
‣ Vertical
‣ Transactions Per Second
‣ Count of All Transactions
‣ Count of Failed Transactions
‣ Count of Successful Transactions
‣ Avg, Min, Max Transaction Time
‣ Horizontal: VU Iteration
‣ Secondary Axis (%)
‣ Failure Percentage
‣ Success Percentage
‣ Resource Monitoring (Choose from CPU, Net, Memory)
‣ Option: Display List of Step Names




help

Licensed for 50 virtual users Transactions Per Second (TPS)
Count of All Transactions

Count of Successful Transactions

Count of Failed Transactions

Avg, Min, Max Transaction Time Average

Secondary Axis
Failure Percentage

Success Percentage

Resource Monitoring CPU

Option: Display List of Step Names
Cancel Save



Transaction Chart Settings
‣ Vertical
‣ Transaction Duration Time
‣ Vertical Options
‣ Successful
‣ Failed
‣ Duration: Avg, Min, Max
‣ Horizontal
‣ Time Offset Into Test
‣ Specific Start/End Time




help


Licensed for 50 virtual users Transaction Duration Time
Successful

Failed

Duration Average

Horizontal Axis
Show standard date/time

Cancel Save



Resource Chart Settings
‣ Vertical
‣ Percentage used
‣ Grouped by CPU, Net, Memory
‣ Horizontal
‣ Time Offset Into Test
‣ Specific Start/End Time




help

Licensed for 50 virtual users Percentage Used

Grouped by CPU

Horizontal Axis

Show standard date/time

Cancel Save



Data Chart Settings
‣ Successful Transactions
‣ Failed Transactions
‣ Option: Show Steps



Data Chart

help

Data Chart Settings
and 10 service monitors Successful Transactions

Failed Transactions

Show Steps

Cancel Save



Live Charts for Drill Downs

help

Live Charts
and 10 service monitors New Chart View Chart Edit Chart Delete Chart

Charts


Scalability Index

Transactions by Avg Duration




help





Click To See Steps

help



Steps In Selected Transaction - soapUI TestSuite

Request 1 175

Request 2 3,201

Groovy Script - Validate 643



Additional Requirements
‣ Keyboard left-right arrows move selected point
‣ User entered Notes
‣ Steps need labeling



Click Step To See Detail
Steps In Selected Transaction - soapUI TestSuite

Request 1 175

Request 2 3,201

Groovy Script - Validate 643

Request Response Error Custom

<soapenv:Envelope xmlns:soapenv="http://
<soapenv:Header/>
<soapenv:Body>
<exam:Change_Price>
<ProductNumber>1033</ProductNumber>
<Type>add_product</Type>
<Quantity>55</Quantity>
<Location>2</Location>



For Java, DS Steps
Steps In Selected Transaction - DesignerScript

Log-in To Application 175

Product Catalog Search 3,201

Buy Products 675

Request Response Error Screenshot

Command: _click
Locator: textbox("company")
Value: "PushToTest"



For Java, DS Steps

Row1_ColumnB: 38178
Row2_ColumnB: 98123
Row3_ColumnB: 18177


Unable To Find Flight Number Exception

java.lang.Throwable
at
boo.hoo.StackTrace$FirstNested$SecondNested.
at boo.hoo.StackTrace$FirstNested.<init>(St
at boo.hoo.StackTrace.<init>(StackTrace.j



Add A Step As A Chart
Steps In Selected Transaction - DesignerScript

Log-in To Application 175

Product Catalog Search 3,201

Buy Products 675

Add This Transaction To Report As New Chart

Add This Step To Report As New Chart



Add More To The Report
PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011

Weekly Scalability Report
Created on September 10, 2011


Step Detail Chart - Transaction 18372 - Step Log-in To Application

Request Command: _click
Locator: textbox("company")
Value: "PushToTest"

Response Row1_ColumnB: 38178
Row2_ColumnB: 98123
Row3_ColumnB: 18177

Error Unable To Find Flight Number Exception

java.lang.Throwable
at
boo.hoo.StackTrace$FirstNested$SecondNested.
at boo.hoo.StackTrace$FirstNested.<init>(St
at boo.hoo.StackTrace.<init>(StackTrace.j

Screenshot



Next let’s talk about

Resource Repository
Design
‣ Resource Plan
‣ User Interface Design
‣ Required Changes to TestMaker 6.0
39



Repository Client Access
‣ Designer
‣ Editor
‣ Controller
‣ EOPs



Resource Repository - Open
Open Test Resource

Repositories

Local Repository




Show Versions Cancel Open



Resource Repository - Save
Save Test Resource

Repositories

Local Repository




Load Test 133 Scenario

Show Versions Cancel Save



Resource Repository
Preferences
Repositories TestCard Server Notiﬁcations

Repositories Local Repository

Local Repository
Type MySQL
Location jdbc:mysql//localhost:1527/testmaker

User testmaker

Add Remove Password testmaker

Save



Learn More
‣ http://www.pushtotest.com
‣ Post your feedback, comments, ideas, designs to PushToTest Forums
‣ Attend the Open Source Test Workshop
‣ http://workshop.pushtotest.com
‣ Send Feedback to fcohen@pushtotest.com

PushToTest TestMaker 6.5 Open Source Test Design Document

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a PushToTest TestMaker 6.5 Open Source Test Design Document

Similar a PushToTest TestMaker 6.5 Open Source Test Design Document (20)

Más de Clever Moe

Más de Clever Moe (20)

Último

Último (20)

PushToTest TestMaker 6.5 Open Source Test Design Document