Invited Talk, ISSTA 2nd International Workshop on End-to-end Test Script Engineering July 16, 2012, Minneapolis. Limitations of x-unit testing framework, MTS testing framework that combined test objects with procedural aspects of TTCN.

1. MTS: Controllable Test Objects
Robert V. Binder
2nd International Workshop on End-to-end Test Script Engineering
July 16, 2012
Minneapolis
Copyright © 2012, Robert V. Binder

2. Overview
• Testing in Flatland
• Overview of MTS
• Lessons Learned
• Q&A
July 16, 2012 MTS - Controllable Test Objects 2

3. July 16, 2012 MTS - Controllable Test Objects 3

4. The xUnit Phenomena
jUnit
cppUnit
July 16, 2012 MTS - Controllable Test Objects 4

5. Incremental Testing Framework «uses»
TestRunner +Contains TestLog -Uses TestObject
1 1 1 1
0..1
TestSuite -Contains TestCase
-Contains
0..1 0..*
0..*
MyTestSuite MyTestObject
July 16, 2012 MTS - Controllable Test Objects 5

6. Testing in Flatland
TestCases TestSuites • Run A
A pqrstv
p s p q B C
• Run B
q t B r s t u
r s C
r u • Run C
C t u
t u
July 16, 2012 MTS - Controllable Test Objects 6

7. Straight Track
July 16, 2012 MTS - Controllable Test Objects 7

8. xUnit’s One Track Strategy
• Define and nest groups • Obstacle to any rich
• Run any sequence of strategy
groups – Select, repeat
• No support for group or – Skip, Halt
member – Time
– Repeat – Broadcast
– Select – Synchronize
July 16, 2012 MTS - Controllable Test Objects 8

9. Composition Isn’t Enough
• atom == test case • How to do ( p | q )
• ( ) group
• * quantification • How to do (t ( p | q ))*
• | select (or)
xUnit is like reg exp without * ? + |
A formal language without alternation and Kleene star
July 16, 2012 MTS - Controllable Test Objects 9

10. Composition Isn’t Enough
Why Repeat? Why Select?
• Iterate over collection • Respond to environment
• Iterate over generation configuration/conditions
strategy • Skip
• Performance test over – Depends on earlier
interval – Don’t halt long complex test
run
• Capacity test to limits
– Count failures for reliability
estimate
July 16, 2012 MTS - Controllable Test Objects 10

11. Straight Track
July 16, 2012 MTS - Controllable Test Objects 11

12. Switching Tracks
July 16, 2012 MTS - Controllable Test Objects 12

13. MTS ARCHITECTURE AND FEATURES
July 16, 2012 MTS - Controllable Test Objects 13

14. MTS: Any App, Any Platform
Console Host Agent Host
MTS Console
MTS Test
TEST RUN
Agent
REPORTS
Agent Host
MTS Remote
Agent
MTS Test
Agent Client
Under Test
Client Host Under Test
MTS Remote
Agent
MTS Remote
Client
Agent
Under Test
Host Under Test may be
Server
Client Host Under Test  Cell Phone Under Test
 PDA
 Desktop Server Host Under Test
MTS Remote
Agent
 Server
 Embedded Processor
Server  Network Equipment
Under Test  Access Point
 Base Station
Server Host Under Test
July 16, 2012 MTS - Controllable Test Objects 14

15. MTS Approach
• Test harness for distributed, end to end
testing
– xUnit: composable test objects
– TTCN: adapters, concurrency, …
– Tcl: defacto standard for test script automation
July 16, 2012 MTS - Controllable Test Objects 15

16. About TTCN-3
• Procedural DSL similar • Well-suited to model-based
to CORBA IDL testing
– Generate TTCN from MSCs,
FSMs, etc.
• Point of Observation – Generate impl source from
TTCN
and Control (adapters) – Compile & run impl
• Modular, but not • Same test suite can be used
necessarily composable on many implementations
July 16, 2012 MTS - Controllable Test Objects 16

17. Tcl
• Open source iut.set {1}
interpreter
iut.set {$x}
• LISP-ish evaluation iut.set {[genX{$y}]}
iut.set {[eval $codeForX]}
July 16, 2012 MTS - Controllable Test Objects 17

18. Tcl
• Open source
interpreter
• LISP-ish evaluation
• Object-oriented
[incr Tcl]
July 16, 2012 MTS - Controllable Test Objects 18

19. MTS::TestObject
MTS::TestObject
MTS Framework +testObjectHeadline
+getTestObjectClassName()
+getTestObjectHeadline()
+getTestObjectFileName()
Adapter +getTestObjectDir()
+getRemoteHostStatus()
+connect() +getTestRunRepeatCount() YourTestObject
+open() +getTestRunIteration()
#require() MyTestObject
+close()
+test() #setup()
+record() #test() +require()
PPC 2002 #cleanup() +setup()
WM5 #ensure() +test()
#check() +cleanup()
+getImage() #run() +ensure()
#submit()
WebService #userLogEntry()
#userLogImage()
+getServiceWSDL() #evaluateCompareFrame()
+parseWSDL() #replayTestActionSequence()
+queryForDictionary()
+queryForXML()
July 16, 2012 MTS - Controllable Test Objects 19

20. TestObject methods
Skip rest of test object when its preconditions aren't met.
require
Automatically called when a TestObject is run.
Create starting conditions for a test; allocate necessary resources. setup
Run tests. The user's implementation is expected to execute test
test
actions leading to one or more test verdicts.
Restore pretest conditions; release resources the user test object
cleanup
has claimed.
Skip or halt rest of a test run when postconditions aren't met.
ensure
Automatically called after cleanup completes.
July 16, 2012 MTS - Controllable Test Objects 20

21. Using require
::itcl::class aTestCase{
inherit TestObject
::itcl::body aTestCase::require {} {
if {[getRemoteHostStatus] eq "Connected"} {
set remoteHostconnection true
} else {
userLogEntry "Remote Host not ready, skipping"
set remoteHostconnection false
}
return $remoteHostconnection
}
::itcl::body aTestCase::test {} { … }
July 16, 2012 MTS - Controllable Test Objects 21

22. Completion Actions
return Current Test object Caller Test object Test objects run stack Counters
keyword
OK No effect, Destruct No effect. No effect. No
(ensure) change.
SKIP Destruct Continue with the No effect. No
next test object. change.
HALT Destruct Immediately exit the Terminate the test run. No
method that called Percolate the HALT to the change.
this test object. run stack.
EXCP Capture current Tcl Immediately exit the Terminate the test run. Increment
errorInfo. Destruct the method that called Percolate the EXCP to all error
current test object. this test object. test objects in the run counter.
July 16, 2012
stack.
MTS - Controllable Test Objects 22

23. Using ensure and Completion Actions
::itcl::class checkDir {inherit TestObject
::itcl::body checkDir::require {} { … }
::itcl::body checkDir::setup {} { … }
::itcl::body checkDir::test {} { … }
::itcl::body checkDir::cleanup {} { … }
::itcl::body checkDir::ensure {
set currentDir [pwd]
if { ! [regexp {mydir} $currentDir]} {
return SKIP
} else {
return OK
}
}
July 16, 2012 MTS - Controllable Test Objects 23

24. MTS run command
run <target> repeat • Hardcoded target
<times> <arg list> run myTestObject
• “Sources in” target file • Target name in a variable
TestObject subclass run $anyTestObject
• Runs this TestObject
• Returns control to the • Function computes target
run [eval whichTestObject]
caller
July 16, 2012 MTS - Controllable Test Objects 24

25. Selection and Repetition in MTS
How to do ( p | q ) How to do (t ( p | q ))*
::itcl::body pq::test{} { ::itcl::body mySuite::test{reps} {
if {[rhStatus]} { run strategy1 -repeat $reps
run remoteTest }
} else {
run localTest ::itcl::body strategy1::test{} {
} run t
} run pq
}
July 16, 2012 MTS - Controllable Test Objects 25

26. TestObject helpers
Start other test objects or create a test suite run <target>
Evaluate and log pass/fail conditions check <expression>
Time any action during a test run Timer <method>
Object interface for test run as a whole testRun <method>
Monitor remote host resource utilization rpm <method>
Check the remote host getRHostStatus myHost
Perform data-driven tests submit <args>
Annotate test run reports userLogEntry <args>
July 16, 2012 MTS - Controllable Test Objects 26

27. IMPLEMENTING
MODAL CLASS TEST PATTERN
July 16, 2012 MTS - Controllable Test Objects 27

28. Modal Class Test PatternTwoPlayerGame
T wo Play erG am e ( )
α
TwoPlayerGame p1 _S tart ( ) / p2 _S tart ( ) / ThreePlayerGame( ) /TwoPlayerGame( )
+TwoPlayerGame() s im u lat eV olle y( )
G am e S tarte d
s im u lat eV olle y( ) α
+p1_Start( )
+p1_WinsVolley( ) p1_Start( ) / p3_Start( )/
-p1_AddPoint( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( ) simulateVolley( ) simulateVolley( )
[th is .p 1_ Sc ore ( ) < 20 ] / [th is .p 2_ Sc ore ( ) < 20 ] / Game Started
+p1_IsWinner( ) th is .p 1A ddP oin t( ) th is .p 2A ddP oin t( )
+p1_IsServer( ) s im u lat eV olle y( ) p1 _W ins V olle y ( ) / s im u lat eV olle y( )
+p1_Points( ) s im u lat eV olle y( ) p2_Start( ) /
+p2_Start( ) P la ye r 1 P la ye r 2 simulateVolley( )
+p2_WinsVolley( ) S erv ed S erv ed p1_WinsVolley( ) /
-p2_AddPoint( ) p2 _W ins V olle y ( ) / simulateVolley( )
s im u lat eV olle y( )
+p2_IsWinner( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( )
+p2_IsServer( ) [th is .p 1_ Sc ore ( ) = = 20] / [th is .p 2_ Sc ore ( ) = = 20] /
+p2_Points( ) th is .p 1A ddP oin t( ) th is .p 1A ddP oin t( ) p2_WinsVolley( )
+~( ) p1_WinsVolley( ) [this.p2_Score( ) < 20] / p3_WinsVolley( )
P la ye r 1 P la ye r 2
[this.p1_Score( ) < 20] / this.p2AddPoint( ) [this.p3_Score( ) < 20] /
p1 _Is W in ner( ) / Won Won p2 _Is W in ner( ) / this.p1AddPoint( ) simulateVolley( ) this.p3AddPoint( )
retu rn T R UE ; retu rn T R UE ;
simulateVolley( ) simulateVolley( )
~( ) ~( ) p1_WinsVolley( ) / p2_WinsVolley( ) /
ω
simulateVolley( ) simulateVolley( )
Player 1 Player 2 Player 3
Served Served Served
p2_WinsVolley( ) / p3_WinsVolley( ) /
simulateVolley( ) simulateVolley( )
ThreePlayerGame p1_WinsVolley( ) p3_WinsVolley( )
[this.p1_Score( ) == 20] / [this.p3_Score( ) == 20] /
T h ree P la y erG a m e ( ) / T wo P la y erG am e ( )
α this.p1AddPoint( ) this.p3AddPoint( )
p 3_ S tart ( ) / p3_WinsVolley( ) /
ThreePlayerGame G a m e S ta rt ed
s im ulat eV o lley ( ) simulateVolley( )
p 3_ W ins V o lle y( ) / p2_WinsVolley( )
+ThreePlayerGame() s im ulat eV o lley ( ) [this.p2_Score( ) == 20] /
+p3_Start( ) p 3_ W ins V o lle y( )
this.p1AddPoint( )
[t his .p 3_ S co re ( ) < 2 0] /
+p3_WinsVolley( ) th is . p3 A dd P oint ( )
T w oP lay erG am e ( )
-p3_AddPoint( ) s im ulat eV o lley ( )
p 1_ W ins V o lle y( ) /
+p3_IsWinner( ) s im ulat eV o lley ( )
+p3_IsServer( ) P la y er 3
+p3_Points( ) S erv e d
p1_IsWinner( ) / p2_IsWinner( ) / p3_IsWinner( ) /
+~( ) p 2_ W ins V o lle y( ) / return TRUE; Player 1 return TRUE; Player 2 Player 3 return TRUE;
s im ulat eV o lley ( )
Won Won Won
p 3_ W ins V o lle y( )
[t his .p 3_ S co re ( ) = = 2 0] /
th is . p3 A dd P oint ( ) ~( )
~( ) ~( )
ω
P la y er 3
W on p 3_ Is W in ne r( ) /
ret urn T R UE ;
~( )
ω
July 16, 2012 MTS - Controllable Test Objects 28

29. Modal Class Test Design
1 ThreePlayerGame( )
2 p1_Start( )
8 P layer 2 Served
3 p2_Start( )
4 p3_Start( )
11 P layer 3 Served
5 p1_WinsVolley( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega
*7
7 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on
14
8 p2_WinsVolley( ) P layer 1 W on
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
*6 P layer 1 Served
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
2 Sneak path tests
*9 P layer 2 Served
not shown
11 P layer 3 Served
1 3
alpha Gam eS tarted P layer 2 Served 17 om ega
* 10
P layer 2 W on
15
P layer 2 W on
Note iterations
5 P layer 1 Served
4
* 12 P layer 3 Served
17 om ega
11 p3_WinsVolley( )
* 13 P layer 3 W on
16
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
P layer 3 Served P layer 3 W on
13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8
P layer 2 Served
14 p1_IsWinner( )
15 p2_IsWinner( )
16 p3_IsWinner( ) 5 P layer 1 Served
17 ~( )
July 16, 2012 MTS - Controllable Test Objects 29

30. Implementing Modal Class Design
xUnit MTS
• TestCase for each transition (20) • TestObject for each transition (20)
• Hand hack TestSuite for each • Top level TestObject uses run to
root to leaf path (17) and one to
run them all call transition test objects
– addTest – Iteration trival
– addTestSuite • Parameterized strategies
• How to iterate transitions? – N+
• Embed strategy in single large test
– Markov
object?
– Very smelly – Etc.
• Develop code generator that writes • Parameterized state spec
test suites?
July 16, 2012 MTS - Controllable Test Objects 30

31. 1 ThreePlayerGame( )
2 p1_Start( )
8 P layer 2 Served
3 p2_Start( )
4 p3_Start( )
11 P layer 3 Served
5 p1_WinsVolley( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega
*7
7 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on
14
Dude, where’s my object?
8 p2_WinsVolley( ) P layer 1 W on
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
*6 P layer 1 Served
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
2
*9 P layer 2 Served
11 P layer 3 Served
1 3
alpha Gam eS tarted P layer 2 Served 17 om ega
* 10
P layer 2 W on
15
P layer 2 W on
5 P layer 1 Served
4
* 12 P layer 3 Served
xUnit MTS
17 om ega
11 p3_WinsVolley( )
* 13 P layer 3 W on
16
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
P layer 3 Served P layer 3 W on
13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8
P layer 2 Served
14 p1_IsWinner( )
15 p2_IsWinner( )
16 p3_IsWinner( ) 5 P layer 1 Served
17 ~( )
1 8 15 1 6 11 1 8 15
2 9 16 2 7 12 2 9 16
3 10 17 8 3 10 17
3 13
11 21
4 11 18 4 18
4 9 14
5 12 19 5 12 19
5 10 15
6 13 20 6 13 20
16
7 14 7 14
TestCase objects TestSuite objects TestObject objects
July 16, 2012 MTS - Controllable Test Objects 31

32. Controllable Test Objects
Design Goal TTCN xUnit MTS
Select, Iterate, any level Yes No Yes
Generate with MBT strategies Yes Yes Yes
Logical/Physical separation Yes No Yes
Platform agnostic/robust Yes ? Yes
Channel agnostic/robust Yes ? Yes
Distributed control Yes No Yes
Minimize Test Agent resources NA No Yes
Minimize brittleness Yes No Yes
Intuitive interaction No Yes Yes
Composable No Yes Yes
July 16, 2012 MTS - Controllable Test Objects 32

33. MTS hard problem queue
• Protocol to drive remote target host IUT from test host
– Bi-directional, lightweight (BEEP ?)
– Platform and stack agnostic
– Target environment control (remote command line)
– Distributed adapter RPC
• Forced reflection to generate adapters
• Logical/physical mapping in generative models
• Scalable asynchronous remote agents
– Control strategy
– Provisioning
– Protocol
July 16, 2012 MTS - Controllable Test Objects 33

34. Question Time
Bob Binder
rvbinder@gmail.com
www.robertvbinder.com
July 16, 2012 MTS - Controllable Test Objects 34