Oleg Bunin presented tips for testing JavaFX user interfaces from the JavaFX quality team. He discussed different approaches to UI testing including manual testing, record and replay automation, and coding tests. Coding tests can be more expensive to create initially but have lower maintenance costs. The Jemmy library was presented as a tool for automating UI tests in JavaFX. Following test-driven development principles and designing reusable test primitives and libraries can improve test automation effectiveness.

1. Олег Бунин
Testing JavaFX UI
Tips from JavaFX quality team
Alexandre (Shura) Iline
Java and JavaFX quality architect
Oracle

2. Agenda
●UI testing overview
●Test automation approaches
●Jemmy library
●Test automation effectiveness
●Test library design approaches

3. UI testing … by Wikipedia
«GUI software testing is the process of
testing a product that uses a graphical user
interface, to ensure it meets its written
specifications.»

4. UI testing … most often ...
«Checking whether usage of a product UI
leads to results expected by the the person
who performs testing»

5. ●Start text editor
●Push «File/Open»
●Verify file chooser directory
●Select some file
●Verify editor area content
●Verify application title
●Verify buttons availabilities
●....

6. UI Testing Manual Automated
Initial step Design test specification. Create tests.
Establish regular runs.
Ongoing Click, click, click Analize results
File bugs
Change the specification Change the tests
Qualification High, low for test executors High
Effectiveness Low High
Other Inexpensive to start Continuous quality monitoring ,
inexpensive to reuse
Fun Bo-o-o-ring Much like programming

7. Development
Continuous build
Commit
Build
Executed
automagically
after commit
Success
Analysis.
Rollback!!!
Code
changes
No
Yes.
Promote
Code is compilableTest further

8. Continuous build with testing
Commit
Build Success
Testing
Is it working?
Passed
Analysis.
Rollback!!!
Code
changes
Test
changes
No
No
Test further
Build is good
Code line is healthy
Go on ...
Yes
Yes.
= Compilation
successful

9. Automation
approach
Record && Replay Coding
Test creation Inexpensive*
Usually just repeating manual test
in special environment
Must be accompanied by other
means
Expensive*
Consists of programming
Test execution Does not depend on approach
Test
maintanance
Higher (in most cases)*
Very much depends on test format
Lower*
Depends on principles of
building test library
Test analysis Does not depend on approach
(*) Much more information closer to the end of this presentation

10. Jemmy

11. Jemmy v2
Started as a tool to tests TeamWare UI (1999)
Used for NetBeans extensions (2000)
Official test tool for NetBeans (2001)
Open-source (2001)
Jemmy v3
Started in (2008) as a proof of concept experiemnt
Extended to support JavaFX (2009)
Opensource with support of JavaFX 1.2 (2009)
Developed in close-source for 1.3 since then

12. Jemmy usages
●V2 (Swing/AWT)
● Internal: Swing, NetBeans, Glassfish
● External:
● Used by: Boeing (x2), agfa.com, compuware.com, attix5.com,
exgate.tek.com, globo.com, gmx.de, non.agilent.com
● Evaluated by: syntelinc.com, covast.com, visus-tt.com,
covast.com, nokia.com, twt.com, ubs.com
●V3 (FX)
● Internal: JavaFX, Authoring Tool, JavaStore

13. Demo...
Finally!

14. Jemmy v3JavaFX
AWT/Swing
SWT
Scenegraph
LCDUI
JemmyCore
Interfaces
AWT robot FX robot
Hierarchy
Input
Scene, node
containers
Scroller, slider,
list, text, ...
Generic
interfaces
Covered in this presentation

15. UI test
Find Do VerifyPass Pass
Pass
Fail Fail Fail
Failure analysis
Find next control
To perform operation
On in
Verify that expected
State reached
Perform necessary
actions

16. Lookup
● By ID
● Easiest but may not be possible
● «open_file_btn»
● By type
● Most common
● Button, text, combo box, etc
● By index
● Unavoidable
●
2nd
button with text «browse»
● By toString()
● Sucks
● projectmanager.Main$Main
$Script$1Scene$ObjLit$20
@4d16318b
● By text, tooltip,
associated label
● Best, if possible
● Button with text «browse»

17. UIcouldbe
verycomplicated

18. UIcouldbe
verycomplicated

19. UIcouldbe
verycomplicated

20. Lookup demo

21. Interfaces
Interface Control types Description
Mouse,
Keyboard,
Drag
Enything Low level input
Parent Containers, list (for its content) Something you could look within
Selectable Toggle button, radio button,
combobox, check box, lists, etc
A control which provides limited
number of choices
CaretOwner Text box, scroll bar, slider A control which has a number
value which changes within some
range
Text Text box Editable text container

22. Interfaces demo

23. Verifications
UI feedback Non UI feedback
Dialog displayed
Test changed
Image updated
Progress bar changed position
File created
Database updated
Sunset happened :)

24. Waiting
●UI is a multy-thread environment
Things happen in background
Test code is in another thread
●Not much could be really verified
●Everything should be waited for

25. Verification demo

26. Test automation
effectiveness

27. TD + *TS
NR
TM * NR
NC*
EA =
NC
*
EA
– automation effectiveness
NR
and NC
are characteristics for a product.
TM
is a characteristic of a test suite
TD
and TS
depend on test automation approach
Smaller TD
and TS
- higher the EA
.

28. 1 2 3 4 5 6 7 8
0
5
10
15
20
25
0
1
2
3
4
Nc = 1 Ratio
Auto Manual Effect
Time
1 2 3 4 5 6 7 8
0
5
10
15
20
25
0
1
2
3
4
Nc = 3 Ratio
Auto Manual Effect
Time
Assumptions: TM
= 1 engineer*week TS
= 0.1 * TM
TD
= 5 * TM
NR
= 8

29. Td
or Ts
– what to minimize
TS
- if (NC
* NR
) is big
Multi-platform
Compatibility with external products (servers, browsers, ...)
Long-living
TD
- if (NC
* NR
) is small
Proof of concept
Preview

30. Tests fail every now and then ...
… because the tested UI is changed

31. Tests fail every now and then ...
Ah! And also because errors are made ...

32. End2End demo

33. Ts
continuends
Time spent on What to do %% of time
Allocating failures Use test harness 1% - 5%
Anderstandint the
failure reason
Use test logging, save
images, save UI state
10% - 80%
Fix the tests Move common code to
the library (*)
90% - 10%
(*) The only way it is different from programming is that there are a lot of tests.
More on this later.

34. Harness
TestNGTestNG

35. Understanding failure
Logging
UI operations
Going to push ... button
Selecting ... menu
UI responces
Found list view: …
UI hierarchy
Scene «…»
Group «…»
Button «…»
UI states
Images

36. Demo
Jemmy logging functionality

37. Fix the tests
One UI, many-many tests.
Do as few test code changes as possible
Ideally … one change in test code for one
change in UI code.

38. Decrease maintenance cost
●Implement reusable operations in a library
openFile(String filename)
●Decrease implementation dependencies
lookup(Button.class, new ByText<Button>(«Low»))
●Increasing stability
.wrap(0).mouse().click();
Thread.sleep(1000);
...

39. Demo
Refining test.

40. Library design approaches

41. Remember the formula?
TD
+ *TS
NR
TM
* NR
NC
*
EA
=
NC
*
EA
– automation effectiveness
To be used for every particular product.
NR
and NC
are unique for a product.
TM
is a characteristic of a test suite.
Smaller TD
and TS
- higher the EA
.
Coefficient depend on the way you write your tests

42. TS
mainly consists of time for test modification.
When product changes, tests need to be changed accordingly.
Many tests! Hundreds.
“Less changes of test code per a change in the product UI.”
Ideally ... no more than one.
But ... how? You are the coders – you know:
Move code to test library.

43. Application domain modelCarrecordCarrecordCarrecord
ColorColor
ModelModel MakeMake
YearYear License plateLicense plate
VINVIN
......
That's ... car catalog of some sort

44. Application UI
Product UIProduct UI

45. Coordinates
● click(134,32) //selects some record
● click(215,122) //hits “Properties”
● sleep(5) //sleeps to let dialog be painted
● click(64,182) //expands color combo
● click(235,182) //selects Gray
● click(235,212) //hit OK
TTdd
~= 1.1 * T~= 1.1 * Tmm
,, TTss
~= 1 * T~= 1 * TmmNever tried, but ...

46. Widgets
● Find “Car records” frame
● Find table
● Select “1abc234” cell
● Push “Properties” button
● Wait for “1abc234” dialog
● Select “Gray” color in combo box
● Push “OK”

47. ProductUIProductUIDomain
model
Domain
model
Car record
ColorColorModelModel
MakeMake YearYear License plateLicense plateVINVIN
Test
Widgets or coordinates
TTDD
~= 3 * T~= 3 * TMM
,, TTSS
~= .5 * T~= .5 * TMM

48. UI Primitives● Find car list frame
CarListFrame list = new CarListFrame()
● Open properties dialog for car “1abc234”
CarDialog propDialog =
list.carProperties(“1abc234”);
● Set color to gray
propDialog.setColor(Color.GRAY);
● Apply changes
propDialog.ok();

49. ProductUIProductUIDomain
model
Domain
model
Car record
Test libraryTest library
Color
Color
Model
Model
Make
Make
Year
Year
License plate
License plate
VIN
VIN
CarListFrame CarDialog
Test
TTDD
~= 5 * T~= 5 * TMM
,, TTSS
~= .2 * T~= .2 * TMM

50. Domain model
● Set col or t o gr ay f or a car
“ 1abc234”
new Car Recor d( “ 1abc234” ) .
set Col or ( Col or . GRAY) ;
Underneath the cover, CarRecord class does
all described earlier

51. Product
UI
Product
UI
Domain
model
Domain
model
Car record
Domain test libraryDomain test library
Color
Color
Model
Model
Make
Make
Year
Year
License plate
License plate
VIN
VIN
UI test libraryUI test libraryCarList CarDialog
CarRecord
Test
TTDD
~= 7.5 * T~= 7.5 * TMM
,, TTSS
~= .05 * T~= .05 * TMM

52. Coordinates Widgets UI Library Domain library
0
1
2
3
4
5
6
7
8
1.1
3
5
7.5
1
0.5
0.1 0.05
Td/Tm
Ts/Tm
Td
and Ts
together

53. Coordinates Widgets UI Library Domain library
0
5
10
15
20
25
30
1.1
3
5
7.5
24
12
2.4
1.2
25.1
15
7.4
8.7
Td
Ts
Td+(Ts*Nc*Nr)
TD
and TS
for NC
=3, NR
=8, TM
=1

54. EA
for NC
=3, NR
=8
Coordinates Widgets UI Library Domain library
0
0.5
1
1.5
2
2.5
3
3.5
0.96
1.6
3.24
2.76
Ea

55. Олег Бунин
Testing JavaFX UI
Tips from JavaFX quality team
Alexandre (Shura) Iline
Java and JavaFX quality architect
Oracle

56. Links
●http://jemmy.dev.java.net