This slide is for a presentation at the 25th International Conference on Software Engineering and Knowledge Engineering (SEKE'13) at Boston. Abstract of the published paper: When developers test modern web applications that use JavaScript, challenging issues lie in their event-driven, asynchronous, and dynamic features. Many researchers have assessed the adequacy of test cases with code coverage criteria; however, in this paper, we show that the code coverage-based approach possibly misses some faults in the applications. We propose a mutation analysis approach for estimating the fault-finding capability of test cases. We assume that developers can find overlooked fault instances and improve the test cases with the estimated capability. To create a set of faulty programs, i.e., mutants, we classify the JavaScript features in web applications and then define a comprehensive set of mutation operators. We conducted a case study on a real-world application and found that our approach supported the improvement of test cases to expose hand-seeded faults by an extra ten percent. You can find the transcript used in the presentation from the link below: https://docs.google.com/document/d/1CsqcYKyHZQL6Z7OkM_LVVyB-aMURMWl37lc03LOa4fI/edit?usp=sharing

1. Boston
USA
Kazuki Nishiuraa
Yuta Maezawaa
Hironori Washizakib
Shinichi Honidena,c
06/27/2013
Mutation Analysis for
JavaScript Web
Application Testing
SEKE
2013 aThe University of Tokyo, Japan
bWaseda University, Japan
cNational Institute for Informatics, Japan

2.  Background
 Mutation Analysis: Test adequacy criteria
 JavaScript Web Applications (JSWAs)
 AjaxMutator: Mutation Analysis tool for JSWAs
(http://knishiura-lab.github.io/AjaxMutator/)
 Feature analysis on JSWA
 Definition of Mutation Operators
 Bug survey/case study using real-world apps.
2
Table of Contents
Improvement of test cases; exposed 10% more faults

3. is a method to evaluate adequacy of test cases
3
Mutation Analysis
Ticket t = issueTicket(true, true);
assertEquals(600, t.price);
Do we have
enough test ?
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
t = {price:500, numBanquet:0};
} else {
t = {price:700, numBanquet:1};
}
if (extraBanquet) {
t.numBanquet = t.numBanquet+1;
t.price = t.price + 100;
}
return t;
}
Ticket t = issueTicket(false, false);
assertEquals(700, t.price);
Program under test Test case #1
Test case #2

4. is a method to evaluate adequacy of test cases
4
Mutation Analysis
Ticket t = issueTicket(true, true);
assertEquals(600, t.price);
Do we have
enough test ?
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
t = {price:500, numBanquet:0};
} else {
t = {price:700, numBanquet:1};
}
if (extraBanquet) {
t.numBanquet = t.numBanquet+1;
t.price = t.price + 100;
}
return t;
}
Ticket t = issueTicket(false, false);
assertEquals(700, t.price);
Program under test Test case #1
Test case #2
Test #1 Test #2
100% coverage

5. is a method to evaluate adequacy of test cases
Mutation Analysis
Ex) Program to calculate 𝑥2 − 𝑦2
int diffOfSquares(int x, int y){
int sum = x + y;
int diff = x – y;
return sum * diff;
}
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
t = {price:500, numBanquet:0};
} else {
t = {price:700, numBanquet:1};
}
if (extraBanquet) {
t.numBanquet = t.numBanquet-1;
t.price = t.price + 100;
}
return t;
}
Mutant #1
by injecting artificial fault
Mutation operator
+ -
• Mutant guides improvement of test cases
• Mutation operator determine mutant
Ticket t = issueTicket(true, true);
assertEquals(600, t.price);
Ticket t = issueTicket(false, false);
assertEquals(700, t.price);
Test case #1
Test case #2
Unkilled by test #1,2

6. is a method to evaluate adequacy of test cases
Mutation Analysis
Ex) Program to calculate 𝑥2 − 𝑦2
int diffOfSquares(int x, int y){
int sum = x + y;
int diff = x – y;
return sum * diff;
}
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
t = {price:500, numBanquet:0};
} else {
t = {price:700, numBanquet:1};
}
if (extraBanquet) {
t.numBanquet = t.numBanquet-1;
t.price = t.price + 100;
}
return t;
}
Mutant #1
by injecting artificial fault
Mutation operator
+ -
Ticket t = issueTicket(true, true);
assertEquals(1, t.numBanquet);
Test case #3
• Mutant guides improvement of test cases
• Mutation operator determine mutant
Ticket t = issueTicket(true, true);
assertEquals(600, t.price);
Ticket t = issueTicket(false, false);
assertEquals(700, t.price);
Test case #1
Test case #2
Unkilled by test #1,2killed by test #3

7. 7
How JavaScript Web Applications Work
Web Page
Async. comm.Event invocation
e.g., click, mouseover
Update page
Send DataDisplay
Server
Client
HTML/CSS
XML/XSLT
DOM
XHR
JavaScript
Can we test JavaScript program adequately?
Target of our research

8. 8
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
No exception doesn’t mean no fault Insufficiency of coverage

9. 9
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
On detail loaded
No exception doesn’t mean no fault Insufficiency of coverage

10. 10
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
On detail loaded
Click
No exception doesn’t mean no fault Insufficiency of coverage

11. 11
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
On detail loaded
Click
No exception doesn’t mean no fault Insufficiency of coverage

12. 12
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
On detail loaded
Click After a while
No exception doesn’t mean no fault Insufficiency of coverage

13. 13
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
Statement coverage：100%
On detail loaded
Click After a while
No exception doesn’t mean no fault Insufficiency of coverage

14. 14
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
Statement coverage：100%
On detail loaded
Click After a while
No exception doesn’t mean no fault Insufficiency of coverage
Fault: wrong property name
// should be data.price
elm.setText(data.value);

15. 15
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
！

16. 16
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
Mutant
！
AjaxMutator
(2) Generate mutants

17. 17
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
Mutant
Unkilled Mutant
！
AjaxMutator
(2) Generate mutants

18. 18
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
(4)Add Test
Mutant
Unkilled Mutant
！
AjaxMutator
(2) Generate mutants

19. (3) Execute
Test
19
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
(4)Add Test
Mutant
Unkilled Mutant
！
AjaxMutator
1. Analyze feature of JSWA
2. Define mutation operators
Approach(2) Generate mutants

20. ①
②
③
Web Page Send DataDisplay
Server
Client
How JSWA works
XHRXML/XSLTDOMHTML, etc..
JavaScript
20
Three Main Characteristics of JSWAs [1]
[1] Justin Gehtland, et al. “Pragmatic Ajax: A Web 2.0 Primer”, O’REILLY, 2006
① ②
③
Async. comm.
Update page
Event invocation
e.g., click, mouseover
JavaScript Web Application
Asynchronous communicationsEvent DOM Manipulation
① ② ③

21. JavaScript Web Application
Asynchronous communications
Feature Diagram for Event Registration
Event Registration
Interval
Callback Function
Timer Event Registration
Repeat?
Target
Event Type
User Event Registration
Callback Function
Window DOM Element Body Return Value
Click Etc…Focus
21
Mandatory Optional
Alternative Or

22. Event Registration
Timer Event Regi
22
detailButton1.click(showDetail1);
Original Program
loginButton.click(showDetail1);
Mutant #1
Target Event Type
User Event Registration
Callback Function
Window DOM Element Body Return ValueClick Etc…Focus
Defining Mutation Operators Based on Features
Click

23. Event Registration
Timer Event Regi
23
detailButton1.click(showDetail1);
Original Program
loginButton.click(showDetail1);
Mutant #1
Mutant #2
Target Event Type
User Event Registration
Callback Function
Window DOM Element Body Return ValueClick Etc…Focus
Defining Mutation Operators Based on Features
Click
detailButton1.mouseover(showDetail1);

24. Event Registration
Timer Event Regi
24
detailButton1.click(showDetail1);
Original Program
loginButton.click(showDetail1);
Mutant #1
Mutant #2
detailButton1.click(showDetail2);
Mutant #3
Target Event Type
User Event Registration
Callback Function
Window DOM Element Body Return ValueClick Etc…Focus
Defining Mutation Operators Based on Features
Click
detailButton1.mouseover(showDetail1);

25. Mutant #1
Event Registration
Timer Event Regi
25
Click
detailButton1.click(showDetail1);
Original Program
loginButton.click(showDetail1);
detailButton1.mouseover(showDetail1);
Mutant #2
detailButton1.click(showDetail2);
Mutant #3
Target Event Type
User Event Registration
Callback Function
Window DOM Element Body Return ValueClick Etc…Focus
Defining Mutation Operators Based on Features
Test in
motivating example
Require additional test cases to find faults

26. 26
Timer event
Mutant #2
setTimeout(showDetail, 1000);
setTimeout(applyDiscount1, 1000);
Original Program Mutant #1
setTimeout(applyDiscount1, 2000);
Event Registration
Interval
Callback Function
Repeat?
nt Registration Timer Event Registration

27. JavaScript Web Application
27
Request
Response
Asynchronous Communication
Callback Function
URL Method Body Return Value
Status Body
Parameter
Event DOM Manipulati
Asynchronous Communication
Mutant #2
$.get(“buy.php”, showButtons);
$.get(“items.php”, showMsg);
$.get(“items.php”, showButtons);
Original Program
Focus on client-side features
Mutant #1

28. ommunication
Web Application
28
Target Selection Manipulation
Relative
Position
Tag Name
Creation Deletion
Insertion
Attribute
Value Attribute Assignment
DOM Manipulation
Mutant #1
Mutant #2
$(“.detailBtn”).parent();
btn.width = 300;
$ (“.detailBtn”);
Original Program #1
Mutant #1
btn.height = 100;btn.width = 100;
Original Program #2
DOM Manipulation

29. 1. Can JavaScript features really cause faults?
2. Can developers improve test cases
with AjaxMutator to find unexposed faults?
3. Can AjaxMutator be used in a reasonable time?
29
Evaluation: Research Questions

30. 30
RQ1: Actual faults in JSWAs related to features
[2] Frolin S. Ocariza Jr. et al. “Autoflox: An automatic fault localizer for client-side javascript”
Ticket # Related Feature Brief Explanation
1895 User Event Don’t handle on-focus event properly
2184 Don’t handle on-resize event properly
8812 DOM
Manipulation
Generating span element where p should be generated
9740 Selecting some unwanted elements
Search from bug repository of
JavaScript features can really cause actual faults
“JavaScript”, “JS”, or “console” [2]Keyword

31. Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Tests to kill
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
(1)Implement Test
* Undergraduate students who has been developed JSWAs for two years in industry
• Normal
• Invalid
RQ2, 3: Case study setup
scenario

32. Faulty
program
×20
Faulty
program
×20
32
Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Tests to kill
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
Faulty
program
×20
(1)Implement Test
Participant*
(3) Seed
fault
* Undergraduate students who has been developed JSWAs for two years in industry
• Normal
• Invalid
RQ2, 3: Case study setup
scenario

33. Faulty
program
×20
Faulty
program
×20
33
Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Tests to kill
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
Faulty
program
×20
(1)Implement Test
Participant*
(3) Seed
fault
* Undergraduate students who has been developed JSWAs for two years in industry
• Normal
• Invalid
(4) Check if test
can find faults
RQ2, 3: Case study setup
scenario

34. ×20×20
34
Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Test to kill
more mutants
Mutant
Unkilled
Mutant
×20
(1)Implement
Test
Participant*
(3) Seed
faults
• Normal use case
• Another invalid scenario
(4) Check if test
can find faults
RQ2: Improvement of test cases
We could improve test cases to find unexposed faults
Test suites # of test cases Coverage Killed mutants Fault finding capability
Initial 2 95% 45.9% 89.5%
Improved +5 100% 67.0% 100%

35. 35
Limitations
5 5 4 4
20
8
4
2 2
1
16
1
1
1
2
17
4
6
2
2
2
0
10
20
30
40
50
60
Semantically Equivalent (5.5%)
Unkilled (27.5%)
Killed by improved test (21.1%)
Killed by initial test (45.9%)
Equivalent mutants
Browser robustness
* target program deal with only click-type event
*
elm.width=100 -1
Browser ignore invalid
value and infer correct one
further challenge
Details of generated mutants
val = 100
elm.width=100 val

36. 36
RQ3: Time for Mutation Analysis
Web Applications
JavaScript
109 Mutants
59 Unkilled
AjaxMutator
20 min.
109 mutants × 2 tests
20 min.
1 hour
Implement Test
3 hour
Add Test
Reasonable time; automation/optimization can be applied
59 mutants × 5 tests

37. 37
Related Work
 For HTML and JSP [Praphamontripong’10]
 For JSWA but limited features [Mirshokraie’13]
 Faults inWeb apps [Marchetto’07][Guo’09]
 JavaScript run-time errors [Ocariza’11]
Survey on real-world faults
Mutation analysis on Web Apps
AjaxMuator: comprehensive mutation operators for features of JSWA
Definition of mutation operators by a top-down way
No bug reports can be used for defining mutation operators

38.  Proposal of mutation operators by feature analysis of JSWAs
 Publicly available implementation: AjaxMutator
 http://knishiura-lab.github.io/AjaxMutator/
 Survey and case study using real-world apps:
 AjaxMutator helps aa
 More case study and survey
 Automated test case generation
38
Conclusion & Future Work
Conclusion
Future Work
Improvement of test cases to expose faults

39. Q&A
39

40. 40
RQ3: Time for Mutation Analysis
Initial Test Additional Test
Preparing test cases
(manual)
1 hour 3 hours
Mutation Analysis
(Automated)
20 minutes
(2 Test cases * 109 mutants)
20 minutes
(5 Test cases×59 mutants)
To reduce manual cost
→ We can utilize automated-test case generation technique
To reduce mutation analysis cost (computational time)
→ We can parallelize execution
• AjaxMutator can be used in reasonable time
• Automation/Optimization can be applied

41. 41
Defined Mutation Operators
JS Features Operation target Mutation Example
User Event Event target cancelBtn.click(requestBuy);
Event type buyBtn.mouseover(requestBuy);
Callback function buyBtn.click(closeModal);
Timer Event Timer interval setTimeout(anotherCallback, 1000);
Callback function setTimeout(callback, 2000);
Asynchronous
Communications
Request target $.get(“server.php”, callback);
Request on-success
callback function
$.get(“server.php”, callback);
DOM Manipulation Selection of element $(“.btn”).id = “btn1”;
Assigned property of DOM $(“#btn”).className = “btn1”;
Assigned value to DOM $(“#btn”).id = “btn0”;

42. Category Mutation Target Ours Mirshokraie’s[3]
User event Target, type, callback ✔ NA
Timer event Interval, callback ✔ NA
Asynchronous
communications
Request destination URL ✔ ✔
Asynchronous or Synchronouns NA(Out of scope) ✔
Callback ✔(on-success
callback)
✔(condition on which
callback is invoked)
DOM Element selection ✔ ✔
Attribute assignment ✔ ✔
Element insertion ✔(Implemented
after submission)
✔
Other Common mistakes for
inexperienced programmers
NA ✔
42
Comparison between Mirshokraie et al.’13[3]
[3] A. Mirshokraie, Shabnam, et al.
“Efficient Javascript Mutation Testing,” Proc. of ICST, 2013