A Critique of the Proposed National Education Policy Reform
Testing for Educational Gaming and Educational Gaming for Testing
1. Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)
Microsoft Research Connections (Judith Bishop)
Testing for
Educational Gaming and
Educational Gaming
for Testing
Tao Xie
North Carolina State University
contact: taoxie@gmail.com
2. Testing Educational Gaming
Dynamic Symbolic Execution (Pex)
Parameterized Unit Testing
Pex for Fun (Pex4Fun)
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com
6. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
7. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Input
null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
8. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Input
null
Execute&Monitor
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
9. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Constraints to solve
a!=null
Input
null
Execute&Monitor
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
10. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Constraints to solve
a!=null
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
11. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
F
TF
a==null
Constraints to solve
a!=null
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
12. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
F
TF
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
13. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
14. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
15. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
16. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
17. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
18. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Done: There is no path left.
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
19. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
20. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
for any list,
for any item,
21. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
… adding 1 item
increases Count by 1
22. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
23. Parameterized Unit Test =
Unit Test with Parameters
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
24. Parameterized Unit Test =
Unit Test with Parameters
Separation of concerns
Specification of behavior
Data to achieve coverage
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
25. Parameterized Unit Testing in Practice
Getting Popular
Parameterized Unit Tests (PUTs) commonly
supported by various test frameworks
• .NET: Supported by .NET test frameworks
• Java: Supported by JUnit 4.X
Generating test inputs for PUTs supported by tools
• .NET: Supported by Microsoft Research Pex
• Java: Supported by Agitar AgitarOne
(http://www.agitar.com/)
27. Limitations
16K characters of code (single file / editor window)*
Single-threaded only
No environment interactions*
No non-determinism*
Try to avoid floating-point computations
Max. 20s total exploration time*
* Limitation only applies to pex4fun.com, but not standalone Pex tool
28. Dynamic Symbolic Execution
Summary
“Ask Pex” sends code to cloud
Code is compiled and analyzed in cloud
Dynamic Symbolic Execution automatically finds
relevant interesting test inputs that achieve high
code coverage
Results are shown in browser
34. Coding Duels
Pex computes “semantic diff” in cloud
code written in browser vs.
secret reference implementation
You win when Pex finds no differences
35. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18
36. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18
37. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18
38. Coding Duels
Fun and Engaging
Iterative gameplay
Adaptive
Personalized
No cheating
Clear winning criterion
39. Example User Feedback
“It really got me *excited*. The part that got me most is
about spreading interest in teaching CS: I do think that it’s
REALLY great for teaching | learning!”
“I used to love the first person shooters and the
satisfaction of blowing away a whole team of
Noobies playing Rainbow Six, but this is far more
fun.”
“I’m afraid I’ll have to constrain myself to spend just an hour
or so a day on this really exciting stuff, as I’m really stuffed
with work.”
Released since 2010
X
41. Coding Duels
Summary
Coding Duel: write function that matches
specification given by another implementation
Semantic equivalence checked in cloud
Student’s tasks:
Infer likely intended behavior
Express behavior as code
http://pex4fun.com/CreatingAndPublishingCodingDuels
http://pex4fun.com/TipsAndTricksForGreatDuels
48. Coding Duels for Training Testing
public static string Puzzle(int[] elems, int capacity, int elem) {
if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))
return "Assumption Violation!";
Stack s= new Stack(capacity);
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
int origSize = s.GetNumOfElements();
//Please fill in below test scenario on the s stack
//The lines below include assertions to assert the program behavior
PexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);
PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());
PexAssert.IsTrue(s.IsMember(elem));
return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “
+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();
}
Set up a stack with some elements
Cache values used in assertions
49. Usage Scenarios of Pex4Fun
• Massive Open Online Courses (MOOC): Challenges
– Grading, addressed by Pex4Fun
– Cheating [Open Challenge]
• Course assignments (students/professionals)
– E.g., intro programming, software engineering
• Student/professional competitions
– E.g., coding-duel competition at ICSE 2011
• Assessment of testing/programming/problem
solving skills for job applicants
– Not just final results of problem solving but also process!
50. Teaching and Learning
Summary
How to become a teacher:
Sign in, choose nickname
Send us email with nickname
Teachers can
Create courses
Reuse existing or author existing pages
Get automated assessments of students
See students code
http://pex4fun.com/TeachingWithPexForFun
52. More Reading
Nikolai Tillmann, Jonathan De Halleux, Tao Xie,
Sumit Gulwani and Judith Bishop
Teaching and Learning Programming and
Software Engineering via Interactive Gaming
In Proceedings of the 35th International
Conference on Software Engineering (ICSE 2013),
Software Engineering Education (SEE), San
Francisco, CA, May 2013.
http://people.engr.ncsu.edu/txie/publications/ic
se13see-pex4fun.pdf
53. Conclusion
Pex
Test generation for .NET
based on dynamic symbolic execution
Enables Parameterized Unit Testing
Pex4Fun on the web
.NET: C#, Visual Basic, F#
Execution, code analysis with Pex
Coding Duels
Great for teaching, learning
http://pex4fun.com
http://research.microsoft.com/pex
54. Testing Educational Gaming
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com
Q & A
Thank you!
contact: taoxie@gmail.com
Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,
a Microsoft Research SEIF Award, and a Microsoft Research Award.