SlideShare una empresa de Scribd logo
1 de 37
Descargar para leer sin conexión
What Test Coverage
Mean to us?
Joseph Yao
Tuesday, July 9, 13
Who I am?
n Agile Coach at Odd-e
n Father, Husband...
n Software Craftsman
n Basketball, Board Game...
Tuesday, July 9, 13
Agenda
n Bias and Truth of Test Coverage
n Classical Test Coverage
n Mutation Test Coverage
n TDD and Test Coverage
Tuesday, July 9, 13
Bias of Test Coverage
Tuesday, July 9, 13
Truth of Test Coverage
I expect a high level of coverage. Sometimes
managers require one. There's a subtle difference.
-- Brian Marick
http://martinfowler.com/bliki/TestCoverage.html
Tuesday, July 9, 13
Truth of Test Coverage
Tuesday, July 9, 13
Classical Test Coverage
Tuesday, July 9, 13
Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)
Tuesday, July 9, 13
Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Tuesday, July 9, 13
Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)
Tuesday, July 9, 13
Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Unsatisfied Test: foo(2, -1)
Tuesday, July 9, 13
Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2, 2), foo(-1, 2)
Tuesday, July 9, 13
Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo(2, 2)
Tuesday, July 9, 13
Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1)
T/F
Tuesday, July 9, 13
Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo(2, 2), foo(-1, 2)
T/F
Tuesday, July 9, 13
Classical Test Coverage
Tuesday, July 9, 13
Classical Test Coverage
Tuesday, July 9, 13
Expectation of Automated Test
Tuesday, July 9, 13
Expectation of Automated Test
Any Possible “Bad” Code
Change can be Detected
Tuesday, July 9, 13
Mutation Test Coverage
Tuesday, July 9, 13
Why Mutation Test
Tuesday, July 9, 13
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
Tuesday, July 9, 13
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
Tuesday, July 9, 13
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
n Missing test to prove the sideEffect call is needed
n If code is redundant, it can’t be learnt via any
classical test coverage
Tuesday, July 9, 13
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
n Test not prevent “bad” code change to happen
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
Other Mutators
n Negate Conditionals Mutator
n Math Mutator
n Increments Mutator
n Invert Negatives Mutator
n Inline Constant Mutator
n Return Values Mutator
n Void Method Calls Mutator
n Non Void Method Calls Mutator
n Constructor Calls Mutator
Tuesday, July 9, 13
Test Driven Development
Tuesday, July 9, 13
TDD and Test Coverage
n You can most likely ignore Classical Test Coverage
n You can’t ignore Mutation Test Coverage
n If not TDD, then you can learn a lot from both
Classical and Mutation Test Coverage
Tuesday, July 9, 13
PokerHands Kata
Tuesday, July 9, 13
True Mutation
Tuesday, July 9, 13
Problematic Test
Tuesday, July 9, 13
False Mutation
Tuesday, July 9, 13
Reference
n Martin Fowler’s blog http://martinfowler.com/bliki/
TestCoverage.html
n Mutation Test Wikipedia http://en.wikipedia.org/
wiki/Mutation_testing
n Mutation Test Analysis Report http://
crestweb.cs.ucl.ac.uk/resources/
mutation_testing_repository/TR-09-06.pdf
Tuesday, July 9, 13
Q & A
新浪微博:姚若舟
微信:yaoruozhou
土豆(Kata视频):姚若舟
Github: JosephYao
Tuesday, July 9, 13

Más contenido relacionado

Último

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 

Último (20)

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 

What test coverage mean to us

  • 1. What Test Coverage Mean to us? Joseph Yao Tuesday, July 9, 13
  • 2. Who I am? n Agile Coach at Odd-e n Father, Husband... n Software Craftsman n Basketball, Board Game... Tuesday, July 9, 13
  • 3. Agenda n Bias and Truth of Test Coverage n Classical Test Coverage n Mutation Test Coverage n TDD and Test Coverage Tuesday, July 9, 13
  • 4. Bias of Test Coverage Tuesday, July 9, 13
  • 5. Truth of Test Coverage I expect a high level of coverage. Sometimes managers require one. There's a subtle difference. -- Brian Marick http://martinfowler.com/bliki/TestCoverage.html Tuesday, July 9, 13
  • 6. Truth of Test Coverage Tuesday, July 9, 13
  • 8. Function Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Satisfied Test: foo(2, 2) Tuesday, July 9, 13
  • 9. Function Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Tuesday, July 9, 13
  • 10. Statement Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Satisfied Test: foo(2, 2) Tuesday, July 9, 13
  • 11. Statement Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Unsatisfied Test: foo(2, -1) Tuesday, July 9, 13
  • 12. Decision Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Satisfied Test: foo(2, 2), foo(-1, 2) Tuesday, July 9, 13
  • 13. Decision Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Unsatisfied Test: foo(2, 2) Tuesday, July 9, 13
  • 14. Condition Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1) T/F Tuesday, July 9, 13
  • 15. Condition Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Unsatisfied Test: foo(2, 2), foo(-1, 2) T/F Tuesday, July 9, 13
  • 18. Expectation of Automated Test Tuesday, July 9, 13
  • 19. Expectation of Automated Test Any Possible “Bad” Code Change can be Detected Tuesday, July 9, 13
  • 22. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } Tuesday, July 9, 13
  • 23. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } Tuesday, July 9, 13
  • 24. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } n Missing test to prove the sideEffect call is needed n If code is redundant, it can’t be learnt via any classical test coverage Tuesday, July 9, 13
  • 25. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  • 26. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  • 27. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Test to cover this Mutator: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, 0)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  • 28. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) n Test not prevent “bad” code change to happen Test to cover this Mutator: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, 0)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  • 29. Other Mutators n Negate Conditionals Mutator n Math Mutator n Increments Mutator n Invert Negatives Mutator n Inline Constant Mutator n Return Values Mutator n Void Method Calls Mutator n Non Void Method Calls Mutator n Constructor Calls Mutator Tuesday, July 9, 13
  • 31. TDD and Test Coverage n You can most likely ignore Classical Test Coverage n You can’t ignore Mutation Test Coverage n If not TDD, then you can learn a lot from both Classical and Mutation Test Coverage Tuesday, July 9, 13
  • 36. Reference n Martin Fowler’s blog http://martinfowler.com/bliki/ TestCoverage.html n Mutation Test Wikipedia http://en.wikipedia.org/ wiki/Mutation_testing n Mutation Test Analysis Report http:// crestweb.cs.ucl.ac.uk/resources/ mutation_testing_repository/TR-09-06.pdf Tuesday, July 9, 13