Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Mutation Testing Workshop at Ericsson, Kista, Sweden

Ad

Finding	the	worst	tested	methods	
in	a	test	suite
Oscar	L.	Vera	
DiverSE	–	Inria	Rennes	
Mutation	Testing	Workshop	
April	...

Ad

Unit	test	case	example
class	VList	{

		private	List	elements;		
		private	int	version;	
		public	void	add(Object	item)	{	...

Ad

Developers
• Do	our	tests	assess	the	behavior	of	the	program?	
• Concrete	improvements
3

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Cargando en…3
×

Eche un vistazo a continuación

1 de 22 Anuncio
1 de 22 Anuncio
Anuncio

Más Contenido Relacionado

Anuncio

Mutation Testing Workshop at Ericsson, Kista, Sweden

  1. 1. Finding the worst tested methods in a test suite Oscar L. Vera DiverSE – Inria Rennes Mutation Testing Workshop April 12th, 2019
  2. 2. Unit test case example class VList {
 private List elements; private int version; public void add(Object item) { elements.add(item); incrementVersion(); } public int size () {
 return elements.size(); } private void incrementVersion () { version ++; } } class VListTest @Test public void testAdd (){ VList l = new VList(); l.add(1); assertEquals(l.size(), 1); } } 2
  3. 3. Developers • Do our tests assess the behavior of the program? • Concrete improvements 3
  4. 4. Introduced in 2016 R. Niedermayr, E. Juergens, and S. Wagner Will my tests tell me if I break this code? Proceedings of the International Workshop on Continuous Software Evolution and Delivery, 2016, pp. 23–29. Pseudo-tested methods 4
  5. 5. • Executed by the test suite • The code can be removed • No test case notice the change public void setValue(int x) { // pass } public void setValue(int x) { if(a + x < 10) a += x; } public int fact(int x) { int result = 0; for(int i=1; i <= x; i++) { result *= i; } return result; } public int fact(int x) { return 42; } Pseudo-tested methods 5
  6. 6. A pseudo-tested method class VList {
 private List elements; private int version; public void add(Object item) { elements.add(item); incrementVersion(); } public int size () {
 return elements.size(); } private void incrementVersion () { version ++; } } class VListTest @Test public void testAdd (){ VList l = new VList(); l.add(1); assertEquals(l.size(), 1); } } Pseudo-tested 6
  7. 7. Can pseudo-tested methods provide concrete hints to improve a test suite? 7
  8. 8. Conceptual replication • Additional set of study subjects • 21 Java open source projects • Custom and dedicated tooling • Descartes https://github.com/STAMP-project/pitest-descartes 8
  9. 9. Data: P /*Program*/, TS /*Test suite*/ Result: pseudo {pseudo-tested methods in P} foreach m in P | covered(m, TS) and ofInterest(m): variants = {extreme variants of m} if returnsValue(m): stripBody(m) checkReturnType(m) variants ! fixReturnValues(m) else: variants ! stripBody(m) failure = false foreach v in variants: P’ ! replace(m, v, P) failure ! failure or fails(TS, P’) if no failure: pseudo add m Create method variants Verify if the test suite fails 9
  10. 10. Excluded methods • Not covered by any test • Compiler generated methods • Returning a literal constant • Simple getters • Simple setters • Deprecated • hashCode • Empty 10 public int m () { return 3; } public int getI () { return this.i; } public void setI (int i) { this.i = i; } public void m () { } @Deprecated public void m () {...} public int hashCode () {...}
  11. 11. Pseudo-tested methods 0 125 250 375 500 AuthZForce PDP Core Apache Commons Codec Apache Commons Lang Jaxen XPath Engine Joda-Time SAT4J Core Spoon 239 28 213 72 473 143 28 2 82 296 476 1110 100 47 29 40 12 2 224 13 11
  12. 12. Ratio of pseudo-tested methods 12 0 % 25 % 50 % 75 % 100 % AuthZForce PDP Core Apache Commons Codec Apache Commons Lang Jaxen XPath Engine Joda-Time SAT4J Core Spoon 91 % 99 % 93 % 69 % 83 %80 % 96 %99 %97 % 90 %88 % 98 %98 %95 %98 %96 %97 %97 %99 % 89 % 96 % 9 %1 %7 % 31 % 17 %20 % 4 %1 %3 % 10 %12 % 2 %2 %5 %2 %4 %3 %3 %1 % 11 % 4 % Pseudo-tested Tested
  13. 13. Real-life examples
  14. 14. Apache Commons Codec public void testIsEncodeEquals() { final String[][] data = { {"Meyer", "Mu00fcller"}, {"Meyer", "Mayr"}, ... {"Miyagi", "Miyako"} }; for (final String[] element : data) { final boolean encodeEqual = this.getStringEncoder().isEncodeEqual(element[1], element[0]); } } No assertion Pseudo-tested 14
  15. 15. Apache Commons IO Weak oracle Result is the same if nothing is written Pseudo-tested public void testTee() { ByteArrayOutputStream baos1 = new ByteArrayOutputStream(); ByteArrayOutputStream baos2 = new ByteArrayOutputStream(); TeeOutputStream tos = new TeeOutputStream(baos1, baos2); ... tos.write(array); assertByteArrayEquals(baos1.toByteArray(), baos2.toByteArray()); } 15
  16. 16. Apache Commons Collection abstract class AbstractHashedMap<K, V> { protected void ensureCapacity(final int newCapacity) { final int oldCapacity = data.length; if (newCapacity <= oldCapacity) { return; } if (size == 0) { threshold = calculateThreshold(newCapacity, loadFactor); data = new HashEntry[newCapacity]; } else { //Rehash code ... threshold = calculateThreshold(newCapacity, loadFactor); data = newEntries; } } ... } Covered by 38 test cases Executed 11593 times 16 Pseudo-tested
  17. 17. Apache Commons Collections class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } class SingletonListIteratorTest { ... @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} ... } No exception is thrown A fail is needed here Pseudo-tested 17
  18. 18. Amazon Web Services SDK class SdkTLSSocketFactory { protected void prepareSocket(SSLSocket s){ ... s.setEnabledProtocols(protocols); ... } } @Test void typical() { SdkTLSSocketFactory f = ...; f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols (String[] protocols) { assertTrue( Arrays.equals(protocols, expected)); } ... }); } Pseudo-tested No assertion is verified if the method is emptied 18
  19. 19. Relevant • Widely used in the code • Relevant for external clients • Supports a core functionality Not relevant • Part of debug functionalities • Not widely used • Deprecated • Simple or trivial • Placeholders 30% of pseudo-tested methods considered as relevant by developers Decision is subjective in many cases 19
  20. 20. Eight selected issues • 4 communicated directly • 1 issue created in the tracker but no action • 2 direct commits with fixes • 4 pull requests fixing the issues • All accepted 20
  21. 21. Conclusions • Pseudo-tested methods may uncover real testing flaws • The testing issues are easy to understand • These flaws can be relevant • Room for automation 21
  22. 22. Current work Can these issues be automatically solved? • State-of-the-art test generation tools • Test improvement tools 22

×