JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

Paul Thwaite – Java 8 Test Lead
Wednesday 3 October 2012

Testing with Bytecode Instrumentation (BCI)
JavaOne 2012 - CON3961

© 2012 IBM Corporation

Important Disclaimers

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.

ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE
DIFFERENCES.

ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE
OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS

2 © 2012 IBM Corporation

Introduction to the speaker – Paul Thwaite

■ 11 years experience developing and deploying Java SDKs

■ Recent work focus:
– Quality Assurance lead
• Leading cross-continent teams
• System Test on IBM SDKs

– Test lead for Java 8
• Focus on testing in OpenJDK
• OpenJDK contributor

■ My contact information:
– paul_thwaite@uk.ibm.com
– linkedin.com/profile/view?id=26327969


What you will learn from this talk

■ Why me and what I do
■ What is BCI?
■ Typical problems encountered during testing
■ Two demos showing how BCI can help with testing
■ Introduction to Java agents
■ How BCI works
■ Limitations of BCI
■ Overview of the different BCI approaches


Why me and what I do

■ Test IBM's Java SDK
– I break Java
■ Focus on system testing in QA
– Multi-threaded load testing
– Third-party applications
■ System test the Java SDK means writing Java
■ Can the VM handle any application or Java code?


Bytecode Instrumentation


What is BCI?
■ Bytecode Instrumentation (BCI) is modifying the bytes of a class at runtime
■ Classes can be modified as the JVM loads them or at any point afterwards
■ Application and SDK classes can be changed (with few restrictions)
■ Abstraction frameworks exist to make BCI easier

Disk Runtime

Snippets
Class JVM
Code
Class
BCI Code
Code

Code


ASM

■ ASM is an all purpose Java bytecode manipulation framework
■ It can be used to modify existing classes or dynamically generate classes, directly in binary
form
■ Many products such as AspectJ and JRuby use ASM
■ The examples in this presentation use ASM to abstract out the complexity of transforming
classes


Problems encountered in testing


Typical problems encountered during testing

■ Test exception handling

■ Test for memory exhaustion

■ Test secure paths through code

■ Test specific code paths in very large systems such as WebSphere Application Server


Problem 1: Testing all exception paths

try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}


java.net.Socket



try {
// more code...
//Handle correctly
//Handle correctly
//Handle correctly
}

■ The test running normally



try {
// more code...
//Handle correctly
//Handle correctly
//Handle correctly
}

■ How do you force the exception?
– Pull the network cable out of the test machine
– Update /etc/hosts to force the exception

■ Not practical
– Manual, test-specific, other tests may run on the same machine


try {
// more code...
//Handle correctly
//Handle correctly
//Handle correctly
}

– Intercept the connection
– IOException is a “catch all” and very hard to create
– You would need to understand the implementation in order to test properly

■ Not practical


try {
// more code...
//Handle correctly
//Handle correctly
//Handle correctly
}

– A SecurityManager would need to be used
– Complicated to setup
– Not practical especially if testing a complex system
– Likely would need two versions of the test


Demo using BCI to help solve
these problems


Problem 1: Demo of UnknownHostException

■ What does the test do?
– Uses java.net.Socket to create a connection to three hosts
– Prints to stdout whether the connection was successful or not
– BCI is used to trigger the UnknownHostException on www.oracle.com

■ Test 1 – the test running normally

java -cp testcase tests.NetworkTest

■ Test 2 – the test running with BCI

java -Xbootclasspath/a:lib/asm-all-4.0.jar:lib/networkTrans.jar
-javaagent:lib/networkTrans.jar=www.oracle.com
-cp testcase tests.NetworkTest

Pass in the host on which to force the UnknownHostException

Problem 1: Demo of UnknownHostException

– Uses java.net.Socket to create a connection to three hosts
– Prints to stdout whether the connection was successful or not
– BCI is used to trigger the UnknownHostException on www.oracle.com


Connected to host: www.example.com
Connected to host: www.oracle.com
Connected to host: www.ibm.com


Connected to host: www.example.com
Couldn't find host: www.oracle.com UnknownHostException thrown
Connected to host: www.ibm.com


Problem 1: Breakdown of the Java command

Append the ASM Append a Java
library to the agent library to
bootclasspath the bootclasspath

java -Xbootclasspath/a:lib/asm-all-
4.0.jar:lib/networkTrans.jar
-javaagent:lib/networkTrans.jar=www.oracle.com -cp
testcase tests.NetworkTest

Enable the Java agent Pass in options to
the agent
Run the test as normal -
the test does not change
Note: The bootclasspath is used because the Socket class is being modified.

Problem 2: Testing Resource Exhaustion

■ A large system, such as WebSphere Application Server, should be able to handle an
OutOfMemoryError
■ An OutOfMemoryError can occur at any time
– It can also happen in different places within the system
■ Hard and complicated to test without side effects
– Changing the application server's environment is not a good idea
– Using a reduced heap will not guarantee all cases are covered


Problem 2: Demo of OutOfMemoryError – Java command

– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown
– It will take a very long time for the OutOfMemoryError exception to be thrown
– BCI is used to set the trigger to just three Integers without having to change the test

java -cp testcase tests.AllocateTest
Option to trigger
OOME at just 3
Integers

java -javaagent:lib/resourceTrans.jar=3
-cp testcase:lib/asm-all-4.0.jar tests.AllocateTest


Problem 2: Demo of OutOfMemoryError – stdout

– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown
– It will take a very long time for the OutOfMemoryError exception to be thrown
– BCI is used to set the trigger to just three Integers without having to change the test
Adding new value 0
Adding new value 1
Adding new value 2...
...Adding new value 450...

Adding new value 0
Adding new value 1
Adding new value 2
Adding new value 3
Exception in thread "main" java.lang.OutOfMemoryError at
transformer.ResourceTransformer.resourceHelper(ResourceTransformer.java:79)
at tests.AllocateTest.increment(AllocateTest.java)
at tests.AllocateTest.main(AllocateTest.java:12)

Problem 2: Breakdown of the Java command

Enable the Java agent Pass in options to
the agent

java -javaagent:lib/resourceTrans.jar=3 -cp
testcase:lib/asm-all-4.0.jar tests.AllocateTest

Add the ASM library to Run the test as normal -
the classpath the test does not change


Recap – why BCI?

■ Some parts of Java applications are hard to test
■ BCI provides a solution to this problem
■ With BCI, you can:
– Replace your method bodies at runtime
– Replace your entire class
– All without having to change your code
■ BCI is performed by inserting new bytecodes into your application at runtime
■ Using the UnknownHostException example, there are two ways to perform BCI

– Change the testcase to use a mock version of java.net.Socket which calls a different
class i.e. MySocketClass

– Modify the java.net.Socket class
■ Whichever you choose, you will be playing with bytecodes


Recap – why BCI?

■ Some parts of Java applications are hard to test
■ BCI provides a solution to this problem
■ With BCI, you can:
– Replace your method bodies at runtime
– Replace your entire class
– All without having to change your code
■ BCI is performed by inserting new bytecodes into your application at runtime
■ Using the UnknownHostException example, there are two ways to perform BCI

– Change the testcase to use a mock version of java.net.Socket which calls a different
class i.e. MySocketClass

– Modify the java.net.Socket class
■ Whichever you choose, you will be playing with bytecodes
■ But don't fear – there are tools available which can help you!


Java agents


Java agent overview

■ A Java agent provides instrumentation capabilities to a Java application
■ Java agents provide the means to perform bytecode transformation
■ An agent must have a premain method
import java.lang.instrument.Instrumentation;
public class MyJavaAgent {
public static void premain(String agentArgument,
Instrumentation instrumentation)
{}
}
Manifest-Version: 1.0
Premain-Class: MyJavaAgent
■ It is packaged inside a JAR file
■ The JAR file manifest defines the agent class
■ The agent has 'no modelling restrictions' i.e. it can do anything any other java code can do
– Start threads, listen on the network etc
■ The Instrumention object is used to perform class transformations


How BCI works


Using BCI to modify java.net.Socket

Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?

ASM
visitCode()
Socket(String host,int port)
insert new code



Socket
visitMethod(info)

ASM
visitCode()
insert new code



Socket
visitMethod(info)

ASM
visitCode()
insert new code
if(hostname.equals(host)) throw new
UnknownHostException();


if (hostname.equals(host)) throw new

ASM logic

mv.visitVarInsn(Opcodes.ALOAD, 1);

mv.visitMethodInsn(INVOKESPECIAL,
"java/net/UnknownHostException", "<init>", "()V");

mv.visitCode();


if (hostname.equals(host)) throw new

Bytecodes

0: aload_1
1: ldc #2; // String host
3: invokevirtual #3; //Method
java/lang/String.equals:(Ljava/lang/Object;)Z
6: ifeq 17
9: new #4; //class java/net/UnknownHostException
12: dup
13: invokespecial #5; //Method
java/net/UnknownHostException."<init>":()V
16: athrow
17: return

Limitations of BCI at runtime

■ Restrictions when retransforming classes at runtime:
– Fields cannot be added or removed
– Methods cannot be added or removed
– Method signatures cannot be modified
■ Effectively only method bodies can be modified
■ Most classes can be modified (including most SDK classes)


BCI Options


Replace a class using a pre-compiled version of your new class

■ Pre-compile a new version of a class
■ When the class to be loaded next is the one you want to replace
■ You can return the bytes for your new class and discard the old one
■ No knowledge of bytecode is required
■ Original source must be available and unchanged
■ Need to match the correct compiler version


Modify bytecodes with a BCI framework

■ Use a framework
– Provides an abstraction layer
– Easier to learn (provides symbolic constants, helper APIs)
– Adds an additional dependency
■ ASM
– Widely used
– Uder active development
– Friendly license
■ ASM gives you a good symbolic framework for writing bytecodes
■ Knowledge of the JVM architecture is required


Use javap to create bytecodes

■ Write some java code that does what you want
■ Compile your class to byte-code
■ Use a de-compiler to examine the class file
■ JVM ships with javap
■ verbose option (-v) decompiles method bytecodes

void setupConnections();
Code:
0: ldc #2 // String www.example.com
2: sipush 1234
5: invokestatic #3 // Method MySocketFactory.newSocket:
(Ljava/lang/String;I)Ljava/net/Socket;
8: astore_1
9: return


ASMifier – Create ASM logic

■ ASMifier
– Part of standard ASM distribution
– Turn a Java method into byte code logic for insertion into an ASM class
– An Eclipse plugin is available

mv.visitLdcInsn("www.example.com");
mv.visitIntInsn(SIPUSH, 1234);
mv.visitMethodInsn(INVOKESTATIC, "transformer/SocketHelper$MySocketFactory",
"newSocket", "(Ljava/lang/String;I)Ljava/net/Socket;");
mv.visitVarInsn(ASTORE, 1);
mv.visitInsn(RETURN);


What can go wrong? (Dealing with class verify problems)

■ If the JVM is unhappy with your modified class, it may throw a java.lang.VerifyError
■ 'stack shape inconsistent'/'stack underflow'
– Attempting to operate on the wrong type of variable/stack is empty
– Often due to failure to push/pop all the correct variables

■ 'Inconsistent stackmap frames'
– Stackmaps are method meta-data used by the JVM
– A map of all the variables on the stack at the target of any branch/goto instruction
– BCI cause targets to move, invalidating the stack maps
– ASM will re-calculate them for you

■ 'stack overflow'
– Maximum stack size specified in method meta data
– Any operations added via BCI may increase required stack
– ASM will calculate that for you too


What can go wrong? (Runtime Problems)

■ No compile time checking
■ Access control
– Calling a private method is valid byte-code, fails at runtime
■ Calling a non-existent method
– Incorrect argument types
– Incorrect argument ordering
– Typos


Conclusion

■ BCI is very powerful
■ BCI is used in many products
■ JVM knowledge is required but...
■ There are many tools available to make it easier
■ BCI allows testing of otherwise impossible or tricky situations

■ Slides will be available on Slideshare after the conference
■ Examples are available on request
– paul_thwaite@uk.ibm.com


References

■ Get Products and Technologies:
– IBM Java Runtimes and SDKs:
• https://www.ibm.com/developerworks/java/jdk/
– IBM Monitoring and Diagnostic Tools for Java:
• https://www.ibm.com/developerworks/java/jdk/tools/

■ Learn:
– IBM Java InfoCenter:
• http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp
■ Discuss:
– IBM Java Runtimes and SDKs Forum:
• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0


References

■ Eclipse Bytecode Outline plugin (allows you to use asmifier in Eclipse)
– http://andrei.gmxhome.de/bytecode/index.html
■ ASM
– http://asm.ow2.org/index.html
■ Java VM Spec
– http://docs.oracle.com/javase/specs/jvms/se7/html/index.html
■ java.lang.instrument package documentation
– http://docs.oracle.com/javase/7/docs/api/index.html?java/lang/instrument/package-summary.html


Copyright and Trademarks

© IBM Corporation 2012. All Rights Reserved.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corp., and registered in many jurisdictions worldwide.

Other product and service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark
information” page at URL: www.ibm.com/legal/copytrade.shtml


JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (9)

Similar a JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

Similar a JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation (20)

Último

Último (20)

JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation