There is hardly a Senior Java developer who has never heard of sun.misc.Unsafe. Though it has always been a private API intended for JDK internal use only, the popularity of Unsafe has grown too fast, and now it is used in many open-source projects. OK.RU is not an exception: its software also heavily relies on Unsafe APIs.
During this session we'll try to understand what is so attractive about Unsafe. Why do people keep using it regardless the warnings of removal from future JDK releases? Are there any safe alternatives to private API or is it absolutely vital? We will review the typical cases when Java developers prefer to go unsafe and discuss major benefits and the drawbacks of it. The report will be supported by the real examples from OK.RU experience.
3. 3
Can I override object with sun.misc.Unsafe?
Is there a way to force unload a class
by using the sun.misc.Unsafe class?
Can one break a security manager with sun.misc.Unsafe?
Rescuing a swallowed Exception in Java
How to free memory using Java Unsafe, using a Java reference?
Create a mutable java.lang.String
Mystic sun.misc.Unsafe
4. 4 Mystic sun.misc.Unsafe
How to free memory using Java Unsafe, using a Java reference?
Player p = (Player) unsafe.allocateInstance(Player.class);
// Is there a way of accessing the memory address
// from the object reference
unsafe.freeMemory(p.hashCode());
http://stackoverflow.com/questions/24429777
5. 5 Mystic sun.misc.Unsafe
Why it is called Unsafe
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002ad221034ee5, pid=3680, tid=1091057984
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64)
# Problematic frame:
# C [libc.so.6+0x6dee5] cfree+0x25
#
# An error report file with more information is saved as:
# /home/apangin/hs_err_pid3680.log
#
6. 6
What is Unsafe actually
• Bridge between Class Library and JVM
• For use inside JDK
- NIO
- AWT
- Reflection
- java.util.concurrent
- MethodHandles
• Exists in many JVMs, even on Android
Mystic sun.misc.Unsafe
7. 7 Mystic sun.misc.Unsafe
public final class Unsafe {
static {
registerNatives();
sun.reflect.Reflection.registerMethodsToFilter(Unsafe.class, "getUnsafe");
}
private Unsafe() {}
private static final Unsafe theUnsafe = new Unsafe();
@CallerSensitive
public static Unsafe getUnsafe() {
Class cc = Reflection.getCallerClass();
if (cc.getClassLoader() != null)
throw new SecurityException("Unsafe");
return theUnsafe;
}
8. 8 Mystic sun.misc.Unsafe
Getting Unsafe
public static Unsafe getUnsafe() throws Exception {
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
return (Unsafe) f.get(null);
}
9. 9
Security hole?
Security Manager is off by default
$ java -Djava.security.manager
java.security.AccessControlException:
access denied
("java.lang.RuntimePermission"
"accessClassInPackage.sun.misc")
Mystic sun.misc.Unsafe
10. 10 Mystic sun.misc.Unsafe
You’ve been warned!
$ javac Test.java
Test.java:2: warning: Unsafe is internal proprietary API and may
be removed in a future release
import sun.misc.Unsafe;
^
$ javac -XDignore.symbol.file Test.java
12. 12 Inside Unsafe
public native long allocateMemory(long bytes);
public native long reallocateMemory(long address, long bytes);
public native void freeMemory(long address);
public native byte getByte(long address);
public native void putByte(long address, byte x);
public native int getInt(Object o, long offset);
public native void putInt(Object o, long offset, int x);
public native long objectFieldOffset(Field f);
public native long staticFieldOffset(Field f);
public native int arrayBaseOffset(Class<?> arrayClass);
public native void copyMemory(Object srcBase, long srcOffset,
Object destBase, long destOffset,
long bytes);
13. 13
package snow.misc;
public class MyUnsafe {
public native int getInt(long address);
#include <jni.h>
JNIEXPORT jint JNICALL
Java_snow_misc_MyUnsafe(JNIEnv* env, jobject myUnsafe, jlong address) {
return *(jint*)address;
}
Inside Unsafe
14. 14 Inside Unsafe
JNI call
1. Create stack frame
2. Move arguments according to ABI
3. Wrap objects into JNI handles
4. Obtain JNIEnv* and jclass
5. Trace method_entry
6. Lock if synchronized
7. Lazy lookup and linking
8. in_java in_native thread transition
9. Call the native function
10. Check for safepoint
11. Switch state to in_java
12. Unlock if synchronized
13. Notify method_exit
14. Unwrap result, reset JNI handles block
15. Handle exceptions
16. Remove stack frame
15. 15
Implementation
• Most methods are JVM intrinsics
- E.g. getInt is a single MOV instruction
• Executed in Java or VM context
- Beware of mapped I/O
(safepoint pauses)
Inside Unsafe
16. 16
public native int getIntVolatile(Object o, long offset);
public native void putIntVolatile(Object o, long offset, int x);
public native void putOrderedInt(Object o, long offset, int x);
public final native boolean compareAndSwapInt(Object o, long offset,
int expected, int x);
/* @since 1.8 */
public native void loadFence();
public native void storeFence();
public native void fullFence();
public final int getAndAddInt(Object o, long offset, int delta) {
int v;
do {
v = getIntVolatile(o, offset);
} while (!compareAndSwapInt(o, offset, v, v + delta));
return v;
}
Inside Unsafe
18. 18 Inside Unsafe
public native Class<?> defineClass(String name, byte[] b, int off, int len,
ClassLoader loader,
ProtectionDomain protectionDomain);
public native Class<?> defineAnonymousClass(Class<?> hostClass, byte[] data,
Object[] cpPatches);
public native Object allocateInstance(Class<?> cls) throws InstantiationException;
public native void park(boolean isAbsolute, long time);
public native void unpark(Object thread);
public native int getLoadAverage(double[] loadavg, int nelems);
public native void throwException(Throwable ee);
19. 19 Inside Unsafe
Deprecated in JDK 8, removed in 9
@Deprecated
public native void monitorEnter(Object o);
@Deprecated
public native void monitorExit(Object o);
@Deprecated
public native boolean tryMonitorEnter(Object o);
// Phantom HotSpot intrinsics
public native void prefetchRead(Object o, long offset);
public native void prefetchWrite(Object o, long offset);
23. 23 Use cases
Off-heap hash map
0 addr 0 … addr 0
hash time next
key (?)
value
hash time 0
key (?)
value
entry
one.nio.mem.SharedMemoryMap
24. 24 OffheapLongList
public OffheapLongList(int initialCapacity) {
capacity = Math.max(initialCapacity, 4);
base = unsafe.allocateMemory(capacity * 8L);
}
public void add(long value) {
if (size >= capacity) {
capacity *= 2;
base = unsafe.reallocateMemory(base, capacity * 8L);
}
unsafe.putLong(base + size * 8L, value);
size++;
}
public long get(int index) {
if (index < 0 || index >= size) {
throw new IndexOutOfBoundsException();
}
return unsafe.getLong(base + index * 8L);
}
25. 25 Use cases
When to free memory?
class OffheapCleaner<T> extends PhantomReference<T> {
static final ReferenceQueue<T> queue = new ReferenceQueue<T>();
final long address;
OffheapCleaner(T referent, long address) {
super(referent, queue);
this.address = address;
}
}
while (true) {
OffheapCleaner<T> cleaner = (OffheapCleaner<T>) queue.remove();
unsafe.freeMemory(cleaner.address);
}
26. 26
Why not ByteBuffers?
• Size limit (Integer.MAX_VALUE)
• Not thread-safe
• No freeMemory
public void forceClean(ByteBuffer buffer) {
((sun.nio.ch.DirectBuffer) buffer)).cleaner().clean();
}
Use cases
27. 27 sun.nio.ch.FileChannelImpl
try {
// If no exception was thrown from map0, the address is valid
addr = map0(imode, mapPosition, mapSize);
} catch (OutOfMemoryError x) {
// An OutOfMemoryError may indicate that we've exhausted memory
// so force gc and re-attempt map
System.gc();
try {
Thread.sleep(100);
} catch (InterruptedException y) {
Thread.currentThread().interrupt();
}
try {
addr = map0(imode, mapPosition, mapSize);
} catch (OutOfMemoryError y) {
// After a second OOME, fail
throw new IOException("Map failed", y);
}
}
28. 28 Cassandra
// Hoping GC will remove some not used tables.
System.gc();
Thread.sleep(60000);
location = estimateFlushPath();
if (location!=null)
return new File(location, ssTableFileName).getAbsolutePath();
// Hoping GC will remove some not used tables - sometimes 2 GC cycles are needed.
logger_.warn("Still Insufficient disk space to flush "+ssTableFileName);
System.gc();
Thread.sleep(60000);
location = estimateFlushPath();
29. 29 Use cases
Direct buffers ↔ Unsafe
Field addressField = Buffer.class.getDeclaredField("address");
addressField.setAccessible(true);
// ByteBuffer address
long address = addressField.getLong(byteBuffer);
// address ByteBuffer
ByteBuffer result = prototype.duplicate();
addressField.setLong(result, address);
42. 42
Chronology
• JEP 200: The Modular JDK
• Mark Reinhold speaks at Devoxx
• Chris Engelbert started a discussion group
• Apocalyptic post at blog.dripstat.com
• JCrete
• JVM Language Summit
• JEP 260: Encapsulate Most Internal APIs
Removal of sun.misc.Unsafe
2014 / 07
2015 / 06
2015 / 07
2015 / 08
43. 43
Migration plan
• Replacement in JDK 8 hide in JDK 9
- sun.misc.BASE64Encoder etc.
- Available via command-line flag
• No replacement in JDK 8 available outside
- sun.misc.Unsafe, sun.reflect.ReflectionFactory
- sun.misc.Cleaner, sun.misc.SignalHandler
• Replacement in JDK 9 hide in JDK 9,
remove in JDK 10
Removal of sun.misc.Unsafe
45. 45
Project Panama
• JEP 191: Foreign Function Interface
• Expected in Java 10
• JNR
Replacements
public interface LibC {
int setuid(@uid_t long uid);
}
LibC libc = LibraryLoader.create(LibC.class).load("c");
libc.setuid(restrictedUser);
46. 46
JEP 193: VarHandles
• Targeted for Java 9
Replacements
class Queue {
int size;
...
}
VarHandle queueSize = MethodHandles.lookup().
findVarHandle(Queue.class, "size", int.class);
queueSize.addAndGet(10);
53. 53 Unsafe bugs
JVM Bugs
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002b9fd9f44898, pid=19692, tid=1096575296
#
# JRE version: 6.0_24-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64)
# Problematic frame:
# J Test.loop(Lsun/misc/Unsafe;J)V
#
# An error report file with more information is saved as:
# /home/apangin/hs_err_pid19692.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
for (int i = 0; i < BUF_SIZE; i += 8)
unsafe.putLong(addr + i,
Long.reverseBytes(i));
57. 57
Development @ OK
• Blog
- http://habrahabr.ru/company/odnoklassniki
• Open source
- https://github.com/odnoklassniki
• Career
- http://v.ok.ru
Thank you
