Why you’re going to fail running java on docker! - Rafael Benavides

1. 1
Why you’re going to FAIL running Java on docker!
bit.ly/javadockerfail
@burrsutter @rafabene

2. @burrsutter
Change History
0.1 - JBCNConf.com 2017 - Barcelona
0.2 - JavaOne 2017 - San Francisco
0.3 - JavaDay UA 2017 - Kiev

10. Big Wins for Developers
● Highly Portable Packaging solution - for microservices, web apps
● Lightweight, Encapsulated OS abstraction - carry your OS with you
● Getting Started (docker run -it centos/wildfly) Instantly
● Dev Environments that more closely match Prod Environments
● Dev Environments that match OTHER Dev Environments (no more...but it
works on my machine)
● No more waiting 3+ weeks for a VM to be provisioned by Ops just so you
can run a series of tests
Linux Container

11. @burrsutter

12. @burrsutter

13. Virtualization vs Containers
Server Machine
Host Operating System
Hypervisor
Guest
OS
Guest
OS
Guest
OS
Bins
Libs
Bins
Libs
Bins
Libs
App 1 App 2 App 3
Server Machine
Host Operating System
Bins
Libs
Bins
Libs
App 1 App 2 App 3
Docker
● Shared bins/libs and Host Operating
System make containers lighter
● You can run multiple versions of app
dependencies (Java 6 and 7)
● “Golden Image” deployment model
● Process isolation

14. Pros vs Cons
Pros:
● Docker containers start very fast, take less memory
● Super easy to try out a new technology (e.g docker pull centos)
● Supports the “immutable image” deployment model
● It caches layers to make builds faster
Cons:
● It is not a VM - it is a process running within the Host OS, do not assume
you can run any flavor of container OS on any host OS
● It is not perfectly portable - you will eventually wish to get out of
boot2docker-vm into a real Linux VM - one that more closely matches your
production environment
● Lineage of docker images at Docker Hub is unknown

15. History of Containers
2000
2010
2005
2015
2000:
JAILS ADDED
TO FREEBSD
2006:
GENERIC PROCESS
CONTAINERS
2008:
KERNEL AND USER
NAMESPACES
2014:
GOOGLE
KUBERNETES
2008:
LINUX CONTAINER
PROJECT (LXC)
2015:
STANDARDS VIA
OCI AND CNCF
2013:
RED HAT
ENTERPRISE LINUX
2013:
DOTCLOUD
BECOMES DOCKER
2007:
GPC RENAMED
CONTROL GROUPS
2003:
SELINUX ADDED TO
LINUX MAINLINE
2015:
RHT CONTAINER
PLATFORM
2001:
LINUX -VSERVER
PROJECT
2013:
DOTCLOUD PYCON
LIGHTNING TALK
2005:
FULL RELEASE OF
SOLARIS ZONES

16. History of Java
Java 1.0.2 - 30 May 1996
J2EE 1.2 - 12 Dec 1999
https://en.wikipedia.org/wiki/Java_EE_version_history#J2EE_1.2_.28December_12.2C_1999.29
https://en.wikipedia.org/wiki/Java_version_history

18. A Challenge
Server Hardware
Operating System
Application Server
.war or .ear
Java Virtual Machine
Custom Configuration
Linux Kernel Version & Distribution
Java 1.6.6_45 or Java 1.7.0_67
Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y
JDBC driver, datasource, JMS queue, users
Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a
JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a
patch for the JVM or app server that broke your code?
Containerize
Your
App

19. Email
MyApp.war has been tested with the following
On my Windows 7 desktop
JDK 1.8.43
Wildfly 9
Configuration:
Datasource: MySQLDS
Tested with: mysql-connector-java-5.1.31-bin.jar
Production Environment
Red Hat Enterprise Linux 6.2
JRE 1.7.3
WebSphere 8.5.5
Oracle 9

20. Dockerfile
Container Guest OS
Java App Server
custom configuration
FROM centos/wildfly
COPY standalone.xml /opt/wildfly/standalone/configuration/
COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/
COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/
COPY myapp/target/your.war /opt/wildfly/standalone/deployments/
your.war
dependencies
Note: There are better ways to handle Java apps, this is for illustration purposes

22. @burrsutter
Behind the Scenes

23. Cgroups and Namespaces
Cgroups
- cpu (cpu shares)
- cpuacct
- cpuset (limit processes to a CPU)
- memory (swap, dirty pages)
- blkio (throttle reads/writes)
- devices
- net_prio (packet class and priority)
- freezer
Namespaces
- pid (processes)
- net (network interfaces, routing)
- ipc (system V ipc)
- mnt (mount points, filesystems)
- uts (hostname)
- user (UIDs)

24. /sys/fs/cgroup/memory
cgroup.clone_children memory.kmem.tcp.limit_in_bytes memory.move_charge_at_immigrate
cgroup.event_control memory.kmem.tcp.max_usage_in_bytes memory.oom_control
cgroup.procs memory.kmem.tcp.usage_in_bytes memory.pressure_level
memory.failcnt memory.kmem.usage_in_bytes memory.soft_limit_in_bytes
memory.force_empty memory.limit_in_bytes memory.stat
memory.kmem.failcnt memory.max_usage_in_bytes memory.swappiness
memory.kmem.limit_in_bytes memory.memsw.failcnt memory.usage_in_bytes
memory.kmem.max_usage_in_bytes memory.memsw.limit_in_bytes memory.use_hierarchy
memory.kmem.slabinfo memory.memsw.max_usage_in_bytes notify_on_release
memory.kmem.tcp.failcnt memory.memsw.usage_in_bytes tasks

25. @burrsutter

26. DevOps Challenges for Multiple Containers
▪ How to scale?
▪ How to avoid port conflicts?
▪ How to manage them on
multiple hosts?
▪ What happens if a host has
trouble?
▪ How to keep them running?
▪ How to update them?
▪ Where are my containers?
Node
Node Node
Node Node
Logger
Node

27. @burrsutter

29. ▪ Greek for “Helmsman,” also the root of the
word “Governor” (from latin: gubernator)
▪ Container orchestrator
▪ Supports multiple cloud and bare-metal
environments
▪ Inspired by Google’s experience with containers
▪ Open source, written in Go
▪ Manage applications, not machines
Meet Kubernetes

30. SCM
(Git/Svn)
Master
API Server
Service Layer
VirtualPhysical Private Public
Persistent
Storage
Node Node
Logger
Node
Node Node Node
Dev
Ops
CI/CD
Automation
Routing Layer
Registry
SDN Overlay Network
Controllers
- Scheduler
- Deployments
- Services
- Builds
- Routes
- DeploymentConfigs
Kubernetes
OpenShift
- Builds
- ImageStreams

31. SCM
(Git/Svn)
Master
API Server
Service Layer
VirtualPhysical Private Public
Persistent
Storage
Node Node
Logger
Node
Node Node Node
Dev
Ops
CI/CD
Automation
Routing Layer
Registry
SDN Overlay Network
Controllers
- Scheduler
- Deployments
- Services
- Builds
- Routes
- DeploymentConfigs
Kubernetes
OpenShift
- Builds
- ImageStreams

32. SCM
(Git/Svn)
Master
API Server
Service Layer
VirtualPhysical Private Public
Persistent
Storage
Node Node
Logger
Node
Node Node Node
Dev
Ops
CI/CD
Automation
Routing Layer
Registry
SDN Overlay Network
Controllers
- Scheduler
- Deployments
- Services
- Builds
- Routes
- DeploymentConfigs
Kubernetes
OpenShift
- Builds
- ImageStreams

33. @burrsutter
Java

34. Why Fail?
8GB
4 cores
64GB
16 cores
Developer Workstation
Production Server

35. @burrsutter
System.out.println("Memory: " +
Runtime.getRuntime().maxMemory()/1024/1024);
System.out.println("CPUs: " +
Runtime.getRuntime().availableProcessors());
Why does Java Fail?
It sees ALL the HOST resources, it is blind to cgroups by default
Note: free also fails
https://fabiokung.com/2014/03/13/memory-inside-linux-containers/

36. @burrsutter

37. @burrsutter
Tomcat
https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/core/ContainerBase.java#L304
https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/util/net/NioEndpoint.java#L138
https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/websocket/AsyncChannelGroupUtil.java#L85
https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/manager/StatusTransformer.java#L224
Wildfly
https://github.com/wildfly/wildfly-common/blob/master/src/main/java/org/wildfly/common/cpu/ProcessorInfo.java#L61
Vert.x
https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/VertxOptions.java#L41
Who uses those APIs?
Tomcat, Wildfly, Vert.x

38. @burrsutter
Memory (max heap defaults to ¼ of available memory)
JIT Optimizations
Mode: Client or Server
Thread Management
Garbage Collector
Do NOT just “java -jar myVertxApp.jar” nor “java -jar mySpringBootApp.jar”
Do try “java -XX:+PrintFlagsFinal -XX:+PrintGCDetails $JAVA_OPTIONS -jar myApp.jar”
JVM Ergonomics
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.html

39. @burrsutter
Memory

40. @burrsutter
Heap (-Xms -Xmx)
------------------------------
Native JRE
Perm (-XX:MaxPermSize), Java8 Meta (-XX:SurvivorRatio, -XX:MaxNewSize, -XX:NewRatio)
JIT Bytecode
JNI
NIO
Threads (defaults to 512K to 1024K depending on platform)
Tip: Either -Xmx at 50% of container constrained memory or
-XX:MaxRam=500m - your heap will be approximately 250mb *
JVM Memory: Heap is about 50%

41. @burrsutter

42. @burrsutter
https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/
https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits
max_memory() {
# High number which is the max limit until which memory is supposed to be
# unbounded.
local max_mem_unbounded="$(cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes)"
local mem_file="/sys/fs/cgroup/memory/memory.limit_in_bytes"
if [ -r "${mem_file}" ]; then
local max_mem="$(cat ${mem_file})"
if [ ${max_mem} -lt ${max_mem_unbounded} ]; then
echo "${max_mem}"
fi
fi
}
Workaround
docker pull fabric8/java-jboss-openjdk8-jdk

43. @burrsutter
Heap Patched: Java 8u131, included in Java 9
https://bugs.openjdk.java.net/browse/JDK-8170888

44. @burrsutter
Try it yourself
Just need Docker for Mac or Windows
docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
vs
docker run -m 100MB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version
UseCGroupMemoryLimitForHeap tells the JVM to look to the cgroup’s value
for memory in /sys/fs/cgroup/memory/memory.limit_in_bytes

45. @burrsutter
Cores

46. @burrsutter
docker run --cpus="1.5" -m 400M openjdk:8u131
*docker run --cpu-quota=150000 --cpu-period=100000 -m 400M
openjdk:8u131
docker run --cpu-shares=1536 -m 400M openjdk:8u131
**docker run --cpuset-cpus=0,2 -m 400M openjdk:8u131
Docker CPU Settings
https://docs.docker.com/engine/admin/resource_constra
ints/#configure-the-default-cfs-scheduler
* Kubernetes uses this model
** JVM properly calculates availableProcessors()

47. @burrsutter
docker run -it --cpus="2" -m 200M java
java -XX:ParallelGCThreads=2
-XX:ConcGCThreads=2
-Djava.util.concurrent.ForkJoinPool.common.parallelism=2
-XX:MaxRam=200m
// Half a core cpu-quota / cpu-period
docker run -it --cpu-quota=50000 --cpu-period=100000 -m 200M openjdk:8u131
java -XX:+UseSerialGC -Xmx=100m
Note: Set Max heap (-Xmx) to about 50% of total container memory
Note: UseSerialGC when core count is below 2
Docker CPU Mapped to Java
Some “rules of thumb”

48. @burrsutter
CPU Set Patched
https://bugs.openjdk.java.net/browse/JDK-6515172

49. @burrsutter
Demo Boom
https://github.com/burrsutter/javadockerfail
Demo Script
https://docs.google.com/document/d/1Td_nnjk12Miq6SpRs3RysDZbpQUjpxXk_p0J526VhcE/edit?usp=sharing

50. @burrsutter
Base Docker Image
FROM fabric8/java-jboss-openjdk8-jdk:1.2.7
Maven Plugin:
- Docker builds leveraging base image
- Deploys to Kubernetes and OpenShift easily
https://fabric8.io/gitbook/mavenPlugin.html
mvn io.fabric8:fabric8-maven-plugin:3.3.5:setup
mvn fabric8:deploy
Hot Tips
https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/

51. @burrsutter
More and More Resources
https://developers.redhat.com/products/cdk/download/
https://github.com/burrsutter/javadockerfail
https://developers.redhat.com/blog/2017/03/14/java-inside-docker/
https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/
https://dzone.com/articles/running-a-jvm-in-a-container-without-getting-kille
https://github.com/redhat-developer-demos/java-container
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-
pods-with-resource-limits-are-run
https://github.com/fabric8io-images/java/tree/master/images/jboss/openjdk8/jdk
https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits
http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015014.html
bit.ly/javadockerfail

52. 52
Why you’re going to FAIL running Java on docker!
bit.ly/javadockerfail
@burrsutter @rafabene

53. @burrsutter
Previously Used Slides

55. Creative Commons/Flikr, SF History Center, SF Public Library, GG Bridge 1935-1936 at
http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/

56. Creative Commons/Flikr, SF History Center, SF Public Library, GG & Bay Bridges 1935 at
http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/

57. Creative Commons Found SF Historical Archive, GG & Bay Bridges, at
http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/

59. Creative Commons Photo ‘For Bankrupt Hanjin, Costs and Containers Pile Up’ at
https://www.workboat.com/news/bluewater/hanjin-costs-containers-pile-up-us-ports/

60. Wikimedia Commons ‘Container Port & Toxic Fire San Francisco’ at
https://commons.wikimedia.org/wiki/File:Containter_port_and_toxic_fire,_San_Francisco.jpg