Presented at JAX London 2013 Per-tenant resource management can help ensure that collocated tenants peacefully share computational resources based on individual quotas. This session begins with a comparison of deployment models (shared: hardware, OS, middleware, everything) to motivate the multitenant approach. The main topic is an exploration of experimental data isolation and resource management primitives in IBM’s JDK that combine to help make multitenant applications smaller and more predictable.

1. Steve Poole IBM
JVM Support for Multitenant Applications

2. Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR
INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A
CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON
HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S
CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM,
WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES
ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR
ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE
EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED
COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS

3. Steve Poole
!
Works at IBM’s Hursley Laboratory in the UK
Involved in IBM Java VM development since before Java was 1
Currently leading IBM’s OpenJDK technical engagement

4. What this talk is about
!
!
!
!
JVM Support for Multi-Tenant Applications
This experimental technology is being developed to help address
important pressures on Java
!
Google ‘IBM Java 8 beta’ for more information
!
■
By the end of this session, you should be able to:
!
– Understand what multitenancy is and what it’s good for
!
– Describe the challenges of multitenant Java deployments
!
– Understand ideas for new JDK features to convert existing
applications into multitenant deployments
!

5. Agenda
1. What Multi-tenancy is all about aka how to simplify to save time
and money
2. The challenges of building a multi-tenant application in Java
3. How to keep your application running in a multi-tenant world dealing with bad behaviour
4. Risk vs. Reward: How dense can we go?
5. Wrap-up: Summary, and next steps

6. Part 1
What is Multitenancy?
!
(simplify to save time and money)

7. Who owns one of these?

8. Who owns one of these?

9. Who owns one of these?

10. Who owns one of these?
At a basic level
Multitenancy is a drive to
reduce unnecessary
complexity and duplication
this is
sophisticated
but expensive
this is simpler
and cheaper

11. Who owns one of these?
At a basic level
Multitenancy is a drive to
reduce unnecessary
complexity and duplication
this is
sophisticated
but expensive
Multitenancy
helps with
finding the right
balance
this is simpler
and cheaper

12. Its not just a hardware story
Simplifying the software stack by removing all extraneous
pieces makes better use of hardware and the people who
run it.
!
Simple == Cheaper == Predictable == Robust

13. Simple
Dont Repeat Yourself
“Every piece of knowledge must have a single, unambiguous,
authoritative representation within a system”
Pragmatic Programmer (Hunt & Thomas)
(or: copy-and-paste encourages problems)
We want to avoid the clone army

14. Are you already using Multitenancy
•Are your choices only
•fully dedicated machine or shared hardware?
•Multitenancy isn’t just a checkbox.
•There are various levels of ‘tenancy’ defined today.

15. SaaS Tenancy Spectrum

16. SaaS Tenancy Spectrum
Your basic laptop, desktop or server
machine.
yours, all yours..

17. SaaS Tenancy Spectrum
A simple shared machine LPAR’d
or VM’d into multiple complete
stacks.

18. SaaS Tenancy Spectrum
Reduced overhead by
running multiple app
servers on one O/S
Keep everything else
separate though.
“just in case”

19. SaaS Tenancy Spectrum
Multiple
applications
running on one
app server sharing db
server and
middleware!

20. SaaS Tenancy Spectrum
sharing
one
app!

21. SaaS Tenancy Spectrum
Sharing everything including database
tables.

22. SaaS Tenancy Spectrum
A Multitenant JVM covers theses
levels of sharing

23. SaaS Tenancy Spectrum
Merging these bubbles saves money, reduces
complexity and can be a real business differentiator.
But it requires engineering effort

24. SaaS Tenancy Spectrum
Remember, from the end user point of view,
they see it like this

25. Efficencies
■
Customer viewpoint
!
– Cost: provider offers the service more cheaply
– Time to Value: up and running fast, typically
upgraded often & quickly
– Quality of Service: focus on SLA needed not your
ability to run infrastructure
– Bypass IT Backlog: streamlined deployment
(handled by provider)
!
■
Provider viewpoint
– Cost: Minimal moving parts / duplication
– Agility: Upgrades, backups, on-boarding
Fewer Parts == Better Density == £££ for the service provider

26. Part 2
Getting there..
!

27. Climbing MT tenant
Increasing density of the software stack
is impacting us all.
!
Cost of hosting is a business
differentiator already
!
Its only going to increase in importance
!
If you’re in the cloud you already know
!
!
Moving from ‘free’ to ‘not free’ hosting is an intensive exercise.
Your software choices are starting to be decided by CPU and memory usage.
With your own hardware you could let it run slow. Can’t do that in the cloud!
!
So how do move forward with Java and Multitenancy?

28. Challenge #1 Isolation
The fear factor is loosing isolation
■ unexpected side effects from other parts
■ This is a reasonable concern
■ Same number of eggs (apps), fewer baskets
■ You want really good baskets arranged carefully
http://circa71.wordpress.com/2010/07/
■ But it’s not a new problem
■ We can reduce the change of failure (more later)
■ We can reduce the impact with good
choreography
■ You need to think about choreography today
■ Today we use clusters to provide fail-over and
arrange clusters to avoid same machine/rack/site
failures
http://bit.ly/e7G1jb
■ Apply the same ideas to Multitenancy
■

29. Challenge #2 Cost of Entry
merge
merge
J Hypervisor sharing only
simplest (hence most
popular) Still have
neighbours but they
are far away
J Port Collisions
J Data Isolation between apps
J File System Collisions
J Control over resource hogs
J Security Challenges
J Easy == No app changes
J JVM can help!!
Saves footprint (GB)
But now we have collisions
Ops guys can help with O/S
JVMs can use -Xshareclasses
More footprint savings
But now we have real isolation concerns
(closer neighbours)
We need control over resource hogs

30. Challenge #2.5 Building isolation
Java Heap consumes 100’s of MB of memory
–Heap objects cannot be shared between JVMs
–GC has a helper thread-per-core by default
Just-in-Time Compiler consumes 10’s of MB of memory
–Generated code is private and big
–Generated code is expensive to produce
• Steals time from application
• Multiple compilation threads by default
No choreography between JVM instances
–Compilation or GC activity can happen at identical (and bad) times

31. Challenge #2.5 Building isolation
We need to fix the following
!
L Data Isolation between applications
!
L Control over resource hogs
Without forcing people to change their applications!

32. Data Isolation Challenges
■
■
Applications embed deployment information like url patterns in code
Wait! What happens if we try to deploy two copies of this
servlet to a single server?

33. ■
Static variables are bad (for sharing)
■
Most libraries are full of static variables
Wait! What happens if
each tenant needs a
different default locale?

34. Wait! What happens if
each tenant needs a
different default locale?
•Usual solutions:
– Wrap the whole thing in a ClassLoader (i.e. rewrite your code)
– Get ride of the static variable (i.e. rewrite your code)
– Use BCI to rewrite the code automatically (erodes robustness)
!
!
We need a silver bullet!

35. Multitenant JDK
■
Concept: Add a single argument (–Xmt for multi-tenant) to
your Java command-line to opt into sharing a runtime with
others.
■
Result: Your application behaves exactly as it if had a
dedicated JVM, but in reality it runs side-by-side with other
applications.
■
Benefits: Smaller, faster, and eventually smarter
–Less duplication: (1 GC, 1 JIT), Heap object sharing
–Fast Startup: JVM is already running and warm when starting apps
■
Required: No code Changes!

36. Part 3
Demo (of a demo)

37. Launch your application
■
Opt-in to multitenancy by adding –Xmt

38. Register with javad daemon
■
JVM will locate/start daemon automatically
locate

39. Create a new tenant
■
New tenant created inside the javad daemon
locate
Tenant1

40. Create a 2nd tenant
■
New tenant created inside the javad daemon
locate
Tenant1

41. Create a 2nd tenant
■
New tenant created inside the javad daemon
locate
Tenant1
Tenant2
One copy of common code
lives in the javad process.
Most runtime structures
are shared.

42. Providing data isolation
■
■
What if … the JVM knew about tenants and provided each one with a different view of
static variables?
Meet the @TenantScope annotation.
Tenant1
…
LocaleSettings.setDefaultLocale(
LocaleSettings.UK );
…
Tenant2
…
LocaleSettings.setDefaultLocale(
LocaleSettings.US );
…
■
@TenantScope Semantics: Static variable values are stored per-tenant
■
Each tenant has their own LocaleSettings.defaultLocale
■
Now many tenants can share a single LocaleSettings class

43. Did I say ‘no code changes?’
■
■
@TenantScope markup gets added automatically as classes are loaded
Tenants see dedicated middleware – but behind the curtains classes (and
JIT’ed code) are actually shared
Application
Changes
merge

44. If it’s invisible - why have @TenantScope?
Allows middleware to opt out
■ Opportunities for even more density.
■
!
■
Basic operations on Tenants available to the middleware
–Data Isolation
–Resource Management (more in this in a minute)
!
■
Ability for the middleware to differentiate between Tenants
–Which one is causing the problem?
!
■
Querying the state of Tenants
–How much free memory do you have?

45. Part 4
Hey, aren’t we done yet?

46. Dealing with bad behaviour
http://bit.ly/ficwkl
images from http://www.rra.memberlodge.org/Neighbourhood-Watch-Reporting

47. Shared environments need resource ctl
The closer your neighbours the better your controls
must be
■ Multitenant JDK provides controls on
■
–CPU time
–Heap size
–Thread count
–File IO: read b/w, write b/w
–Socket IO: read b/w, write b/w
!

48. Resource Control Ergonomics
■
Simple command-line switches for new resources
– -Xlimit:cpu=10-30 // 10% minimum CPU, 30% max
// 30% max CPU
– -Xlimit:cpu=30
–-Xlimit:netIO=20M // Max bandwidth of 20 Mbps
■
Existing options get mapped for free
– -Xms8m –Xmx64m
■
// Initial 8M heap, 64M max
Plus some JMX beans to see how much of each resource you are
using
– i.e. understand how your code uses resources by wrapping in a tenant

49. Building on JSR 284
JSR-284 Resource Consumption Mgmt API
■
Throttling at Java layer for portability
■
Or, leveraging OS WLM directly for efficiency (Linux & AIX)
– Note: many WLMs tend to like processes, not groups of threads
Tenant
Tenant
Tenant
Tenant
Tenant
JVM Resource Management
JSR 284 API
JVM
Memory
CPU
GC
(Heap Mgmt)
Thread
File I/O
Socket I/O
CPU
File I/O
Socket I/O
Resource Throttle Layer
OS Level Resources Management
OS resources
Resource native API
OS
Thread
Handler
OS Workload Manager (WLM)
Socket
Hardware resources
CPU: XXX GHZ
Memory: XXX GB
DISK: XXXKB/S
Network: XXXKB/S

50. CPU Throttling
Round
1239s
1452s
1390s
1094s
1122s
1139s
1123s
6
1244s
1134s
Average
1243s
1212s
cpu throttling in jvm controller
90.00
60.00
67.50
cpu%
80.00
cpu%
1167s
5
cpu throttling in os controller
1267s
4
Accuracy
1362s
3
• Duration comparison: Linux AMD64, run a CPU-intensive app
with 10 threads with 100% CPU quota, each thread doing the
same Fibonacci calculation, benchmark the duration
• Accuracy comparison: Linux AMD64, run two CPU-intensive
apps each doing the same Fibonacci calculation, but with
different CPU quota: 60% vs 30%, benchmark the accuracy
JVM as controller
2
Benchmark setting
OS as controller
1
Duration
40.00
20.00
45.00
60% throttling
30% throttling
22.50
0.00
0.00
00:0000:3001:0001:3002:0002:3003:0003:3004:0004:3005:0005:3006:0006:3007:0007:3008:0008:3009:0009:30
00:0100:3101:0101:3102:0102:3103:0103:3104:0104:3105:0105:3106:0106:3107:0107:3108:0108:3109:0109:31
00:0200:3201:0201:3202:0202:3203:0203:3204:0204:3205:0205:3206:0206:3207:0207:3208:0208:3209:0209:32
00:0300:3301:0301:3302:0302:3303:0303:3304:0304:3305:0305:3306:0306:3307:0307:3308:0308:3309:0309:33
00:0400:3401:0401:3402:0402:3403:0403:3404:0404:3405:0405:3406:0406:3407:0407:3408:0408:3409:0409:34
00:0500:3501:0501:3502:0502:3503:0503:3504:0504:3505:0505:3506:0506:3507:0507:3508:0508:3509:0509:35
00:0600:3601:0601:3602:0602:3603:0603:3604:0604:3605:0605:3606:0606:3607:0607:3608:0608:3609:0609:36
00:0700:3701:0701:3702:0702:3703:0703:3704:0704:3705:0705:3706:0706:3707:0707:3708:0708:3709:0709:37
00:0800:3801:0801:3802:0802:3803:0803:3804:0804:3805:0805:3806:0806:3807:0807:3808:0808:3809:0809:38
00:0900:3901:0901:3902:0902:3903:0903:3904:0904:3905:0905:3906:0906:3907:0907:3908:0908:3909:0909:39
00:1000:4001:1001:4002:1002:4003:1003:4004:1004:4005:1005:4006:1006:4007:1007:4008:1008:4009:1009:40
00:1100:4101:1101:4102:1102:4103:1103:4104:1104:4105:1105:4106:1106:4107:1107:4108:1108:4109:1109:41
00:1200:4201:1201:4202:1202:4203:1203:4204:1204:4205:1205:4206:1206:4207:1207:4208:1208:4209:1209:42
00:1300:4301:1301:4302:1302:4303:1303:4304:1304:4305:1305:4306:1306:4307:1307:4308:1308:4309:1309:43
00:1400:4401:1401:4402:1402:4403:1403:4404:1404:4405:1405:4406:1406:4407:1407:4408:1408:4409:1409:44
00:1500:4501:1501:4502:1502:4503:1503:4504:1504:4505:1505:4506:1506:4507:1507:4508:1508:4509:1509:45
00:1600:4601:1601:4602:1602:4603:1603:4604:1604:4605:1605:4606:1606:4607:1607:4608:1608:4609:1609:46
00:1700:4701:1701:4702:1702:4703:1703:4704:1704:4705:1705:4706:1706:4707:1707:4708:1708:4709:1709:47
00:1800:4801:1801:4802:1802:4803:1803:4804:1804:4805:1805:4806:1806:4807:1807:4808:1808:4809:1809:48
00:1900:4901:1901:4902:1902:4903:1903:4904:1904:4905:1905:4906:1906:4907:1907:4908:1908:4909:1909:49
00:2000:5001:2001:5002:2002:5003:2003:5004:2004:5005:2005:5006:2006:5007:2007:5008:2008:5009:2009:50
00:2100:5101:2101:5102:2102:5103:2103:5104:2104:5105:2105:5106:2106:5107:2107:5108:2108:5109:2109:51
00:2200:5201:2201:5202:2202:5203:2203:5204:2204:5205:2205:5206:2206:5207:2207:5208:2208:5209:2209:52
00:2300:5301:2301:5302:2302:5303:2303:5304:2304:5305:2305:5306:2306:5307:2307:5308:2308:5309:2309:53
00:2400:5401:2401:5402:2402:5403:2403:5404:2404:5405:2405:5406:2406:5407:2407:5408:2408:5409:2409:54
00:2500:5501:2501:5502:2502:5503:2503:5504:2504:5505:2505:5506:2506:5507:2507:5508:2508:5509:2509:55
00:2600:5601:2601:5602:2602:5603:2603:5604:2604:5605:2605:5606:2606:5607:2607:5608:2608:5609:2609:56
00:2700:5701:2701:5702:2702:5703:2703:5704:2704:5705:2705:5706:2706:5707:2707:5708:2708:5709:2709:57
00:2800:5801:2801:5802:2802:5803:2803:5804:2804:5805:2805:5806:2806:5807:2807:5808:2808:5809:2809:58
00:2900:5901:2901:5902:2902:5903:2903:5904:2904:5905:2905:5906:2906:5907:2907:5908:2908:5909:29
00:0000:4201:2402:0602:4803:3004:1204:5405:3606:1807:0007:4208:2409:0609:48
00:0100:4301:2502:0702:4903:3104:1304:5505:3706:1907:0107:4308:2509:0709:49
00:0200:4401:2602:0802:5003:3204:1404:5605:3806:2007:0207:4408:2609:0809:50
00:0300:4501:2702:0902:5103:3304:1504:5705:3906:2107:0307:4508:2709:0909:51
00:0400:4601:2802:1002:5203:3404:1604:5805:4006:2207:0407:4608:2809:1009:52
00:0500:4701:2902:1102:5303:3504:1704:5905:4106:2307:0507:4708:2909:1109:53
00:0600:4801:3002:1202:5403:3604:1805:0005:4206:2407:0607:4808:3009:1209:54
00:0700:4901:3102:1302:5503:3704:1905:0105:4306:2507:0707:4908:3109:1309:55
00:0800:5001:3202:1402:5603:3804:2005:0205:4406:2607:0807:5008:3209:1409:56
00:0900:5101:3302:1502:5703:3904:2105:0305:4506:2707:0907:5108:3309:1509:57
00:1000:5201:3402:1602:5803:4004:2205:0405:4606:2807:1007:5208:3409:1609:58
00:1100:5301:3502:1702:5903:4104:2305:0505:4706:2907:1107:5308:3509:17
00:1200:5401:3602:1803:0003:4204:2405:0605:4806:3007:1207:5408:3609:18
00:1300:5501:3702:1903:0103:4304:2505:0705:4906:3107:1307:5508:3709:19
00:1400:5601:3802:2003:0203:4404:2605:0805:5006:3207:1407:5608:3809:20
00:1500:5701:3902:2103:0303:4504:2705:0905:5106:3307:1507:5708:3909:21
00:1600:5801:4002:2203:0403:4604:2805:1005:5206:3407:1607:5808:4009:22
00:1700:5901:4102:2303:0503:4704:2905:1105:5306:3507:1707:5908:4109:23
00:1801:0001:4202:2403:0603:4804:3005:1205:5406:3607:1808:0008:4209:24
00:1901:0101:4302:2503:0703:4904:3105:1305:5506:3707:1908:0108:4309:25
00:2001:0201:4402:2603:0803:5004:3205:1405:5606:3807:2008:0208:4409:26
00:2101:0301:4502:2703:0903:5104:3305:1505:5706:3907:2108:0308:4509:27
00:2201:0401:4602:2803:1003:5204:3405:1605:5806:4007:2208:0408:4609:28
00:2301:0501:4702:2903:1103:5304:3505:1705:5906:4107:2308:0508:4709:29
00:2401:0601:4802:3003:1203:5404:3605:1806:0006:4207:2408:0608:4809:30
00:2501:0701:4902:3103:1303:5504:3705:1906:0106:4307:2508:0708:4909:31
00:2601:0801:5002:3203:1403:5604:3805:2006:0206:4407:2608:0808:5009:32
00:2701:0901:5102:3303:1503:5704:3905:2106:0306:4507:2708:0908:5109:33
00:2801:1001:5202:3403:1603:5804:4005:2206:0406:4607:2808:1008:5209:34
00:2901:1101:5302:3503:1703:5904:4105:2306:0506:4707:2908:1108:5309:35
00:3001:1201:5402:3603:1804:0004:4205:2406:0606:4807:3008:1208:5409:36
00:3101:1301:5502:3703:1904:0104:4305:2506:0706:4907:3108:1308:5509:37
00:3201:1401:5602:3803:2004:0204:4405:2606:0806:5007:3208:1408:5609:38
00:3301:1501:5702:3903:2104:0304:4505:2706:0906:5107:3308:1508:5709:39
00:3401:1601:5802:4003:2204:0404:4605:2806:1006:5207:3408:1608:5809:40
00:3501:1701:5902:4103:2304:0504:4705:2906:1106:5307:3508:1708:5909:41
00:3601:1802:0002:4203:2404:0604:4805:3006:1206:5407:3608:1809:0009:42
00:3701:1902:0102:4303:2504:0704:4905:3106:1306:5507:3708:1909:0109:43
00:3801:2002:0202:4403:2604:0804:5005:3206:1406:5607:3808:2009:0209:44
00:3901:2102:0302:4503:2704:0904:5105:3306:1506:5707:3908:2109:0309:45
00:4001:2202:0402:4603:2804:1004:5205:3406:1606:5807:4008:2209:0409:46
00:4101:2302:0502:4703:2904:1104:5305:3506:1706:5907:4108:2309:0509:47
time
time
os throttling
jvm throttling
Result: JVM control achieves comparable performance, but less accuracy.

51. Part 5
Performance so far

52. Current Performance Data
■
Environment: Measure standard benchmarks in a 1 GB + 1
core VirtualBox guest
– Advantage: Easy to control, highly reproducible
■
Methodology: Add applications until the system swaps, then
it’s ‘full’
– More applications is better
– Per tenant cost is amount of RAM / # tenants

53. Standard JVM
20
15
10
5
0
instances
Standard JVM
Hand tuned Standard JVM
MT support

54. Best hand-tuned JVM config
60
45
30
15
0
instances
Standard JVM
Hand tuned Standard JVM
MT support

55. Simple out of the box -Xmt
300
4MB
tenant
225
150
75
0
instances
Standard JVM
Hand tuned Standard JVM
MT support

56. Part 6
Wrap up

57. This is still ‘experimental’
■
Focus to date has been ‘zero application changes’
– We can do even better with tenant-aware middleware
■
API’s used to provide isolation & throttling are available to stack
products
– JSR-284 (Resource Management)
– JSR-121 (Isolates)
– @TenantScope fields
■
Java language and frameworks (EclipseLink) are evolving to have
first-class multitenant support
■
Stay tuned for progress: watch the IBM Java 8 beta program

58. This is still ‘experimental’
Simplifying the software stack by removing all extraneous pieces
makes better use of hardware (and people who run it).
!
Multitenancy can make us more efficient:
–Trades isolation for footprint and agility
–JVM support makes multitenancy safer and easier
–Measuring resource usage and load patterns is critical
–Multitenant JDK primitives give us room for future growth

59. Conclusion
Now that you’ve completed this session, you are able to:
– Understand what multitenancy is and what it’s good for
• Per-tenant costs measured in single-digit MB are possible
– Describe challenges of multitenant Java deployments
• Hard for VM guys, should be easy for you
• Choreography of load / deployment is up to you
– Understand new JDK features to convert existing applications into
multitenant deployments
• Are we on the right track? Could you use this in your business?
Thank you - any questions?