Rakuten Ichiba is the biggest online shopping mall in Japan and use GlassFish as core APIs of Rakuten Ichiba. These slide decks show the exprience how to improve the performance of GlassFish at the peak time of Rakuten Ichiba's heavily-loaded sale.

1. GlassFish Community Feedback
GlassFish usage in Rakuten, Inc.

2. About me
 Name: Makito Hashiyama(@capyogu)
 Age: 29
 Role: Team manager / In charge of APIs for Rakuten Ichiba
 Like: GlassFish, Tomcat, KVS(memcached, Coherence), GAE,
Android…
 E-mail: makito.hashiyama@mail.rakuten.com,
hashiyaman@gmail.com

3. What is Rakuten?
 E-commerce and Internet company based in Tokyo, Japan
 B2B2C e-commerce platform
Head Office
E-Commerce
eBook
Travel
Other services & businesses
Rakuten Institute of Technology
Development center
Head Office / Regional Headquarters

4. How We Are Using GlassFish
 Usage
 One of core API service for Rakuten Ichiba
 Require high availability
External
APIs
 Environment
 Use on production environment
 GlassFish version 3.1.2.2
SOAP/
REST
Client side
Our API
GlassFish
Clusters
Oracle
Coherence

5. Benefits gleaned from Glassfish
 Reference implementation of Java EE
 Only GlassFish supported JAX-WS as standard in 2007
 It was an advantage to evaluate new features earlier
 Easy to manage with low cost
 We need to manage huge cluster without stopping
 Cost saving(Weblogic -> GlassFish)

6. What Worked with GlassFish
 Community support(create patches)
 https://java.net/jira/browse/GLASSFISH-5200
(If use jvmRoute JSESSIONID cookie is not secure even in
HTTPS)
 https://java.net/jira/browse/GRIZZLY-1333
(NetworkAddressValidator will fail when passed property
substitution values)
We have contributed patched to GlassFish community.

7. Improvement to handle a huge traffic
 Rakuten Super Sale
 Biggest online sales in Japan
 A lot of doorbuster deals(It causes a huge amount of traffic)
 Performance bottleneck
 External APIs called by our API were slow down
 We needed to improve the system at the peek time
Client side
delay
Slow down
Our API
External
APIs

8. Improvement to handle a huge traffic
Worker Thread
Worker Thread
Request
Client
side
Task
Queue
Worker Thread
Worker Thread
Worker Thread
GlassFish
CPU load
was high
delay
delay
delay
delay
delay
External
APIs

9. Improvement to handle a huge traffic
Worker Thread
Request
Client
side
Task
Queue
Worker Thread
External
APIs
Worker Thread
GlassFish
(1)According to vmstat, ‘run queue’ was very high
(2)Decrease worker threads to keep ‘run queue’ low
(3)As a result, latency increased but throughput was improved

10. Improvement to handle a huge traffic
 As a result…
 Our API could process over 12,000 transactions / minute
 The result showed the high reliability and availability of
GlassFish

11. Resolve issues & challenge to upper goal
 Some issues
 Instance down due to the full of task queue
 Unknown exception on server.log
org.glassfish.flashlight.impl.client.ReflectiveClientInvoker
java.lang.reflect.InvocationTargetException
CAUSE: java.lang.NullPointerException
id=101
target=org.glassfish.web.admin.monitor.HttpServiceStatsProvider@193e1fc
method=public void org.glassfish.web.admin.monitor.HttpServiceStatsProvider.
connectionAcceptedEvent(java.lang.String,int,java.lang.String)
paramNames=[listenerName, connection, address]
probeIndices=[0, 1, 2]
useProbeArgs=true
hasComputedParams=false
 Challenge to upper goal