Most of the modern cloud web services execute on top of runtime environments like .NET's Common Language Runtime or Java Runtime Environment. On the one hand, runtime environments provide several off-the-shelf benefits like code security and cross-platform execution. On the other hand, runtime’s features such as just-in-time compilation and automatic memory management add a non-deterministic overhead to the overall service time, increasing the tail of the latency distribution. In this context, the Garbage Collector (GC) is among the leading causes of high tail latency. To tackle this problem, we developed the Garbage Collector Control Interceptor (GCI) – a request interceptor algorithm, which is agnostic regarding the cloud service language, internals, and its incoming load. GCI is wholly decentralized and improves the tail latency of cloud services by making sure that service instances shed the incoming load while cleaning up the runtime heap. We evaluated GCI’s effectiveness in a stateful service prototype, varying the number of available instances. Our results showed that using GCI eliminates the impact of the garbage collection on the service latency for small (4 nodes) and large (64 nodes) deployments with no throughput loss.

1. Improving Tail Latency of
Stateful Cloud Services via
GC Control and Load Shedding
Daniel Fireman danielfireman@gmail.com
João Brunet, Raquel Lopes, David Quaresma, Thiago Emmanuel Pereira

2. Runtime Environments (RTEs)
Manage/Control/Support the program execution
● Platform independence
● Abstractions
○ Memory model
○ Execution
○ Concurrency
● Safety
○ Checks, checks and more checks
○ Automatic memory management
● ...

3. Cloud loves RTEs

4. Garbage Collection (GC) impact on latency
CPU Competition
+
Pauses
Overhead
distribution tail:
0.1% of the
requests were
impacted

5. Scaled out view of the latency impact
1 - 0.99 100
= 63%
Number of endpoints
reached to serve one
request
Considered percentile (when
does a peak happen?)

6. Which ends up harming
● User experience
● Predictability
● SLAs

7. You could try
● Tune the GC configuration
● Investigate and fix spots of GC pressure
● Go off-heap
● Switch to manual memory management
● Buy comercial implementations and support

8. If all that is too hard or expensive
Avoid collecting garbage while processing requests

9. Garbage Collector Control
Interceptor (GCI)

10. Overview

11. Components: Proxy
● Intercepts all requests
● Service/RTE agnostic
● Decides when to check heap
● Decides which requests to shed
github.com/gcinterceptor/gci-proxy

12. Components: Request Processor (a.k.a Agent)
● Exec. commands from the proxy
● RTE Specific
● Calls RTE’s APIs
○ Check Heap
○ GC
github.com/gcinterceptor/gci-{java,ruby, nodejs, go}

13. How GCI works: part 1

14. How GCI works: part 2

15. How GCI works: part 3

16. And more!
● Plug and Play
● Adaptive
● Fully decentralized
● Runtime agnostic
○ uses available APIs to trigger garbage collection and check the heap
● Transport agnostic
○ uses the available interception and load shedding mechanisms to avoid
receiving request during collection

17. Evaluation

18. Research Question
Does GCI shorten the tail of the latency distribution of
stateful services without significantly penalizing the service
throughput?
Small (4 nodes) and Large (64 clusters)

19. Stateful services matter

20. Simulator
github.com/gcinterceptor/gci-simulator

21. Results - Large Cluster
Does GCI shorten the tail of the latency distribution of
stateful services without significantly penalizing the service
throughput?

22. Results - Large Cluster
● 99th → 30%
● 99.999th → 47%
● No throughput loss
Yes!

23. Conclusions and Next Steps
● Avoid processing requests while GC’ing is effective to improve tail latency of
stateful cloud services
GCI is an easy-to-use mechanism to achieve that
● Furthermore, it is adaptive, low overhead and fully distributed!
Next we would like to:
● Decrease the proxy overhead
● Experiment a more realistic application (e.g. Hazelcast) and load (e.g. YCSB)
● Investigate stateless and DSP cloud applications

24. Thank you
@daniellfireman@danielfireman
github.com/gcinterceptor