Keeping Infinispan In Shape: Highly-Precise, Scalable Data Eviction

1. galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

2. Keeping Infinispan In Shape: Highly-Precise, Scalable Data Eviction Galder Zamarreño & Vladimir Blagojevic Senior Engineer, Red Hat 7th October 2010, JUDCon - Berlin galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

3. Who is Galder? • R&D engineer (Red Hat Inc): • Infinispan developer • JBoss Cache developer • Contributor and committer: • JBoss AS, Hibernate, JGroups, JBoss Portal,...etc • Blog: zamarreno.com • Twitter: @galderz galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

4. Agenda • Eviction in: • Java Collections Framework • JBoss Cache • Infinispan 4.0 • New in Infinispan 4.1: • LIRS • Batching Updates galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

5. Java Collections Framework • Key building block of any Java app • Introduced in Java 1.2 • Extended with concurrent collections in Java 1.5 • Collection element eviction ?? galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

6. Collections cannot grow forever galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

7. If using a collection as cache • Forces clients to either: • Remove elements proactively • Or run a periodic cleanup process • Which can be a PITA... galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

8. Eviction in JBoss Cache days galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

9. JBoss Cache eviction issues • Under heavy load, eviction queues could fill up - bottleneck • Possibly a side effect of MVCC’s non-blocking reads • Separating data into regions could alleviate the issue • But it’s really a hack! galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

10. Original Infinispan 4.0 eviction • Tried a different approach: • Avoid using queues to hold events • Taking advantage of tree to map change: • Attempt to maintain map ordered as per eviction rules • Could we use ConcurrentSkipListMap ? • No. O(1) desired for all map operations galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

11. Doubly Linked ConcurrentHashMap • Based on H.Sundell and P.Tsigas paper: • Lock-Free Deques and Doubly Linked Lists (2008) • With each cache access or update update links • Eviction just a matter of walking the linked list one way galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

12. Issues under heavy load • Algorithm always uses same node (the tail) to append stuff • With high concurrency, CAS stress lead to loads of retrying • So much retrying, we’re getting infinite loops • Before 4.0.0.Final, reverted to a more conservative algorithm galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

13. 4.0.0.Final eviction galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

14. LRU eviction algorithm issues • Weak access locality • One-time accessed keys not evicted timely • In loops, soon to be accessed keys might get evicted first • With distinct access frequencies, frequently accessed keys can unfortunately get evicted • LRU’s working set limited to cache size galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

15. Enter the room... LIRS • Eviction algorithm that can cope with weak access locality • Based on S.Jiang and X.Zhang’s 2002 paper: • LIRS: An efficient low inter-reference recency set replacement policy to improve buffer cache performance • LIRS based around two concepts: IRR and Recency galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

16. IRR and Recency • Inter-Reference Recency (IRR): • Number of other unique keys accessed between two consecutive accesses to same key • Recency (R) • Number of other unique keys accessed from last reference until now galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

17. How does LIRS work? • If key has a high IRR, it’s next IRR is likely to be high again • Keys with highest IRR are considered for eviction • Once IRR is out of date, we start relying on Recency • LIRS = Low Inter-reference Recency Set galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

18. How is LIRS implemented? • Low IRR (LIR) area • Holds hot keys! • High IRR (HIR) area • Holds recently accessed keys • Keys here might get promoted to LIR area galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

19. Cache Hit - LIR area galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

20. Cache Hit - LIR area galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

21. Hit - HIR area and in LIR Q ... galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

22. Hit - HIR area and in LIR Q galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

23. Hit - HIR area and not in LIR Q galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

24. Cache Miss galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

27. LIRS implementation hurdles • LIRS requires a lot of key shifting around • It can lead to high contention • Unless you can implement it in scalable way, it’s useless • Low contended way to implement a high precision eviction algorithm? Is it possible? galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

28. Batching Eviction Updates • Original idea: X.Ding, S.Jiang and X.Zhang’s 2009 paper: • BP-Wrapper: A System Framework Making Any Replacement Algorithms (Almost) Lock Contention Free • Keeping cache access per thread in a queue • If queue reaches a threshold: • Acquire locks and execute eviction as per algorithm • Batching updates significantly lowers lock contention galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

29. Batching Updates in Infinispan • Decided against recording access per thread • 100s threads could be hitting cache; some short lived • Created BoundedConcurrentHashMap • Based on Doug Lea's ConcurrentHashMap • Records accesses in a lock-free queue in each segment • When threshold passed, acquire lock for segment and evict galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

30. Precision and Performance • Segment level eviction does not affect overall precision • Community member run some performance tests: • After swapping their own LRU cache with BoundedCHM: • Berlin SPARQL Benchmark performance increased 55-60% for both cold and hot caches galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

31. Summary • In JBoss Cache, eviction can become a bottleneck • Infinispan 4.0 uses conservative eviction • Infinispan 4.1 has more precise eviction algorithm (LIRS) • Batching updates, present in 4.1, significantly lowers lock contention • Result = Highly-concurrent, highly-precise implicit eviction galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010

32. Questions? infinispan.org blog.infinispan.org twitter.com/infinispan #infinispan #judcon galder@jboss.org | twitter.com/galderz | zamarreno.com Thursday, October 7, 2010