How we broke Apache Ignite by adding persistence
•Descargar como PPTX, PDF•
0 recomendaciones•94 vistas
Apache Ignite™ is a rapidly changing platform: if you were to look at 3 years ago, you would see a completely different product. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and In-memory computing platform. We will examine technical tasks and decisions that were driving the transformations (as an example - how we added native persistence to Apache Ignite™) and will wrap up the talk with the outstanding problems that are going to be solved for Apache Ignite™.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a How we broke Apache Ignite by adding persistence
Similar a How we broke Apache Ignite by adding persistence (20)
Último
Último (20)
How we broke Apache Ignite by adding persistence
- 1. How we broke Apache Ignite by adding persistence Stephen Darlington 16 December, 2019 2018 © GridGain Systems
- 2. 2019 © GridGain Systems GridGain Company Confidential2 (spoiler: already fixed)
- 3. 2019 © GridGain Systems GridGain Company Confidential What is Ignite? Distributed memory-centric storage Combines the performance and scale of in- memory computing together with the disk durability and strong consistency in one system Co-located Computations Brings the computations to the servers where the data actually resides, eliminating need to move data over the network Distributed Key-Value Read, write and transact with fast key-value APIs Distributed SQL ACID Transactions Machine and Deep Learning Horizontally, fault-tolerant distributed SQL database that treats memory and disk as active storage tiers Supports distributed ACID transactions for key-value as well as SQL operations Set of simple, scalable and efficient tools that allow building predictive machine learning models without costly data transfers (ETL)
- 4. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsStreamingMessaging Transactio ns SQLKey-Value Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store
- 5. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite’s History 5 Data Grid Local Store Transactional Persistence ?
- 6. 2019 © GridGain Systems GridGain Company Confidential Circa 2011 6 Local Store Transactional Persistence ?Data Grid
- 7. 2019 © GridGain Systems GridGain Company Confidential Circa 2014 7 Transactional Persistence ?Data Grid Local Store
- 8. 2019 © GridGain Systems GridGain Company Confidential Circa 2017 • Start time does not depend on the data volume • Can store more data than memory • Crash recovery • Single in-memory & native persistence architecture
- 9. 2019 © GridGain Systems GridGain Company Confidential Ignite 2.0: What we wanted “We will just save everything to disk”
- 10. 2019 © GridGain Systems GridGain Company Confidential Circa 2017 10 ?Data Grid Local Store Transactional Persistence
- 11. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory
- 12. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory • ARIES Architecture • Page-based • Write-ahead log (when persistence is enabled) • Everything is off heap
- 13. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory • PK Index: how to replace a HashMap • Concurrent B+ Tree: a well-known data structure • Separate PK Index per each partition • Compare key hash first • Bonus: guaranteed iteration order in a hash map
- 14. 2019 © GridGain Systems GridGain Company Confidential Baseline Topology 16 • [16:21:01] Ignite node started OK (id=326bab44) • [16:21:01] >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate. • [16:21:01] Topology snapshot [ver=1, locNode=326bab44, servers=1, clients=0, state=INACTIVE, CPUs=8, offheap=3.2GB, heap=3.6GB] • [16:21:01] ^-- Baseline [id=11, size=3, online=1, offline=2] • [16:21:01] ^-- 2 nodes left for auto-activation [6213b7af-23bb-4c8d-a045-157d7f2d7718, db969788- fc01-41f4-a91c-c03f2d201f76] • [16:21:19] Joining node doesn't have encryption data [node=89b6ef6c-1055-4678-bcfa-00fb222208ce] • [16:21:19] Topology snapshot [ver=2, locNode=326bab44, servers=2, clients=0, state=INACTIVE, CPUs=8, offheap=6.4GB, heap=7.1GB] • [16:21:19] ^-- Baseline [id=11, size=3, online=2, offline=1] • [16:21:19] ^-- 1 nodes left for auto-activation [6213b7af-23bb-4c8d-a045-157d7f2d7718] • [16:21:37] Joining node doesn't have encryption data [node=dd55ff24-da61-42cd-bbaf-c7940fab07d3] • [16:21:37] Topology snapshot [ver=3, locNode=326bab44, servers=3, clients=0, state=INACTIVE, CPUs=8, offheap=9.6GB, heap=11.0GB] • [16:21:37] ^-- Baseline [id=11, size=3, online=3, offline=0] • [16:21:37] ^-- All baseline nodes are online, will start auto-activation
- 15. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Disks are slow (even NVMe) • At peak load naïve implementation steps on it’s tail easily • Sudden performance drops to 0
- 16. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • So, we need to… make Ignite slower • Throttle input load depending on • How fast we produce “dirty” pages • How fast we write to disk • How free the Copy-On-Write buffer is
- 17. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Page cache: what can go wrong • We already have one-page cache in Ignite (durable memory) • OS-level page cache • Effectively doubles the memory consumption
- 18. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Page cache: solution is Direct IO • Available in Java 10, but we build on Java 8 • Need to have native/platform specific calls or Java- dependent module
- 19. 2019 © GridGain Systems GridGain Company Confidential The future? 21 Data Grid Local Store Transactional Persistence ?
- 20. 2019 © GridGain Systems GridGain Company Confidential22 Questions?
- 21. 2019 © GridGain Systems GridGain Company Confidential More information 23 • Main landing page: https://ignite.apache.org • Documentation: https://apacheignite.readme.io/docs • Please complete our survey on how Apache Ignite should evolve: https://docs.google.com/forms/d/e/1FAIpQLSdUveEVXer3lpkyiqfFw4175T vZzGHUOS4snPfnkO0NDku0eQ/viewform • Realtime data loading: https://www.imcsummit.org/2019/us/session/best- practices-loading-real-time-data-distributed-systems-change-data-capture
- 22. 2019 © GridGain Systems GridGain Company Confidential24 Stephen Darlington Senior Consultant, GridGain Systems @sdarlington
Notas del editor
- First, I’m not Alexey Second… we didn’t break it
- Alternate title: How and why Apache Ignite™ is Changing from an In-Memory Data Grid into an In-Memory Database. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and an in-memory computing platform. We will examine technical tasks and decisions that were driving the transformations. Step back: who knows about Ignite?
- Traditional databases don’t scale. Buy bigger and bigger boxes until you run out of money. Traditional compute grids have to copy data across the network, which at modern scale is just impractical. Ignite scales horizontally and sends compute to the data rather than the other way around. In memory for speed. Disk persistence for volume.
- Memory data grid Optionally backed by the “Cache Store” SQL works as fork-join – SQL executed on every node, results joined together Resuts correct results in limited use cases SQL only in-memory Restart means loading all data Centralised store – scalablity issues
- LocalStore designed to allow faster restarts Local Store is an append-only structure with periodical compactions Still need to load data from local store to memory (on datasets of 100s of GBs warmup could take 3+ hours) Keys are still residing in memory, so the data set size is limited SQL functionality is improved (two-step queries like plain ORDER BY, GROUP BY, some cases of distributed joins are implemented, but still there are restrictions, not all queries return correct results, a user must check collocation rules) So better but still missing stuff. What are our requirements for The Best system?
- Algorithms for Recovery and Isolation Exploiting Semantics Ignite 1.x had options for on- and off-heap storage. 2.x only off-heap Unlike LocalStore (and competition), it uses a WAL… like legacy databases. Crash recovery
- ABA problem -- thread synchronization (https://en.wikipedia.org/wiki/ABA_problem)
- We’ve mostly talked about the Good Stuff.. But there are complications What’s happening here? Baseline topology
- … and the discussion about Java 10 brings us to … the futur
- Improve SQL What’s wrong with H2? H2 is not a distributed database with a very simple planner. Ignite uses H2 internal APIs and hacks to integrate with H2. No query execution graph. A query is effectively executed based on AST with minimum transformations H2 planner does not know about distributed SQL, it is almost impossible to execute complex SQL queries (subqueries, complex aggregations, non-colocated tables) effectively No memory control Currently investigating using Apache Calcite as a query optimizer + custom execution engine. See Apache Developers list (+IEP) Improve Split-Brain handling out-of-the-box Require users to implement TopologyValidator Should work without users writing a single line of code Modularisation don’t download everything Dependencies for stuff like Spark / Kafka with different release schedules Even “internal” stuff like ML