Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Apache Ignite - Distributed SQL Database Capabilities
Similar a Apache Ignite - Distributed SQL Database Capabilities (20)
Último
Último (20)
Apache Ignite - Distributed SQL Database Capabilities
- 1. © 2017 GridGain Systems, Inc. In-Memory Performance Durability of Disk
- 2. © 2017 GridGain Systems, Inc. Harnessing the 21st Century with a Distributed Memory-Centric SQL Database Denis Magda Ignite PMC Chair GridGain Director of Product Management
- 3. © 2017 GridGain Systems, Inc. • SQL Capabilities • Connectivity • Data Definition Language • Data Manipulation Language • Collocated Processing • Memory Architecture • Q&A Agenda
- 4. © 2017 GridGain Systems, Inc. SQL Capabilities
- 5. © 2017 GridGain Systems, Inc. Distributed SQL JDBC ODBC SQL API Java .NET C++ BI SELECT, UPDATE, INSERT, MERGE, DELETE, CREATE and ALTER DDL, DML Support Cross-platform Compatibility Indexes in RAM or Disk Dynamic Scaling Server Node Server NodeServer Node Apache Ignite Cluster DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Tools
- 6. © 2017 GridGain Systems, Inc. Connectivity • JDBC • ODBC • REST • Java, .NET and C++ APIs // Register JDBC driver. Class.forName("org.apache.ignite.IgniteJdbcThinDriver"); // Open the JDBC connection. Connection conn = DriverManager.getConnection("jdbc:ignite:thin://192.168.0.50"); ./sqlline.sh --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1/
- 7. © 2017 GridGain Systems, Inc. Data Definition Language • CREATE/DROP TABLE • CREATE/DROP INDEX • ALTER TABLE • Changes Durability • Ignite Native Persistence CREATE TABLE `city` ( `ID` INT(11), `Name` CHAR(35), `CountryCode` CHAR(3), `District` CHAR(20), `Population` INT(11), PRIMARY KEY (`ID`, `CountryCode`) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode";
- 8. © 2017 GridGain Systems, Inc. Data Manipulation Language • ANSI-99 specification • Fault-tolerant and consistent • INSERT, UPDATE, DELETE • SELECT • JOINs • Subqueries SELECT country.name, city.name, MAX(city.population) as max_pop FROM country JOIN city ON city.countrycode = country.code WHERE country.code IN ('USA','RUS','CHN') GROUP BY country.name, city.name ORDER BY max_pop DESC LIMIT 3;
- 9. © 2017 GridGain Systems, Inc. 1. Initial Query 2. Query execution over local data 3. Reduce multiple results in one 1. Initial Query 2. Query execution (local + remote data) 3. Potential data movement 4. Reduce multiple results in one 2 2 1 Collocated Joins Non-Collocated Joins Server Node ON-DISK Server Node ON-DISK Client Node 3 2 2 1 Server Node ON-DISK Server Node ON-DISK Client Node 4 3
- 10. © 2017 GridGain Systems, Inc. Collocated Processing
- 11. © 2017 GridGain Systems, Inc. 1. Initial Request 2. Fetch data from remote nodes 3. Process entire data-set 3 1 Data 1 2 2 Data 2 Client-Server Processing Co-located Processing Server Node ON-DISK Server Node ON-DISK 1. Initial Request 2. Co-located processing with data 3. Reduce multiple results in one 2 2 1Client Node Server Node ON-DISK Server Node ON-DISK Client Node 3
- 12. © 2017 GridGain Systems, Inc. Memory Architecture
- 13. © 2017 GridGain Systems, Inc. Durable Memory Off-heap Removes noticeable GC pauses Automatic Defragmentation Stores Superset of Data Predictable memory consumption Fully Transactional (Write-Ahead Log) DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Server Node Server Node Server Node Ignite Cluster Instantaneous Restarts
- 14. © 2017 GridGain Systems, Inc. Ignite Native Persistence 1. Update RAM 2. Persist Write-Ahead Log Partition File 1 3. Ack 4. Checkpointing Partition File N Server Node
- 15. © 2017 GridGain Systems, Inc.
- 16. © 2017 GridGain Systems, Inc. Regions and Segments • Memory split into regions • Regions split into segments • Segments include pages
- 17. © 2017 GridGain Systems, Inc. B+Tree • Self-balancing tree • Memory & Disk • Sorted Index • Secondary Indexes • Hash Index • Primary Keys • Hash code based sorting
- 18. © 2017 GridGain Systems, Inc. Free Lists • Tracks pages of ~ equal free space • 25% free • 75% free • Essential for updates • Gives page with min size needed • Reduces fragmentation • Lowers pages compaction activity
- 19. © 2017 GridGain Systems, Inc. Any Questions? Thank you for joining us. Follow the conversation. http://ignite.apache.org #apacheignite #denismagda
Notas del editor
- Apache Ignite incorporates distributed SQL database capabilities as a part of its platform. The database is horizontally scalable, fault tolerant and SQL ANSI-99 compliant. It supports all SQL, DDL, and DML commands including SELECT, UPDATE, INSERT, MERGE, and DELETE queries. It also provides support for a subset of DDL commands relevant for distributed databases. Data sets as well as indexes can be stored both in RAM and on disk thanks to the durable memory architecture. This allows executing distributed SQL operations across different memory layers achieving in-memory performance with durability of disk. You can interact with Apache Ignite using SQL language via natively developed APIs for Java, .NET and C++, or via the Ignite JDBC or ODBC drivers. This provides a true cross-platform connectivity from languages such as PHP, Ruby and more.
- https://apacheignite-sql.readme.io/docs/distributed-joins
- https://ignite.apache.org/collocatedprocessing.html Collocation of computations with data allow for minimizing data serialization within network and can significantly improve performance and scalability of your application. Whenever possible, you should always make best effort to colocate your computations with the cluster nodes caching the data that needs to be processed. Let's assume that a blizzard is approaching New York City. You, as a telecommunication company has to warn all the people sending a message to everyone with precise instructions on how to behave during such weather conditions. There are around 8 million New Yorkers in your database that have to receive the text message. With the client-server approach the company has to connect to the database, move all 8 million (!) records from there to a client application that will text to everyone. This is highly inefficient that wastes network and computational resources of company's IT infrastructure. However, if the company initially collocates all the cities it covers with the people who live there then it can send a single computation (!) to the cluster node that stores information about all New Yorkers and send the text message from there. This approach avoids 8 million records movement over the network and helps utilizing cluster resources for computation needs. That's the collocated processing in action!
- Apache Ignite memory-centric platform is based on the Durable Memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Persistent Store feature is enabled. The memory architecture helps achieve in-memory performance with durability of disk using all the available resources of the cluster. Ignite's durable memory is built and operates in a way similar to the Virtual Memory of operating systems such as Linux. However, one significant difference between these two types of architectures is that Durable Memory always keeps the whole data set and indexes on disk if the Ignite Persistent Store is used, while Virtual Memory uses the disk for swapping purposes only. In-Memory • Off-Heap memory • Removes noticeable GC pauses • Automatic Defragmentation • Predictable memory consumption • Boosts SQL performance On Disk • Optional Persistence • Support of flash, SSD, Intel 3D Xpoint • Stores superset of data • Fully Transactional ◦ Write-Ahead-Log (WAL) • Instantaneous Cluster Restarts
- Ignite Native Persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite's Durable Memory as an optional disk layer storing data and indexes on SSD, Flash, 3D XPoint, and other types of non-volatile storages. With the Ignite Persistence enabled, you no longer need to keep all the data and indexes in memory or warm it up after a node or cluster restart because the Durable Memory is tightly coupled with persistence and treats it as a secondary memory tier. This implies that if a subset of data or an index is missing in RAM, the Durable Memory will take it from the disk.
- Ignite Native Persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite's Durable Memory as an optional disk layer storing data and indexes on SSD, Flash, 3D XPoint, and other types of non-volatile storages. With the Ignite Persistence enabled, you no longer need to keep all the data and indexes in memory or warm it up after a node or cluster restart because the Durable Memory is tightly coupled with persistence and treats it as a secondary memory tier. This implies that if a subset of data or an index is missing in RAM, the Durable Memory will take it from the disk.
- B-tree is a self-balancing tree data structure that keeps data sorted and allows searches, sequential access, insertions, and deletions in logarithmic time. B+Tree is a central part of the whole Ignite Virtual memory architecture because even basic key-value operations work via it (cache.get and cache.put)! Move to the next slide.
- On the previous slide we explained how to look up a value inside of the virtual memory. However, how does the virtual memory know where to put a new value? In fact, Ignite uses a special data structure called Free List to support this. Basically, a free list is a doubly linked list that stores references to pages of approximately equal free space. For instance, there is a free list that stores all the data pages that have up to 75% free space and a list that keeps track of the index pages with 25% capacity left. Data and index pages are tracked in separate free lists.