Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

MariaDB Platform for hybrid transactional/analytical workloads

178 visualizaciones

Publicado el

OpenWorks 2019 Session
In order to provide data-driven customers with more historical data and real-time analytics, MariaDB Platform can be configured for hybrid transactional/analytical workloads by leveraging row storage for current data transactions and columnar storage for historical data and analytics. In this session Shane Johnson, Senior Director of Product Marketing at MariaDB, shows how change-data-capture and query routing, both available out of the box, can be used to bring scalable analytics to customer-facing applications without changing their code – and without depending on a separate data warehouse.

Publicado en: Software
  • Sé el primero en comentar

MariaDB Platform for hybrid transactional/analytical workloads

  1. 1. MariaDB Platform for hybrid transactional/analytical workloads Shane K Johnson Senior Director of Product Marketing MariaDB Corporation
  2. 2. Agenda 1. Hybrid workloads and use cases 2. MariaDB Platform X3 3. Scalability 4. Real-world use cases and hybrid cloud 5. Implementation: change-data-capture 6. Implementation: workload-based query routing 7. Getting started with Docker
  3. 3. Hybrid workloads
  4. 4. Database workloads Transactional Current data Range queries Known queries Historical data Aggregate queries Unkown queries Analytical
  5. 5. Database workloads Transactional Current data Range queries Known queries Row-based storage Indexes Clustered/Replicated Historical data Aggregate queries Unkown queries Columnar storage No indexes Distributed Analytical
  6. 6. MariaDB TX MariaDB Server MariaDB MaxScale InnoDB/MyRocks MariaDB AX MariaDB Server MariaDB MaxScale ColumnStore App/dev MariaDB TX BI/data science MariaDB AX
  7. 7. Database (OLTP) Data warehouse (OLAP) Application development BI/reporting + data science
  8. 8. Data warehouse (OLAP) Database (OLTP) Hybrid workloads: the problem Transactions App/dev Analytics BI/reporting + data science
  9. 9. Data warehouse (OLAP) Database (Hybrid) Hybrid workloads: the solution Transactions Analytics App/dev Analytics BI/reporting + data science
  10. 10. MariaDB Platform X3
  11. 11. MariaDB TX 3.0 MariaDB Server 10.3 MariaDB MaxScale 2.2 InnoDB/MyRocks MariaDB AX 2.0 MariaDB Server 10.2 MariaDB MaxScale 2.2 ColumnStore 1.2 MariaDB Platform X3 MariaDB MaxScale 2.3 MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3
  12. 12. The database proxy inspects queries and routes them to transactional and/or analytical database instances. The change-data-capture stream replicates all writes from transactional databases to analytical databases within microbatches. MariaDB Platform X3 MariaDB MaxScale 2.3 CDC MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3 Transactional Analytical
  13. 13. Applications Containers MariaDB Platform X3 MariaDB MaxScale 2.3 CDC MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3 Transactional Analytical Kubernetes (Helm) Docker (Compose) C JDBC ODBC Node.js Ingest streaming data Kafka connector Administration SQL Diagnostic Manager SQLyog MariaDB Backup MariaDB Flashback Import bulk data Spark connector C/Java/Python API
  14. 14. Scalability
  15. 15. Hybrid workloads: why scalability is needed Applications have transactional and analytical queries 1. Constrained by limited, lightweight analytics 2. Need full analytics to create competitive features Outgrowing OLTP Applications with lots of customers, lots of transactions 1. Limited to current or recent transaction data (months) 2. Need access to all historical data (years) Using historical data SaaS customers are becoming data-driven organizations 1. They don’t have access to their own data 2. They need to analyze it in unknown/unexpected ways Exposing analytics
  16. 16. Hybrid workloads: options ● Oracle ○ In-Memory Column Store ● Microsoft SQL Server ○ ColumnStore Indexes (Clustered or Nonclustered) ● IBM Db2 ○ Shadow Tables ● MySQL Enterprise ○ None ● EnterpriseDB Postgres Platform ○ None
  17. 17. Hybrid workloads: full comparison * IBM Db2 Shadow Tables, ** Oracle IM column store can read from on-disk row storage if needed Oracle Microsoft IBM * MariaDB MySQL Postgres Row storage (OLTP) Yes Yes Yes Yes Yes Yes Sharded Yes No Yes Yes No No Columnar storage (OLAP) Yes Yes Yes Yes No No Disk-based No ** Yes Yes Yes - - Distributed Yes (RAC) No No Yes - -
  18. 18. Application Connection 1 MariaDB Server Row storage MariaDB Server Columnar storage MariaDB MaxScale Transactional Analytical
  19. 19. MariaDB Server (Spider) Application Connection 1 MariaDB Server 1 Row storage MariaDB Server 2 Row storage MariaDB Server n Row storage MariaDB Server (ColumnStore) Sharding MariaDB MaxScale Transactional Analytical MariaDB Server Columnar storage
  20. 20. MariaDB Server (Spider) Application Connection 1 MariaDB Server (ColumnStore) Node 1 Columnar storage Node 2 Columnar storage Node n Columnar storage Distributed storage MariaDB MaxScale Transactional Analytical MariaDB Server Row storage
  21. 21. MariaDB Server (Spider) Application Connection 1 MariaDB Server 1 Row storage MariaDB Server 2 Row storage MariaDB Server n Row storage MariaDB Server (ColumnStore) CS node 1 Columnar storage CS node 2 Columnar storage CS node n Columnar storage Sharding Distributed storage MariaDB MaxScale Transactional Analytical
  22. 22. Real-world use cases
  23. 23. Retail (market research) Database (Hybrid) Transactions Analytics Transactional Capture daily product prices Update product information Show current prices Store current pricing data Analytical Show prices over time Self-service analytics Store historical pricing data
  24. 24. Telecommunications (IP telephony – SaaS) Database (Hybrid) Transactions Analytics Transactional Capture call detail records Charge by call/message Generate bills Analytical Monitor usage Identify peak periods Estimate costs Self-service analytics
  25. 25. Finance (trading) Database (Hybrid) Transactions Analytics Transactional Capture trades Analytical Analyze trading data Archiving for regulator compliance (historical data) Self-service analytics
  26. 26. Hybrid workloads: perfect for hybrid cloud ● MariaDB Platform separates and isolates different workloads ○ Run different workloads on different infrastructure ○ Place different workloads closer to different users ○ Scale different workloads on different hardware
  27. 27. MariaDB Platform MariaDB Server MariaDB Server MariaDB Server MariaDB Server MariaDB Server InnoDB/MyRocks MariaDB Server ColumnStore On premises for transactions Keep transactions close to the business Public cloud for analytics Incremental path to the cloud, places analytics close to customers
  28. 28. Implementation: change-data-capture
  29. 29. MariaDB Server (primary) InnoDB/MyRocks MariaDB MaxScale Binlog Binlog router MariaDB Server (secondary) InnoDB/MyRocks MariaDB Server (secondary) InnoDB/MyRocks Binlog server
  30. 30. App/Service/Script (backend) Bulk data adapter MariaDB Server (transactions) InnoDB/MyRocks MariaDB MaxScale Binlog Binlog router Avro router CDC protocol CDC client ColumnStore (storage) Write engine MariaDB Server (analytics) ColumnStore (interface) CDC server
  31. 31. ColumnStore (storage) CDC data adapter MariaDB Server (analytics) ColumnStore (interface) MariaDB MaxScale Binlog routerMySQL protocol Avro router CDC protocol MariaDB Server (transactions) InnoDB/MyRocks
  32. 32. MariaDB MaxScale [replication-router] type = service router = binlogrouter user = repl password = pass server_id = 5 master_id = 1 Binlogdir = /var/lib/maxscale Mariadb10-compatibility = 1 filestem = mariadb-bin [avro-router] type = service router = avrorouter source = replication-router avrodir = /var/lib/maxscale [replication-listener] type = listener service = replication-router protocol = MySQLClient port = 6603 Binlog router (service) MySQL protocol (listener) Avro router (service) CDC protocol (listener) [avro-listener] type = listener service = avro-router protocol = cdc port = 4001 mxs_adapter -c /etc/ColumnStore.xml -u {user} -p {password} -h {host} -P {port} -r 2 -d -n -z {schema} {table}
  33. 33. Implementation: workload-based query routing
  34. 34. Database Database MariaDB MaxScale Server (server -> IP) Server (server -> IP) Listener (port -> service) Service (request -> server) Server (server -> IP) Filter(s)Protocol Monitor Applications Database
  35. 35. Server 1 (transactional) [sw-db1] type = server address = <MariaDB-Server-1-IP> port = 3306 protocol = MariaDBBackend Server 2 (transactional) [sw-db2] type = server address = <MariaDB-Server-2-IP> port = 3306 protocol = MariaDBBackend Server 3 (transactional) [sw-db3] type = server address = <MariaDB-Server-3-IP> port = 3306 protocol = MariaDBBackend Service 1 (transactional) [MDB-Service] type = service router = readwritesplit servers = sw-db1,sw-db2, sw-db3 user = maxuser password = maxpwd Listener 1 (transactional) [MDB-Listener] type = listener service = MDB-Service protocol = MariaDBClient port = 4009
  36. 36. Server 4 (analytical) [sw-mcs-um2] type = server address = <MariaDB-UM-1-IP> port = 3306 protocol = MariaDBBackend Server 5 (analytical) [sw-mcs-um2] type = server address = <MariaDB-UM-2-IP> port = 3306 protocol = MariaDBBackend Service 2 (analytical) [CS-Service] type = service router = readconnroute router_options = running servers = sw-mcs-um1, sw-mcs-um2 user = maxuser password = maxpwd Listener 2 (analytical) [CS-Listener] type = listener service = CS-Service protocol = MariaDBClient port = 4010
  37. 37. Listener 2 (analytical) Listener 1 (transactional) [MDB-Listener] type = listener service = MDB-Service protocol = MariaDBClient port = 4009 [CS-Listener] type = listener service = CS-Service protocol = MariaDBClient port = 4010 Server 6 (transactional) Server 7 (analytical) [CS-Service-as-server] type = server address = 127.0.0.1 port = 4010 protocol = MariaDBBackend [MDB-Service-as-server] type = server address = 127.0.0.1 port = 4009 protocol = MariaDBBackend Service 3 (HTAP) Listener 3 (HTAP) [HTAP-Service] type = service router = schemarouter ignore_databases_regex = .* servers = MDB-Service-as-server,CS-Service-as-server Preferred_server = MDB-Service-as-server user = maxuser password = maxpwd filters = target-selector [HTAP-Listener] type = listener service = HTAP-Service protocol = MariaDBClient port = 4011
  38. 38. Service 3 (HTAP) [HTAP-Service] type = service router = schemarouter ignore_databases_regex = .* servers = MDB-Service-as-server,CS-Service-as-server Preferred_server = MDB-Service-as-server user = maxuser password = maxpwd filters = target-selector [target-selector] type = filter module = namedserverfilter match01 = (?i)SELECT.*loan target01 = CS-Service-as-server match02 = .* target02 = MDB-Service-as-server Filter 1 Regex filter (by query syntax) Schema router (by table/view) Named server filter (by query syntax) Hint filter Options
  39. 39. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases --maxscale route to analytics MariaDB MaxScale Query routing: hint filter
  40. 40. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases --maxscale route to analytics MariaDB MaxScale Query routing: hint filter
  41. 41. Database instance (analytical) Database instance (transactional) Database instance (transactional) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases_history MariaDB MaxScale Query routing: schema router tbl_purchases view_purchase_history tbl_purchases
  42. 42. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases_history MariaDB MaxScale Query routing: combo 1. Named server filter (attach hint) select avg(total) from tbl_purchases 2. Regex filter
  43. 43. Getting started with Docker
  44. 44. MariaDB Platform X3 in a container ● MariaDB Platform “in-a-box” ● Fully-configured and ready to use out of the box ● Intended as a quickstart for development ● Launched with a Docker command ● Runs on a laptop https://github.com/mariadb-corporation/mariadb-platform-docker/tree/master/single-container
  45. 45. MariaDB Platform X3 in a container
  46. 46. THANK YOU!

×