SlideShare una empresa de Scribd logo
1 de 44
Descargar para leer sin conexión
Managing High Performance Data with vFabric SQLFire
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
We Challenge the traditional RDBMS design NOT SQL ,[object Object],[object Object],[object Object],[object Object],Confidential First write to LOG Second write to Data files Buffers primarily tuned for IO
Confidential Common themes in next-gen DB architectures NoSQL, Data Grids, Data Fabrics, NewSQL “ Shared nothing” commodity clusters focus shifts to memory, distributing data and clustering Scale by partitioning the data and move behavior to data nodes HA within cluster and across data centers Add capacity to scale dynamically
What is different ? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Low scale  High scale  Very high scale  STRICT – Full ACID (RDB) Tunable Consistency Eventual
What is our take with SQLFire?
So, what is vFabric SQLFire? ,[object Object],[object Object],[object Object]
Show me a picture
Comparing with NoSQL, Object Data Grids ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comparing with NoSQL, Object Data Grids ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Features in 1.0 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Flexible Deployment Topologies Confidential Java Application cluster can host an embedded clustered database by just changing the URL jdbc:sqlfire:;mcast-port=33666;host-data=true
Flexible Deployment Topologies Confidential
Partitioning & Replication
Explore features through example Assume, thousands of flight rows, millions of flightavailability records
Creating Tables Table CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ); SQLF SQLF SQLF
Replicated Tables CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ) REPLICATE ; Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
Partitioned Tables CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL ) PARTITION BY COLUMN( FLIGHT_ID ); Table Partitioned Table Partitioned Table Partitioned Table Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
Partition Redundancy CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL)   PARTITION BY COLUMN (FLIGHT_ID)   REDUNDANCY 1 ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
Partition Colocation CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition
Persistent Tables CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS)  PERSISTENT persistentStore ASYNCHRONOUS ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition sqlf backup /export/fileServerDirectory/sqlfireBackupLocation Data dictionary is always persisted in each server
Demo  default partitioned tables, colocation, persistent tables
Scaling data with Partitioned tables
Hash partitioning for linear scaling Key Hashing provides single hop access to its partition But, what if the access is not based on the key … say, joins are involved
Hash partitioning only goes so far ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],EquiJOIN of rows across multiple nodes is not supported in SQLFire 1.0
Partition aware DB design ,[object Object],[object Object],[object Object],[object Object]
Partition aware DB design ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Partition aware DB design Entity Groups Table  FlightAvailability partitioned by FlightID colocated with Flights FlightID is the  entity group Key
Partition Aware DB design ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Scaling Application logic with Parallel  “Data Aware procedures”
Procedures Java Stored Procedures may be created according to the SQL Standard SQLFabric also supports the JDBC type Types.JAVA_OBJECT. A parameter of type JAVA_OBJECT supports an arbitrary Serializable Java object.  In this case, the procedure will be executed on the server to which a client is connected (or locally for Peer Clients) CREATE PROCEDURE getOverBookedFlights  (IN argument OBJECT, OUT result OBJECT)   LANGUAGE JAVA PARAMETER STYLE JAVA  READS SQL DATA DYNAMIC RESULT SETS 1  EXTERNAL NAME com.acme.OverBookedFLights;
Data Aware Procedures Parallelize procedure and prune to nodes with required data Extend the procedure call with the following syntax: Hint the data the procedure depends on CALL getOverBookedFlights( <bind arguments> ON TABLE FLIGHTAVAILABILITY  WHERE FLIGHTID = <SomeFLIGHTID> ; If table is partitioned by columns in the where clause the procedure execution is pruned to nodes with the data (node with <someFLIGHTID> in this case) CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name ] [ { ON TABLE table_name [ WHERE whereClause ] }  | { ON {ALL | SERVER GROUPS  (server_group_name [, server_group_name ]*) }} ] Fabric Server 2 Fabric Server 1 Client
Parallelize procedure then aggregate (reduce) Fabric Server 2 Fabric Server 1 Client Fabric Server 3 CALL SQLF.CreateResultProcessor( processor_name, processor_class_name); register a Java Result Processor  (optional in some cases) : CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name  ] [ { ON TABLE table_name [ WHERE whereClause ] }  | { ON {ALL | SERVER GROUPS  (server_group_name [, server_group_name ]*) }} ]
Consistency model
Consistency Model without Transactions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Distributed Transactions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Scaling disk access with shared nothing disk files and a  “journaling” store design
Disk persistence in SQLF ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance benchmark
How does it perform? Scale? ,[object Object],[object Object],[object Object]
How does it perform? Scale? ,[object Object]
Is latency low with scale? ,[object Object],[object Object],[object Object]
Q & A ,[object Object],[object Object]
Built using GemFire object data fabric + Derby

Más contenido relacionado

La actualidad más candente

MS SQL SERVER: Programming sql server data mining
MS SQL SERVER: Programming sql server data miningMS SQL SERVER: Programming sql server data mining
MS SQL SERVER: Programming sql server data miningDataminingTools Inc
 
Killer Scenarios with Data Lake in Azure with U-SQL
Killer Scenarios with Data Lake in Azure with U-SQLKiller Scenarios with Data Lake in Azure with U-SQL
Killer Scenarios with Data Lake in Azure with U-SQLMichael Rys
 
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...Michael Rys
 
Graph db as metastore
Graph db as metastoreGraph db as metastore
Graph db as metastoreHaris Khan
 
Database Architecture and Basic Concepts
Database Architecture and Basic ConceptsDatabase Architecture and Basic Concepts
Database Architecture and Basic ConceptsTony Wong
 
Oracle Text in APEX
Oracle Text in APEXOracle Text in APEX
Oracle Text in APEXScott Wesley
 
Partitioning tables and indexing them
Partitioning tables and indexing them Partitioning tables and indexing them
Partitioning tables and indexing them Hemant K Chitale
 
Partitioning Tables and Indexing Them --- Article
Partitioning Tables and Indexing Them --- ArticlePartitioning Tables and Indexing Them --- Article
Partitioning Tables and Indexing Them --- ArticleHemant K Chitale
 
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresArchitectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresGleicon Moraes
 
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...Beat Signer
 
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)Michael Rys
 
Partitioning on Oracle 12c - What changed on the most important Oracle feature
Partitioning on Oracle 12c - What changed on the most important Oracle featurePartitioning on Oracle 12c - What changed on the most important Oracle feature
Partitioning on Oracle 12c - What changed on the most important Oracle featureLuis Marques
 
20140908 spark sql & catalyst
20140908 spark sql & catalyst20140908 spark sql & catalyst
20140908 spark sql & catalystTakuya UESHIN
 
Analytic Views in Oracle 12.2
Analytic Views in Oracle 12.2Analytic Views in Oracle 12.2
Analytic Views in Oracle 12.2Kim Berg Hansen
 
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2Marco Gralike
 
Dynamic Publishing with Arbortext Data Merge
Dynamic Publishing with Arbortext Data MergeDynamic Publishing with Arbortext Data Merge
Dynamic Publishing with Arbortext Data MergeClay Helberg
 
Introduction and overview ArangoDB query language AQL
Introduction and overview ArangoDB query language AQLIntroduction and overview ArangoDB query language AQL
Introduction and overview ArangoDB query language AQLArangoDB Database
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at HuaweiHBaseCon
 
Oracle Table Partitioning - Introduction
Oracle Table Partitioning  - IntroductionOracle Table Partitioning  - Introduction
Oracle Table Partitioning - IntroductionMyOnlineITCourses
 

La actualidad más candente (20)

MS SQL SERVER: Programming sql server data mining
MS SQL SERVER: Programming sql server data miningMS SQL SERVER: Programming sql server data mining
MS SQL SERVER: Programming sql server data mining
 
Killer Scenarios with Data Lake in Azure with U-SQL
Killer Scenarios with Data Lake in Azure with U-SQLKiller Scenarios with Data Lake in Azure with U-SQL
Killer Scenarios with Data Lake in Azure with U-SQL
 
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...
U-SQL Killer Scenarios: Custom Processing, Big Cognition, Image and JSON Proc...
 
Graph db as metastore
Graph db as metastoreGraph db as metastore
Graph db as metastore
 
Database Architecture and Basic Concepts
Database Architecture and Basic ConceptsDatabase Architecture and Basic Concepts
Database Architecture and Basic Concepts
 
Oracle Text in APEX
Oracle Text in APEXOracle Text in APEX
Oracle Text in APEX
 
Hive and Shark
Hive and SharkHive and Shark
Hive and Shark
 
Partitioning tables and indexing them
Partitioning tables and indexing them Partitioning tables and indexing them
Partitioning tables and indexing them
 
Partitioning Tables and Indexing Them --- Article
Partitioning Tables and Indexing Them --- ArticlePartitioning Tables and Indexing Them --- Article
Partitioning Tables and Indexing Them --- Article
 
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresArchitectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
 
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...
Structured Query Language (SQL) - Lecture 5 - Introduction to Databases (1007...
 
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)
The Road to U-SQL: Experiences in Language Design (SQL Konferenz 2017 Keynote)
 
Partitioning on Oracle 12c - What changed on the most important Oracle feature
Partitioning on Oracle 12c - What changed on the most important Oracle featurePartitioning on Oracle 12c - What changed on the most important Oracle feature
Partitioning on Oracle 12c - What changed on the most important Oracle feature
 
20140908 spark sql & catalyst
20140908 spark sql & catalyst20140908 spark sql & catalyst
20140908 spark sql & catalyst
 
Analytic Views in Oracle 12.2
Analytic Views in Oracle 12.2Analytic Views in Oracle 12.2
Analytic Views in Oracle 12.2
 
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2
OPP2010 (Brussels) - Programming with XML in PL/SQL - Part 2
 
Dynamic Publishing with Arbortext Data Merge
Dynamic Publishing with Arbortext Data MergeDynamic Publishing with Arbortext Data Merge
Dynamic Publishing with Arbortext Data Merge
 
Introduction and overview ArangoDB query language AQL
Introduction and overview ArangoDB query language AQLIntroduction and overview ArangoDB query language AQL
Introduction and overview ArangoDB query language AQL
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
 
Oracle Table Partitioning - Introduction
Oracle Table Partitioning  - IntroductionOracle Table Partitioning  - Introduction
Oracle Table Partitioning - Introduction
 

Destacado

Afrocentricity conference in Paris 11-12 May 2012
Afrocentricity conference in Paris 11-12 May 2012Afrocentricity conference in Paris 11-12 May 2012
Afrocentricity conference in Paris 11-12 May 2012Antumi (Toasijé) Pallas
 
Application management for hybrid cloud
Application management for hybrid cloudApplication management for hybrid cloud
Application management for hybrid cloudVMware vFabric
 
Panafricanismo ante ataques a soberanías de los estados africanos
Panafricanismo ante ataques a soberanías de los estados africanosPanafricanismo ante ataques a soberanías de los estados africanos
Panafricanismo ante ataques a soberanías de los estados africanosAntumi (Toasijé) Pallas
 
vFabric Data Director - DB as a Service
vFabric Data Director - DB as a ServicevFabric Data Director - DB as a Service
vFabric Data Director - DB as a ServiceVMware vFabric
 
vFabric - Ideal Platform for SaaS Apps
vFabric - Ideal Platform for SaaS AppsvFabric - Ideal Platform for SaaS Apps
vFabric - Ideal Platform for SaaS AppsVMware vFabric
 
VMware vFabric gemfire for high performance, resilient distributed apps
VMware vFabric gemfire for high performance, resilient distributed appsVMware vFabric gemfire for high performance, resilient distributed apps
VMware vFabric gemfire for high performance, resilient distributed appsVMware vFabric
 
Data fabric and VMware
Data fabric and VMwareData fabric and VMware
Data fabric and VMwareVMware vFabric
 
Water Quality Management
Water Quality ManagementWater Quality Management
Water Quality Managementjcayayan
 
Introduction to Cloud Foundry
Introduction to Cloud FoundryIntroduction to Cloud Foundry
Introduction to Cloud FoundryVMware vFabric
 
VMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric
 
VMware vFabric - CIO Webinar - Al Sargent
VMware vFabric - CIO Webinar - Al SargentVMware vFabric - CIO Webinar - Al Sargent
VMware vFabric - CIO Webinar - Al SargentVMware vFabric
 
vFabric for i ISVs and MSPs
vFabric for i ISVs and MSPsvFabric for i ISVs and MSPs
vFabric for i ISVs and MSPsVMware vFabric
 
Introduction to Cloud Application Platform
Introduction to Cloud Application PlatformIntroduction to Cloud Application Platform
Introduction to Cloud Application PlatformVMware vFabric
 
Spring One 2012 Presentation – Effective design patterns with NewSQL
Spring One 2012 Presentation – Effective design patterns with NewSQLSpring One 2012 Presentation – Effective design patterns with NewSQL
Spring One 2012 Presentation – Effective design patterns with NewSQLVMware vFabric
 
Literacy breaking the cycle
Literacy breaking the cycleLiteracy breaking the cycle
Literacy breaking the cycleZifan Wang
 
Rawlerフレームワーク(全体)
Rawlerフレームワーク(全体)Rawlerフレームワーク(全体)
Rawlerフレームワーク(全体)Takaichi Ito
 

Destacado (20)

New trends in data
New trends in dataNew trends in data
New trends in data
 
Afrocentricity conference in Paris 11-12 May 2012
Afrocentricity conference in Paris 11-12 May 2012Afrocentricity conference in Paris 11-12 May 2012
Afrocentricity conference in Paris 11-12 May 2012
 
Application management for hybrid cloud
Application management for hybrid cloudApplication management for hybrid cloud
Application management for hybrid cloud
 
Panafricanismo ante ataques a soberanías de los estados africanos
Panafricanismo ante ataques a soberanías de los estados africanosPanafricanismo ante ataques a soberanías de los estados africanos
Panafricanismo ante ataques a soberanías de los estados africanos
 
vFabric Data Director - DB as a Service
vFabric Data Director - DB as a ServicevFabric Data Director - DB as a Service
vFabric Data Director - DB as a Service
 
vFabric - Ideal Platform for SaaS Apps
vFabric - Ideal Platform for SaaS AppsvFabric - Ideal Platform for SaaS Apps
vFabric - Ideal Platform for SaaS Apps
 
VMware vFabric gemfire for high performance, resilient distributed apps
VMware vFabric gemfire for high performance, resilient distributed appsVMware vFabric gemfire for high performance, resilient distributed apps
VMware vFabric gemfire for high performance, resilient distributed apps
 
Data fabric and VMware
Data fabric and VMwareData fabric and VMware
Data fabric and VMware
 
Water Quality Management
Water Quality ManagementWater Quality Management
Water Quality Management
 
Introduction to Cloud Foundry
Introduction to Cloud FoundryIntroduction to Cloud Foundry
Introduction to Cloud Foundry
 
VMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a Service
 
VMware vFabric - CIO Webinar - Al Sargent
VMware vFabric - CIO Webinar - Al SargentVMware vFabric - CIO Webinar - Al Sargent
VMware vFabric - CIO Webinar - Al Sargent
 
vFabric for i ISVs and MSPs
vFabric for i ISVs and MSPsvFabric for i ISVs and MSPs
vFabric for i ISVs and MSPs
 
Introduction to Cloud Application Platform
Introduction to Cloud Application PlatformIntroduction to Cloud Application Platform
Introduction to Cloud Application Platform
 
Pr.seminar
Pr.seminarPr.seminar
Pr.seminar
 
Spring One 2012 Presentation – Effective design patterns with NewSQL
Spring One 2012 Presentation – Effective design patterns with NewSQLSpring One 2012 Presentation – Effective design patterns with NewSQL
Spring One 2012 Presentation – Effective design patterns with NewSQL
 
Catedra Fotos Carnaval 2011
Catedra Fotos Carnaval 2011Catedra Fotos Carnaval 2011
Catedra Fotos Carnaval 2011
 
Jha
JhaJha
Jha
 
Literacy breaking the cycle
Literacy breaking the cycleLiteracy breaking the cycle
Literacy breaking the cycle
 
Rawlerフレームワーク(全体)
Rawlerフレームワーク(全体)Rawlerフレームワーク(全体)
Rawlerフレームワーク(全体)
 

Similar a vFabric SQLFire for high performance data

vFabric SQLFire Introduction
vFabric SQLFire IntroductionvFabric SQLFire Introduction
vFabric SQLFire IntroductionJags Ramnarayan
 
Azure Databricks is Easier Than You Think
Azure Databricks is Easier Than You ThinkAzure Databricks is Easier Than You Think
Azure Databricks is Easier Than You ThinkIke Ellis
 
Sql azure dec_2010 Lynn & Ike
Sql azure dec_2010 Lynn & IkeSql azure dec_2010 Lynn & Ike
Sql azure dec_2010 Lynn & IkeIke Ellis
 
Tech Days09 Sqldev
Tech Days09 SqldevTech Days09 Sqldev
Tech Days09 Sqldevllangit
 
SQL Server 2008 for Developers
SQL Server 2008 for DevelopersSQL Server 2008 for Developers
SQL Server 2008 for Developersllangit
 
SQL Server 2008 for .NET Developers
SQL Server 2008 for .NET DevelopersSQL Server 2008 for .NET Developers
SQL Server 2008 for .NET Developersllangit
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQLAhmed Helmy
 
02 data warehouse applications with hive
02 data warehouse applications with hive02 data warehouse applications with hive
02 data warehouse applications with hiveSubhas Kumar Ghosh
 
Graph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemGraph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemMarco Parenzan
 
SQL Server 2008 Performance Enhancements
SQL Server 2008 Performance EnhancementsSQL Server 2008 Performance Enhancements
SQL Server 2008 Performance Enhancementsinfusiondev
 
Microsoft Database Options
Microsoft Database OptionsMicrosoft Database Options
Microsoft Database OptionsDavid Chou
 
Microsoft Zend webcast on Azure
Microsoft Zend webcast on AzureMicrosoft Zend webcast on Azure
Microsoft Zend webcast on AzureJosh Holmes
 
Microsoft/Zend Webcast on Cloud Computing
Microsoft/Zend Webcast on Cloud ComputingMicrosoft/Zend Webcast on Cloud Computing
Microsoft/Zend Webcast on Cloud ComputingJosh Holmes
 
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the CloudSQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the CloudMark Kromer
 
Introduction à DocumentDB
Introduction à DocumentDBIntroduction à DocumentDB
Introduction à DocumentDBMSDEVMTL
 
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit Jain
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit JainApache Hadoop India Summit 2011 talk "Hive Evolution" by Namit Jain
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit JainYahoo Developer Network
 
Hadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologiesHadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologiesappaji intelhunt
 

Similar a vFabric SQLFire for high performance data (20)

vFabric SQLFire Introduction
vFabric SQLFire IntroductionvFabric SQLFire Introduction
vFabric SQLFire Introduction
 
Azure Databricks is Easier Than You Think
Azure Databricks is Easier Than You ThinkAzure Databricks is Easier Than You Think
Azure Databricks is Easier Than You Think
 
Sql azure dec_2010 Lynn & Ike
Sql azure dec_2010 Lynn & IkeSql azure dec_2010 Lynn & Ike
Sql azure dec_2010 Lynn & Ike
 
Tech Days09 Sqldev
Tech Days09 SqldevTech Days09 Sqldev
Tech Days09 Sqldev
 
SQL Server 2008 for Developers
SQL Server 2008 for DevelopersSQL Server 2008 for Developers
SQL Server 2008 for Developers
 
SQL Server 2008 for .NET Developers
SQL Server 2008 for .NET DevelopersSQL Server 2008 for .NET Developers
SQL Server 2008 for .NET Developers
 
HadoopDB
HadoopDBHadoopDB
HadoopDB
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 
02 data warehouse applications with hive
02 data warehouse applications with hive02 data warehouse applications with hive
02 data warehouse applications with hive
 
Graph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemGraph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft Ecosystem
 
SQL Server 2008 Performance Enhancements
SQL Server 2008 Performance EnhancementsSQL Server 2008 Performance Enhancements
SQL Server 2008 Performance Enhancements
 
Microsoft Database Options
Microsoft Database OptionsMicrosoft Database Options
Microsoft Database Options
 
Microsoft Zend webcast on Azure
Microsoft Zend webcast on AzureMicrosoft Zend webcast on Azure
Microsoft Zend webcast on Azure
 
Microsoft/Zend Webcast on Cloud Computing
Microsoft/Zend Webcast on Cloud ComputingMicrosoft/Zend Webcast on Cloud Computing
Microsoft/Zend Webcast on Cloud Computing
 
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the CloudSQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
 
HTAP Queries
HTAP QueriesHTAP Queries
HTAP Queries
 
Introduction à DocumentDB
Introduction à DocumentDBIntroduction à DocumentDB
Introduction à DocumentDB
 
Big data concepts
Big data conceptsBig data concepts
Big data concepts
 
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit Jain
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit JainApache Hadoop India Summit 2011 talk "Hive Evolution" by Namit Jain
Apache Hadoop India Summit 2011 talk "Hive Evolution" by Namit Jain
 
Hadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologiesHadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologies
 

Último

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 

Último (20)

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 

vFabric SQLFire for high performance data

  • 1. Managing High Performance Data with vFabric SQLFire
  • 2.
  • 3.
  • 4. Confidential Common themes in next-gen DB architectures NoSQL, Data Grids, Data Fabrics, NewSQL “ Shared nothing” commodity clusters focus shifts to memory, distributing data and clustering Scale by partitioning the data and move behavior to data nodes HA within cluster and across data centers Add capacity to scale dynamically
  • 5.
  • 6. What is our take with SQLFire?
  • 7.
  • 8. Show me a picture
  • 9.
  • 10.
  • 11.
  • 12. Flexible Deployment Topologies Confidential Java Application cluster can host an embedded clustered database by just changing the URL jdbc:sqlfire:;mcast-port=33666;host-data=true
  • 15. Explore features through example Assume, thousands of flight rows, millions of flightavailability records
  • 16. Creating Tables Table CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ); SQLF SQLF SQLF
  • 17. Replicated Tables CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ) REPLICATE ; Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
  • 18. Partitioned Tables CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL ) PARTITION BY COLUMN( FLIGHT_ID ); Table Partitioned Table Partitioned Table Partitioned Table Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
  • 19. Partition Redundancy CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL) PARTITION BY COLUMN (FLIGHT_ID) REDUNDANCY 1 ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF
  • 20. Partition Colocation CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition
  • 21. Persistent Tables CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) PERSISTENT persistentStore ASYNCHRONOUS ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition sqlf backup /export/fileServerDirectory/sqlfireBackupLocation Data dictionary is always persisted in each server
  • 22. Demo default partitioned tables, colocation, persistent tables
  • 23. Scaling data with Partitioned tables
  • 24. Hash partitioning for linear scaling Key Hashing provides single hop access to its partition But, what if the access is not based on the key … say, joins are involved
  • 25.
  • 26.
  • 27.
  • 28. Partition aware DB design Entity Groups Table FlightAvailability partitioned by FlightID colocated with Flights FlightID is the entity group Key
  • 29.
  • 30. Scaling Application logic with Parallel “Data Aware procedures”
  • 31. Procedures Java Stored Procedures may be created according to the SQL Standard SQLFabric also supports the JDBC type Types.JAVA_OBJECT. A parameter of type JAVA_OBJECT supports an arbitrary Serializable Java object. In this case, the procedure will be executed on the server to which a client is connected (or locally for Peer Clients) CREATE PROCEDURE getOverBookedFlights (IN argument OBJECT, OUT result OBJECT) LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA DYNAMIC RESULT SETS 1 EXTERNAL NAME com.acme.OverBookedFLights;
  • 32. Data Aware Procedures Parallelize procedure and prune to nodes with required data Extend the procedure call with the following syntax: Hint the data the procedure depends on CALL getOverBookedFlights( <bind arguments> ON TABLE FLIGHTAVAILABILITY WHERE FLIGHTID = <SomeFLIGHTID> ; If table is partitioned by columns in the where clause the procedure execution is pruned to nodes with the data (node with <someFLIGHTID> in this case) CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name ] [ { ON TABLE table_name [ WHERE whereClause ] } | { ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }} ] Fabric Server 2 Fabric Server 1 Client
  • 33. Parallelize procedure then aggregate (reduce) Fabric Server 2 Fabric Server 1 Client Fabric Server 3 CALL SQLF.CreateResultProcessor( processor_name, processor_class_name); register a Java Result Processor (optional in some cases) : CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name ] [ { ON TABLE table_name [ WHERE whereClause ] } | { ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }} ]
  • 35.
  • 36.
  • 37. Scaling disk access with shared nothing disk files and a “journaling” store design
  • 38.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44. Built using GemFire object data fabric + Derby

Notas del editor

  1. Original design rooted in good principles of data independence, durability and consistency of data Designers naturally focused on disk IO performance and maintaining strict data consistency through many locking/latching techniques Buffer management is all about using memory for caching contiguous disk blocks
  2. Driven by the desire to scale, be HA and offer lower latencies ... clusters of commodity servers .... ... focus shifts to distributing data and clustering ... shared nothing including the disk ... avoid disk seeks as much as possible .. ... memory is cheap and reliable ... Pool memory across cluster and use highly optimized concurrent data structures ... partitioning with consistent hashing ... dynamically increase cluster capacity ... Move and parallelize behavior to data (MR) ... High availability within cluster and across data centers
  3. Entity groups defined in SQLFire using “colocation” clause Entity group guaranteed to be collocated in presence of failures or rebalance Now, complex queries can be executed without requiring excessive distributed data access