My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"

1. Avoiding CRUD operations lock-in
in NoSQL databases: extension of
the CPIM library
Candidato: Fabio Arcidiacono (799001)
Relatore: Prof.ssa Elisabetta Di Nitto
Correlatore: Ing. Marco Scavuzzo
Scuola di Ingegneria Industriale e dell'Informazione
Corso di Laurea Magistrale in
Ingegneria Informatica
Anno Accademico 2013 - 2014

2. Tesi di Laurea Magistrale – Fabio Arcidiacono
Data management systems
2
Well structured data
Relational model
ACID transactions
Vertical scaling
SQL
RDBMS
Non-structured data
Various data models
BASE properties
Horizontal scaling
Proprietary API
NoSQL

3. Tesi di Laurea Magistrale – Fabio Arcidiacono
NoSQL Common language approaches
3
Meta-model
• Apache MetaModel
• SOS platform
SQLification
• Apache Phoenix
• UnQL
• Native support
ORM
• Kundera
• PlayORM
• Spring-data
• Apache Gora

4. Tesi di Laurea Magistrale – Fabio Arcidiacono
Work objectives
4
Integrate Kundera in the CPIM library
Contribute to the open source project Kundera
Integrate the migration and synchronization system Hegira
Evaluation

5. Tesi di Laurea Magistrale – Fabio Arcidiacono
Work objectives
5
Integrate Kundera in the CPIM library
Contribute to the open source project Kundera
Integrate the migration and synchronization system Hegira
Evaluation

6. Tesi di Laurea Magistrale – Fabio Arcidiacono
Kundera
A JPA 2.1 ORM Library for NoSQL databases
6
ORM operation (through EntityManager interface)
JPQL queries (DELETE and UPDATE)
On-premises databases:
• Cassandra
• HBase
• MongoDB
• Oracle NoSQL
• Redis
• Neo4j
• Couchdb
• Elastic Search
• MySQL

7. Tesi di Laurea Magistrale – Fabio Arcidiacono
Why Kundera
• Open source
• Developed with extensibility as primary goal
• Support to many different NoSQL databases
• Polyglot persistency
• In the field since 2010 with an active community
• Already used in production
7

8. Tesi di Laurea Magistrale – Fabio Arcidiacono
Contributions to Kundera
Two newly developed clients
• Azure Tables1
• GAE Datastore2
Paradigm shift
• Off-premises databases à DaaS solutions
• Bug fix Kundera deploy on PaaS
8
1: https://github.com/deib-polimi/kundera-azure-table
2: https://github.com/deib-polimi/kundera-gae-datastore

9. Tesi di Laurea Magistrale – Fabio Arcidiacono
Developed clients
9
Exploit consistency mechanisms as much
as possible
GAE Datastore
à no Ancestor Path support
Azure Tables
à manage partition key and row key
master
Limited support to consistency
mechanisms but achieve interoperability
GAE Datastore
à no Ancestor Path support
Azure Tables
à fix partition key to table name
migration

10. Tesi di Laurea Magistrale – Fabio Arcidiacono
Work objectives
10
Integrate Kundera in the CPIM library
Contribute to the open source project Kundera
Integrate the migration and synchronization system Hegira
Evaluation

11. Tesi di Laurea Magistrale – Fabio Arcidiacono
CPIM
Abstract application logic from the specific PaaS Provider to overcome the vendor lock-in
11
Many supported services:
• Blob
• NoSQL
• Memcache
• Queue
• Mail
• SQL

12. Tesi di Laurea Magistrale – Fabio Arcidiacono
Original CPIM NoSQL service implementation
12
• Many JPA providers
• Duplicated code
• No complete code portability
• Choice of the NoSQL database strictly bounded to the cloud
provider (e.g. App Engine à Datastore)
• Limited NoSQL databases support
CloudEntityManager
jpa4Azure
SimpleJPA
Google JPA
Azure EntityManager
Azure EntityManagerFactory
AWS EntityManager
AWS EntityManagerFactory
GAE EntityManager
GAE EntityManagerFactory
CloudEntityManagerFactory

13. Tesi di Laurea Magistrale – Fabio Arcidiacono
Kundera integration
• Single persistence provider
• Complete code portability
• NoSQL support inherited by Kundera
• Easier Configuration through standard persistence.xml
13
CloudEntityManager
Kundera
CloudEntityManagerFactory

14. Tesi di Laurea Magistrale – Fabio Arcidiacono
Work objectives
14
Integrate Kundera in the CPIM library
Contribute to the open source project Kundera
Integrate the migration and synchronization system Hegira
Evaluation

15. Tesi di Laurea Magistrale – Fabio Arcidiacono
Data migration
15
• move application to another cloud provider
• move data to a database that better fit requirements
• load balancing, system expansion, failure recovery, costs, etc.
• modern computer systems are expected to be up continuously
• data synchronization between the two involved systems

16. Tesi di Laurea Magistrale – Fabio Arcidiacono
Hegira support
16
• Intercept transparently user operations (DMQ)
• Translate operations to SQL statements
• Send them to the Hegira commit-log

17. Tesi di Laurea Magistrale – Fabio Arcidiacono
Work objectives
17
Integrate Kundera in the CPIM library
Contribute to the open source project Kundera
Integrate the migration and synchronization system Hegira
Evaluation

18. Tesi di Laurea Magistrale – Fabio Arcidiacono
Cloud Serving Benchmark
18
Compare Kundera client w.r.t. the use of low-level API for the same operations
• Development of new adapter for operations through Kundera
• Development of new adapter for operations through the low-level API
Workload
100.000 entities
Transaction
phase
(read)
Load
phase
(write)
Write
operation
report
Read
operation
report
produces
produces
Framework for evaluating the performance of different NoSQL databases

19. Tesi di Laurea Magistrale – Fabio Arcidiacono
19
Environment setup
YCSB
+
YCSB adapters
+
Kundera GAE
Datastore client Datastore
YCSB
+
YCSB adapters
+
Kundera Azure
Tables client
Azure
Tables
4 core
7 GB RAM
4 core
3.6 GB RAM

20. Tesi di Laurea Magistrale – Fabio Arcidiacono
Results comparison
20
Azure Tables
Read latency Read throughput Write latency Write throughput
Kundera 42,44 ms 689,67 ops/sec 40,701 ms 707,12 ops/sec
low-level API 36,74 ms 787,22 ops/sec 38,809 ms 758,54 ops/sec
overhead 13,43 % 12,39 % 4,75 % 6,78 %
Google Datastore
Read latency Read throughput Write latency Write throughput
Kundera 139,13 ms 212,74 ops/sec 151,159 ms 194,64 ops/sec
low-level API 132,36 ms 222,5 ops/sec 150,018 ms 198,67 ops/sec
overhead 4,36 % 4,39 % 0,76 % 2,03 %

21. Tesi di Laurea Magistrale – Fabio Arcidiacono
Conclusions
21
Contributions:
● Integration of Kundera in CPIM library
● New Kundera clients to support Google Datastore and Azure Tables
● Hegira integration in the CPIM library
Future work:
● Compare developed client performance with the ones of the other
client developed by Kundera team

22. Tesi di Laurea Magistrale – Fabio Arcidiacono
THANK YOU
42