SlideShare una empresa de Scribd logo
1 de 78
Descargar para leer sin conexión
Page 1Gunther Pippèrr © 2014 http://www.pipperr.de
ORACLE NOSQL – A NEW ALTERNATIVE
FOR THE ORACLE RDBMS DATABASE?
Just another Key/Values store?
Java Meeting – 10/04/2014
Page 2Gunther Pippèrr © 2014 http://www.pipperr.de
Gunther Pippèrr – IT-Architect – Consultant
• High-Tech
• Real Estate
• Utility
• Communications
• IT Systems Architect
• Technical project manager
• Design and implementation
of database applications
• Design and implementation
of IT infrastructures for data
management
Gunther Pippèrr is working intensively for more than 15 years with
the products of Oracle in database / application server and
document management.
Mr. Pippèrr has deep knowledge about the structure of complex IT
architecture and successfully implemented this in practice.
Mr. Pippèrr has a degree in computer science as Dipl. Ing.
Technische Informatik (FH) - FH Weingarten.
Software manufacture telecommunication
- Architect, database design and implementation of a mass data
processing system for communication data
German technology group
- Architect and project management, database design and
implementation of an order management for control of external
contractors for the language service provider of a German technology
group.
Utility Consulting
- Architect and technical project responsibility for a smart metering
portal for meter data collection and asset management
Architect and technical project responsibility for IT infrastructure
projects, such as:
- Central data storage for Munich hotel group with over 25
hotels worldwide
- Data collector for monitoring Russian cable operator
- Redundant database cluster infrastructure for various Web
applications such as insurance portals , fond platforms, loyalty cards
Huge Munich building contractor
- CRM and tendering portal
Industry Expertise
Background
Functional Expertise
Selected Experience
• German: http://www.pipperr.de
• English: http://orapowershell.codeplex.com
Web
Page 3Gunther Pippèrr © 2014 http://www.pipperr.de
LinkedIn Contact
https://www.linkedin.com/pub/gunther-pipp%C3%A8rr/5b/623/23
Page 4Gunther Pippèrr © 2014 http://www.pipperr.de
Use Case
Our challenge to solve
Page 5Gunther Pippèrr © 2014 http://www.pipperr.de
Our Use Case – Internet Simple Password Store
 We like to build:
– A secure password store for identity credentials
App
REST
Interface
PWD Save
User Id PWD Name Username
Password
URL
Access Token URL
PWD Name
PWD Name
Value
Value
Value
Value
Key
Mayor Minor
Page 6Gunther Pippèrr © 2014 http://www.pipperr.de
How we can store the Data?
Performance, Security, Availability
Page 7Gunther Pippèrr © 2014 http://www.pipperr.de
OracleRDBMSOracleNoSQL
Shared Everything
Shared Nothing
Instance
SQL*Net
Central Storage
Storage
Node
Storage
Node
Storage
Node
Distributed Storage
Client
SQL
Client
Java
API
RMI
Page 8Gunther Pippèrr © 2014 http://www.pipperr.de
Use for each Problem the right tool
 NoSQL – General Concept
– More a approach for data management and processing then a
special technologies
 Basic Concept
– Distributed, scalable solutions (Scale-out Concept)
• CAP Theorem – Consistency – Availability - Partition Tolerance
• BASE - Basically Available, Soft-state, Eventually
versus
ACID – Atomicity, Consistency, Isolation, Durability
– Highly specialized solutions for exactly one problem with the
handling of very large or very quickly required data
• Dozens of solutions available in the market
Page 9Gunther Pippèrr © 2014 http://www.pipperr.de
NOSQL – Definition
 A real NoSQL database is:
– none relational
– Schema free
– distributed multi-node architecture
• Goal: a shared nothing cluster solution
– Simple replications mechanism
– A easy API ( Not only SQL = NoSQL)
– Not necessarily a ACID Consistency Model
– Open Source
But – the major disadvantage
=> Usually no easy query language
=> Complex joins are not so easy to implement
Page 10Gunther Pippèrr © 2014 http://www.pipperr.de
Classification Oracle NoSQL in the NoSQL World
 Key-Value
– One key points to a set of data
• Oracle
 Document oriented
– All data of a data set is organized in one document
• Lotus Notes
 Graph
– networked structures
• Neo4J
Page 11Gunther Pippèrr © 2014 http://www.pipperr.de
Idea - Key-Value store database
– Oracle NoSQL – a pure Key Value Store
• A distributed and scalable high available database
• Consistent hashing approach
Key Value
Key Value
Key Value
Key Value
Data store is distributed over all nodes
Intelligent driver
• Implementation of
the Hash
Mechanisms
• Knows the status of
the store
• Read the partition
Map at the first
connect to the store
Client
Software
Key Value
Key Value
Key Value
Key Value
Key Value
Key Value
Key Value
Key Value
One ACID Transaction is possible for all records of a mayor key with one Single API call.
Page 12Gunther Pippèrr © 2014 http://www.pipperr.de
The Mayor –Minor Key Concept
 Key – Composed of two components
Data
Key
Mayor Minor
String Byte Array
Partitioning over
the mayor key
Example:
User identifier
Password Name / User Name
Access Token / URL
Value user name
Token value
The structure of the data is NOT self-describing
Page 13Gunther Pippèrr © 2014 http://www.pipperr.de
Implementation
 Key = Hash Mayor Key => Partition Mapping
– When you create the store in the first step a fixed count of
partitions will be created
• Each Partition corresponds essentially to his own Berkeley DB
„Database“
• The Hash of the key determine the partition
• The partition will be stored redundant over the store
Client Driver : Hashing the ID
Store over n Storage Nodes
Partition 1
Partition 2
Partition 3
Partition …
Partition N
Replication Group
Replication Group
Replication Group
MD5(major key) % n Partitions
→Partition ID
Replication Group =
Master + Replicates
Page 14Gunther Pippèrr © 2014 http://www.pipperr.de
Query the store
 Java API
– Alternative C API (Enterprise Feature!)
– Command Line Interface for maintenance
 The key and ONLY the Key can be queried
– For a search in the values all the data in the store have to be
read
– For a count you have to iterate over all values
Page 15Gunther Pippèrr © 2014 http://www.pipperr.de
Variants / versions of the Oracle NoSQL
 CC – Community Edition
– Open Source Code
• Open Source with the GNU Affero General Public License
– Support is possible on a annual basis
 EE – Enterprise Edition
– Full support
– Support of Oracle external tables - exchange data with the
normal Oracle database
– Oracle Event Processing
– RDF Adapter
– SNMP Monitoring
– Jena Adapter (Graph/SPARQL)
– Since Version 3 - Security options with the Oracle Wallet
Page 16Gunther Pippèrr © 2014 http://www.pipperr.de
Integration in the Hadoop Ecosystem
Hadoop Map Reduce Read the Data from the NoSQL Store
Hadoop File system
Store results
11g
12c
Read as External Table
pure Java
Hive
Read as External Table
Only EE Edition of the Oracle NoSQL!
Page 17Gunther Pippèrr © 2014 http://www.pipperr.de
Evolution form Key-Value / Avro to table
 Release 1 – Key Value
– Pure Key-Value concept
– No „Data Dictionary“ – Entire logic in the application
 Release 2 – Avro
– Schema Definition with Apache Avro Standard
– Declaration of the structure of the Key-Value Data inside the
database AND in the application necessary
 Release 3 – Tables
– Structure definition as table
• A secondary index on columns of the table is possible
Value
Page 18Gunther Pippèrr © 2014 http://www.pipperr.de
The architecture of the Oracle
NoSQL database in detail
A distributed key value store
Page 19Gunther Pippèrr © 2014 http://www.pipperr.de
The store = the database
 Store – all nodes together over all storage nodes
Server 1
Storage
Node 1
Server 2
Storage
Node 2
Server 4
Storage
Node 4
Server 3
Storage
Node 3
Example of a store with a replication factor of 3
1 1 13332 2 24 44
Master 1
Replika sn1 Replika sn1
The replication factor is defined as „master + count replicates“.
In our example one master and two replicates = replication factor 3
Zone = one store node environment
Master Master Master
Replication Group =
Master + Replicate
Shared A Shared D
Zone
Page 20Gunther Pippèrr © 2014 http://www.pipperr.de
Overview
 Depending on the replications factor the data will
duplicated over the store
 The count of partitions will be distributed over all master
nodes
 A topology describes the whole store
 Example– 4 nodes - replication factor 3 – 800 partitions
– One Master per node + two replica per storage node
– 800 partitions distributed over the four master Nodes
• 200 for each master
– Inserts only over the master node
• The Master Node distribute the data to the replicas
– It is possible to read the data from each node
The count of partition can not be changed after the create of the store!
Page 21Gunther Pippèrr © 2014 http://www.pipperr.de
The distribution of the data in the store
 The client API hashes the key und distributes the Key
over the count of partitions
Java Code
Java API
Key
7
B1A1
1
2 8
1
1
D1C1
9 1
0
1
2
88 B2B3
Insert
The hash of the key defines the partition
AND
The hash defines the master node holds
the appropriate partition.
Page 22Gunther Pippèrr © 2014 http://www.pipperr.de
The distribution of the data in the store
 Read is possible from each replicate
Java Code
Java API
Key
read
7
B1A1
1
2 8
1
1
D1C1
9 1
0
1
2
88 B2B3
The hash of the key defines the partition
AND
The hash defines the master node, holds
the appropriate partition.
Page 23Gunther Pippèrr © 2014 http://www.pipperr.de
Internal management
 Admin Service for the internal management
– Separate admin database
– Separate port
 The Admin Service is responsible for implementing of
the Master – Slave concept
– If one Node fails a new master will be defined
Page 24Gunther Pippèrr © 2014 http://www.pipperr.de
Consistency => all nodes see the same data at the same time
Availability => a guarantee that every request receives a response about whether it was successful or failed
Partition tolerance => the system continues to operate despite arbitrary message loss or failure of part of the system
Availability
 What happens in case of a failure of a storage node?
– If the Master is unavailable – it can now not longer be written
to the store?
• Admin Process supervise the whole store
– Has his own Admin Database
– Recognizes errors
– Select from the remaining Replicas the replica with the highest LSN (log
sequence number) for a new master
If the developer decide to use ReplicaAckPolicy.ALL you will get an error message if a node fails!
The developer has to choose :
• Consistency and Availabitlity
OR
• Availabitlity and Partition Tolerance
Page 25Gunther Pippèrr © 2014 http://www.pipperr.de
The engine (1)
 The Java Version of the Berkeley DB
– JE Version 5.0.83 - NoSQL 2.2.18
– JE Version 6.0.14 - NoSQL 3.0.14
– Each partition is a database
– Berkeley DB Replication for the master slave concept
– The DB files can be analyzed with the Berkeley DB classes
Tipp:
You can configure the original Berkeley DB Mechanism per Storage Node
See: http://www.pipperr.de/dokuwiki/doku.php?id=nosql:log_file_verhalten_oracle_nosql_db_11gr2
Show the version with:
java -classpath ".libkvclient.jar;.libkvstore.jar;.libje.jar" com.sleepycat.je.util.DbVerify -V
Page 26Gunther Pippèrr © 2014 http://www.pipperr.de
The engine (2)
 Particularity of the Berkeley DB:
• Redo Log entries and the data are stored to the same „data“ file
• Background Jobs take care about the cleaning of the unnecessary
entries
je.log.bufferSize
Minimal Size of the log
je.log.fileMax
Maximal Size of the log
je.cleaner.minUtilization
% must be used of this file
je.cleaner.minFileUtilization
Clean Logfile, if this value
is reached
je.cleaner.bytesInterval
Start cleaning if x Bytes written
je.cleaner.minAge
Only clean if x files are older then the latest
je.cleaner.threads
Number of Cleaner Threads
Page 27Gunther Pippèrr © 2014 http://www.pipperr.de
Example – Analyze the Data files
 Read the size of the DB with the Berkeley classes:
– Ensure the correct class path to the jar files
• export KVLIB=/opt/oracle/produkt/11.2.0/kv-2.1.8/lib
• export
KVCLASS=$KVLIB/kvclient.jar:$KVLIB/kvstore.jar:$KVLIB/je.jar
– Path to the data files
• export JEENV=/opt/oracle/kvdata/GPIDB/sn1/rg1-rn1/env/
– Analyze with:
java -classpath $KVCLASS com.sleepycat.je.util.DbSpace -h $JEENV
File Size (KB) % Used
-------- --------- ------
00000000 110808 21
TOTALS 110808 21
(LN size correction factor: NaN)
Page 28Gunther Pippèrr © 2014 http://www.pipperr.de
Installation
Unzip Java archive, create directory ,
ready?
Download Software from :
http://www.oracle.com/technetwork/database/database-technologies/nosqldb/overview/index.html
Page 29Gunther Pippèrr © 2014 http://www.pipperr.de
The planning of a NoSQL Installation - Network
 Network Infrastructure
– Physic
• High performante internal communication between the nodes
necessary
– pay attention for low latencies in the network
» Keyword Infiband
» Own 10Gbit Network?
– Define TCP Ports
• For the internal communication of the Storage nodes
• For the communication of the Clients with the Master nodes
– One Port for external connection + one Port for the Admin Console +
ServicePortRange for the RMI Connection from the client
– Maintainability
• SSH between the den Storage Nodes
– Script for maintenance over many nodes
Define some company standard , with the time you will get a lot of environments!
Page 30Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL installation - Ports
 Necessary ports - Network Infrastructure
Ports for one Storage Node
5000
Application
Admin Client
-port
5001
HTTP Admin Client
-admin
5010-5020
Replication
-harange
…
-servicerange
RMI Port for the
Client communication
5021-5040
…
Parameter when equipped with the "makebootconfig"
haPortRange servicePortRange
Parameter for changing with the "change-policy -params "
registry portApplication
Page 31Gunther Pippèrr © 2014 http://www.pipperr.de
Oracle NoSQL Client Connect
 Access from the Client to the „registry port“
– For example Port 5000 on node A
 The query is take through over the Java RMI protocol
on the auf den respective node
– Define enough remote ports!
-servicerange
-port
Node A
Node B
Node C
Client
Port 5000
ServiceRange
Port 5021
ServiceRange
Port 5021
ServiceRange
Port 5021
Firewall!
Page 32Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL Installation - Hardware
 CPU and Memory
– Cache Size can be adjusted from 2 until n GB
• Watch carefully GC effects of Java
– Oracle NoSQL is explicit optimized for clean, small objects sizes to not
overwhelm the Java garbage collector
 Storage Infrastructure
– One Store Node can server also more then one Master und
replicates
• If you have more then one Storage node on a server you should use
separate disks/luns to have best I/O on the server
– In the Background the “Cleaner” processes need some I/O resources
Page 33Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL Installation – Disk space
 Disk space
– Calculation is not entirely trivial
• The Berkeley DB store all actions like the redo data in the same file
• Not until some Thresholds are reached the data files will be optimized
• => The delete of data enlarge the store in the first step!
Rule of thumb:
Pro Master/ Replika => Count of Transactions (Insert/Update/Delete) * (Net Data Size + Overhead)
=> Filling of the data files per Default optimized to 40%
Page 34Gunther Pippèrr © 2014 http://www.pipperr.de
Installation
 Preparation on each Node
– Java JDK installation and create Store User
– Setup SSH Connection
– Create the KVROOT (Data Storages!)
 Oracle NoSQL installation
– Extract the code – create the KVHOME
– Create the Store Node Base directory (KVROOT)
– Create the Boot configuration
– Start the Admin Node
– Define a plan for the new store
• Define the database topologies
– Rollout the plan
– test
Per Script: 5min
Page 35Gunther Pippèrr © 2014 http://www.pipperr.de
Install our own local test system
 Download from oracle.com and extract the zip file to a
directory
 Create a local store and start the NoSQL Database
 Start a admin Shell
REM go to your DB directory
cd R:KVSTORE
REM set the libraries Path
set KVHOME=D:kv-3.0.14
REM Create the Store
java -jar %KVHOME%libkvstore.jar kvlite -root R:KVSTORE
-store kvstore -host localhost -port 5000 -admin 5001
java -jar %KVHOME%libkvstore.jar runadmin -port 5000 -host localhost
kv-> ping
… Pinging components of store kvstore based upon topology sequence #14
… Rep Node [rg1-rn1] Status: RUNNING,MASTER at sequence number: 39 ha
Page 36Gunther Pippèrr © 2014 http://www.pipperr.de
Work with the NoSQL Store
Maintenance and operation
Page 37Gunther Pippèrr © 2014 http://www.pipperr.de
Maintenance (1) – command Line
 Kv – For the Store Administration
 kvshell – Select / insert data
kvshell-> put -key "/Gunther/name" -value "Pipperr"
Operation successful, record inserted.
kvshell-> get -key "/Gunther/name"
Pipperr
Page 38Gunther Pippèrr © 2014 http://www.pipperr.de
Maintenance (2) – Web interface
– Web interface based on a internal jetty server
http://nosqldb03:5005/
Page 39Gunther Pippèrr © 2014 http://www.pipperr.de
JMX Interface
 Start JMC – Java Mission Control
– JAVA_HOME/bin/jmc.exe
– Connect over the „registry port“
Page 41Gunther Pippèrr © 2014 http://www.pipperr.de
Reliability
Availability and Backup
Page 42Gunther Pippèrr © 2014 http://www.pipperr.de
Backup per Snapshot possible
 Backup:
– A complete Store can be saved consistent per Snapshot
– pay attention for enough disk space!
 Restore
– A Store can be completely reset
• Create a Store snapshot
• Stop the Store
• Provide the Snapshot (Link in the File system for example)
• Start the Store, Storage Nodes read die Data, create the data again
and DELETE the snapshot!
– Use as a kind of import
• The generated files can be read from an other store
Page 43Gunther Pippèrr © 2014 http://www.pipperr.de
Performance
I/O and Network
Page 44Gunther Pippèrr © 2014 http://www.pipperr.de
Performance considerations
 read
– Network
• All records must be read over the network
 write
– I/O
• Background process need enough recourse for the optimization of the
data files!
ALL record are processed on the client side!
A simple Count(*) have to read all data!
Page 45Gunther Pippèrr © 2014 http://www.pipperr.de
Security?
Oracle NoSQL and Security?
Page 46Gunther Pippèrr © 2014 http://www.pipperr.de
Security?
 Version 2 ?
 The developer has to care about it – not implemented
 Since Version 3
– SSL encoding
– Password security
• But Wallet usage is a EE feature!
Page 47Gunther Pippèrr © 2014 http://www.pipperr.de
Read and write to the store
with Java
No SQL necessary – all is done in Java
Page 48Gunther Pippèrr © 2014 http://www.pipperr.de
Client
Software
Client Driver : Hashing der ID
Store over n Storage Nodes
Partition 1
Partition 2
Partition 3
Partition …
Partition N
Replication Group
Replication Group
Replication Group
MD5(major key) % n Partitions
→Partition ID
Replication Group =
Master + Replication Node
Key Value
Key Value
Partition …
Intelligent driver
• Implementation of
the Hash
Mechanisms
• Knows the status of
the store
• Read the partition
Map at the first
connect to the store
Overview
Page 49Gunther Pippèrr © 2014 http://www.pipperr.de
First – Connect to the store (1)
 Create the connection to the store
• Set the store properties with the class KVStoreConfig
Store Name
// Node liste
String[] kvhosts = {"nosqldb01:5000","nosqldb02:5000","nosqldb03:5000"}
// Config erzeugen
KVStoreConfig kvconfig = new KVStoreConfig("GPIDB", kvhosts);
Port
Node
Page 50Gunther Pippèrr © 2014 http://www.pipperr.de
First – Connect to the store (2)
 Create the connection to the store
• Define the Read and Write Rules
// Define Consistency
kvconfig.setConsistency(Consistency.NONE_REQUIRED);
// Define Durabiltiy
kvconfig.setDurability(Durability.COMMIT_NO_SYNC);
// Connect to the store
KVStore kstore = KVStoreFactory.getStore(kvconfig);
Rule read
Rules write
Open connection to the
Store of a Factory
Do not forget to close the Store!
kstore.close();
Page 51Gunther Pippèrr © 2014 http://www.pipperr.de
First Example - Write the data
 Address the DB object over the key
 Create the key
 Write the data
Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME");
Mayor Key
Minor Key
// Wert als Byte Array anlegen
byte[] b = ("gpipperr@googelmail.com").getBytes();
// Wert in den Store schreiben
kvstore.put(k, Value.createValue(b));
Write the data to the store
Page 52Gunther Pippèrr © 2014 http://www.pipperr.de
First Example - Read the data over the key
 Create the key
 Read data
Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME");
Mayor Key
Minor Key
// Read the Data from the Store with the key
ValueVersion vv = kvstore.get(k);
// Read the Daten Value
Value vdata = vv.getValue();
// Recreate the Values
String username= new String(vdata.getValue());
// Work with the data
System.out.println(username);
read
Page 53Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 54Gunther Pippèrr © 2014 http://www.pipperr.de
TRANSACTION CONCEPTS
Are we reading the last version of data?
Page 55Gunther Pippèrr © 2014 http://www.pipperr.de
The consistency of the data – Transactionality
 In one session
transactionality for write one
mayor key is possible
 The developer has to decide
when a record is apply as
written!
read
 The developer has to decide
if the read record is actual
write
FastSave
Durability Consistency
Performance Performance
Latest Version first available
answer
On all nodes
to disk
On one node
to the cache
FastSave
Page 56Gunther Pippèrr © 2014 http://www.pipperr.de
The write consistency in the Store - Durability
 The developer is responsible for the write consistency!
 Will be defined at the connect to the store
– Class oracle.kv.Durability
• Commit-Policy for the Master
• Commit-Policy for the replicas
• Replica Acknowledgement-Policy
FastSave
Performance
On all nodes
to disk
On one node
to the cache
Page 57Gunther Pippèrr © 2014 http://www.pipperr.de
Durability Policies
 Commit-Policy for the Master and the replicates
– SyncPolicy.SYNC
• Wait until the Storage Node write the transaction to the log file and
synchronized to the disk
– SyncPolicy.WRITE_NO_SYNC
• Wait until the Storage Node write the transaction to the log file in the
cache (not written to disk!)
– SyncPolicy.NO_SYNC
• Don’t wait
 Replica Acknowledgement
– ReplicaAckPolicy.ALL
• Wait until all replicates confirm, that all transaction has been written
– ReplicaAckPolicy.SIMPLE_MAJORITY
• Wait until the simple majority the replicates confirm the transaction
– ReplicaAckPolicy.NONE
• Don’t wait for the answers of the replicates
Page 58Gunther Pippèrr © 2014 http://www.pipperr.de
The read consistency in the store - Consistency
 The developer is responsible for the read consistency!
 Each key object owns a version information
– A kind of transaction ID
 The behavior will be defined with the connect to the
store
– read – Class oracle.kv.Consistency
Performance
FastSave
Latest Version first available
answer
Page 59Gunther Pippèrr © 2014 http://www.pipperr.de
Consistency
 Consistency.ABSOLUTE
– The session can read only from the master for this key.
– This ensures that the data is relay valid
 Consistency.Time
– The session can read from a replica
– With the Parameter you can define how „old“ the data can be
to accepted as valid data
 Consistency.Version
– The session can read from a replica
– The version information will be used instead of the time
 Consistency.NONE_REQUIRED
– Very result is valid, can be any old
Page 60Gunther Pippèrr © 2014 http://www.pipperr.de
THE TABLE IS BACK
Structured data access
Page 61Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1)
 Our Key Model
 Our Table Model
User Id PWD Name Username
Password
URL
Access Token URL
PWD Name
PWD Name
Value
Value
Value
Value
USER_ID | PWD_NAME | USER_NAME | PASSWORD| URL
USER_ID | ACESS_TOCKEN | URL | Value
Table USER_PASSWORDS
Table USER_ACESS_TOCKENS
Page 62Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1.1)
 Create a table definition in the Store with the client
utility
set KVHOME=D:entwicklunglibrarieskv-ee-3.0.14
java -jar %KVHOME%libkvcli.jar -host localhost -port 5000
-store kvstore
#Create table
table create -name USER_PASSWORDS
# Add table fields
add-field -name USER_ID -type STRING
add-field -name PWD_NAME -type STRING
add-field -name USER_NAME -type STRING
add-field -name PASSWORD -type STRING
add-field -name URL -type STRING
Page 63Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1.2)
# Add the primary Key
primary-key -field USER_ID
#Leave table Mode
exit
#create the Table in the Store
plan add-table -name USER_PASSWORDS -wait
#check the Table Definition
show table -name USER_PASSWORDS
#insert first Record of data in to the table with the client tools
put table -name USER_PASSWORDS -json
"{"USER_ID":"gunther@pipperr.de","PWD_NAME":"google.de","USER_NA
ME":"gpipperr","PASSWORD":"Password01","URL":"www.google.de"}"
#create index on the table
plan add-index -name IDX_USER_PASSWORDS -table
USER_PASSWORDS -field USER_ID –wait
#read the data
get table -name USER_PASSWORDS -index IDX_USER_PASSWORDS
Page 64Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1)
 Write the source code to use the table defintion
// get the Table API
TableAPI tableAPI = kvstore.getTableAPI();
//get a reference to the table
Table empTable = tableAPI.getTable("USER_PASSWORDS");
// Get a Row instance
Row empRow = empTable.createRow();
// put data in the row
empRow.put("empno", 10);
// write the data to the store
tableAPI.put(empRow, null, null);
Page 65Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (2)
 Read from this table again
// get the Table API
TableAPI tableAPI = kvstore.getTableAPI();
//get a reference to the table
Table empTable = tableAPI.getTable("emp");
//Create a primary key and assign the field value
PrimaryKey key = table.createPrimaryKey();
key.put("userID", 1);
//Get the matching row
Row row = tableAPI.get(key, new ReadOptions(null, 0, null));
if (row == null) {
throw new Error("No matching row found");
}
//Print the full row as JSON
System.out.println(row.toJsonString(true));
//Access a specific field
System.out.println("firstName field as JSON: " +
row.get("firstName").toJsonString(false));
Page 66Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 67Gunther Pippèrr © 2014 http://www.pipperr.de
INTEGRATION INTO THE HADOOP DATA
WAREHOUSE
Archive your data
Page 68Gunther Pippèrr © 2014 http://www.pipperr.de
Oracle NoSQL – MapReduce Hadoop
 Traditional MapReduce Pattern
 Input Format Class – KVInputFormat.class
Mapper
Mapper
Mapper
Mapper
Reducer HDFSOracle NoSQL
Store
KVInputFormat
oracle.kv.Key.toString()
lib/kvclient.jar must in the class path!
Like:
hadoop jar secure-pwd-store.jar gpi.secstore.PWDStoreHadoop
-libjars $KVHOME/lib/kvclient.jar
/tmp/nosqlTest2
Error: java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
oracle.kv.hadoop.KVInputFormat not found at
Better solution -put all in one jar fie
Page 69Gunther Pippèrr © 2014 http://www.pipperr.de
Main Part – Job Configuration
 Define the Input Format Class in the Job Configuratoin
// set the Input Format Classe
job.setInputFormatClass(KVInputFormat.class);
//Parameter for the Input Format Class
KVInputFormat.setKVStoreName("KVStore");
String [] kvhostList = {"StoreDB01:5000", "StoreDB02:5000",};
KVInputFormat.setKVHelperHosts(kvhostList);
Set the Input Class for the
MapReduce Job
Define the connection to the Store
Default: Key and Value are read as Text Format
Page 70Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 71Gunther Pippèrr © 2014 http://www.pipperr.de
Conclusion
Are the benefits worth the effort?
Page 72Gunther Pippèrr © 2014 http://www.pipperr.de
Why we should use the Oracle NoSQL? (1)
 What is really important for a strategic development?
– Maintainability
• Is the product easy to use?
– Long-term strategy of the manufacturer
• Is a product cycle from 4-5 year possible?
– Release and Update Problematic
• Java 8 and higher? Linux 7?
Page 73Gunther Pippèrr © 2014 http://www.pipperr.de
Why we should use the Oracle NoSQL? (2)
 What is for a strategic development really important?
– Fits the base concept to my technical needs?
• Are my requirements changing with the time?
– Costs
Oracle promises this:
– Version 3 ready
– The Software Berkeley DB since year in use
Page 74Gunther Pippèrr © 2014 http://www.pipperr.de
Use Cases
 Use Cases
– Personalization of web pages
– Authentication
– Stream Processing
• machine data
Goal:
Get for one Key fast as possible the results
Requirement: Provide large amounts of data with low latency – Persistent Cache
Page 75Gunther Pippèrr © 2014 http://www.pipperr.de
Case Study: Web Application
 In a Web application each user can have a very high
number of different "bonus offers, vouchers"
 This bonus offers are calculated individually according
to the profiles of a user on a weekly base
– The NoSQL Store holds the UserID as key with a list of all
metadata for the bonus offers
• Avoid complex joins over the DWH with the user profiles
• Immediate display results on the smart phone or in the Web without a
time delay – very low latencies
Requirement: Provide large amounts of data with low latency – Persistent Cache
Page 76Gunther Pippèrr © 2014 http://www.pipperr.de
Architecture of a NoSQL Stores
3 Storage Nodes (SN) on three servers
with a Replication Factor of three = three Replication Groups (RG)
Server 1
Storage Node 1
SN 1
Server 2
Storage Node 2
SN 2
Server 3
Storage Node 3
SN 3
Rg1-
n1
Master 1
Replication Node
Replica 1
Replication Node
Replica 2
Replication Node
Shared A
Store
Shared B Shared C
Zone A
Admin Srv Admin Srv Admin Srv
Storage Node Agent (SNA) Storage Node Agent (SNA) Storage Node Agent (SNA)
Rg2-
n1
Rg3-
n1
Rg1-
n2
Rg2-
n2
Rg3-
n2
Rg1-
n3
Rg2-
n3
Rg3-
n3
Page 77Gunther Pippèrr © 2014 http://www.pipperr.de
Direct Comparison Oracle RDBMS NoSQL
 ~35 years of development
 Powerful SQL API
 Joins
 Constraints
 Many Features
 OLTP
 Shared anything approach
 A general all purpose
weapon
Oracle NoSQL
 ~2-3 years
 Simple Java API
 Joins only over the
application
 ?
 Pure data store
 Mostly only read operation
 Shared Nothing Cluster
 Specialized niche solution
Oracle RDBMS
Page 78Gunther Pippèrr © 2014 http://www.pipperr.de
Fragen
Oracle NoSQL
NO:SQL
Page 79Gunther Pippèrr © 2014 http://www.pipperr.de
Sources
 Oracle - ORACLE
– English
http://www.oracle.com/technetwork/database/database-
technologies/nosqldb/documentation/index.html
– German
http://www.oracle.com/webfolder/technetwork/de/community/dojo/index.ht
ml
 See also:
– German http://www.pipperr.de/dokuwiki/doku.php?id=nosql_datenbank

Más contenido relacionado

La actualidad más candente

MySQL High-Availability and Scale-Out architectures
MySQL High-Availability and Scale-Out architecturesMySQL High-Availability and Scale-Out architectures
MySQL High-Availability and Scale-Out architectures
FromDual GmbH
 
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
Yahoo Developer Network
 
High Availability for HBase Tables - Past, Present, and Future
High Availability for HBase Tables - Past, Present, and FutureHigh Availability for HBase Tables - Past, Present, and Future
High Availability for HBase Tables - Past, Present, and Future
DataWorks Summit
 

La actualidad más candente (20)

Maintainable cloud architecture_of_hadoop
Maintainable cloud architecture_of_hadoopMaintainable cloud architecture_of_hadoop
Maintainable cloud architecture_of_hadoop
 
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in HadoopKudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL
 
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...
 
Apache ignite Datagrid
Apache ignite DatagridApache ignite Datagrid
Apache ignite Datagrid
 
Introduction to Cassandra and CQL for Java developers
Introduction to Cassandra and CQL for Java developersIntroduction to Cassandra and CQL for Java developers
Introduction to Cassandra and CQL for Java developers
 
Announcing Amazon EC2 F1 Instances with Custom FPGAs
Announcing Amazon EC2 F1 Instances with Custom FPGAsAnnouncing Amazon EC2 F1 Instances with Custom FPGAs
Announcing Amazon EC2 F1 Instances with Custom FPGAs
 
MySQL High-Availability and Scale-Out architectures
MySQL High-Availability and Scale-Out architecturesMySQL High-Availability and Scale-Out architectures
MySQL High-Availability and Scale-Out architectures
 
Introduction to Apache Kudu
Introduction to Apache KuduIntroduction to Apache Kudu
Introduction to Apache Kudu
 
Kudu - Fast Analytics on Fast Data
Kudu - Fast Analytics on Fast DataKudu - Fast Analytics on Fast Data
Kudu - Fast Analytics on Fast Data
 
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
 
Ozone - Evolution of hdfs scalability
Ozone - Evolution of hdfs scalabilityOzone - Evolution of hdfs scalability
Ozone - Evolution of hdfs scalability
 
HBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and SparkHBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and Spark
 
HBaseCon 2015 General Session: State of HBase
HBaseCon 2015 General Session: State of HBaseHBaseCon 2015 General Session: State of HBase
HBaseCon 2015 General Session: State of HBase
 
HBaseCon 2015 General Session: Zen - A Graph Data Model on HBase
HBaseCon 2015 General Session: Zen - A Graph Data Model on HBaseHBaseCon 2015 General Session: Zen - A Graph Data Model on HBase
HBaseCon 2015 General Session: Zen - A Graph Data Model on HBase
 
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
 
High Availability for HBase Tables - Past, Present, and Future
High Availability for HBase Tables - Past, Present, and FutureHigh Availability for HBase Tables - Past, Present, and Future
High Availability for HBase Tables - Past, Present, and Future
 
dplyr Interfaces to Large-Scale Data
dplyr Interfaces to Large-Scale Datadplyr Interfaces to Large-Scale Data
dplyr Interfaces to Large-Scale Data
 
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
 

Similar a Oracle nosql twjug-oktober-2014_taiwan_print_v01

Scalable Deployment Patterns in WSO2 API Manager
Scalable Deployment Patterns in WSO2 API Manager Scalable Deployment Patterns in WSO2 API Manager
Scalable Deployment Patterns in WSO2 API Manager
WSO2
 

Similar a Oracle nosql twjug-oktober-2014_taiwan_print_v01 (20)

The Analytic Platform behind IBM’s Watson Data Platform by Luciano Resende a...
 The Analytic Platform behind IBM’s Watson Data Platform by Luciano Resende a... The Analytic Platform behind IBM’s Watson Data Platform by Luciano Resende a...
The Analytic Platform behind IBM’s Watson Data Platform by Luciano Resende a...
 
Ceph on 64-bit ARM with X-Gene
Ceph on 64-bit ARM with X-GeneCeph on 64-bit ARM with X-Gene
Ceph on 64-bit ARM with X-Gene
 
Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...
Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...
Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...
 
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
 
HPE Solutions for Challenges in AI and Big Data
HPE Solutions for Challenges in AI and Big DataHPE Solutions for Challenges in AI and Big Data
HPE Solutions for Challenges in AI and Big Data
 
Saviak lviv ai-2019-e-mail (1)
Saviak lviv ai-2019-e-mail (1)Saviak lviv ai-2019-e-mail (1)
Saviak lviv ai-2019-e-mail (1)
 
IBM: The Linux Ecosystem
IBM: The Linux EcosystemIBM: The Linux Ecosystem
IBM: The Linux Ecosystem
 
Scalable Deployment Patterns in WSO2 API Manager
Scalable Deployment Patterns in WSO2 API Manager Scalable Deployment Patterns in WSO2 API Manager
Scalable Deployment Patterns in WSO2 API Manager
 
Se training storage grid webscale technical overview
Se training   storage grid webscale technical overviewSe training   storage grid webscale technical overview
Se training storage grid webscale technical overview
 
Big SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor LandscapeBig SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor Landscape
 
Oracle NoSQL Database -- Big Data Bellevue Meetup - 02-18-15
Oracle NoSQL Database -- Big Data Bellevue Meetup - 02-18-15Oracle NoSQL Database -- Big Data Bellevue Meetup - 02-18-15
Oracle NoSQL Database -- Big Data Bellevue Meetup - 02-18-15
 
Pivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream AnalyticsPivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream Analytics
 
New Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference ArchitecturesNew Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference Architectures
 
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
 
Inter connect2016 yps-2749_02232016_aspresented
Inter connect2016 yps-2749_02232016_aspresentedInter connect2016 yps-2749_02232016_aspresented
Inter connect2016 yps-2749_02232016_aspresented
 
Konsolidace Oracle DB na systémech s procesory M7
Konsolidace Oracle DB na systémech s procesory M7Konsolidace Oracle DB na systémech s procesory M7
Konsolidace Oracle DB na systémech s procesory M7
 
Geode Meetup Apachecon
Geode Meetup ApacheconGeode Meetup Apachecon
Geode Meetup Apachecon
 
Oracle big data appliance and solutions
Oracle big data appliance and solutionsOracle big data appliance and solutions
Oracle big data appliance and solutions
 
Cloud-Native .Net des applications containerisées .Net sur Linux, Windows e...
 Cloud-Native .Net des applications containerisées .Net sur Linux, Windows e... Cloud-Native .Net des applications containerisées .Net sur Linux, Windows e...
Cloud-Native .Net des applications containerisées .Net sur Linux, Windows e...
 
NetApp Se training storage grid webscale technical overview
NetApp Se training   storage grid webscale technical overviewNetApp Se training   storage grid webscale technical overview
NetApp Se training storage grid webscale technical overview
 

Más de Gunther Pippèrr

Oracle no sql-doag-datenbank_konferenz_juni_2014
Oracle no sql-doag-datenbank_konferenz_juni_2014Oracle no sql-doag-datenbank_konferenz_juni_2014
Oracle no sql-doag-datenbank_konferenz_juni_2014
Gunther Pippèrr
 

Más de Gunther Pippèrr (12)

DBSAT – Die Oracle DATENBANK bzgl. PII Daten analysieren
DBSAT – Die Oracle DATENBANK bzgl. PII Daten analysierenDBSAT – Die Oracle DATENBANK bzgl. PII Daten analysieren
DBSAT – Die Oracle DATENBANK bzgl. PII Daten analysieren
 
Über Rechte/Rollen und den sicheren Betrieb der Datenbank
Über Rechte/Rollen und den sicheren Betrieb der DatenbankÜber Rechte/Rollen und den sicheren Betrieb der Datenbank
Über Rechte/Rollen und den sicheren Betrieb der Datenbank
 
Archiving Oracle Primavera project plans with software development tools
Archiving Oracle Primavera project plans with software development toolsArchiving Oracle Primavera project plans with software development tools
Archiving Oracle Primavera project plans with software development tools
 
Introduction into Oracle Data Pump 11g/12c - Export and Import Data
Introduction into Oracle Data Pump 11g/12c - Export and Import DataIntroduction into Oracle Data Pump 11g/12c - Export and Import Data
Introduction into Oracle Data Pump 11g/12c - Export and Import Data
 
Ldap sqlnet
Ldap sqlnetLdap sqlnet
Ldap sqlnet
 
Oracle oem 12c_plugin_development-doag-konferenz_11_2014_print_gunther_pipperr
Oracle oem 12c_plugin_development-doag-konferenz_11_2014_print_gunther_pipperrOracle oem 12c_plugin_development-doag-konferenz_11_2014_print_gunther_pipperr
Oracle oem 12c_plugin_development-doag-konferenz_11_2014_print_gunther_pipperr
 
01 sqlplus
01 sqlplus01 sqlplus
01 sqlplus
 
Der oracle dba_und_seine_passwoerter
Der oracle dba_und_seine_passwoerterDer oracle dba_und_seine_passwoerter
Der oracle dba_und_seine_passwoerter
 
Doag 2104 manuskript_hadoop_oracle_integration_gunther_pipperr_v02
Doag 2104 manuskript_hadoop_oracle_integration_gunther_pipperr_v02Doag 2104 manuskript_hadoop_oracle_integration_gunther_pipperr_v02
Doag 2104 manuskript_hadoop_oracle_integration_gunther_pipperr_v02
 
Oracle hadoop doag-big-data_09_2014_gpi
Oracle hadoop doag-big-data_09_2014_gpiOracle hadoop doag-big-data_09_2014_gpi
Oracle hadoop doag-big-data_09_2014_gpi
 
Oracle connection manager_cman_doag_sig_security_mai_2015
Oracle connection manager_cman_doag_sig_security_mai_2015Oracle connection manager_cman_doag_sig_security_mai_2015
Oracle connection manager_cman_doag_sig_security_mai_2015
 
Oracle no sql-doag-datenbank_konferenz_juni_2014
Oracle no sql-doag-datenbank_konferenz_juni_2014Oracle no sql-doag-datenbank_konferenz_juni_2014
Oracle no sql-doag-datenbank_konferenz_juni_2014
 

Último

Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...
shambhavirathore45
 
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
shivangimorya083
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
Lars Albertsson
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
amitlee9823
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 

Último (20)

Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...
 
Introduction-to-Machine-Learning (1).pptx
Introduction-to-Machine-Learning (1).pptxIntroduction-to-Machine-Learning (1).pptx
Introduction-to-Machine-Learning (1).pptx
 
Data-Analysis for Chicago Crime Data 2023
Data-Analysis for Chicago Crime Data  2023Data-Analysis for Chicago Crime Data  2023
Data-Analysis for Chicago Crime Data 2023
 
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
 
Week-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionWeek-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interaction
 
Sampling (random) method and Non random.ppt
Sampling (random) method and Non random.pptSampling (random) method and Non random.ppt
Sampling (random) method and Non random.ppt
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
 
Best VIP Call Girls Noida Sector 39 Call Me: 8448380779
Best VIP Call Girls Noida Sector 39 Call Me: 8448380779Best VIP Call Girls Noida Sector 39 Call Me: 8448380779
Best VIP Call Girls Noida Sector 39 Call Me: 8448380779
 
100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptx100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptx
 
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
 
Mature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptxMature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptx
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
 
CebaBaby dropshipping via API with DroFX.pptx
CebaBaby dropshipping via API with DroFX.pptxCebaBaby dropshipping via API with DroFX.pptx
CebaBaby dropshipping via API with DroFX.pptx
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
Midocean dropshipping via API with DroFx
Midocean dropshipping via API with DroFxMidocean dropshipping via API with DroFx
Midocean dropshipping via API with DroFx
 
VidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptxVidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptx
 
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
 
Best VIP Call Girls Noida Sector 22 Call Me: 8448380779
Best VIP Call Girls Noida Sector 22 Call Me: 8448380779Best VIP Call Girls Noida Sector 22 Call Me: 8448380779
Best VIP Call Girls Noida Sector 22 Call Me: 8448380779
 

Oracle nosql twjug-oktober-2014_taiwan_print_v01

  • 1. Page 1Gunther Pippèrr © 2014 http://www.pipperr.de ORACLE NOSQL – A NEW ALTERNATIVE FOR THE ORACLE RDBMS DATABASE? Just another Key/Values store? Java Meeting – 10/04/2014
  • 2. Page 2Gunther Pippèrr © 2014 http://www.pipperr.de Gunther Pippèrr – IT-Architect – Consultant • High-Tech • Real Estate • Utility • Communications • IT Systems Architect • Technical project manager • Design and implementation of database applications • Design and implementation of IT infrastructures for data management Gunther Pippèrr is working intensively for more than 15 years with the products of Oracle in database / application server and document management. Mr. Pippèrr has deep knowledge about the structure of complex IT architecture and successfully implemented this in practice. Mr. Pippèrr has a degree in computer science as Dipl. Ing. Technische Informatik (FH) - FH Weingarten. Software manufacture telecommunication - Architect, database design and implementation of a mass data processing system for communication data German technology group - Architect and project management, database design and implementation of an order management for control of external contractors for the language service provider of a German technology group. Utility Consulting - Architect and technical project responsibility for a smart metering portal for meter data collection and asset management Architect and technical project responsibility for IT infrastructure projects, such as: - Central data storage for Munich hotel group with over 25 hotels worldwide - Data collector for monitoring Russian cable operator - Redundant database cluster infrastructure for various Web applications such as insurance portals , fond platforms, loyalty cards Huge Munich building contractor - CRM and tendering portal Industry Expertise Background Functional Expertise Selected Experience • German: http://www.pipperr.de • English: http://orapowershell.codeplex.com Web
  • 3. Page 3Gunther Pippèrr © 2014 http://www.pipperr.de LinkedIn Contact https://www.linkedin.com/pub/gunther-pipp%C3%A8rr/5b/623/23
  • 4. Page 4Gunther Pippèrr © 2014 http://www.pipperr.de Use Case Our challenge to solve
  • 5. Page 5Gunther Pippèrr © 2014 http://www.pipperr.de Our Use Case – Internet Simple Password Store  We like to build: – A secure password store for identity credentials App REST Interface PWD Save User Id PWD Name Username Password URL Access Token URL PWD Name PWD Name Value Value Value Value Key Mayor Minor
  • 6. Page 6Gunther Pippèrr © 2014 http://www.pipperr.de How we can store the Data? Performance, Security, Availability
  • 7. Page 7Gunther Pippèrr © 2014 http://www.pipperr.de OracleRDBMSOracleNoSQL Shared Everything Shared Nothing Instance SQL*Net Central Storage Storage Node Storage Node Storage Node Distributed Storage Client SQL Client Java API RMI
  • 8. Page 8Gunther Pippèrr © 2014 http://www.pipperr.de Use for each Problem the right tool  NoSQL – General Concept – More a approach for data management and processing then a special technologies  Basic Concept – Distributed, scalable solutions (Scale-out Concept) • CAP Theorem – Consistency – Availability - Partition Tolerance • BASE - Basically Available, Soft-state, Eventually versus ACID – Atomicity, Consistency, Isolation, Durability – Highly specialized solutions for exactly one problem with the handling of very large or very quickly required data • Dozens of solutions available in the market
  • 9. Page 9Gunther Pippèrr © 2014 http://www.pipperr.de NOSQL – Definition  A real NoSQL database is: – none relational – Schema free – distributed multi-node architecture • Goal: a shared nothing cluster solution – Simple replications mechanism – A easy API ( Not only SQL = NoSQL) – Not necessarily a ACID Consistency Model – Open Source But – the major disadvantage => Usually no easy query language => Complex joins are not so easy to implement
  • 10. Page 10Gunther Pippèrr © 2014 http://www.pipperr.de Classification Oracle NoSQL in the NoSQL World  Key-Value – One key points to a set of data • Oracle  Document oriented – All data of a data set is organized in one document • Lotus Notes  Graph – networked structures • Neo4J
  • 11. Page 11Gunther Pippèrr © 2014 http://www.pipperr.de Idea - Key-Value store database – Oracle NoSQL – a pure Key Value Store • A distributed and scalable high available database • Consistent hashing approach Key Value Key Value Key Value Key Value Data store is distributed over all nodes Intelligent driver • Implementation of the Hash Mechanisms • Knows the status of the store • Read the partition Map at the first connect to the store Client Software Key Value Key Value Key Value Key Value Key Value Key Value Key Value Key Value One ACID Transaction is possible for all records of a mayor key with one Single API call.
  • 12. Page 12Gunther Pippèrr © 2014 http://www.pipperr.de The Mayor –Minor Key Concept  Key – Composed of two components Data Key Mayor Minor String Byte Array Partitioning over the mayor key Example: User identifier Password Name / User Name Access Token / URL Value user name Token value The structure of the data is NOT self-describing
  • 13. Page 13Gunther Pippèrr © 2014 http://www.pipperr.de Implementation  Key = Hash Mayor Key => Partition Mapping – When you create the store in the first step a fixed count of partitions will be created • Each Partition corresponds essentially to his own Berkeley DB „Database“ • The Hash of the key determine the partition • The partition will be stored redundant over the store Client Driver : Hashing the ID Store over n Storage Nodes Partition 1 Partition 2 Partition 3 Partition … Partition N Replication Group Replication Group Replication Group MD5(major key) % n Partitions →Partition ID Replication Group = Master + Replicates
  • 14. Page 14Gunther Pippèrr © 2014 http://www.pipperr.de Query the store  Java API – Alternative C API (Enterprise Feature!) – Command Line Interface for maintenance  The key and ONLY the Key can be queried – For a search in the values all the data in the store have to be read – For a count you have to iterate over all values
  • 15. Page 15Gunther Pippèrr © 2014 http://www.pipperr.de Variants / versions of the Oracle NoSQL  CC – Community Edition – Open Source Code • Open Source with the GNU Affero General Public License – Support is possible on a annual basis  EE – Enterprise Edition – Full support – Support of Oracle external tables - exchange data with the normal Oracle database – Oracle Event Processing – RDF Adapter – SNMP Monitoring – Jena Adapter (Graph/SPARQL) – Since Version 3 - Security options with the Oracle Wallet
  • 16. Page 16Gunther Pippèrr © 2014 http://www.pipperr.de Integration in the Hadoop Ecosystem Hadoop Map Reduce Read the Data from the NoSQL Store Hadoop File system Store results 11g 12c Read as External Table pure Java Hive Read as External Table Only EE Edition of the Oracle NoSQL!
  • 17. Page 17Gunther Pippèrr © 2014 http://www.pipperr.de Evolution form Key-Value / Avro to table  Release 1 – Key Value – Pure Key-Value concept – No „Data Dictionary“ – Entire logic in the application  Release 2 – Avro – Schema Definition with Apache Avro Standard – Declaration of the structure of the Key-Value Data inside the database AND in the application necessary  Release 3 – Tables – Structure definition as table • A secondary index on columns of the table is possible Value
  • 18. Page 18Gunther Pippèrr © 2014 http://www.pipperr.de The architecture of the Oracle NoSQL database in detail A distributed key value store
  • 19. Page 19Gunther Pippèrr © 2014 http://www.pipperr.de The store = the database  Store – all nodes together over all storage nodes Server 1 Storage Node 1 Server 2 Storage Node 2 Server 4 Storage Node 4 Server 3 Storage Node 3 Example of a store with a replication factor of 3 1 1 13332 2 24 44 Master 1 Replika sn1 Replika sn1 The replication factor is defined as „master + count replicates“. In our example one master and two replicates = replication factor 3 Zone = one store node environment Master Master Master Replication Group = Master + Replicate Shared A Shared D Zone
  • 20. Page 20Gunther Pippèrr © 2014 http://www.pipperr.de Overview  Depending on the replications factor the data will duplicated over the store  The count of partitions will be distributed over all master nodes  A topology describes the whole store  Example– 4 nodes - replication factor 3 – 800 partitions – One Master per node + two replica per storage node – 800 partitions distributed over the four master Nodes • 200 for each master – Inserts only over the master node • The Master Node distribute the data to the replicas – It is possible to read the data from each node The count of partition can not be changed after the create of the store!
  • 21. Page 21Gunther Pippèrr © 2014 http://www.pipperr.de The distribution of the data in the store  The client API hashes the key und distributes the Key over the count of partitions Java Code Java API Key 7 B1A1 1 2 8 1 1 D1C1 9 1 0 1 2 88 B2B3 Insert The hash of the key defines the partition AND The hash defines the master node holds the appropriate partition.
  • 22. Page 22Gunther Pippèrr © 2014 http://www.pipperr.de The distribution of the data in the store  Read is possible from each replicate Java Code Java API Key read 7 B1A1 1 2 8 1 1 D1C1 9 1 0 1 2 88 B2B3 The hash of the key defines the partition AND The hash defines the master node, holds the appropriate partition.
  • 23. Page 23Gunther Pippèrr © 2014 http://www.pipperr.de Internal management  Admin Service for the internal management – Separate admin database – Separate port  The Admin Service is responsible for implementing of the Master – Slave concept – If one Node fails a new master will be defined
  • 24. Page 24Gunther Pippèrr © 2014 http://www.pipperr.de Consistency => all nodes see the same data at the same time Availability => a guarantee that every request receives a response about whether it was successful or failed Partition tolerance => the system continues to operate despite arbitrary message loss or failure of part of the system Availability  What happens in case of a failure of a storage node? – If the Master is unavailable – it can now not longer be written to the store? • Admin Process supervise the whole store – Has his own Admin Database – Recognizes errors – Select from the remaining Replicas the replica with the highest LSN (log sequence number) for a new master If the developer decide to use ReplicaAckPolicy.ALL you will get an error message if a node fails! The developer has to choose : • Consistency and Availabitlity OR • Availabitlity and Partition Tolerance
  • 25. Page 25Gunther Pippèrr © 2014 http://www.pipperr.de The engine (1)  The Java Version of the Berkeley DB – JE Version 5.0.83 - NoSQL 2.2.18 – JE Version 6.0.14 - NoSQL 3.0.14 – Each partition is a database – Berkeley DB Replication for the master slave concept – The DB files can be analyzed with the Berkeley DB classes Tipp: You can configure the original Berkeley DB Mechanism per Storage Node See: http://www.pipperr.de/dokuwiki/doku.php?id=nosql:log_file_verhalten_oracle_nosql_db_11gr2 Show the version with: java -classpath ".libkvclient.jar;.libkvstore.jar;.libje.jar" com.sleepycat.je.util.DbVerify -V
  • 26. Page 26Gunther Pippèrr © 2014 http://www.pipperr.de The engine (2)  Particularity of the Berkeley DB: • Redo Log entries and the data are stored to the same „data“ file • Background Jobs take care about the cleaning of the unnecessary entries je.log.bufferSize Minimal Size of the log je.log.fileMax Maximal Size of the log je.cleaner.minUtilization % must be used of this file je.cleaner.minFileUtilization Clean Logfile, if this value is reached je.cleaner.bytesInterval Start cleaning if x Bytes written je.cleaner.minAge Only clean if x files are older then the latest je.cleaner.threads Number of Cleaner Threads
  • 27. Page 27Gunther Pippèrr © 2014 http://www.pipperr.de Example – Analyze the Data files  Read the size of the DB with the Berkeley classes: – Ensure the correct class path to the jar files • export KVLIB=/opt/oracle/produkt/11.2.0/kv-2.1.8/lib • export KVCLASS=$KVLIB/kvclient.jar:$KVLIB/kvstore.jar:$KVLIB/je.jar – Path to the data files • export JEENV=/opt/oracle/kvdata/GPIDB/sn1/rg1-rn1/env/ – Analyze with: java -classpath $KVCLASS com.sleepycat.je.util.DbSpace -h $JEENV File Size (KB) % Used -------- --------- ------ 00000000 110808 21 TOTALS 110808 21 (LN size correction factor: NaN)
  • 28. Page 28Gunther Pippèrr © 2014 http://www.pipperr.de Installation Unzip Java archive, create directory , ready? Download Software from : http://www.oracle.com/technetwork/database/database-technologies/nosqldb/overview/index.html
  • 29. Page 29Gunther Pippèrr © 2014 http://www.pipperr.de The planning of a NoSQL Installation - Network  Network Infrastructure – Physic • High performante internal communication between the nodes necessary – pay attention for low latencies in the network » Keyword Infiband » Own 10Gbit Network? – Define TCP Ports • For the internal communication of the Storage nodes • For the communication of the Clients with the Master nodes – One Port for external connection + one Port for the Admin Console + ServicePortRange for the RMI Connection from the client – Maintainability • SSH between the den Storage Nodes – Script for maintenance over many nodes Define some company standard , with the time you will get a lot of environments!
  • 30. Page 30Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL installation - Ports  Necessary ports - Network Infrastructure Ports for one Storage Node 5000 Application Admin Client -port 5001 HTTP Admin Client -admin 5010-5020 Replication -harange … -servicerange RMI Port for the Client communication 5021-5040 … Parameter when equipped with the "makebootconfig" haPortRange servicePortRange Parameter for changing with the "change-policy -params " registry portApplication
  • 31. Page 31Gunther Pippèrr © 2014 http://www.pipperr.de Oracle NoSQL Client Connect  Access from the Client to the „registry port“ – For example Port 5000 on node A  The query is take through over the Java RMI protocol on the auf den respective node – Define enough remote ports! -servicerange -port Node A Node B Node C Client Port 5000 ServiceRange Port 5021 ServiceRange Port 5021 ServiceRange Port 5021 Firewall!
  • 32. Page 32Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL Installation - Hardware  CPU and Memory – Cache Size can be adjusted from 2 until n GB • Watch carefully GC effects of Java – Oracle NoSQL is explicit optimized for clean, small objects sizes to not overwhelm the Java garbage collector  Storage Infrastructure – One Store Node can server also more then one Master und replicates • If you have more then one Storage node on a server you should use separate disks/luns to have best I/O on the server – In the Background the “Cleaner” processes need some I/O resources
  • 33. Page 33Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL Installation – Disk space  Disk space – Calculation is not entirely trivial • The Berkeley DB store all actions like the redo data in the same file • Not until some Thresholds are reached the data files will be optimized • => The delete of data enlarge the store in the first step! Rule of thumb: Pro Master/ Replika => Count of Transactions (Insert/Update/Delete) * (Net Data Size + Overhead) => Filling of the data files per Default optimized to 40%
  • 34. Page 34Gunther Pippèrr © 2014 http://www.pipperr.de Installation  Preparation on each Node – Java JDK installation and create Store User – Setup SSH Connection – Create the KVROOT (Data Storages!)  Oracle NoSQL installation – Extract the code – create the KVHOME – Create the Store Node Base directory (KVROOT) – Create the Boot configuration – Start the Admin Node – Define a plan for the new store • Define the database topologies – Rollout the plan – test Per Script: 5min
  • 35. Page 35Gunther Pippèrr © 2014 http://www.pipperr.de Install our own local test system  Download from oracle.com and extract the zip file to a directory  Create a local store and start the NoSQL Database  Start a admin Shell REM go to your DB directory cd R:KVSTORE REM set the libraries Path set KVHOME=D:kv-3.0.14 REM Create the Store java -jar %KVHOME%libkvstore.jar kvlite -root R:KVSTORE -store kvstore -host localhost -port 5000 -admin 5001 java -jar %KVHOME%libkvstore.jar runadmin -port 5000 -host localhost kv-> ping … Pinging components of store kvstore based upon topology sequence #14 … Rep Node [rg1-rn1] Status: RUNNING,MASTER at sequence number: 39 ha
  • 36. Page 36Gunther Pippèrr © 2014 http://www.pipperr.de Work with the NoSQL Store Maintenance and operation
  • 37. Page 37Gunther Pippèrr © 2014 http://www.pipperr.de Maintenance (1) – command Line  Kv – For the Store Administration  kvshell – Select / insert data kvshell-> put -key "/Gunther/name" -value "Pipperr" Operation successful, record inserted. kvshell-> get -key "/Gunther/name" Pipperr
  • 38. Page 38Gunther Pippèrr © 2014 http://www.pipperr.de Maintenance (2) – Web interface – Web interface based on a internal jetty server http://nosqldb03:5005/
  • 39. Page 39Gunther Pippèrr © 2014 http://www.pipperr.de JMX Interface  Start JMC – Java Mission Control – JAVA_HOME/bin/jmc.exe – Connect over the „registry port“
  • 40. Page 41Gunther Pippèrr © 2014 http://www.pipperr.de Reliability Availability and Backup
  • 41. Page 42Gunther Pippèrr © 2014 http://www.pipperr.de Backup per Snapshot possible  Backup: – A complete Store can be saved consistent per Snapshot – pay attention for enough disk space!  Restore – A Store can be completely reset • Create a Store snapshot • Stop the Store • Provide the Snapshot (Link in the File system for example) • Start the Store, Storage Nodes read die Data, create the data again and DELETE the snapshot! – Use as a kind of import • The generated files can be read from an other store
  • 42. Page 43Gunther Pippèrr © 2014 http://www.pipperr.de Performance I/O and Network
  • 43. Page 44Gunther Pippèrr © 2014 http://www.pipperr.de Performance considerations  read – Network • All records must be read over the network  write – I/O • Background process need enough recourse for the optimization of the data files! ALL record are processed on the client side! A simple Count(*) have to read all data!
  • 44. Page 45Gunther Pippèrr © 2014 http://www.pipperr.de Security? Oracle NoSQL and Security?
  • 45. Page 46Gunther Pippèrr © 2014 http://www.pipperr.de Security?  Version 2 ?  The developer has to care about it – not implemented  Since Version 3 – SSL encoding – Password security • But Wallet usage is a EE feature!
  • 46. Page 47Gunther Pippèrr © 2014 http://www.pipperr.de Read and write to the store with Java No SQL necessary – all is done in Java
  • 47. Page 48Gunther Pippèrr © 2014 http://www.pipperr.de Client Software Client Driver : Hashing der ID Store over n Storage Nodes Partition 1 Partition 2 Partition 3 Partition … Partition N Replication Group Replication Group Replication Group MD5(major key) % n Partitions →Partition ID Replication Group = Master + Replication Node Key Value Key Value Partition … Intelligent driver • Implementation of the Hash Mechanisms • Knows the status of the store • Read the partition Map at the first connect to the store Overview
  • 48. Page 49Gunther Pippèrr © 2014 http://www.pipperr.de First – Connect to the store (1)  Create the connection to the store • Set the store properties with the class KVStoreConfig Store Name // Node liste String[] kvhosts = {"nosqldb01:5000","nosqldb02:5000","nosqldb03:5000"} // Config erzeugen KVStoreConfig kvconfig = new KVStoreConfig("GPIDB", kvhosts); Port Node
  • 49. Page 50Gunther Pippèrr © 2014 http://www.pipperr.de First – Connect to the store (2)  Create the connection to the store • Define the Read and Write Rules // Define Consistency kvconfig.setConsistency(Consistency.NONE_REQUIRED); // Define Durabiltiy kvconfig.setDurability(Durability.COMMIT_NO_SYNC); // Connect to the store KVStore kstore = KVStoreFactory.getStore(kvconfig); Rule read Rules write Open connection to the Store of a Factory Do not forget to close the Store! kstore.close();
  • 50. Page 51Gunther Pippèrr © 2014 http://www.pipperr.de First Example - Write the data  Address the DB object over the key  Create the key  Write the data Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME"); Mayor Key Minor Key // Wert als Byte Array anlegen byte[] b = ("gpipperr@googelmail.com").getBytes(); // Wert in den Store schreiben kvstore.put(k, Value.createValue(b)); Write the data to the store
  • 51. Page 52Gunther Pippèrr © 2014 http://www.pipperr.de First Example - Read the data over the key  Create the key  Read data Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME"); Mayor Key Minor Key // Read the Data from the Store with the key ValueVersion vv = kvstore.get(k); // Read the Daten Value Value vdata = vv.getValue(); // Recreate the Values String username= new String(vdata.getValue()); // Work with the data System.out.println(username); read
  • 52. Page 53Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  • 53. Page 54Gunther Pippèrr © 2014 http://www.pipperr.de TRANSACTION CONCEPTS Are we reading the last version of data?
  • 54. Page 55Gunther Pippèrr © 2014 http://www.pipperr.de The consistency of the data – Transactionality  In one session transactionality for write one mayor key is possible  The developer has to decide when a record is apply as written! read  The developer has to decide if the read record is actual write FastSave Durability Consistency Performance Performance Latest Version first available answer On all nodes to disk On one node to the cache FastSave
  • 55. Page 56Gunther Pippèrr © 2014 http://www.pipperr.de The write consistency in the Store - Durability  The developer is responsible for the write consistency!  Will be defined at the connect to the store – Class oracle.kv.Durability • Commit-Policy for the Master • Commit-Policy for the replicas • Replica Acknowledgement-Policy FastSave Performance On all nodes to disk On one node to the cache
  • 56. Page 57Gunther Pippèrr © 2014 http://www.pipperr.de Durability Policies  Commit-Policy for the Master and the replicates – SyncPolicy.SYNC • Wait until the Storage Node write the transaction to the log file and synchronized to the disk – SyncPolicy.WRITE_NO_SYNC • Wait until the Storage Node write the transaction to the log file in the cache (not written to disk!) – SyncPolicy.NO_SYNC • Don’t wait  Replica Acknowledgement – ReplicaAckPolicy.ALL • Wait until all replicates confirm, that all transaction has been written – ReplicaAckPolicy.SIMPLE_MAJORITY • Wait until the simple majority the replicates confirm the transaction – ReplicaAckPolicy.NONE • Don’t wait for the answers of the replicates
  • 57. Page 58Gunther Pippèrr © 2014 http://www.pipperr.de The read consistency in the store - Consistency  The developer is responsible for the read consistency!  Each key object owns a version information – A kind of transaction ID  The behavior will be defined with the connect to the store – read – Class oracle.kv.Consistency Performance FastSave Latest Version first available answer
  • 58. Page 59Gunther Pippèrr © 2014 http://www.pipperr.de Consistency  Consistency.ABSOLUTE – The session can read only from the master for this key. – This ensures that the data is relay valid  Consistency.Time – The session can read from a replica – With the Parameter you can define how „old“ the data can be to accepted as valid data  Consistency.Version – The session can read from a replica – The version information will be used instead of the time  Consistency.NONE_REQUIRED – Very result is valid, can be any old
  • 59. Page 60Gunther Pippèrr © 2014 http://www.pipperr.de THE TABLE IS BACK Structured data access
  • 60. Page 61Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1)  Our Key Model  Our Table Model User Id PWD Name Username Password URL Access Token URL PWD Name PWD Name Value Value Value Value USER_ID | PWD_NAME | USER_NAME | PASSWORD| URL USER_ID | ACESS_TOCKEN | URL | Value Table USER_PASSWORDS Table USER_ACESS_TOCKENS
  • 61. Page 62Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1.1)  Create a table definition in the Store with the client utility set KVHOME=D:entwicklunglibrarieskv-ee-3.0.14 java -jar %KVHOME%libkvcli.jar -host localhost -port 5000 -store kvstore #Create table table create -name USER_PASSWORDS # Add table fields add-field -name USER_ID -type STRING add-field -name PWD_NAME -type STRING add-field -name USER_NAME -type STRING add-field -name PASSWORD -type STRING add-field -name URL -type STRING
  • 62. Page 63Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1.2) # Add the primary Key primary-key -field USER_ID #Leave table Mode exit #create the Table in the Store plan add-table -name USER_PASSWORDS -wait #check the Table Definition show table -name USER_PASSWORDS #insert first Record of data in to the table with the client tools put table -name USER_PASSWORDS -json "{"USER_ID":"gunther@pipperr.de","PWD_NAME":"google.de","USER_NA ME":"gpipperr","PASSWORD":"Password01","URL":"www.google.de"}" #create index on the table plan add-index -name IDX_USER_PASSWORDS -table USER_PASSWORDS -field USER_ID –wait #read the data get table -name USER_PASSWORDS -index IDX_USER_PASSWORDS
  • 63. Page 64Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1)  Write the source code to use the table defintion // get the Table API TableAPI tableAPI = kvstore.getTableAPI(); //get a reference to the table Table empTable = tableAPI.getTable("USER_PASSWORDS"); // Get a Row instance Row empRow = empTable.createRow(); // put data in the row empRow.put("empno", 10); // write the data to the store tableAPI.put(empRow, null, null);
  • 64. Page 65Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (2)  Read from this table again // get the Table API TableAPI tableAPI = kvstore.getTableAPI(); //get a reference to the table Table empTable = tableAPI.getTable("emp"); //Create a primary key and assign the field value PrimaryKey key = table.createPrimaryKey(); key.put("userID", 1); //Get the matching row Row row = tableAPI.get(key, new ReadOptions(null, 0, null)); if (row == null) { throw new Error("No matching row found"); } //Print the full row as JSON System.out.println(row.toJsonString(true)); //Access a specific field System.out.println("firstName field as JSON: " + row.get("firstName").toJsonString(false));
  • 65. Page 66Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  • 66. Page 67Gunther Pippèrr © 2014 http://www.pipperr.de INTEGRATION INTO THE HADOOP DATA WAREHOUSE Archive your data
  • 67. Page 68Gunther Pippèrr © 2014 http://www.pipperr.de Oracle NoSQL – MapReduce Hadoop  Traditional MapReduce Pattern  Input Format Class – KVInputFormat.class Mapper Mapper Mapper Mapper Reducer HDFSOracle NoSQL Store KVInputFormat oracle.kv.Key.toString() lib/kvclient.jar must in the class path! Like: hadoop jar secure-pwd-store.jar gpi.secstore.PWDStoreHadoop -libjars $KVHOME/lib/kvclient.jar /tmp/nosqlTest2 Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class oracle.kv.hadoop.KVInputFormat not found at Better solution -put all in one jar fie
  • 68. Page 69Gunther Pippèrr © 2014 http://www.pipperr.de Main Part – Job Configuration  Define the Input Format Class in the Job Configuratoin // set the Input Format Classe job.setInputFormatClass(KVInputFormat.class); //Parameter for the Input Format Class KVInputFormat.setKVStoreName("KVStore"); String [] kvhostList = {"StoreDB01:5000", "StoreDB02:5000",}; KVInputFormat.setKVHelperHosts(kvhostList); Set the Input Class for the MapReduce Job Define the connection to the Store Default: Key and Value are read as Text Format
  • 69. Page 70Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  • 70. Page 71Gunther Pippèrr © 2014 http://www.pipperr.de Conclusion Are the benefits worth the effort?
  • 71. Page 72Gunther Pippèrr © 2014 http://www.pipperr.de Why we should use the Oracle NoSQL? (1)  What is really important for a strategic development? – Maintainability • Is the product easy to use? – Long-term strategy of the manufacturer • Is a product cycle from 4-5 year possible? – Release and Update Problematic • Java 8 and higher? Linux 7?
  • 72. Page 73Gunther Pippèrr © 2014 http://www.pipperr.de Why we should use the Oracle NoSQL? (2)  What is for a strategic development really important? – Fits the base concept to my technical needs? • Are my requirements changing with the time? – Costs Oracle promises this: – Version 3 ready – The Software Berkeley DB since year in use
  • 73. Page 74Gunther Pippèrr © 2014 http://www.pipperr.de Use Cases  Use Cases – Personalization of web pages – Authentication – Stream Processing • machine data Goal: Get for one Key fast as possible the results Requirement: Provide large amounts of data with low latency – Persistent Cache
  • 74. Page 75Gunther Pippèrr © 2014 http://www.pipperr.de Case Study: Web Application  In a Web application each user can have a very high number of different "bonus offers, vouchers"  This bonus offers are calculated individually according to the profiles of a user on a weekly base – The NoSQL Store holds the UserID as key with a list of all metadata for the bonus offers • Avoid complex joins over the DWH with the user profiles • Immediate display results on the smart phone or in the Web without a time delay – very low latencies Requirement: Provide large amounts of data with low latency – Persistent Cache
  • 75. Page 76Gunther Pippèrr © 2014 http://www.pipperr.de Architecture of a NoSQL Stores 3 Storage Nodes (SN) on three servers with a Replication Factor of three = three Replication Groups (RG) Server 1 Storage Node 1 SN 1 Server 2 Storage Node 2 SN 2 Server 3 Storage Node 3 SN 3 Rg1- n1 Master 1 Replication Node Replica 1 Replication Node Replica 2 Replication Node Shared A Store Shared B Shared C Zone A Admin Srv Admin Srv Admin Srv Storage Node Agent (SNA) Storage Node Agent (SNA) Storage Node Agent (SNA) Rg2- n1 Rg3- n1 Rg1- n2 Rg2- n2 Rg3- n2 Rg1- n3 Rg2- n3 Rg3- n3
  • 76. Page 77Gunther Pippèrr © 2014 http://www.pipperr.de Direct Comparison Oracle RDBMS NoSQL  ~35 years of development  Powerful SQL API  Joins  Constraints  Many Features  OLTP  Shared anything approach  A general all purpose weapon Oracle NoSQL  ~2-3 years  Simple Java API  Joins only over the application  ?  Pure data store  Mostly only read operation  Shared Nothing Cluster  Specialized niche solution Oracle RDBMS
  • 77. Page 78Gunther Pippèrr © 2014 http://www.pipperr.de Fragen Oracle NoSQL NO:SQL
  • 78. Page 79Gunther Pippèrr © 2014 http://www.pipperr.de Sources  Oracle - ORACLE – English http://www.oracle.com/technetwork/database/database- technologies/nosqldb/documentation/index.html – German http://www.oracle.com/webfolder/technetwork/de/community/dojo/index.ht ml  See also: – German http://www.pipperr.de/dokuwiki/doku.php?id=nosql_datenbank