The document summarizes Oracle's Big Data Appliance and solutions. It discusses the Big Data Appliance hardware which includes 18 servers with 48GB memory, 12 Intel cores, and 24TB storage per node. The software includes Oracle Linux, Apache Hadoop, Oracle NoSQL Database, Oracle Data Integrator, and Oracle Loader for Hadoop. Oracle Loader for Hadoop can be used to load data from Hadoop into Oracle Database in online or offline mode. The Big Data Appliance provides an optimized platform for storing and analyzing large amounts of data and is integrated with Oracle Exadata.

1. <Insert Picture Here>
Oracle Big Data Appliance and Solutions
Downlload this slide
http://ouo.io/OtuIkv

2. The following is intended to outline our general product
direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing
decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remain at the
sole discretion of Oracle.

3. Case: On-line Ads and Content
NoSQL
DB
Expert
System
Real-time: Determine
best ad to place
on page for this user
Input into
Lookup user
profile
Add user
if not present
Web
logs
HDFS
Profiles
NoSQL DB
High scale
data reductions BI and
Analytics
Billing
Predictions
on browsing
Actual
ads
served
Low
Latency
Batch

4. Agenda
• Big Data Technology
• Oracle Big Data Appliance
• Big Data Applications
• Summary
• Q&A

5. <Insert Picture Here>
Big Data Technology

6. • Deep Analytics
• Agile Development
• Massive Scalability
• Real Time Results
• High Throughput
• In-Place Preparation
• All Data Sources/Structures
• Low, predictable Latency
• High Transaction Volume
• Flexible Data Structures
Big Data: Infrastructure Requirements
Acquire Organize Analyze

7. Divided Solution Spectrum
Acquire AnalyzeOrganize
MapReduce
Solutions
DBMS
(DW)
DBMS
(OLTP)
Advanced
Analytics
Distributed
File Systems
Transaction
(Key-Value)
Stores
ETL
NoSQL
Flexible
Specialized
Developer
Centric
SQL
Trusted
Secure
Administered
Dynamic
Schema
Data
Variety
Schema

8. 8 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Integrated Software Solution Stack
Acquire AnalyzeOrganize
Oracle
Database
(DW)
Oracle
Database
(OLTP)
In-DB
Analytics
“R”
Mining
Text
Graph
Spatial
Oracle
BI EE
Oracle NoSQL
DB
HDFS Hadoop
Oracle
Data Integrator
Oracle Loader
for Hadoop
Dynamic
Schema
Data
Variety
Schema

9. 9 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Engineered Solutions
Oracle
Database
(DW)
Oracle
Database
(OLTP)
In-DB
Analytics
“R”
Mining
Text
Graph
Spatial
Oracle
BI EE
Oracle NoSQL
DB
HDFS Hadoop
Oracle
Data Integrator
Oracle Loader
for Hadoop
Big Data Appliance
• Hadoop
• NoSQL Database
• Oracle Loader for hadoop
• Oracle Data Integrator
Oracle Exadata
• OLTP & DW
• Data Mining & Oracle R
• Semantics
• Spatial
Exalytics
• Speed of
Thought
Analytics
Acquire AnalyzeOrganize
Dynamic
Schema
Data
Variety
Schema

10. Big Data Appliance
Batch Usage Model
Oracle
Big Data Appliance
Oracle
Exadata
InfiniBand
Acquire Organize Analyze
Oracle
Exalytics
InfiniBand

11. 11 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Why build a Hadoop Appliance?
• Time to Build?
• Required Expertise?
• Cost and Difficulty Maintaining?

12. •18 Sun X4270 M2 Servers
– 48 GB memory per node = 864 GB memory
– 12 Intel cores per node = 216 cores
– 24 TB storage per node = 432 TB storage
•40 Gb p/sec InfiniBand
•10 Gb p/sec Ethernet
Oracle Big Data Appliance Hardware

13. Big Data Appliance
Cluster of industry standard servers for Hadoop and NoSQL Database
• Focus on Scalability and Availability at low cost
Compute and Storage
• 18 High-performance low-cost
servers acting as Hadoop
nodes
• 24 TB Capacity per node
• 2 6-core CPUs per node
• Hadoop triple replication
• NoSQL Database triple
replication
10GigE Network
• 8 10GigE ports
• Datacenter connectivity
InfiniBand Network
• Redundant 40Gb/s switches
• IB connectivity to Exadata

14. Scale Out to Infinity
Scale out by connecting racks
to each other using Infiniband
• Expand up to eight racks without
additional switches
• Scale beyond eight racks by adding
an additional switch

15. •Oracle Linux 5.6
•Java Hotspot VM
•Apache Hadoop Distribution v0.20.x
•R Distribution
•Oracle NoSQL Database Enterprise
Edition
•Oracle Data Integrator Application
Adapter for Hadoop
•Oracle Loader for Hadoop
Oracle Big Data Appliance Software

16. Why Open-Source Apache Hadoop?
• Fast evolution in critical features
• Built by the Hadoop experts in the community
• Practical instead of esoteric
• Focus on what is needed for large clusters
• Proven at very large scale
• In production at all the large consumers of Hadoop
• Extremely stable in those environments
• Well-understood by practitioners

17. Software Layout
• Node 1:
• M: Name Node, Balancer & HBase Master
• S: HDFS Data Node, NoSQL DB Storage Node
• Node 2:
• M: Secondary Name Node, Management,
Zookeeper, MySQL Slave
• S: HDFS Data Node, NoSQL DB Storage Node
• Node 3:
• M: JobTracker, MySQL Master, ODI Agent,
Hive Server
• S: HDFS Data Node, NoSQL DB Storage Node
• Node 4 – 18:
• S: HDFS Data Nodes, Task Tracker, HBase
Region Server, NoSQL DB Storage Nodes
• Your MapReduce runs here!

18. Big Data Appliance
Big Data for the Enterprise
• Optimized and Complete
• Everything you need to store and integrate
your lower information density data
• Integrated with Oracle Exadata
• Analyze all your data
• Easy to Deploy
• Risk Free, Quick Installation and Setup
• Single Vendor Support
• Full Oracle support for the entire system and
software set

19. <Insert Picture Here>
Oracle NoSQL Database

20. Key-Value Store Workloads
• Large dynamic schema based data repositories
• Data capture
• Web applications
• Online retail
• Sensor/statistics/network capture/Mobile Devices
• Data services
• Scalable authentication
• Real-time communication (MMS, SMS, routing)
• Personalization / Localization
• Social Networks

21. Oracle NoSQL DB
A distributed, scalable key-value database
• Simple Data Model
• Key-value pair with major+sub-key paradigm
• Read/insert/update/delete operations
• Scalability
• Dynamic data partitioning and distribution
• Optimized data access via intelligent driver
• High availability
• One or more replicas
• Disaster recovery through location of replicas
• Resilient to partition master failures
• No single point of failure
• Transparent load balancing
• Reads from master or replicas
• Driver is network topology & latency aware
Storage Nodes
Data Center A
Storage Nodes
Data Center B
NoSQLDB Driver
Application
NoSQLDB Driver
Application

22. • Operation result
• New Partition Map
• RepNodeStorageTable
information
Resolving a Request
Hash Major Key to determine
Partition id
Use Partition Map to map Partition
id to a Rep Group
Use State Table to determine eligible
Storage Node(s) within Rep Group
Use Load Balancer to select best
eligible Rep Node
Contact Rep Node directly
Client
Operation + Key[M,m] + Value + Transaction Policy

23. ACID Transactions
Transaction Policy
Write Durability
• Configurable per-operation,
application can set defaults
• Write Transaction Durability consists
of both
a) Sync policy (on Master and
Replica)
• Sync – force to disk
• Write No Sync – force to OS
buffer
• No Sync – write to local log buffer,
flush when convenient
b) Replica Acknowledgement Policy
• All
• Simple Majority
• None
Transaction Policy
Read Consistency
• Configurable per-operation,
application can set defaults
• Read Consistency specified as
Absolute, Time-based, Version or
None
• Absolute  Read from the master
• Time-based  Read from any
replica that is within <time-
interval> of master or better
• Version  Read from any replica
that is current with <transaction-
token> or higher
• None  Read from any replica

24. Oracle NoSQL DB Differentiation
• Commercial Grade Software and Support
• General-purpose
• Reliable – Based on proven Berkeley DB JE HA
• Easy to install and configure
• Scalable throughput, bounded latency
• Simple Programming and Operational Model
• Simple Major + Sub key and Value data structure
• ACID transactions
• Configurable consistency & durability
• Easy Management
• Web-based console, API accessible
• Manages and Monitors: Topology; Load; Performance; Events; Alerts
• Completes Oracle large scale data storage offerings

25. Try NoSQL Database on OTN
Oracle NoSQL Database:
• Community Edition is available as a software
only distribution
• Enterprise Edition is available as a separately
licensable product or as part of Big Data Appliance

26. <Insert Picture Here>
Oracle Loader for Hadoop

27. 27 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle Loader for Hadoop Features
• Load data into a partitioned or non-partitioned table
– Single level, composite or interval partitioned table
– Support for scalar datatypes of Oracle Database
– Load into Oracle Database 11g Release 2
• Runs as a Hadoop job and supports standard options
• Pre-partitions and sorts data on Hadoop
• Online and offline load modes

28. 28 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle Loader for Hadoop
SHUFFLE
/SORT
SHUFFLE
/SORT
MAP
MAP
MAP
MAP
SHUFFLE
/SORT
REDUCE
REDUCE
SHUFFLE
/SORT
SHUFFLE
/SORT
REDUCE
REDUCE
REDUCE
INPUT
2
INPUT
1
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
ORACLE LOADER FOR HADOOP

29. 29 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle Loader for Hadoop: Online Option
SHUFFLE
/SORT
SHUFFLE
/SORT
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
ORACLE LOADER FOR HADOOP Connect to the database
from reducer nodes, load
into database partitions in
parallel
Read target table metadata
from the database
Perform partitioning,
sorting, and data
conversion

30. 30 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle Loader for Hadoop: Offline Option
SHUFFLE
/SORT
SHUFFLE
/SORT
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
ORACLE LOADER FOR HADOOP
Read target table metadata
from the database
Perform partitioning,
sorting, and data
conversion
Write from reducer nodes to
Oracle Data Pump files
Import into the database in
parallel using external table
mechanism

31. 31 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle Loader for Hadoop Advantages
• Offload database server processing to Hadoop:
– Convert input data to final database format
– Compute table partition for row
– Sort rows by primary key within a table partition
• Generate binary datapump files
• Balance partition groups across reducers

32. 32 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Input and Output Formats
Input Formats
• Delimited text
• Hive tables
– Managed and external tables
– Native and non-native tables
• Write your own input format
Output Formats
Online Mode
• Load directly from Hadoop nodes to
Oracle database
– JDBC
– Parallel direct path
Offline Mode
• Datapump format
– Create binary files for external tables
– Import data into the database from the
external table with a SQL statement
• CSV, delimited text
– Load through SQL*Loader or external
table mechanism

33. 33 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Selection Output Option for Use Case
Oracle Loader for Hadoop
Output Option
Use Case Characteristics
Online load with JDBC The simplest use case for non
partitioned tables
Online load with Direct Path Fast online load for partitioned
tables
Offline load with datapump files Fastest load method for external
tables
On Oracle Big Data Appliance
Direct HDFS
Leave data on HDFS
Parallel access from database
Import into database when
needed

34. 34 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Invoking Oracle Loader for Hadoop
• Command line
$ hadoop jar oraloader.jar oracle.hadoop.loader.OraLoader
-libjars <library jar files>
-D <configuration properties>
$HADOOP_HOME/bin/hadoop jar oraloader.jar oracle.hadoop.loader.oraLoader
-libjars avro-1.4.1.jar, commons-math-2.2.jar
-conf connection.xml
-D mapreduce.inputformat.class=oracle.hadoop.loader.lib.input.DelimitedTextInputFormat
-D mapreduce.outputformat.class=oracle.hadoop.loader.lib.output.JDBCOutputFormat

35. 36 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Automate Usage of Oracle Loader for Hadoop
• ODI has knowledge modules to
– Generate data transformation code to run on Hive/Hadoop
– Invoke Oracle Loader for Hadoop
• Use the drag-and-drop interface in ODI to
– Include invocation of Oracle Loader for Hadoop in any ODI
packaged flow
Oracle Data Integrator (ODI)

36. 37 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.

37. <Insert Picture Here>
Summary

38. Big Data Appliance
Big Data for the Enterprise
• Optimized and Complete
• Everything you need to store and integrate your lower
information density data
• Integrated with Oracle Exadata
• Analyze all your data
• Easy to Deploy
• Risk Free, Quick Installation and Setup
• Single Vendor Support
• Full Oracle support for the entire system and software
set

39. Big Data Appliance and Exadata
Big Data for the Enterprise
NoSQL DB

HDFS

Hadoop

RDBMS 

40. Questions