Oracle Exalyics + TimesTen Best Practices (OOW 2012)

Oracle Exalytics and TimesTen Best Practices
Mark Rittman, Technical Director, Rittman Mead
Oracle Openworld 2012, San Francisco, September 2012

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

Mark Rittman

• Mark Rittman, Co-Founder of Rittman Mead
• Oracle ACE Director, specialising in Oracle BI&DW
• 14 Years Experience with Oracle Technology
• Regular columnist for Oracle Magazine
• Author of two Oracle Press Oracle BI books
‣ Oracle Business Intelligence Developers Guide
‣ Oracle Exalytics Revealed
• Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
• Email : mark.rittman@rittmanmead.com
• Twitter : @markrittman


About Rittman Mead

• Oracle BI and DW gold partner
• World leading specialist partner for technical excellence, solutions delivery and
innovation in Oracle BI
• Approximately 30 consultants worldwide
• All expert in Oracle BI and DW
• UK based
• Offices in US, Europe (Belgium) and India
• Skills in broad range of supporting Oracle tools:
‣ OBIEE
‣ OBIA
‣ ODIEE
‣ Essbase, Oracle OLAP
‣ GoldenGate
‣ Exadata


What is Oracle Exalytics?

• Hardware Element
‣ Sun Fire X4470 M2 server
‣ 1TB RAM, 40 Cores, 3.6TB HDD
• Software Element
‣ OBIEE 11.1.1.6 with Exalytics Enhancements
‣ Oracle Essbase 11.1.2 with Exalytics Enhancements
‣ Oracle TimesTen 11.2.2.2 for Exalytics
‣ New in v1.1 : Support for Oracle Endeca
Information Discovery, Golden Gate, ODI
‣ Runs on 64-bit Oracle Linux
(Exalogic distribution)
• OBIEE and Essbase are licensed as
Oracle BI Foundation
• Exalytics features can only be used in
conjunction with Exalytics hardware

Rittman Mead and Oracle Exalytics

• Rittman Mead are the first Oracle Partner in the UK to buy an Exalytics box
• Ordered February 2012 through Arrow (reseller), delivered end-April 2012
• Hosted in a secure datacenter in London Docklands, access via VPN
• To be used for customer PoCs, training, testing and possible cloud hosting


Installation and Delivery Process

• Installed by Oracle ACS over two days
• Day 1 involved unpacking, racking, connecting to network etc
• Day 2 was for installing the software (only ships with Oracle Linux)
‣ Oracle WebLogic
‣ Oracle BI EE 11g
‣ Oracle TimesTen for Exalytics
• Post-install config from ACS for
‣ Creating initial TimesTen DB
‣ Configuring BI EE for Exalytics
‣ Configuring BI Server > TimesTen connection
• Post-install config from Rittman Mead included:
‣ Configuring RPD for Usage Tracking &
Summary Statistics
‣ Connecting RPD to TimesTen DB


Exalytics as the Exa-Machine for BI

• Runs the BI layer on a high-performance, multi-core, 1TB server
• In-memory cache used to accelerate the BI part of the stack
• If Exadata addresses 80% of the query performance,
Exalytics addresses the remaining 20% Oracle BI

‣ Consistent response times for queries
‣ In-memory caching of aggregates
‣ 40 cores for high concurrency In-Memory DB/Cache
‣ Re-engineered BI and OLAP software
that assumes 40 cores and 1TB RAM

ERP/Apps DW


Also Supports Essbase, and Endeca Information Discovery

• In-Memory Essbase for planning, budgeting and sales analysis-style OLAP applications
• Endeca Information Discovery for search/analytic applications against diverse data

Oracle
Exalytics
In-Memory
Machine


Enables High-Density Analysis of Big Data

• BI tier is rarely the bottleneck, but it can be if very dense visualizations are used
‣ Sparklines, grid of charts etc
• Exalytics’ 40 cores and 1TB RAM make higher density presentation viable
‣ Single query sent to the database
‣ Exalytics breaks data up to create microcharts
• Also helps support high numbers of concurrent users (100+)


Exalytics Architecture

• Developer tools used for defining Developer Workstation

and maintaining aggregate definitions Oracle BI Administration Tool

Summary Advisor Wizard
• Client tools for OBIEE repository,
nqcmd
TimesTen and Oracle Database
TimesTen Client

SQL Developer

Oracle Database 11g Client

Oracle Business Intelligence

• Exalytics layer contains OBIEE,
Oracle TimesTen for Exalytics
WLS Admin
Oracle BI
Server

TimesTen and Essbase
System Comp.
Aggregate
WLS Managed BI Server Schema
(not pictured) Server

• Database tier contains FMW
Database for RCU Schemas
database repository, including Enhanced Usage Tracking
Summary Statistics
usage tracking and summary statistics

Exalytics Under the Covers - How Does it Work?

• Exalytics brings together different technologies, which are still standalone products in
their own right
• To harmonise and optimise their use within Exalytics, it utilises the following techniques:
‣ In-Memory Adaptive Data Mart
‣ In-Memory Intelligent Result Cache
‣ In-Memory Cubes
• Some of these are genuine "secret sauce"
‣ New functionality and algorithms
‣ You can only get them through licensing Exalytics
• Others are descriptions of DW/BI strategies, or existing product functionality, extended
to take advantage of the capacity for processing in memory that Exalytics has


In-Memory Adaptive Data Mart

• Commonly-used aggregates are copied into Oracle TimesTen for Exalytics
• Past query patterns are analyzed and suitable aggregates recommended
• Oracle BI Server then uses these aggregates to make queries run faster
• Aggregates change over time in
response to changes in query patterns TimesTen BI Server
• Tools are provided for managing

Exalytics
and populating these aggregates Aggregates

Data Warehouse
Detail-level
Data


In-Memory Intelligent Results Cache

• OBIEE has an advanced result cache, which can support:
‣ Queries matching previous ones
‣ Queries matching a subset of the results of a cached query resultset
‣ Queries aggregating the results of a cached query resultset
• Stored in cache files on the OBIEE server filesystem
• With Exalytics, the 1TB of RAM typically means
that these files are held in memory
• RAMdisk can also be created for
BI Server & Presentation Server TimesTen BI Server
working area files

Exalytics
In-Memory
Result
Cache


Oracle TimesTen for Exalytics

• New version of TimesTen specifically for Exalytics (and only available with Exalytics)
• Support for analytic functions
‣ Perform all the processing at source
‣ Combine with being in-memory = should be very fast
• Column compression
‣ Whitepaper cites 5x
‣ Given the hardware capacity, we could seriously contemplate loading the whole Data
Warehouse into memory
‣ Opens up lots of interesting design potential
• We can load aggregates into TimesTen, leave base data at source, and use OBIEE’s
Vertical Federation capability to seamlessly report across both
‣ All hidden from the end-user, all they will know is that their reports run fast!


TimesTen and OBIEE Architecture

• Oracle BI Server communicates with TimesTen through TimesTen Client
• Summary Advisor, and nqcmd use Oracle BI Server to access TimesTen
• Typical single TimesTen database per Exalyics machine
‣ Max TimesTen database size around 300MB
- Due to need to set aside equal Summary Oracle BI nqcmd
Advisor Server
Temp size for the Perm size selected
• Clustered Exalytics boxes can be daisy-chained ODBC
together using InfinBand connections Checkpoint
Files
‣ For HA scenarios, does not increase
available RAM TimesTen
Memory-Resident
‣ Summary advisor scripts write to both TimesTen Database

databases, replicating aggregates
‣ TimesTen databases can be “wired together”
for failover/HA purposes
Log
Files


Summary Advisor for Aggregate Recommendation & Creation

• Utility within Oracle BI Administrator tool that recommends aggregates
• Bases recommendations on usage tracking and summary statistics data
• Captured based on past activity
• Runs an iterative algorithm that searches,
each iteration, for the best aggregate


Recommendations Based on Enhanced Usage Tracking Data

• Historically, usage tracking has been tracked using S_NQ_ACCT
‣ Holds basic usage tracking statistics + logical SQL query
• Now supplemented by S_NQ_DB_ACCT
‣ Extra usage tracking information, includes physical SQL
• Exalytics Summary Advisor uses S_NQ_SUMMARY_ADVISOR
‣ Contains summary statistics, execution time etc
‣ Gathered at same time as usage tracking when
Exalytics is enabed
‣ Contents can be derived from usage tracking if needed


Oracle Exalytics Best Practices

• Installation, Configuration for Multiple Environments, Management
• Choosing What Goes in the Adaptive Data Mart
• Where Does Exalytics Add Value?
• Measuring Exalytics Performance and Metrics
• Maintaining and Refreshing the Adaptive Data Mart


Configuring Multiple Environments (Consolidation)

• Exalytics main value proposition is around performance
‣ In-memory aggregates speeding up query performance
• However many customers wish to use Exalytics for consolidation
‣ Run multiple environments (test, dev, prod) on a single server
‣ Consolidate multiple applications on a single server
• Desire is usually down to cost (multiple physical servers, per-processor licensing)

Single Exalytics Server Exalytics Svr 1 Exalytics Svr 2 Exalytics Svr 3

MWHOME1 MWHOME2 MWHOME3 MWHOME1 MWHOME1 MWHOME1

DEV DEV DEV DEV TEST PROD

vs.


Common Consolidation Approach - Multiple Installs

• Up until recently, customers wishing to consolidate usually performed multiple
Middleware Home installs
• OBIEE element usually works fine (installer detects existing ports use, configures OK)
• However TimesTen element may not work as expected with multiple installs
‣ Checkpointing clashes between TT installs, single TT install vs. multiple installs etc
‣ No resource caging between MWHome environments
• Approach is therefore not recommended for consolidation use-cases

Single Exalytics Server

MWHOME1 MWHOME2 MWHOME3

DEV DEV DEV


Best Practice for Consolidation : Oracle Virtual Machine

• Exalytics is now certified for use with Single Exalytics Server
Oracle Virtual Machine, Oracle’s
OVM Guest 1 OVM Guest 2 OVM Guest 3
hard-partitioning virtualization platform
MWHOME1 MWHOME1 MWHOME1
• OVM is installed first, on Exalytics “bare
metal”, with OVM Repository on external DEV DEV DEV

DB server (as with BIPLATFORM, MDS)
• Exalytics VM templates then used to create
required environments (dev, test etc)
• Also allows for licensing in smaller
OVM Hypervisor (Oracle VM Server)
per-processor license blocks (5, 10, 15 etc)
• OVM provides for resource caging, Oracle VM
Repository
isolating environments from each other
Oracle VM
Manager


High Availability using Clustered Exalytics Servers

• Multiple Exalytics servers can be clustered for HA and performance (load balancing)
‣ Requires hardware load balancer above clustered servers
• Linked together via InfiniBand (one for clustering, one for connection to Exadata)
• Note: does not double available RAM
- aggs are replicated in each TT database
‣ Note that Summary Advisor / nqcmd
does not replicate agg tables -
manual replication required
• HA is between TT instances, and OBIEE
managed server + system components


Management Best Practices

• Exalytics is primarily an OBIEE system, and usual best practices apply
• May want to consider cloning tool (CloneZilla etc) for cloning “clean” server build
• New : Oracle Enterprise Manager 12cR2 and BI Management Pack
‣ OEM 12cR2 prerequisite for sub-capacity licensing using OVM


OEM 12cR2 Additional Capabilities vs. Fusion Middleware Control

• Ability to report on DAC runs for BI Apps,
plus BI configuration settings etc
• Usage tracking reporting of dashboard usage,
analysis run-times etc
• Manage Essbase and other BI / DB /
Middleware components
• Long-term persistence of metrics, SLAs, alerts
• Middleware cloning/
management


What Should the TimesTen Adaptive Data Mart be Used For?

• Standard approach is to store aggregates in the TimesTen datamart
‣ Aggregated by the source DB, aggregates then cached in TT database
• Other approaches could be used, however
‣ Store whole detail-level dataset in the TT database
‣ Store just recent detail-level data in TT, and use OBIEE’s fragmentation feature
‣ Use Essbase to hold the aggregates instead
‣ Use Essbase to hold detail-level + aggregates
• Where does Exalytics add value?
• What should go in the TimesTen database?


Potential TimesTen Use Case : In-Memory Database Cache

• Automatically replicate “hot” transactional data from Oracle into TimesTen
• Use OBIEE fragmentation to enable automatic navigation between sources
• Aggregation performed by both TimesTen, and by source DB (as appropriate)
Cache Group is defined
on TimesTen

Transactional data Transactional data

LTS Fragmentation in
DATE = TODAY DATE = TODAY the RPD specifies the
data cutoff
In-Memory DataBase
Cache
DATE < TODAY

No change to Oracle OBIEE chooses the
source server, except correct source based
minor schema config on date range
BI Server


Conclusions and Best Practices Around Detail/Aggregate Storage

• Best use of OBIEE + TT is how the product was intended to be used
‣ Use TimesTen for storing (caching) pre-created aggregates
‣ Aggregates created by source DB (Oracle, SQL Server etc)
‣ Aggregation not performed by TimesTen (as it would be if it held detail-level data)
• IMDB interesting approach, but relies on TT performing aggregation (slow)
• Same goes for holding whole detail-level source in TimesTen
• Also, bear in mind that majority of performance benefit may come from re-hosting
OBIEE application on the Exalytics hardware
‣ Particularly with Exadata sources, source DB may even be fastest host for aggs
‣ Test for your data, do not pre-suppose outcome


Testing Exalytics

• Need to quantify the performance benefit of various Exalytics deployment options
• Take a baseline, and then repeat the same test for incremental changes
• Several test framework options
‣ Manual
‣ Homebrew - Logical SQL + nqcmd
‣ Performance test tool - JMeter, OATS, etc

Extract Logical
SQL, run with TimesTen Run dashboards
nqcmd manually, and time
with a stopwatch

Logical
SQL nqcmd BI Server

Fully automated
Presentation web user
Services performance test
JMeter tool


Plan your testing

• Tempting to go in all guns blazing
‣ 40 cores!! 1TB memory!! w00t!11!
• Decide on the aim of the test. For example:
‣ Validate a configuration option
‣ Determine the response time for a dashboard
‣ Test the performance under concurrent load
• For configuration option testing, emphasis should be on absolute repeatability
• If testing to determine system capacity, make sure the workload model is representative


Setting up JMeter

• JMeter can run on Windows or *nix
• No installation, just unzip
• Record and customize test script
• Execute from JMeter running from suitable location
‣ Consider network, and host server size

Option 1 Option 2 Option 3

BI Server BI Server BI Server Run remotely,
JMeter
across
TimesTen TimesTen TimesTen appropriate
network link to
Run locally on JMeter Virtual Exalytics
Exalytics Machine
JMeter

Run within a VM
on Exalytics


Analyzing the test output

• What’s the response time on a dashboard?

How long did the
dashboard take to
run?

• How does the response time
change if there are concurrent
users?

Response Time
vs
Number of users


Other metrics

• “Will the system support xxxx users?” is
a common question
• JMeter is good at simulating multiple
users OBI metrics

‣ Be careful with your workload model
though
• For system capacity in particular, monitor
OS and OBIEE performance metrics
• Consider EM12cR2 for long-term metrics OS metrics


Refreshing Aggregates Within the TimesTen Data Mart

• Aggregates are built in TimesTen to support sub-second response times
• Summary Advisor tool suggests aggregate dimensionality and grain, generates script
• BI Server’s Aggregate Persistence executes script:
1. Create TimesTen aggregate table
2. Populate TimesTen aggregate
3. Update RPD online with new aggregate metadata
• Handles supporting dimensions too
Base data Aggregates
OBIEE Aggregate
Persistence
Aggregate tables
created

Aggregate tables TimesTen
loaded from base
data RPD updated with
new aggregate
mappings
RPD


Aggregate Persistence limitations

• To refresh an aggregate, it is deleted and rebuilt from scratch
• The RPD is edited directly on the BI Server each time an aggregate is created or rebuilt
• Build failures can be difficult to debug
• If a rebuild fails it can leave the RPD in an inconsistent state with TimesTen

This is that the
error the user
sees

After a failed build, the aggregates are nqquery.log suggests possible errors
still in the RPD, but no longer exist in but no clear root cause
TimesTen


Alternatives to using Aggregate Persistence

• However aggregates are refreshed, they must be included
in the RPD
‣ Can be done manually
‣ Aggregate Persistence is useful for this
• Complete refresh of aggregate data
‣ Extract the SQL that OBIEE generates in Aggregate
Persistence, run this through ODI
‣ Write bespoke aggregate refresh code in ODI
• Incremental refresh using GoldenGate and ODI
‣ Instead of rebuilding aggregates in their entirety each
time, only update the part of the aggregate that has
changed on the base data


Incremental refresh of aggregates using GoldenGate and ODI

• Changes to base data are replicated by GoldenGate onto Staging database
• ODI Interfaces apply deltas to the TimesTen aggregates

Aggregates
Base data GoldenGate CDC staging data ODI

TimesTen
(any supported DB) Oracle

TimesTen ODI
GoldenGate GoldenGate JDBC client agent


Considerations if using Alternatives to Aggregate Persistence

• Exalytics “Out of the Box” is designed around using Aggregate Persistence
‣ Fair assumption that future developments and optimizations will be around this
model
• If rate of change in aggregate requirements is high, then manual RPD changes can be
time-consuming
• Hybrid approach - use Summary Advisor and Aggregate Persistence as an accelerator
for the initial RPD and Aggregate build, and then handle refreshes manually
• There is no one answer -- It Depends!
‣ Do you have time & skills available to write bespoke code?
‣ Rate of change in your aggregate requirements
‣ Existing deployment landscape and technologies available
‣ Risk of missing out on Exalytics optimizations if too far from OOTB deployment?


Summary

• Exalytics is a powerful platform for hosting OBIEE, optimized for a BI workload
• New supported options for hosting multiple environments on a single server
• Oracle VM now permitted for multiple environments and sub-capacity licensing
• TimesTen is best used as a cache store for aggregate tables
• Typically, use source DBs for initial aggregation and storage of detail-level data
• Use JMeter or other metrics capture tools to measure performance before and after
• Manage Exalytics (+ OVM) using EM12cR2 Cloud Control
• Options to use ODI + GoldenGate as alternatives to Summary Advisor for agg refreshes


More Information

• Thank you for attending this presentation
• More information can be found at http://www.rittmanmead.com
• Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
• Look out for our book, “Oracle Business Intelligence Developers Guide” due Q3 2012
• Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)


Oracle Exalyics + TimesTen Best Practices (OOW 2012)

Recomendados

Recomendados

Más contenido relacionado

Más de Mark Rittman

Más de Mark Rittman (20)

Último

Último (20)

Oracle Exalyics + TimesTen Best Practices (OOW 2012)

Notas del editor