Oracle Exadata Performance: Latest Improvements and Less Known Features

1. blog.tanelpoder.com
1
©
2015
Tanel
Poder
Exadata
Performance:
Latest
Improvements
and
Less
Known
Features
Tanel
Poder

2. blog.tanelpoder.com
2
©
2015
Tanel
Poder
Intro:
About
me
• Tanel
Põder
• Consultant,
Trainer,
Speaker,
Computer
Performance
Geek
• 17+
years…
• Oracle
Database
Performance
• Exadata
Performance
• Hadoop
&
Linux
Performance
• Director
of
R&D
at
Accenture
Enkitec
Group
• We
rock!
J
Expert
Oracle
Exadata
book
(with
Kerry
Osborne
and
Randy
Johnson
of
Enkitec)

3. blog.tanelpoder.com
3
©
2015
Tanel
Poder
Prerequisites
• This
is
a
random
list
of
recent
Exadata
improvements
and
less
known
features
• You
already
should
know
Exadata
fundamentals
(what
is
a
smart
scan,
flash
cache,
storage
index
etc)

4. blog.tanelpoder.com
4
©
2015
Tanel
Poder
Agenda
1. Oracle
12c
SQL
Monitoring
Improvements
for
Exadata
2. Storage
Index
Improvements
• IO
pruning
with
Bloom
filters
• Min/Max
IO
pruning
3. Recent
Flash
Cache
Improvements
• Columnar
Flash
Caching
4. Metrics
Improvements
• Storage
Cell
Metrics
in
AWR
5. Q&A
6. Download
all
my
scripts,
tools
from:
• blog.tanelpoder.com/files/

5. blog.tanelpoder.com
5
©
2015
Tanel
Poder
Latest
Cellsrv
Documenta^on?
• Stored
in
storage
cell
filesystem:
• /opt/oracle/cell/doc/doc/
• Also
available
from
MOS
as
a
patch:

6. blog.tanelpoder.com
6
©
2015
Tanel
Poder
SQL
Monitoring
Improvements

7. blog.tanelpoder.com
7
©
2015
Tanel
Poder
Exadata
Metrics
in
SQL
Monitoring
reports
(11g)

8. blog.tanelpoder.com
8
©
2015
Tanel
Poder
SQL
Monitoring
Improvements
in
Oracle
12c
• Execu^on
plan
line
level
addi^onal
metrics
in
the
Other
column!!!

9. blog.tanelpoder.com
9
©
2015
Tanel
Poder
SQL
Monitoring
Improvements
in
Oracle
12c
HASH
JOIN
row
source
JOIN
FILTER
CREATE
row
source
Only
one
row
was
used
for
building
the
filter,
resul^ng
in
one
bit
set

10. blog.tanelpoder.com
10
©
2015
Tanel
Poder
SQL
Monitoring
Improvements
in
Oracle
12c
TABLE
ACCESS
FULL
on
orders
TABLE
ACCESS
FULL
on
customers
Only
115kB
was
returned
out
of
760MB
of
smart
IO
issued.
All
IO
from
Flash
Cache
Storage
Indexes
helped
to
skip
424
MB
of
IOs

11. blog.tanelpoder.com
11
©
2015
Tanel
Poder
v$sql_monitor_statname
in
Oracle
12c
SQL> @sqlmon_sn
NAME DESCRIPTION
---------------------------------------- -------------------------------------------------------------------------
Build Size Size of the build input in bytes
Build Row Count Number of rows for the build
Fan-out Number of partitions used to split both inputs
Slot Size Size of an in-memory hash-join slot
Total Build Partitions Total number of build partitions
Total Cached Partitions Total number of build partitions left in-memory before probing
Multi-pass Partition Pairs Total number of partition pairs processed multi-pass
Total Spilled Probe Rows Total number of rows from the probe spilled to disk (excluding buffering)
Bitmap Size Size of the bitmap in bytes
...
Folded Bitmap Size Folded size of the bitmap in bytes
Bits Set Number of bits set in the bitmap
Total Creator Rows Total number of rows from the creator side
Total user Rows Total number of rows from the user side
Total Filtered Rows Total number of rows filtered by the bloom filter
Times hash func called Number of times hash func is called
Slow metadata bytes Size of slow metadata in bytes
Eligible bytes Total bytes eligible for offload
Filtered bytes Total bytes returned after offload
SI saved bytes Total bytes saved by storage index
Flash cache bytes Total bytes fetched from flash cache
Partial flash cache and disk bytes Total bytes where IO was partially fetched from flash cache and disk
Cell passthru IO bytes Total cell passthru IO bytes
Block IO bytes Total block IO bytes
Slow metadata bytes Size of slow metadata in bytes
This
is
just
a
glance,
the
script
produces
more
output
of
available
plan
line
level
metrics

12. blog.tanelpoder.com
12
©
2015
Tanel
Poder
Flash
Cache
Improvements

13. blog.tanelpoder.com
13
©
2015
Tanel
Poder
Oracle
11.2.0.4
Cell
Flash
Cache
Write
Hit
Metrics
SQL> @ses2 "select sid from v$session where program like '%DBW0%'" flash
SID NAME VALUE
---------- ---------------------------------------------------------------- ----------
296 cell writes to flash cache 50522
296 cell overwrites in flash cache 43998
296 cell flash cache read hits 36
SQL> @ses2 "select sid from v$session where program like '%DBW0%'" optimized
SID NAME VALUE
---------- ---------------------------------------------------------------- ----------
296 physical read requests optimized 36
296 physical read total bytes optimized 491520
296 physical write requests optimized 25565
296 physical write total bytes optimized 279920640
Previously
the
flash
write
hits
were
also
accumulated
under
the
cell
flash
read
hits
metric!
The
cell
write
hits
(of
IO
requests)
metrics
can
be
2-­‐3x
higher
than
DB
write
requests
due
to
ASM
mirroring
(each
cell
reports
its
own
hits
back)
hop://blog.tanelpoder.com/2013/12/04/cell-­‐flash-­‐cache-­‐read-­‐hits-­‐vs-­‐cell-­‐writes-­‐to-­‐flash-­‐cache-­‐sta^s^cs-­‐
on-­‐exadata/

14. blog.tanelpoder.com
14
©
2015
Tanel
Poder
Comments
on
Flash
Cache
(pre-­‐11.2.3.3.0
cellsrv)
• ALTER
TABLE
t
STORAGE
(CELL_FLASH_CACHE
KEEP);
• DEFAULT
• KEEP
• NONE
• Smart
Scans
bypass
the
Flash
Cache
regardless
of
I/O
size,
unless
CELL_FLASH_CACHE
=
KEEP
• (even
if
the
table
data
is
already
cached
there)
• The
Flash
Cache
I/O
path
is
a
FIFO,
no
IORM
for
Flash
Cache
before
11.2.3.3.0
• With
DEFAULT,
buffered
I/O
opera^ons
will
cache
data
of
small
IOs
• Small
I/O
<
128kB
• This
includes
TEMP
IO
requests
smaller
than
128kB!
• Flash
cache
popula^on
is
done
asynchronously
by
separate
jobs
in
cellsrv
The
KEEP
setng
works
at
the
table
level
(not
par^^on
level).
Kept
segments
get
automa^cally
unkept
auer
24
hrs
of
no
access:
(
_cell_object_expira9on_hours
=
24
)

15. blog.tanelpoder.com
15
©
2015
Tanel
Poder
Automa^c
expiry
of
KEEP
objects
from
Flash
Cache
CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName='DEMO' AND hoursToExpiration < 24 DETAIL;
cachedKeepSize: 23345496064
cachedSize: 23364829184
cachedWriteSize: 327680
dbID: 1538629110
dbUniqueName: DEMO
hitCount: 14116
hoursToExpiration: 14
missCount: 35955
objectNumber: 88330
tableSpaceNumber: 27
CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName='DEMO' AND objectNumber = 88330 DETAIL;
cachedKeepSize: 24099815424
cachedSize: 24119345152
cachedWriteSize: 327680
dbID: 1538629110
dbUniqueName: DEMO
hitCount: 20797
hoursToExpiration: 24
missCount: 36700
objectNumber: 88330
tableSpaceNumber: 27
The
KEEP
object
hasn't
been
accessed
for
a
while
(~10
hours
since
last
access)
The
^mer
back
to
24
hours
auer
a
simple
COUNT(*)
query
on
the
table

16. blog.tanelpoder.com
16
©
2015
Tanel
Poder
Flash
Cache
on
Cellsrv
11.2.3.3.0+
• Automa^c
–
even
the
CELL_FLASH_CACHE
DEFAULT
objects
can
be
Smart
Scanned
from
Flash
Cache
• IORM
now
throoles
Flash
Cache
IOs
on
the
flash
disks
too
• Previously
only
IOs
on
hard
disks
and
disks
carved
out
of
flash
were
IORM-­‐ed
• Reduces
commit
latency
as
Smart
Flash
(Redo)
Logs
are
on
the
same
devices

17. blog.tanelpoder.com
17
©
2015
Tanel
Poder
Cellsrv
12.1.2.1.0
(January
2015)
• Lots
of
improvements,
but
let's
look
into
columnar
flash
cache
• Independent
dual
format
caching
in
storage
cell
flash
• Smart
Scans
use
Columnar
Flash
Cache
(if
all
required
columns
are
cached)
• Block
IO
uses
tradi^onal
Flash
Cache
• Conceptual
overview
tradi^onal
vs.
columnar
Flash
Cache
CU
1
col
1
col
2
col
3
col
3
col
4
col
4
col
5
col
5
col
6
CU
2
col
1
col
2
col
3
col
3
col
4
col
4
col
5
col
5
col
6
CU
x
col
1
col
2
col
3
col
3
col
4
col
4
col
5
col
5
col
6
CU
1
..
x
col
1
col
2
col
3
col
4
col
5
col
6
col
2
col
2
col
3
col
3
col
4
col
4
col
5
col
5
col
5
col
6
compact
data
–
less
flash
IO
for
a
column
scan

18. blog.tanelpoder.com
18
©
2015
Tanel
Poder
Columnar
Flash
Cache
metrics
–
1
SQL> @snapper all 5 1 1234
--------------------------------------------------------------------------------------------
TYPE, STATISTIC , DELTA, HDELTA/SEC
--------------------------------------------------------------------------------------------
STAT, physical read IO requests , 6416, 1.26k
STAT, physical read bytes , 6717366272, 1.32G
STAT, cell physical IO bytes eligible for predicate offload , 6717366272, 1.32G
STAT, cell physical IO interconnect bytes returned by smart scan, 445177024, 87.68M
STAT, cell physical IO bytes saved by columnar cache , 5128519680, 1.01G
STAT, cell blocks processed by cache layer , 250366, 49.31k
STAT, cell blocks processed by txn layer , 250366, 49.31k
STAT, cell blocks processed by data layer , 50327, 9.91k
STAT, cell flash cache read hits , 6416, 1.26k
WAIT, enq: KO - fast object checkpoint , 747, 147.12us
WAIT, cell smart table scan , 315077, 62.05ms
WAIT, SQL*Net message to client , 8, 1.58us
WAIT, SQL*Net message from client , 2668093, 525.48ms
WAIT, events in waitclass Other , 1136, 223.73us
Stats snap 1, end=2015-02-11 10:57:34, seconds=5.1

19. blog.tanelpoder.com
19
©
2015
Tanel
Poder
Columnar
Flash
Cache
metrics
-­‐
2

20. blog.tanelpoder.com
20
©
2015
Tanel
Poder
Lis^ng
Columnar
Flash
Cache
usage
in
cell
CellCLI> LIST METRICCURRENT FC_COL_BY_USED DETAIL;
name: FC_COL_BY_USED
alertState: normal
collectionTime: 2015-02-10T23:24:44-06:00
metricObjectName: FLASHCACHE
metricType: Instantaneous
metricValue: 2,059 MB
objectType: FLASHCACHE
CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName = 'DW' AND objectNumber = 101821 DETAIL;
cachedKeepSize: 0
cachedSize: 4418445312
cachedWriteSize: 18243584
columnarCacheSize: 2159017984
columnarKeepSize: 0
dbID: 1996233620
dbUniqueName: DW
hitCount: 48176
missCount: 5238
objectNumber: 101821
tableSpaceNumber: 8
"Number
of
megabytes
used
in
Columnar
FlashCache"
You
can
experiment
with
it
using
_enable_columnar_cache
session
parameter

21. blog.tanelpoder.com
21
©
2015
Tanel
Poder
Storage
Index
Improvements

22. blog.tanelpoder.com
22
©
2015
Tanel
Poder
Skipping
IOs
via
Bloom
Filters
+
Storage
Indexes
• hops://blog.tanelpoder.com/2014/05/17/combining-­‐bloom-­‐filter-­‐
offloading-­‐and-­‐storage-­‐indexes-­‐on-­‐exadata/
SQL> SELECT
2 *
3 FROM
4 soe.customers c
5 , soe.orders o
6 WHERE
7 o.customer_id = c.customer_id
8 AND c.cust_email = 'florencio@ivtboge.com'
9 /
We
have
a
direct
filter
only
on
the
CUSTOMERS
table,
but
a
Bloom
filter
constructed
from
it
may
allow
to
skip
IOs
on
the
ORDERS
table
too!
1. Remember
the
join
column's
MIN/MAX
in
the
driving
row-­‐source
2. Push
the
MIN/MAX
of
the
join
column
to
storage
with
Bloom
filter
3. Compare
the
MIN/MAX
value
to
the
storage
index
to
skip
IOs
on
the
joined
table

23. blog.tanelpoder.com
23
©
2015
Tanel
Poder
Skipping
IOs
for
MIN/MAX
calcula^on
• If
both
projected
and
filtering
column
are
in
storage
indexes…
• …skip
storage
regions
where
it's
known
no
bigger
MAX
(or
min)
values
exist
• Works
if
you
don't
actually
need
to
access
the
values
(COUNT,
SUM,
AVG…)
• Controlled
by
_cell_storidx_minmax_enabled
parameter
SELECT MAX(cust_postal_code)
FROM customers_nocache
WHERE cust_id > 1234;

24. blog.tanelpoder.com
24
©
2015
Tanel
Poder
What
else?
• MOS
Note
1947476.1
• Exadata
12.1.2.1.0
release
and
patch
(17885582)
• Sparse
Grid
Disks
• Snapshot
Databases
for
Dev/Test
(Thin
Cloning
in
Exadata!)
• V$ASM_DISKGROUP_SPARSE,
V$ASM_DISK_SPARSE,
etc
• Limi^ng
Flash
Cache
space
usage
per
DB
• IORM
interdatabase
resource
plan
• 12c
AWR
reports
contain
storage
cell
performance
data!
• V$CELL_DB,
DBA_HIST_CELL_DB
and
many
others

25. blog.tanelpoder.com
25
©
2015
Tanel
Poder
Further
Reading
–
Smart
Scans
• My
Exadata
blog
entries:
• hop://blog.tanelpoder.com/category/exadata/
• Frits
Hoogland's
Exadata
ar^cles:
• hop://fritshoogland.wordpress.com/category/oracle/exadata/
• The
Expert
Oracle
Exadata
book
• We
are
working
on
a
2nd
edi^on
of
this
book
(Andy,Karl,Mar^n,Frits)

26. blog.tanelpoder.com
26
©
2015
Tanel
Poder
Thank
You!
Ques^ons?
tanel@tanelpoder.com
blog.tanelpoder.com
@tanelpoder
www.enkitec.com