11. Journaling
CPU Index
Operations Maintenance
Data
Processing
I/O Constraint
Operations Enforcement
Trigger
Locking
Processing
12. Server
Attributes
Server OS Version
Configuration
Server SMP
Performance
Job, Query System i Database
attributes Design
DBMS
SQL
Table sizes
Request
SQL Views &
Interfaces Indexes
Work
Management
13. Memory
Utilization
Fixed/Calc
Work Prestart
Managemen Jobs
t QZDASOINIT
Data SQL
Providers Statements
Query Index
Options Query Recomme-
QAQQINI ndations
Journaling Compilation
Analysis Options
Unused
System
Indexes &
Values
Views
Query
Engines
(SQE/CQE)
14. Indexes
Advised
Print SQL
SQE Plan
Information
Cache
Messages
Visual
Explain
Debug Tools & SQE Plan
Job log Cache
Messages Methods Snapshots
Detailed DB
Views &
Monitor
Indexes
Data
Summarized
DB Monitor
Data
15. Time spent in data access:
• In the processor
• Waiting for disk I/O
• Waiting for communications I/O
• Waiting for some resource that some other job is using
In most environments the first two are the largest factors.
17. SQL Interface
SQL Request processing
Static SQL
Requester Syntax Gather Build internal query Dynamic SQL
Communication Validation Attributes structures Extended Dynamic
SQL
SQL Optimizer
Create & implement the data manipulations
Query
Query Query Query methods Query
implementation
validation dispatching costing plan creation
ODP
Database Engine
executing the query plan that the optimizer provides
Build the structures Build the structures Build & activate Generate any
Debug messages in Gather database Use DB2 Visual
needed for query for any temporary query cursor feedback
the job log monitor records Explain
cursor indexes (if needed) (ODP) requested
18.
19. • Static SQL
SQL Code hardcoded (embedded ) into the application.
SQL that has been pre-compiled into a package or plan may be executed
directly by DB2 without additional preparation.
• Dynamic SQL
SQL text is provided by the user or generated by the application at execution
time SQL that must be prepared for execution every time at run time.
• Extended Dynamic SQL
Prepare once then reference.
Host Server CLI/JDBC
Static Extended
Dynamic
Compiled
Prepare every
Dynamic
Embedded Prepare once then
time
Statements reference
20.
21.
22. • Check whether this is a new request.
• If this request has not been optimized previously, no plan currently exists, thus full optimization is required.
• If this request has previously been optimized, the saved plan must be validated to ensure that it is viable for the current request.
• If the plan is valid, optimization can be eliminated or minimized.
Query • If the plan is not valid, full optimization is required. A plan can be invalidated when the query environment changes or the database
objects change.
validation
• Determine which query engine can complete the processing.
• The goal is to let the SQE optimize the query.
• Depending on the DB2 release level, the SQE might be incapable of running a particular query.
Query
dispatcher
• Determine what methods are available to access the data.
• Choose the best strategy to be employed.
Query Costing • The methods and strategy are assembled into a Access Plan that the database engine executes.
Plan Creation
• DB2 optimizer creates the ODP for the query.
• The ODP consists of the cursor, cursor behavior, and data mapping constructs.
• This phase is also where the optimizer manifests any feedback information about the query and query plan.
Query • Finally, the plan is handed off to the database engine for execution.
Implementation
23.
24. • Two SQL/Query engines.
• The Optimizer now determines which engine will handle the query request through the query dispatcher:
• Classic Query Engine (CQE)
• SQL Query Engine (SQE)
• There is still only one interface into the optimizer.
25. ODBC/JDBC/ADO/DRDA/XDA
Network
Host Server CLI/JDBC
Static Dynamic Extended
Compiled Prepare every Dynamic
Embedded time Prepare once then
Statements reference
SQL
Native
Record I/O
Optimizer
Machine Interface (MI)
DB2 UDB (Data Storage & Management)
SLIC (System Licensed Internal Code)
CQE Database Engine SQE Database Engine
26. ODBC/JDBC/ADO/DRDA/XDA
Network
The Only way for HLL Host Server CLI/JDBC
programs to use the
Static Dynamic Extended
new engine (SQE) is via Compiled Prepare every Dynamic
embedded SQL Embedded time Prepare once then
Statements reference
SQL
Optimizer
Query Dispatcher
Native
Record I/O
CQE Optimizer SQE Optimizer
Machine Interface (MI) The optimizer &
database engine
SLIC (System Licensed Internal Code)
merged, to form the
DB2 UDB (Data Storage & Management) Database Engine SQL Query Engine &
much of the work was
SQE Optimizer moved to
SLIC
SQE Statistics Manager
CQE Database Engine SQE Data Access Primitives
27. SQL Options V5R2 V5R2 V5R3 V5R3 V5R4 V5R4 V6R1 V6R1
CQE SQE CQE SQE CQE SQE CQE SQE
LIKE Predicates Y Y Y Y
Logical File references Y Y Y Y
UDTFs Y Y Y Y
LOB columns Y Y Y Y
LOWER, TRANSLATE or UPPER scalar Y Y Y Y
CHARACTER_LENGTH, POSITION, or Y Y Y Y
SUBSTRING scalar using UTF-8/16
Alternate sort sequences Y Y Y Y
Derived Logical Files over Physical (S/O) Y Y Y Y
Non-SQL : QQQQry API, Query/400, OPNQRYF Y Y Y Y
ALWCPYDTA(*NO) Y Y Y Y
Sensitive Cursor Y Y Y Y
VIEWS, UNIONS, SubQueries Y Y Y Y
INSERT, UPDATE, DELETE Y Y Y Y
Star Schema Join queries Y Y Y Y
28.
29. Query Dispatcher,
• Determines which engine optimizes and processes query requests
• Considers SQL requests only for the query engines
• Serves as the initial step for all query optimization in the i5/OS environment
• Provides support to use the classic SQE when encountering nonstandard indexes during optimization
Remaining SQE
Older Technolo Restrictions as of V5R4:
DDS LF References, DDS LF References,
DDS Select/Omit LF, DDS Select/Omit LF,
Non-SQL Interfaces Non-SQL Interfaces
Optimizer
Query Dispatcher
CQE Optimizer SQE Optimizer
DB2 UDB (Data Storage & Management) Database Engine
CQE SQE
30. Query Request is Dispatched to CQE if:
• Logical file references
• UDTFs
• LOWER, TRANSLATE, or UPPER scalar function
• CHARACTER_LENGTH, POSITION, or SUBSTRING scalar function
using UTF-8/UTF-16
• Sort sequences and CCSID translation between columns
• DB2 Multisystem
• Non-SQL queries (QQQQry API, Query/400, OPNQRYF)
SQE now (V5R4) optimizes:
• LIKE predicates
• LOB columns
• Sensitive cursors
• ALWCPYDTA(*NO)
31. Revert from SQE to CQE if the optimizer encounters:
• Select/omit logical file
• Logical file over multiple members
• Join logical file
• Derived key(s)
• Native logical files that perform some intermediate mapping of the fields
referenced in the key: renaming fields, adding a translate, or only selecting a
subset of the columns
• Specifying an alternate collating sequence (ACS) on a field used for a key will also
make a derived key(an implied map occurs within the index).
• Sort sequence (NLS) specified for index or logical file
• Cost to back up and revert to CQE adds about 15% to the total optimization time.
• QAQQINI parameter to ignore unsupported logical files
IGNORE_DERIVED_INDEX = *YES
32.
33. We can affect
THIS!
Query Query
dispatcher • Choose most efficient execution
• Validate the query
request access method
• Validate existing query • Determine which query • Build access plan
access plan engine can complete
the processing • Build the structures needed for
query cursor
• Build internal query
structures • Build the structures for any
temporary indexes (if needed)
Query Query • Builds-and activate query cursor
Parsing & (ODP)
optimization
validation
• Generate any feedback
requested
• Debug messages in the job log
• Gather database monitor records
• Use DB2 Visual Explain
36. • Non-dynamic SQL statements embedded in application programs.
• Validation:
• SQL pre-compiler validates and parses the SQL statements in the HLL
program.
• Turns them into host language statements.
• Optimization:
• An access plan is then created for each SQL statement and is stored in the
program object or module object.
• Since all optimization parameters are not known prior to actual execution,
the optimizer creates only a generic optimization plan during pre-compile.
• Execution:
• Binding: Host language statements interface with the database manager
upon execution of the program.
• The plan receives detailed analysis on the first execution of the statement,
and then the optimizer updates the original access plan.
37. • Languages Supported:
• RPG
• COBOL
• C, C++
• PL/I
• SQLJ
• Embedded SQL is the most efficient SQL interface to use on System i
since SQL Access Plan is stored in the program objects.
• When the program is run, the validation and access-plan creation steps
can be skipped. thus, improving performance and making the entire
query effort much faster.
38. CRTSQLxxx
Create
Executable
Parse
&
Access Executable
Plan Validate
Optimize &
Create
Access Plan
* Generic plan quickly generated at compile time
* Complete, optimized plan first execution time
39.
40. • With dynamic execution, the SQL statement is completely ignored at program preparation time and
constructs SQL statements on the fly – at runtime.
• So, at runtime the application has to communicate more information with the database, before it
executes the statement.
• The operational form of the statement (ODP) persists for the duration of the connection or until the
last SQL program leaves the call stack.
• Access plans associated with dynamic SQL might not persist after a database connection or job is
ended.
• DB2 for i interfaces that utilize Dynamic SQL:
• CLI • Embedded Dynamic SQL
• JDBC • RUNSQLSTM
• ODBC, OLEDB, .NET • Interactive SQL (STRSQL)
• PHP • System i Navigator SQL requests
• SQLJ • DB2 Web Query
• REXX • Net.Data
• Greater performance overhead.
• Less sharing & reuse of resources – using JOB Working Memory
41. PREPARE and EXECUTE
EXECUTE IMMEDIATE
Create
Executable
(Prepare)
Parse
&
Access Validate
Plan
Optimize &
Create
Access Plan
* Generic plan quickly generated at prepare time
* Complete, optimized plan first execution time
42. One simple way to think of this is that static vs. dynamic SQL is similar to compiled
vs. interpretive applications in that the executable code is generated before
execution time rather than during execution time. Because there is a CPU cost
associated with analyzing an SQL statement and creating an access plan, dynamic
SQL can hinder application performance.
Cost of
Application
PREAPARE &
Requirements
OPTIMIZATION
• CPU cost of Preparation • Needs of the application
• Cost of doing repetitive Optimization • Skill set of developers
• Cost of caching options • Supporting infrastructure
43.
44. • An extended dynamic SQL statement is neither fully static nor fully dynamic.
• Permanent system object (SQL package) is used to store the access plans.
• SQL packages are used to make the implementation of the dynamic SQL
statements similar to that of static SQL.
• SQLPKG allows Dynamic SQL access plans to be shared across
users and to remain intact after a job or connections ends.
45. QSQPRCED API
ODBC/JDBC
Has this Dynamic
request been
Create
previously
Executable
executed?
(Prepare)
Parse
&
Access
Plan Validate
Optimize &
Create
Access Plan
* Generic plan quickly generated at prepare time
* Complete, optimized plan first execution time
46. • SQL packages are permanent objects with the object type *SQLPKG used to store information related to
prepared, extended dynamic SQL statements.
• SQL package contains all the necessary information to execute the prepared statement.
• When using embedded SQL, no separate SQL package is created, but the access plan is integrated into
the program or service program object.
• For environments which make use of the SQL Query Engine (SQE), SQL packages no longer contain
the entire access plan which is instead stored in the SQL Plan Cache. Even in these environments, the
information in the SQL package provides better reuse of SQL statement information.
• An SQL statement goes into the package only if one of the following is true:
• The statement contains parameter markers.
• It is an INSERT with sub-select (INSERT INTO table1 SELECT FROM table2 WHERE...)
• It is a positioned UPDATE or DELETE.
• It is a SELECT FOR UPDATE
47. • Any SQL statement that can be prepared is eligible
• Unlike SQL programs where all access plans are rebuilt when the associated DB2 table definitions are
changed, there are times when the plans stored in an SQL Package are not automatically rebuilt by DB2
UDB for iSeries. So, recreate your SQL packages any time the definitions for your database objects have
changed.
• Size limitations
• Current size limit is 500 MB, about 16K statements
• Package can grow without new statements being added.
• Access plan rebuilds require additional storage
• DB2 does try to perform package compression in the background to increase life & usefulness of
package objects
48. Advantages of extended dynamic SQL packages:
• Shared resource available to all users
• Access information reuse eliminates need for others to "relearn" SQL statement
• Permanent object saves information across job/system termination (IPL)
• Can be saved/restored to other systems
• Improved performance decisions since statistical information accumulates for each SQL statement
The statistics kept in the object include:
• number of times the package is used,
• number of times a packaged statement is executed,
• number of rows fetched by a statement.
With these pieces of information, statements in an SQL package tend to go into reusable ODP mode
after the first execution, which rapidly improves performance, automatically!
49. System API — QSQPRCED
• API user responsible for creating package
• API user responsible for preparing and describing statement into package
• API user responsible for checking existence of statement and executing statements in package
XDA API set
• Abstraction layer built on top of QSQPRCED for local and remote access
IBM Client Access ODBC driver & IBM Toolbox for Java JDBC driver Drivers
• Handle package creation
• Drivers automate the process of adding statements into package
• Drivers automate process of checking for existing statement and executing statements in package
50. • Choose most efficient
access method
• Build access plan
Query
optimization
51.
52. • An access plan is a control structure that describes the actions necessary to satisfy each query request.
• Think of this plan as the program that the database engine runs.
• An access plan or query plan is the output of the query optimization process.
• An access plan includes all of the optimized information that is necessary to accomplish the query.
Statement
Name &
Text
Associated Internal
Tables & parse tree
fields of statement
Query
Access Plan
isolation
level &
Statistics
commitment
control level
53. The access plan is validated when the query is opened. In validation the following is verified:
• Different File Or Member(change in library list or default schema)
• More Than Ten Percent Change In Number Of Rows
• New Access Path Found(Index created)
• Access Path No Longer Found Or Valid(Index deleted)
• Different CCSID
• Different Date Or Time Format
• Different Sort Sequence Table
• Different Storage Pool or Paging Option(Optimizer‘s fair share of memory pool change)
• Symmetric Multi Processing Change
• QAQQINI Change
• Different Isolation Level or Scroll Option
• New Release
54. The DB2 for i Optimizer performs ―Cost Based" optimization.
The goal for the DB2 for i optimizer is to produce an Access Plan that will
allow the query to execute in the shortest time period possible.
The optimizer chooses an optimal access method for the query by calculating an
implementation cost based on the current state of the database.
The optimizer uses 2 costs when making decisions:
• I/O cost
• CPU cost.
The goal of the optimizer is to minimize both I/O and CPU cost.
The optimizer has the ability and freedom to ―Rewrite the Query”.
55. Determining cost of using existing indexes:
The Optimizer orders the indexes:
• For SQE, the indexes are ordered in general such that the indexes that access the smallest number
of entries are examined first.
• For CQE, the indexes are generally ordered from mostly recently created to oldest.
For each index available, the optimizer does the following:
• Determines if the index meets the selection criteria.
• Determines the cost of using the index by estimating the number of I/Os and the CPU cost that will be
needed to perform the Index Probe (or the Index Scan and the possible Table Probes).
• Compares the cost of using this index with the previous cost (current best).
• Picks the cheaper one.
• Continues to search for best index until the optimizer decides to look at no more indexes:
• For SQE, since the indexes are ordered so that the best indexes are examined first, once an index
that is more expensive than the previously chosen best index, the search is ended.
• For CQE, the time limit controls how much time the optimizer spends choosing an implementation.
56. A given query plan can be thought of as an
intersection of all the factors that affect cost based
optimization on a given server with a given
database design.
To really understand a given implementation plan
and it performance, one must know and understand
all the various factors and settings in effect at the
time of query optimization and execution.
Change any one or more of the factors and the
implementation plan and performance may change.
57. Job &
Query
Server QUERY
ACCESS Database
Software
Hardware PLAN
Request
58. Server
Attributes
QPFRADJ,
Server QQRYDEGREE
Configuration Server
Sub-Systems, Performance
Pools,
Activity Levels
CPU
Work
No. of Management
Processors QUERY
ACCESS
PLAN
Memory
OS
Main Storage
Version
Size
Server Model
& SMP
Architecture
59. Query Time
Limit
Allow Copy
Data
QAQQINI
ALWCPYDTA
Naming
CCSID
Convention
QUERY
ACCESS
PLAN
JOB
Optimization Description
Goal
*LIBL, Priority,…
Data Source
Driver
Configuration
JDBC,ODBC
60. Views
Indexes
MQTs
Radix, EVI
QUERY
Table
Statistics ACCESS Sizes
PLAN
Cached
Results Database
Design
MTI
Optimizer
CQE/SQE
61. Optimization
Goal
SQL Result t set
Statement Size
QUERY
Commitment
Static ACCESS Control
PLAN
Sensitivity
Dynamic
(Live Data)
Extended
Dynamic
62. How does the optimizer know which choice to make?
63. Many data access methods can satisfy a query (each with its own strengths
and weaknesses)
What data access method can be used to find the rows that contain the
CITY TELAVIV within a 1-million-row database table?
... WHERE CITY = ‘TELAVIV’
When...
• 1 row contains the city ‗TELAVIV‘.
• 1000 rows contain the city ‗TELAVIV‘.
• 100,000 rows contain the city ‗TELAVIV‘.
• 1,000,000 rows contain the city ‗TELAVIV‘.
How does the optimizer know which choice to make?
TABLE SCAN, INDEX, …
64. • If the correct statistics are collected and available, the cost-based optimizer more accurately
estimates the number of rows to process.
• Better estimates allow for better query optimization and the selection of the best query plan
• All query optimizers rely on statistics to make plan decisions.
• DB2 for i5/OS has always relied on indexes as its source for statistics.
• Other databases rely on manual statistics collection for their source.
• SQE offers a hybrid approach.
• Column statistics are automatically collected when indexes do not already exist.
65. ...WHERE Customer_No > 112358 GROUP BY Customer_No...
• No Index • No Index • Index
• No Statistics • Statistics • Statistics
• Q1 optimizes with • Q1 optimizes with • Q1 optimizes with
defaults statistics statistics & index
• runs without index • runs without index • runs with index
• Queues up statistics
request for Customer_No
Generate Create
Statistics for Index for
Customer_No Customer_No
66. Table Radix EVI
CQE Optimizer
Statistics
Analysis Access
SQL Query
Plan
Costing
67. Table
With Radix EVI
Statistics
Statistics
Manager
Q&A
SQE
SQL Query Optimizer Access
Plan
Costing
69. • More accurately describes multicolumn key value BEST
• Available immediately as index maintenance occurs
• Selectivity estimates from radix by reading n keys
• Selectivity from EVI by reading symbol table values
Existing Indexed
(Radix or EVI)
• SQE only
• Column cardinality, histograms, and frequent values list
• Constructed over a single column in a table
• Stored internally as a part of the table object after created
• Collected automatically by default for the system
• Statistics are not immediately maintained as the table changes
Column Statistics
• Statistics are refreshed as they become stale over time
• No representation of actual values in columns
Default sources WORST
70. • i5/OS statistics collection job - QDBFSTCCOL
• This job is reactive, based on query requests.
• Automatic collection runs in this background job at a very low priority.
• The statistics manager continuously analyzes entries in the plan cache and queues
up requests for the collection job.
• Options:
• *ALL Allow both user & system requested stats collections
• *NONE No stats collection is allowed
• *USER Only user requested stats collection allowed
• *SYSTEM Only system requested stats collection allowed
• iSeries Navigator GUI helps manage statistics collected by the system.
• There are also APIs to manage the statistics.
71. How does the optimizer know which choice to make?
72. If the goal is: First vehicle to the 10 meter mark, who will win?
The goal
determines
the plan
10 Meter
If the goal is: First vehicle to the 10 kilometer mark, who will win?
10 kilometer
73. The optimization goal:
• Tells the optimizer how many rows you expect to fetch per transaction.
• Optimizer builds a plan that is optimal for returning n or all rows expected
• Affects the query startup time and overall runtime
All rows
Next n rows
Next n rows
Next n rows
First n rows
First I/O All I/O
Read by Build and
key thru an use a hash
index table
74. Optimization goal will affect the optimizer's decisions, such as Use of
indexes, SMP, temporary intermediate results like hash tables
• Set via optional SQL statement clause
• OPTIMIZE FOR n ROWS
• OPTIMIZE FOR ALL ROWS
• Set via QAQQINI options file
• *FIRSTIO
• *ALLIO
• Default for dynamic interfaces is First I/O
• ODBC, JDBC, STRSQL, dynamic SQL in programs
• CQE - 3% of expected rows
• SQE - 30 rows
• Otherwise default is ALL I/O
• Extended dynamic, RUNSQLSTM, INSERT + subSELECT, CLI, static SQL in programs
• All expected rows
75. The Open Data Path is the actual pipe for moving data between the database and applications.
The creation of the Open Data Path is very expensive, in terms of performance, on iSeries servers.
Avoiding the creation of ODPs (Full Open) is the key to delivering high-performing SQL solutions on
iSeries server.
ODPs live in working memory of the job associated with the SQL request.
76.
77. • The SQE Plan Cache is always on, automatic and has no Database Monitor Overhead.
• Initially created with an overall size of 512 MB.
• The purposes of the Plan Cache are to:
• Reuse of a query access plan when the same query is re-executed
• Store runtime information for subsequent use in future query optimizations
• Once an access plan is created, it is available for use by all users and all queries, regardless
of where the query originates.
• Plans are optimized on-demand as new statistics or indexes become available.
• Foundation for a self-learning query optimizer to interrogate the plans to make wiser costing.
• Caches all access plans optimized by the SQE Optimizer.
• Access plans generated by CQE are not stored in the Plan Cache, instead, they are stored in
SQL Packages, the system-wide statement cache, and job cache.
• Works in conjunction with the system wide statement cache and the SQL programs,
packages and service programs.
78. • Multiple access plans (3) can be maintained for a single SQL statement, processed by SQE.
• Although the SQL statement is the primary hash key to the plan cache, a different
environmental setting can cause different access plans to be stored in the plan cache,
each one matching the specific environment.
Examples of these environmental settings include:
• Different SMP degree settings for the same query
• Different library lists specified for the same SQL request
• Different settings for the job‘s share of available memory in the current pool
• Different ALWCPYDTA settings
• Plan cache is automatically maintained to keep most active queries available for reuse.
As new access plans are created for the same SQL statement, older and less frequently
used access plans are discarded to make room for the new access plans.
• Conditions that can cause an existing access plan to be invalidated.
Examples:
• Specifying REOPTIMIZE_ACCESS_PLAN(*YES) or (*FORCE) in the QAQQINI
• Deleting or recreating the table that the access plan refers to
• Deleting an index that is used by the access plan
Plan cache is cleared during an IPL.
79.
80.
81. • DB2 for i also caches access plans for Dynamic SQL requests in the System Wide Statement Cache
(SWSC)
• Only access plans are reused (No ODP reuse)
• SWSC requires no administration
• Cache storage allocation & management handled by DB2
• Cache is created from scratch each IPL
• Cache contents cannot be viewed, max of 165,000+ statements
• SWSC cache does interact with the job cache
82.
83. • With Dynamic interfaces, full opens are avoided by using a "PREPARE once, EXECUTE many―.
statement
• A PREPARE does NOT automatically create a new statement and full open on each execution
• DB2 UDB performs caching on Dynamic SQL PREPAREs within a job
• DB2 UDB caching is not perfect (and subject to change), good application design is the only way to
guarantee ODP reuse
84. Perform Parameter
Marker Conversion
No MATCH (ODP or access plan)
NOT FOUND
found in job cache
Search the
SWSC Add statement
and its plan to
Search JOB Cache
for ODP‘s or
SWSC
Access Plans that
can be reused FOUND Match
in SWSC
Check that job
attributes match
Update the JOB Cache
entry for the statement SWSC attributers
to point at the plan in
the SWSC Attributes did
NOT match
Job & SWSC Create new plan
attributes
MATCHED for statement and
add plan to the
JOB Cache
Use SWSC
access plan
88. • Maintained Temporary Index (MTI) = Autonomic Indexes = Result Set Cache
• Classic Query Engine (CQE) has had the ability to create a temporary index but its usage is restricted
to a single job (not across queries) and single query only.
• MTIs created by SQE can be shared across queries and jobs like permanent indexes.
• The amount of temporary storage used by the database can be noticeably more than in previous
releases.
• SQE automatically controls the creation, maintenance and deletion of MTIs.
• Creation occurs when SQE can justify that a query's performance can be enhanced sufficiently by
using an MTI.
• An MTI is updated as the base table is updated.
The system cost of maintaining an MTI is similar to the cost of maintaining a permanent index.
• An MTI is deleted when:
• The last access plan, in the system plan cache, that refers to that MTI is removed.
• On IPL:
• MTIs are deleted at IPL time, so you may experience a post-IPL warm-up effect for
the queries that leverage MTIs.
• When a permanent index is created that covers the same columns as the MTI
89. How does the optimizer know which choice to make?
90. Indexes
Advised
Print SQL
SQE Plan
Information
Cache
Messages
Visual
Explain
System i
Debug Query SQE Plan
Job log Cache
Messages Optimization Snapshots
Tools
Views & Detailed DB
Indexes Monitor Data
Summarized
DB Monitor
Data
93. • Work management influences SQL performance.
• Size of pool determines which algorithms are used by the query optimizer.
• Use of Expert Cache *CALC helps database optimization and runtime.
• Lower activity levels let SQL use more resources.
Subsystem
Memory Pool Activity
Pool Size Tuning Level
94. CPU:
• SQE uses threads instead of tasks for splitting up a job
• Then, SQE uses fewer tasks.
• SQE can use SMP for executing the threads within an access plan
• SMP can also be used by the statistics engine to collect statistics for SQE
Memory:
• SQE uses a slightly smaller main storage footprint for a query compared to CQE.
• SQE can make its best computation only when the associated memory pool is
defined with a pool paging option of *CALC (expert cache)
• SQE Optimizer responds differently to changes in the memory pool size in
terms of rebuilding an access plan.
Disk Storage:
• SQE performs asynchronous I/O far more aggressively than CQE and fully uses
parallel pre-fetching of data.
• Therefore, any under configuration of the number of disk drives on a system is
accentuated by queries that use SQE, particularly those that access a large
number of table rows.
95. How does the optimizer know which choice to make?
96. FAIR SHARE MEMORY is:
Maximum amount of memory that can be allocated to a
query without having a detrimental effect on other jobs
running in the same memory pool.
97. Plan 1
Index Probe into Index
Index
Memory
Footprint
Query‘s
Fair Share
Plan 2
Hash probe into hash table
Hash Table
Memory
Footprint
98. • CQE fair share = memory pool size / MAX activity level value.
• CQE Optimizer rebuilds an access plan if there is a 2-fold change in size.
• SQE fair share = memory pool size / AVG activity level value
• SQE looks for a 10-fold change.
• AVG = min { max-active, max(avg-active, 5) }
• if Pool paging = *CALC THEN
• 15 minute AVERAGE number of users
avg-active =
• ELSE IF Pool paging = *FIXED THEN
• MAX activity level value
• If query degree is set to *MAX, then fair share = entire pool size
99.
100. • Enabling SMP for a given job allows that job to use multiple tasks or threads to
perform the work.
• Those multiple tasks or threads consume more resources with the goal of
faster response times.
• Multiple processors allow the tasks or threads to run in parallel.
In other words, more work is accomplished in the same unit of time.
102. 64 BIT Processors
Job A Thread J
Job B Thread I
Job C Thread H
Job D Thread G
Job E Job F
• n processors can work on several jobs or threads at one time without any special programming.
• No one job is running on more than one processor.
103. Job A 64 BIT Processors
Thread J
Thread Aa
Thread Ab
Thread Ac Thread I
Thread Ad
Thread Ae
Thread H
Thread Af
Thread Ag
Thread Ah
Thread G
Job B Job C
•The system automatically divides the query work into multiple tasks or threads.
• Multiple processors can work on one job’s tasks or threads.
104. Parallel processing allows a user to specify that queries should be able to
use either I/O or CPU parallel processing as determined by the optimizer.
• I/O parallelism allows for the accessing of data in parallel, but the processing of that data does not
occur in parallel.
• Processor/CPU parallelism allows for both the accessing of data and the processing of that data in
parallel.
• CPU parallelism is only available when DB2 Symmetric Multiprocessing is installed
• CPU parallelism does not necessarily require multiple processors
• Parallel processing is set on a per-job basis:
• The parameter DEGREE on the CHGQRYA CL command.
• The parameter PARALLEL_DEGREE in the QAQQINI file.
• The system value QQRYDEGREE.
• Each job will default to the system value (*NONE is the default).
• SMP can be used for parallel index creation on restore or recovery at IPL.
System value QQRYDEGREE controls rebuild of access paths on restore and IPL.
• DB2 Symmetric Multiprocessing is option 26 of System i
105.
106. • *NONE
No parallel processing is allowed for database query processing.
• *IO
Any number of tasks may be used when the database query optimizer
chooses to use I/O parallel processing for queries. CPU parallel processing is
not allowed. SQE always considers IO parallelism.
• *OPTIMIZE
The query optimizer can choose to use any number of tasks or threads for either
I/O or CPU parallel processing to process the query. Use of parallel processing
and the number of tasks or threads used will be determined with respect to the
number of processors available in the system, this job's share of the amount of
active memory available in the pool which the job is run, and whether the
expected elapsed time for the query is limited by CPU processing or I/O
resources.
107. • *MAX
The query optimizer can choose to use either I/O or CPU parallel processing
to process the query. The optimizer will assume that all active memory in the
pool can be used to process the query.
• *SYSVAL
Use current value of the system value QQRYDEGREE.
• *NBRTASKS nn
• Specifies the number of tasks or threads to be used when the query
optimizer chooses to use CPU parallel processing to process a query. I/O
parallelism will also be allowed.
• Used to manually control the degree value
108. • Application environments that can use and benefit from parallelism
• SQL requests that use methods that are parallel enabled
• Longer running or complex SQL queries
• Longer running requests like index creation
• Few or no concurrent users running in the same memory pool
• Willing to dedicate most or all the resources to the specific SQL request(s)
• Native, record level access from within HLL programs is not enabled for SMP.
• Computing resources
• > 1 (physical) CPUs
• 4-8GB memory per CPU
• 10-20 disk units per CPU
• 60% or less average CPU utilization during the time interval of the request
• Setting the memory pool‘s paging option to *CALC allows the database engine to be
more intuitive and more aggressive with I/O requests.
• The optimization goal "ALL I/O" tends to allow SMP, while "FIRST I/O" does not.
109. • Parallel Data load fully utilizes SMP capabilities
• CPYFRMIMPF and CPYTOIMPF CL commands
• Works with fixed format and delimited files
• Import from stream files (IFS), source files, tape files and more
CPYFRMIMPF
FROMSTMF('~mydir/myimport.txt')
TOFILE(MYLIB/MYTABLE)
DTAFMT(*DLM) FLDDLM(',’)
110. • Lazy Close
• Reuse open connections
• Good for applications such as MS Access
• Data Compression
• Enabled by default
• For clients not CPU bound
• Block with a fetch of 1 row
• Advanced option
• Test, incompatible with some applications
• Record blocking
• Default 32kb
• For read only increase dramatically
• Query Optimization Goal (V5R4)
• *ALLIO or *FIRSTIO
• Extended Dynamic
• For subsequent requests of the same query
• Connection Pooling
111. How does the optimizer know which choice to make?
112. Previous environment
Data Warehouse Load programs written in RPG using non-SQL I/O
Add/Update 3 tables, total refresh of an additional 5 summary tables
Read logical file, chain to another file to get ―group by‖, and add/update
7 Hours to complete
Current Environment
RPG altered to use embedded SQL for 3 table add/updates
5 Summary tables replaced by MQTs (Materialized Query Tables)
Refreshed with ―Refresh Table‖ SQL Command
Binary Radix and Encoded Vector Index techniques used
Result
7 Hour job reduced to…..
113.
114. DESCRIPTION OF PROBLEM FIXED FOR APAR MA : -----------------------------------------------Users report drastic
spikes in CPU when the SQE plan cache reaches it's maximum size. When the max size of the SQE plan cache is reached
two things happen First, the plan cache pruner task is no longer allowed to sleep. Second, no more plans are allowed to be
inserted in the plan cache. This results in excessive CPU in full opens as plans are removed and no new plans are allowed
in. The observations on the system are When plan cache Memory usage becomes near the max over size, it starts
decreasing When plan cache Memory usage starts decreasing, the rate of "Number of times Hit" starts decreasing too.
" Total size of all MTIs" starts decreasing too. " . Full Optimizations" starts increasing at the same time. These symptoms
can cause CPU usage to increase significantly. In reviewing the code a couple of changes were decided on. ) The pruner
task not sleeping and no inserts action in the plan cache are always controlled together when the over max size is
exceeded and turn this combination off much sooner than is done currently. CORRECTION FOR APAR MA : ------
-----------------------The operating system code will be changed to limit the high CPU occurances that occur when the
percent over maximum memory usage for the plan cache is met. Specifically, the amount of time that plans are not allowed
to be insert in the plan cache and the amount of time that plans are forcefully removed from the plan cache, will be limited
MF40965 - LIC-DB CPU SPIKES WHEN PLAN CACHE REACHES MAXIMUM SIZE
115.
116. What’s wrong with the DataSet?
I‘m not saying that there‘s anything inherently wrong with the DataSet object. But it‘s like any other tool—
you need to understand how to use it appropriately. Although it‘s a useful tool for Windows Forms
applications, it‘s much less useful for Web application development.
Let‘s look at a simple example. Suppose you use a DataSet to return a set of 1,000 products to display in
a DataGrid on a form. Since you might want to sort or filter the data later, you choose to save the DataSet
in a session variable. Not knowing any better, you also leave the default page ViewState turned on. When
a user navigates to this page, there are three copies of the data somewhere in memory. It‘s on the server
saved in a session-level variable. It‘s in the ViewState stored as the contents of the DataGrid. And it‘s in
the rendered HTML stream in the form of HTML table directives that render the table. Now multiply the
server memory by the number of users to assess impact on server memory, and multiply the two copies of
the data by the number of users to assess the impact on bandwidth utilization. You can quickly overload a
server and its available network bandwidth on a high-traffic site.
The answer: Use the DataReader
Though not as sexy, the DataReader is much more functional for a Web application. Because the DataSet
object‘s cursor is designed to iterate in a forward-only, read-only fashion over the results of a query, it‘s
very fast. Moreover, the DataReader only holds the current record in memory at any one time—never the
entire results set. The DataSet object can be bound to ASP.NET Server Controls (like the DataGrid). More
importantly, server resources and connection resources are released as soon as you‘re finished traversing
them. Build your data-bound pages using DataReaders to retrieve data from an underlying database
whenever it‘s important for the data to be as fresh as possible.
http://articles.techrepublic.com.com/5100-
10878_11-1045330.html
117.
118.
119.
120.
121.
122.
123.
124.
125.
126.
127.
128.
129. <add name="GAL_SM_KS―
connectionString="Provider=IBMDA400; Password=OPTODBC; Persist Security Info=True;
User ID=OPTODBC; Data Source=192.168.240.1; Force Translate=0"/>
<add name="ODBC2_GAL_SM_KS―
connectionString="Driver={Client Access ODBC Driver (32-bit)}; System=192.168.240.1;
Uid=OPTODBC; Pwd=OPTODBC; Cursor Sensitivity=0; QueryTimeOut=0; TRANSLATE=1"/>
<add name="ODBC_GAL_SM_KS―
connectionString="Driver={Client Access ODBC Driver (32-bit)}; System=192.168.240.1;
Uid=optodbc; Pwd=optodbc; QueryTimeOut=0; TRANSLATE=1; XDYNAMIC=1;
DFTPKGLIB=OPTODBC; PKG=LIBRARY/DEFAULT(IBM),2,0,0,0,0; CMT=0; LAZYCLOSE=1;
QAQQINILIB=OPTODBC; COMPRESSION=1; PREFETCH=1; Cursor Sensitivity=0;
DEBUG=4;TRACE=12"/>
<add name="IDB2_GAL_SM_KS―
connectionString="DataSource=192.168.240.1; UserID=OPTIDB2; Password=OPTIDB2;
DataCompression=true; Pooling=true; MaximumPoolSize=-1; MaximumUseCount=1000;
MinimumPoolSize=2; CheckConnectionOnOpen=false; Trace=StartDebug, PrintJobLog;"/>
<add name="DB2_GAL_SM_KS―
connectionString="Database=S653afa2; Pooling=true; Min Pool Size=2;"/>
<add name="DD_GAL_SM_KS"
connectionString="Charset For 65535=37; Host=192.168.240.1; Port=446;
Database Name=S653afa2; User ID=OPTDD; Password=OPTDD; Persist Security Info=True;
Pooling=True; Min Pool Size=2; Connection Reset=False; Statement Cache Mode=Auto;
Max Statement Cache Size=20;"/>
130. Before:
Dim con As OdbcConnection
Dim cmd As OdbcCommand
Dim da As OdbcDataAdapter
Dim dt As DataTable
con = New OdbcConnection(ConnectionString)
cmd = New OdbcCommand(SqlString, con)
da.Fill(dt)
After:
Dim con As iDB2Connection
Dim cmd As iDB2Command
Dim da As iDB2DataAdapter
Dim dt As DataTable
con = New iDB2Connection(ConnectionString)
iDB2ProviderSettings.DecNumericAsString = True
iDB2ProviderSettings.CharBitDataAsString = True
cmd = New iDB2Command(SqlString, con)
da.Fill(dt)
132. The memory-sharing algorithms discussed above provide balanced performance for all the jobs running in
a memory pool. Running short transactional queries in the same memory pool as long running, data
intensive queries is acceptable. However, if it is desirable to get maximum performance for long-running,
data-intensive queries it may be beneficial to run these types of queries in a memory pool dedicated to
this type of workload. Executing long-running, data-intensive queries in the same memory pool with a
large volume of short transactional queries will limit the amount of memory available for execution of the
long-running query. The plan choice and engine execution of the long-running query will be tuned to run
in the amount of memory comparable to that available to the jobs running the short transactional queries.
In many cases, data-intensive, long-running queries will get improved performance with larger amounts
of memory. With more memory available the optimizer is able to consider access plans which may use
more memory, but will minimize runtime. The query engine will also be able to take advantage of
additional memory by keeping more data in memory potentially eliminating a large number of DASD
I/Os. Also, for a job executing long-running performance critical queries in a separate pool, it may be
beneficial to set QQRYDEGREE=*MAX. This will allow all memory in the pool to be used by the job to
process a query. Thus running the longer-running, data intensive queries in a separate pool may
dramatically reduce query runtime.
133. LAN Performance Considerations
The following parameters for the line description (LIND) and the controller
description (CTLD) have a significant impact on performance. Select one of the
parameters for performance related information.
MAXFRAME (LIND and CTLD)
For a more detailed discussion on iSeries LAN support and for additional LAN
performance considerations, refer to the following manuals:
· iSeries Communications Management, SC41-3406
· iSeries Local Area Network Support, SC41-3404
134. TCP/IP Performance Considerations
When connecting to the server using TCP/IP, you may be able to improve performance with a few simple changes to the server TCP/IP
configuration.
TCP/IP Interface and Route Configuration
Currently the server defaults to a Maximum Transmission Unit (MTU) of 576 when you add a route to the configuration (through CFGTCP
option 2 or ADDTCPRTE). This value ensures packets will not be dropped over this route because all TCP/IP implementations have to
support at least a 576-byte transmission unit.
In many cases, however, this value is unnecessarily small because this route will only be used on this Ethernet or token-ring, and there
are no intermediate hops that only support a 576-byte packet. If this is the case, you should change the Route Maximum Transmission
Unit size to *IFC. This will change the MTU on the Route to the Interface MTU size which defaults to the Line Description Frame Size.
This defaults to approximately 2000 for token-ring and 1500 for Ethernet. There are also cases where the 576-byte size can cause
adapter overruns that put TCP/IP in retransmit mode and slow things down considerably.
This change often results in a considerable increase in your throughput over TCP/IP, especially when installing products.
TCP/IP Send and Receive Buffer Size
Use CFGTCP option 3, Change TCP/IP attributes to change the defaults used for the following:
· TCP/IP receive buffer size
Consider increasing the TCP receive buffer size from the default size of 8192 bytes to a larger value, for example, 64384 bytes. This
value specifies the amount of data the remote system can send before being read by the local application. If a large number of buffer
overruns are occurring on the network adapter, decreasing this value, instead of increasing it, could help performance.
· TCP/IP send buffer size
Consider increasing the TCP send buffer size from the default size of 8192 bytes to a larger value, for example, 64384 bytes. This value
provides a limit on the number of outgoing bytes that are buffered by TCP. If TCP has to resend the data many times due to a buffer
overrun condition of a network adapter, decreasing this value, instead of increasing it, could help performance.
Data processing Optimization: Prepared Statements, Connection Pooling, Blocking, ...Rendering Optimization: Data Readers,…Communication Optimization: iSeries Access, IP/Port Mapping, Signing On
Components of Work Let’s look at why there may be issues with database performance on any database server, including iSeries servers. Simply put, if you look at the work that a computer has to do, there are four basic areas where performance degradation can occur: • Time spent running on the processor • Time spent performing (waiting) for disk I/O • Time spent performing (waiting) for communication, and • Time spent waiting for some other resource or piece of work to complete. In most environments, processor time is where an application spends a great deal of its time. The application is running on the processor, which is the fastest part of the computer, and to go faster, you either need to execute on a faster processor or split the work to run on multiple processors. Typically, for database performance issues, time spent waiting for the disk I/O is the problem.
Query phases During query execution, there are generally four phases: 1. The query is validated. If the query has run previously, a plan might already exist. If so, and if that plan is still valid, the engine uses it. If not, the request must be optimized again. 2. If optimization is required, the query dispatcher is invoked to determine which optimizer and query engine will be used — CQE or SQE. 3. The request goes through optimization. In this phase, the optimizer determines the best plan by examining the available methods and strategies. 4. The query is ready for execution. At this point, the optimizer might open the cursors to get ready to run (open the data path) and gives the plan to the query engine for execution. During optimization, you can generate feedback to illuminate the methods and strategy chosen by the optimizer and used by the query engine. For example, the optimizer can tell you what objects the engine will be using and why. This feedback can take many forms, both textual, row and columns, as well as pictorial or graphic. There is not much assistance that you as the developer can provide for either the validation or execution phases of your query. However, you can have a positive effect on the query optimization phase by studying the optimizer feedback and providing DB2 for i5/OS with the proper statistics and indexes.
Database architecture: High-level view This chart represents a high-level view of the architecture of the DBMS. • The bottom layer represents the integrated database engine, which is implemented within the system licensed internal code (SLIC). • The native record level I/O interface represents non-SQL access to DB2 for i5/OS. • The query optimizer sits above the database engine and is invoked for the SQL interface (static, dynamic, and extended dynamic SQL requests). • The server-side call level (ODBC) and JDBC interfaces, as well as the host server code for the external database clients all sit above the SQL interface layer. • As an open DBMS, DB2 for i5/OS supports connections from ODBC, JDBC, .NET, DRDA, and XDA clients. Starting with IBM OS/400® V5R2, the operating system has been phasing in a redesigned and reengineered database engine and query optimizer. Two query engines As IBM phases in the new database engine and query optimizer on the System i platform, it is easier to refer to each query engine specifically. So, what is in a name? The original query engine is referred to as the Classic Query Engine (CQE). The new engine first arrived in OS/400 V5R2 and is referred to as the SQL Query Engine (SQE). Both engines sit side by side in the operating system. As part of query optimization, the DBMS decides which engine to use. The long-term goal is to use SQE for all SQL requests. As the new engine becomes more sophisticated and can handle more queries, the DBMS automatically sends those queries down to the new engine. However, there is still only one interface to the query optimizer, and it requires no specific or special user programming.
User Defined Table Functions (UDTFs)– Translation (UPPER, LOWER, etc.)– Alternate sort sequences– Select/Omit Logical Files on the table queried ignored by default• Remaining SQE restrictions:– Logical File references– Non-SQL interfaces
The Query Dispatcher The mission of the Query Dispatcher is to decide whether the SQE can process the SQL request. If not, the Query Dispatcher sends the request down to the original engine (CQE). The query also has the ability to retreat from the SQE Optimizer and use the CQE Optimizer. This occurs if there is something about the query environment that the SQE cannot handle during the optimization phase.
Query phases During query execution, there are generally four phases: 1. The query is validated. If the query has run previously, a plan might already exist. If so, and if that plan is still valid, the engine uses it. If not, the request must be optimized again. 2. If optimization is required, the query dispatcher is invoked to determine which optimizer and query engine will be used — CQE or SQE. 3. The request goes through optimization. In this phase, the optimizer determines the best plan by examining the available methods and strategies. 4. The query is ready for execution. At this point, the optimizer might open the cursors to get ready to run (open the data path) and gives the plan to the query engine for execution. During optimization, you can generate feedback to illuminate the methods and strategy chosen by the optimizer and used by the query engine. For example, the optimizer can tell you what objects the engine will be using and why. This feedback can take many forms, both textual, row and columns, as well as pictorial or graphic. There is not much assistance that you as the developer can provide for either the validation or execution phases of your query. However, you can have a positive effect on the query optimization phase by studying the optimizer feedback and providing DB2 for i5/OS with the proper statistics and indexes.
In extended dynamic SQL, the SQL statements are dynamic. However, SQL packages are used to make the implementation of the dynamic SQL statements similar to that of static SQL.The QSQPRCED application program interface (API) is used to prepare the dynamic SQL statements and store the access plan in an SQL package. The SQL statements contained in the resulting SQL package persist until the SQL package is deleted or the SQL statement is explicitly dropped.The iSeries Access Open Database Connectivity (ODBC) driver and Java™ Database Connectivity (JDBC™) driver both have extended dynamic SQL options available. They interface with the QSQPRCED API on behalf of the application program.
In extended dynamic SQL, the SQL statements are dynamic. However, SQL packages are used to make the implementation of the dynamic SQL statements similar to that of static SQL.The QSQPRCED application program interface (API) is used to prepare the dynamic SQL statements and store the access plan in an SQL package. The SQL statements contained in the resulting SQL package persist until the SQL package is deleted or the SQL statement isexplicitly dropped.The iSeries Access Open Database Connectivity (ODBC) driver and Java™ Database Connectivity (JDBC™) driver both have extended dynamic SQL options available. They interface with the QSQPRCED API on behalf of the application program.
Access plan validationAn access plan is a control structure that describes the actions necessary to satisfy each query request. Itcontains information about the data and how to extract it. For any query, whenever optimization occurs,the query optimizer develops an optimized plan of how to access the requested data.To improve performance, an access plan is saved (see exceptions below) once it is built so as to beavailable for potentially future runs of the query. However, the optimizer has dynamic replan capability.This means that even if previously built (and saved) plan is found, the optimizer may rebuild it if itdetermines that a more optimal plan is possible. This allows for maximum flexibility while still takingadvantage of saved plans.v For dynamic SQL, an access plan is created at prepare or open time. However, optimization uses thehost variable values to determine an optimal plan. Therefore, a plan built at prepare time may berebuilt the first time the query is opened (when the host variable values are present).v For an iSeries program that contains static embedded SQL, an access plan is initially created at compiletime. Again, since optimization uses the host variable values to determine an optimal plan, the compiletime plan may be rebuilt the first time the query is opened.v For Open Query File (OPNQRYF), an access plan is created but is not saved. A new access plan iscreated each time the OPNQRYF command is processed.v For Query/400, an access plan is saved as part of the query definition object.In all cases above where a plan is saved above, including static SQL, dynamic replan can still apply asthe queries are run over time.The access plan is validated when the query is opened. Validation includes the following:v Verifying that the same tables are referenced in the query as in the access plan. For example, the tableswere not deleted and recreated or that the tables resolved by using *LIBL have not changed.v Verifying that the indexes used to implement the query, still exist.v Verifying that the table size or predicate selectivity has not changed significantly.v Verifying that QAQQINI options have not changed.
Most efficient data access method Here is a simple example of a query to help you see what the query optimizer goes through and to assist you in understanding why different methods are important. A database table has one million rows. The query needs to find all of the rows: WHERE COLOR = ‘Purple’ • What methods are available? • What method is best for locating and accessing the rows that match the criteria? • How does the optimizer know which method is the fastest? If no index or column statistic on the color column is available, the optimizer does not understand the expected selectivity of the query. To perform the costing, a default value is assigned to represent the selectivity. This value might or might not represent the actual number of rows where color = ‘purple’. If no index on the column color is available, the optimizer has limited choice for locating and accessing the rows. Basically, only a full table scan is available, regardless of the expected selectivity of the query. Without indexes and column statistics, the optimizer does not understand the data, and is limited in the methods available. If a suitable index is provided, the optimizer uses statistical information to cost and compare various methods of access. Even with an index available, the expected selectivity of the query might cause the optimizer to choose a full table scan. In other words, if the optimizer believes a large number of rows will be selected, using a table scan might be the fastest strategy to employ. On the other hand, if the expected selectivity is high (that is, few rows are selected) then using an index to locate and access the rows is the fastest strategy.
Statistics and CQE query optimization This diagram illustrates how CQE works. Your query comes into the optimizer. The optimizer first wants to determine what the data looks like; that is, determine what statistics are available. The optimizer can easily learn the size of the table and the total number of rows. It also knows how many I/O operations it will take to read the table. If there is a radix index or an encoded--vector index (EVI), then CQE can also ask those indexes about the data. The CQE optimizer analyzes those statistics and then uses them to determine the least-cost plan and then it builds the access plan for the engine to run.
Statistics and SQE query optimization (continued) SQE is different; it has a new component, called the statistics manager. The statistics manager and the optimizer components talk to each other through a question-and-answer process. When your SQL query comes in, the optimizer is only responsible for costing and building the plan. It is not responsible for statistics. The optimizer asks the statistics manager questions such as, “How many rows match the color purple?” The statistics manager determines how it can answer the question. It cannot say, “I do not know.” It has to give an answer. That answer depends on what sources of statistics and metadata it has. If it has good statistics, such as a column statistic in the table, it can use that to form an answer. Or, it can use existing indexes (as does the CQE optimizer). However, notice the difference: SQE can use a column statistic; CQE cannot. If there is no source of statistics available to help determine the selectivity of a particular predicate (or combination of predicates), both CQE and SQE make up a number referred to as the default filter factor. This default value is used for query optimization.
Sources for answers Here are the sources for the answers to the statistics-gathering questions discussed in the previous chart. These sources are presented here from best to worst in terms of relative value. In the best case, the optimizer can use existing radix or an EVIs to answer the optimizer’s questions about the data. Furthermore, indexes can have multiple columns for the keys (for example, color and size, or country and city) Multiple, contiguous key columns can provide compound values, allowing the optimizer to understand any correlations between the columns. Farther down, column statistics can only be build for a single column (for example, color or size). For example, if you need to find all of the rows for people living in Kuala Lumpur, Malaysia, an index on country and city provides the correlation. However, it would be necessary to combine two column statistics to find Kuala Lumpur and Malaysia. Another distinction is that indexes are maintained as the data changes. Column statistics are not immediately maintained; rather they are refreshed periodically as the optimizer recognizes that the information is stale. Column statistics are only available to the SQE. CQE does not take advantage of this source. In the worst case, there is no source for statistics. In these situations, the optimizer uses default values that might, or might not, reflect the actual column information.
SQE automatic statistics collection Starting in OS/400 V5R2, there is a never-ending job called QDBFSTCCOL. This job is responsible for collecting the SQE column statistics automatically as a byproduct of query optimization. By default, this is turned on and collects statistics by reacting to the queries that you run. The system has the ability to collect column statistics on demand and learn about the data over time. You can view a table’s column statistics through iSeries Navigator. Application programming interfaces (APIs) are also provided to interface with the column statistics; this allows your applications to view and work with this information programmatically.
The goal determines the plan This simple illustration proves the point that the goal determines the most appropriate vehicle. For example, if your goal is to reach 10 meters as fast as possible, a bicycle is a more effective vehicle. However, if your goal is to reach 100 kilometers as fast as possible, a plane will get you there faster. The bike gets up to speed faster, and is therefore better for shorter distances. The plane takes longer to ramp up to speed, but when it achieves that speed, goes much faster. Some query strategies are similar to the bicycle metaphor; they are very fast to get started, but their top speed is not fast. Other techniques are like the airplane; they are slow to get started but they are very fast at full speed. The one that the optimizer chooses depends on your application’s goal. What behavior do you want? This is referred to as the optimization goal. Another way to talk about the optimization goal is, “How many rows does your application expect to fetch from the result and process at a time?” For instance, if your application only wants the first two rows, you need a plan that provides the first two rows of the result set as fast as possible.
The optimization goal If an application producing a report requires processing all the rows in a 10,000-row result set, the optimization goal for this query is all I/O and the optimizer needs to provide a plan that is best for returning the entire result set as fast as possible. It might seem that reading by key is the fastest way. In fact, the database engine can find your first n rows very fast by key. However, for the next set of n rows, it is not very fast, because it is merely a constant phase. A different, and more cost-effective, plan might build a hash table, which takes a little time to get started, even for the first row (because of the upfront calculation effort). However, after the hash table is built, the database engine can go extremely fast. This chart illustrates the difference in time required to access the first n rows and the time to process the entire 10,000-row result set. This is why it is important to let the optimizer know what your optimization goal is.
The optimization goal (continued) To provide the optimizer with the optimization goal, you can rely on SQL syntax or query attributes for your job, or use the default setting. On the SQL statement, all the products in the DB2 family of products allow you to specify how many rows you expect to process at a time. Using the OPTIMIZE FOR n ROWS clause (where n is either ALL or the number of rows the application will process at one time) to specify the optimization goal directly. To set the optimization goal in the job, use the QAQQINI file OPTIMIZATION_GOAL parameter with a value of *ALLIO or *FIRSTIO. • For CQE, *FIRSTIO is equivalent to 3% of the expected result set. • For SQE, *FIRSTIO is equivalent to 30 rows. The default optimization goal varies by SQL interface. It is important to consider the optimization goal when developing and testing SQL requests, as a change in the SQL interface might cause a change in the optimization. Changing the optimization goal might cause a significant change in what strategy the optimizer chooses for the SQL request.
The Plan Cache is a repository that contains the access plans for queries that were optimizedby SQE.Access plans generated by CQE are not stored in the Plan Cache; instead, they are stored inSQL Packages, the system-wide statement cache, and job cache). The purposes of the PlanCache are to:Facilitate the reuse of a query access plan when the same query is re-executedStore runtime information for subsequent use in future query optimizationsOnce an access plan is created, it is available for use by all users and all queries, regardlessof where the query originates. Furthermore, when an access plan is tuned, when creating anindex for example, all queries can benefit from this updated access plan. This eliminates theneed to reoptimize the query, resulting in greater efficiency.There is a separate plan cache for each IASP for the system, so varying the IASP will alsocause the plan cache to be cleared.
There are several considerations for system configuration in the SQE environment.ProcessorsProcessing capacity is still important to ensure that complex queries are processed in a timelyand efficient manner. For the complex queries that do not use SQE starting in OS/400 V5R2,the CPU power needed by CQE-handled queries is similar to V5R1. For SQE, some queriessee an improvement in processor utilization.Multiple processors (iSeries n-way support) are an important consideration. When SQE usesSQL non-indexed access methods starting in V5R2, multiple processors are a factor becauseSQE can run all or part of an access plan tree in parallel. Since SQE uses threads instead oftasks for splitting up a job, it uses fewer tasks. In addition to SQE using multiple processorsfor executing the threads within an access plan, SMP can also be used by the statisticsengine to collect statistics for SQE.Main storageSQE uses a slightly smaller main storage footprint for a query compared with CQE. Thisreduction is not significant enough to encourage smaller configurations to be deployed.However, there is a change in the way that the optimizer calculates a fair share of memory fora query. This is the maximum amount of memory that can be allocated to a query withouthaving a detrimental effect on other jobs running in the same memory pool.CQE continues to compute a job’s fair share of memory by dividing the memory pool size bythe maximum activity level of that pool. SQE, in contrast, tries to compute the fair share ofmemory calculation by dividing by the average number of active jobs instead of the maximumactivity level. SQE can make this computation only when the associated memory pool isdefined with a pool paging option of *CALC (also known as expert cache). If the pool pagingoption is not *CALC, then it too divides by the maximum activity level. Because of the waySQE calculates the fair share of memory, memory-constrained configurations may degradeperformance more severely than may be the case with CQE.The degree of parallelism is controlled by the DEGREE parameter of the CHGQRYAcommand or by the PARALLEL_DEGREE value within the query attribute table (QAQQINI) inuse. Disk storageWe continually emphasize the need for a balanced system by ensuring that sufficient diskarms are configured. SQE performs asynchronous I/O far more aggressively than CQE andfully uses parallel pre-fetching of data. Therefore, any under configuration of the number ofdisk drives on a system is accentuated by queries that use SQE, particularly those thataccess a large number of table rows.
When the query optimizer takes an SQL statement and converts it into one of these black box programs, it looks at the work management configuration. The size of the pool either encourages or discourages certain approaches or algorithms frombeing used. The larger the memory pool, the more options the optimizer has at its disposal and therefore in theory it should be able to choose the fastest method to implement the SQL statement.The second important factor is the use of Expert Cache. We find that quite a few shops have adopted the use of the *CALC setting on their shared pools but there are still systems out there using *FIXED. If you are using much SQL, it makes sense to use *CALC because it not only is more friendly to database work, but it also give the optimizer additional information it can use to make decisions.The third factor that weighs in here, is activity level. The activity level as configured influences the old query optimizer (CQE) by helping it determine what is a fair share of the pool for a particular request.
Fair share of memory Given that the i5/OS environment is built from the ground up as a multiuser operating system, the optimizer cannot dedicate all the computing resources to an SQL request. During optimization, the optimizer calculates an expected fair share of memory for the SQL request. This keeps the optimizer from overcommitting memory for a given request or allows the optimizer to consider more memory-intensive and aggressive methods. The fair-share value affects what query plans are chosen. A small fair-share value eliminates certain methods and strategies for consideration. A large fair-share value allows the optimizer to consider more aggressive and CPU-intensive methods and strategies. In this example, the method that uses an index has a small memory footprint that fits within the job’s fair-share value. The method that uses a hash table has a larger memory footprint that does not fit within the job’s fair-share value.
SQE uses a slightly smaller main storage footprint for a query compared with CQE. Thisreduction is not significant enough to encourage smaller configurations to be deployed.However, there is a change in the way that the optimizer calculates a fair share of memory fora query. This is the maximum amount of memory that can be allocated to a query withouthaving a detrimental effect on other jobs running in the same memory pool.CQE continues to compute a job’s fair share of memory by dividing the memory pool size bythe maximum activity level of that pool. SQE, in contrast, tries to compute the fair share ofmemory calculation by dividing by the average number of active jobs instead of the maximumactivity level. SQE can make this computation only when the associated memory pool isdefined with a pool paging option of *CALC (also known as expert cache). If the pool pagingoption is not *CALC, then it too divides by the maximum activity level. Because of the waySQE calculates the fair share of memory, memory-constrained configurations may degradeperformance more severely than may be the case with CQE.Fair share of memory (continued) The CQE and SQE calculate the fair-share value for a job’s SQL request slightly differently. The CQE calculates the fair-share value by using a simple equation of the job’s memory pool size divided by the maximum active value for the pool. The SQE first determines an average number of truly active jobs in the pool over a period, then uses this number as the divisor. In both cases, a larger memory pool size, or a smaller maximum active value results in a larger fair share of memory for the optimizer to consider. To allow the optimizer to consider all the memory in the job’s pool while building the plan, set the query parallel degree to the *MAX value. This is sometimes called the bad-neighbor policy, as it ignores the fact that multiple users might be sharing the memory pool.
i5/OS and System i architectureThe architecture and technology of the System i platform, when running the i5/OS operatingsystem, provides the foundation for database parallelism. Starting at the bottom of this diagram,consider the following explanations:• The System i independent I/O subsystem, along with i5/OS storage management, allows forsynchronous and asynchronous database I/O requests. These requests can make use ofparallel operations to access data on multiple disk units, simultaneously, through the I/Oprocessors (IOPs) and I/O adapters (IOAs) — without using the main System i processors.• DB2 for i5/OS uses storage management to spread database objects across all theavailable disk units. For example, as a table is populated, the space is automaticallyallocated on the disk units for optimal performance. This spreading of data minimizescontention on any single disk unit and also provides the basis for parallel I/O processing.• As the disk units are accessed, the data is brought into memory. The design of the System ihardware supports a very large memory system. As true 64-bit systems, i5/OS and DB2 fori5/OS can take full advantage of all the available memory. This provides the benefit of usingmain memory as if it were a cache resource for database objects.• The System i platform has a unique way of addressing storage, by viewing the disk spaceand main memory as one large storage area. This way of addressing storage is known assingle-level storage. The concept of single-level storage means that the knowledge of theunderlying characteristics of hardware devices (in this case, main storage and disk storage)resides in the System Licensed Internal Code (SLIC). The system automatically manages allthe storage. No user intervention is needed to take full advantage of any storagetechnology. Programs work with objects; and objects are accessed by name — not byaddress.• The System i platform supports multiple 64-bit IBM POWER™ processors (currently, up to64). i5/OS can take advantage of multiple processors by automatically dispatching work toone or more processors.• As the query request is processed, the integrated database takes advantage of theadvanced server and operating-system technologies to exploit SMP and, thus, achievedatabase parallelism.
N-way processingWithin i5/OS, a unit of work is defined as a job, thread or task. Built into the operating system isthe ability to dispatch this work to any one of the available processors. This concept has theadvantage of allowing more requests to be serviced as more processors are made available.Any individual job, thread or task can only run on a single processor. If additional processors areavailable, they provide little or no help. For example, if one job runs on a system or LPAR witheight processors available, this job only takes advantage of one of the processors while theother seven are idle. To use the other processors, you must apply additional techniques andstrategies. This is where DB2 SMP is important.
Since SQE uses threads instead oftasks for splitting up a job, it uses fewer tasks. In addition to SQE using multiple processorsfor executing the threads within an access plan, SMP can also be used by the statisticsengine to collect statistics for SQE.
