My SQL Skills Killed the Server

1. MY SQL SKILLS
KILLED THE SERVER
Dave Ferguson
@dfgrumpy
dev.Objective() 2015

2. WHO AM I?
I am an Adobe Community Professional
I started building web applications a long time ago
Contributor to Learn CF in a week
I have a ColdFusion podcast called
CFHour w/ Scott Stroz (@boyzoid)
(please listen)
3x California State Taekwondo
Weapons Champion

3. WHAT WILL WE COVER?
Running Queries
When good SQL goes bad
Bulk processing
Large volume datasets
Indexes
Outside influences

5. (I KNOW SQL)
“WHY AM I HERE?”

6. Because you have probably written something like
this…

7. select * from myTable

8. “I can write SQL in my
sleep”
select * from myTable where id = 2

9. “I can write joins and
other complex SQL”
Select mt.* from myTable mt
join myOtherTable mot
on mt.id = mot.id
where mot.id = 2

10. “I might even create a table”
CREATE TABLE `myFakeTable` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`someName` varchar(150) NOT NULL DEFAULT '',
`someDescription` text,
`type` varchar(50) DEFAULT NULL,
`status` int(11) NOT NULL,
PRIMARY KEY (`id`)
);

11. But, how do you know if what you did
was the best / most efficient way to do
it?

12. Did the
internet tell
you it was
right?

13. Did you get
some advice
from a
someone?

14. “My app works fine. It has
thousands of queries and we
only see slowness every once in
a while. ”

15. Have you ever truly looked at
what your queries are doing?

16. Most developers don't bother.
They leave all that technical
database stuff up to the DBA.
But what if you are the
developer AND the DBA?

17. Query Plan
Uses Execution Contexts
Created for each degree of parallelism for
a query
Execution Context
Specific to the query being executed.
Created for each query
QUERY EXECUTION

18. Execution Context & Query Plan

19. Have you ever looked
at a query plan?
Do you know what a query plan is?

20. Query Plan, In the event you were
curious…

21. WHAT A QUERY PLAN WILL TELL
YOU
Path taken to get data
 Almost like a Java stack trace
Indexes usage
How the indexes are being used
Cost of each section of plan
Possible suggestions for performance
improvement
Whole bunch of other stuff

22. How long are plans / contexts kept?
 1 Hour
 1 Day
 ‘Til SQL server restarts
 Discards it immediately
 The day after forever
 Till the server runs out of cache space

23. What can cause plans to be flushed from
cache?
 Forced via code
 Memory pressure
 Alter statements
 Statistics update
 auto_update statistics on

24. HOW CAN WE KEEP THE
DATABASE FROM
THROWING AWAY THE
PLANS?

25. MORE IMPORTANTLY,
HOW CAN WE GET THE
DATABASE TO USE THE
CACHED PLANS?

26. • Force it
• Use params
• Use stored procedures
• Get more ram
• Use less queries
SIMPLE ANSWER

27. HOW DOES SQL
DETERMINE IF THERE
IS A QUERY PLAN?

28. Something
Like this…

29. THIS QUERY WILL
CREATE A EXECUTION
CONTEXT..
select id, name from myTable where id = 2
THAT…

30. WILL NOT BE USED BY
THIS QUERY.
select id, name from myTable where id = 5

31. WHY IS THAT?
Well, the queries are
not the same.

32. According to the SQL
optimizer,
select id, name from myTable where id = 2
select id, name from myTable where id = 5
this query…
and this query…
are not the same.
So, they each get their own execution plan.

33. PLANS CAN BECOME DATA HOGS
select id, name from myTable where id = 2
If the query above ran 5,000 times over the
course of an hour (with different ids), you could
have that many plans cached.
That could equal around 120mb of cache space!

34. TO RECAP…
EXECUTION CONTEXTS
ARE GOOD
TOO MANY ARE BAD

35. USING QUERY PARAMS…
The secret sauce to plan reuse

36. <cfquery name="testQuery">
select a.ARTID, a.ARTNAME from ART a
where a.ARTID = <cfqueryparam value="5”
cfsqltype="cf_sql_integer">
</cfquery>
Using a simple query… let’s add a param for the
id.

37. select a.ARTID, a.ARTNAME from ART a
where a.ARTID = ?
THE QUERY OPTIMIZER SEES THIS…

38. testQuery (Datasource=cfartgallery, Time=1ms, Records=1)
in /xxx/x.cfm
select a.ARTID, a.ARTNAME from ART a
where a.ARTID = ?
Query Parameter Value(s) -
Parameter #1(cf_sql_integer) = 5
THE DEBUG OUTPUT LOOKS LIKE
THIS…

39. testQuery (Datasource=cfartgallery, Time=8ms, Records=5) in
/xxx/x.cfm
select a.ARTID, a.ARTNAME from ART a
where a.ARTID in (?,?,?,?,?)
Query Parameter Value(s) -
Parameter #1(CF_SQL_CHAR) = 1
Parameter #2(CF_SQL_CHAR) = 2
Parameter #3(CF_SQL_CHAR) = 3
Parameter #4(CF_SQL_CHAR) = 4
Parameter #5(CF_SQL_CHAR) = 5
IT EVEN WORKS ON LISTS…

40. testQuery (Datasource=cfartgallery, Time=3ms, Records=1) in
/xxx/x.cfm
select a.ARTID, a.ARTNAME, (
select count(*) from ORDERITEMS oi where oi.ARTID = ?
) as ordercount
from ART a
where a.ARTID in (?)
Query Parameter Value(s) -
Parameter #1(cf_sql_integer) = 5
Parameter #2(cf_sql_integer) = 5
MORE ACCURATELY, THEY WORK ANYWHERE
YOU WOULD HAVE DYNAMIC INPUT...

41. When can plans cause more harm
then help?
► When your data structure changes
► When data volume grows quickly
► When you have data with a high
degree of cardinality.

42. How do I deal
with all this data?

43. What do I mean by large data
sets?
► Tables over 1 million rows
► Large databases
► Heavily denormalized data

44. Ways to manage large data
► Only return what you need (no “select *”)
► Try and page the data in some fashion
► Optimize indexes to speed up where
clauses
► Avoid using triggers on large volume
inserts
► Reduce any post query processing as
much as possible

45. Inserting / Updating large datasets
► Reduce calls to database by combining
queries
► Use bulk loading features of your
Database
► Use XML/JSON to load data into
Database

46. Combining Queries: Instead of doing
this…

47. Do this…

48. Gotcha’s in query combining
► Errors could cause whole batch to fail
► Overflowing allowed query string size
► Database locking can be problematic
► Difficult to get any usable result from
query

49. Upside query combining
► Reduces network calls to database
► Processed as a single batch in database
► Generally processed many times faster
than doing the insert one at a time
► I have used this to insert over 50k rows
into mysql in under one second.

50. Indexes
The secret art
of a faster select

51. Index Types
► Unique
► Primary key or row ID
► Covering
► A collection of columns indexed in an order that
matches where clauses
► Clustered
► The way the data is physically stored
► Table can only have one
► NonClustered
► Only contain indexed data with a pointer back to
source data

52. Seeking and Scanning
► Index SCAN (table scan)
► Touches all rows
► Useful only if the table contains small amount of
rows
► Index SEEK
► Only touches rows that qualify
► Useful for large datasets or highly selective
queries
► Even with an index, the optimizer may still opt to
perform a scan

53. To index or not to index…
► DO INDEX
► Large datasets where 10 – 15% of the data is
usually returned
► Columns used in where clauses with high
cardinality
► User name column where values are unique
► DON’T INDEX
► Small tables
► Columns with low cardinality
► Any column with only a couple values

54. Do I really need an index?

55. It Depends.

56. Really… it Depends!

57. Outside influences

58. Other things that can effect
performance
► Processor load
► Memory pressure
► Hard drive I/O
► Network

59. Processor
► Give SQL Server process CPU priority
► Watch for other processes on the server using
excessive CPU cycles
► Have enough cores to handle your database
activity
► Try to keep average processor load below
50% so the system can handle spikes
gracefully

60. Memory (RAM)
► Get a ton (RAM is cheap)
► Make sure you have enough RAM to keep
your server from doing excess paging
► Make sure your DB is using the RAM in the
server
► Allow the DB to use RAM for cache
► Watch for other processes using excessive
RAM

61. Drive I/O
► Drive I/O is usually the largest bottle neck on the
server
► Drives can only perform one operation at a time
► Make sure you don’t run out of space
► Purge log files
► Don’t store all DB and log files on the same physical
drives
► On windows don’t put your DB on the C: drive
► If possible, use SSD drives for tempdb or other highly
transactional DBs
► Log drives should be in write priority mode
► Data drives should be in read priority mode

62. Network
► Only matters if App server and DB server are on
separate machines (they should be)
► Minimize network hops between servers
► Watch for network traffic spikes that slow data
retrieval
► Only retrieving data needed will speed up retrieval
from DB server to app server
► Split network traffic on SQL server across multiple NIC
cards so that general network traffic doesn’t impact
DB traffic

63. Some Important
Database Statistics

64. Important stats
► Recompiles
► Recompile of a proc while running shouldn’t occur
► Caused by code in proc or memory issues
► Latch Waits
► Low level lock inside DB; Should be sub 10ms
► Lock Waits
► Data lock wait caused by thread waiting for
another lock to clear
► Full Scans
► Select queries not using indexes

65. Important stats continued..
► Cache Hit Ratio
► How often DB is hitting memory cache vs Disk
► Disk Read / Write times
► Access time or write times to drives
► SQL Processor time
► SQL server processor load
► SQL Memory
► Amount of system memory being used by SQL

66. Where SQL goes wrong
(Good examples of bad SQL)

67. Inline queries that well… shouldn’t be

68. Over joining data

69. Transactions – Do you see the issue?

71. THAT IS ALL
THANK YOU
Dave Ferguson
@dfgrumpy
dave@dkferguson.com
www.cfhour.com
dev.Objective() 2015Don’t forget to fill out the survey