Visit http://sqlactions.com/2015/01/18/jan-14th-2015-pass-dba-vc-presentation-on-sql-server-dump-analysis/ for the complete webinar and demos. While dealing with SQL Server administration, you might have come across scenarios when a session terminates abruptly, SQL Server instance crashes, SQL Server cluster fails over without a graceful message. Most often, it leaves a dump file behind which is unfortunately not in a human readable format. In this one hour session, you will learn the basics of a dump file, the tools available for reading thru a dump, the different types of dumps and demos on various debugger commands, how to analyze a dump file to establish the root cause.

1. SQL Server Dump Analysis
PRASHANT KUMAR
H T T P : / / S Q L A C T I O N S . C O M
P R A S H A N T @ S Q L A C T I O N S . C O M
@ P R K U M A

2. Agenda
1. Introduction
2. Windows concepts
3. Tools
4. Windbg Setup and Configuration
5. Symbols
6. Open a dump and load symbols (Demo)
7. SQL Server dump files
8. Analysing SQL Server dump files (Demo)
9. Resources
10. Q n A

3. Introduction

4. What is a dump file?
A dump is an image of a process’ memory space at a given point of time
written to a file for future verification.
Dump Analysis requires understanding of Windows Memory Management
and Programming languages.
Managed Vs Non-managed Application.
Dump
User Mode
Kernel Mode

5. The Art of debugging…
The process of ‘Debugging’ is not limited to
just using a debugger.
Debugging
(Identify and
dissect the
problem)
Knowledge
of the code
and
expected
behaviour
Inspectio
n of logs
Using other
tools e.g.
Perfmon,
Netmon,
eventviewer
etc.
Establish a
hypothesis.
Test the
hypothesis.

6. Windows Concepts

7. 32-bit Address Space Layout
Windows provides a page-based virtual memory
management scheme that allows applications to
realize a 32-bit linear address space for 4 GB
of memory.
Each application can address 2 GB of available
memory, regardless of how much physical
memory actually exists.
Windows employs the PC's hard disk as the
memory-backing store, and has a practical
limit imposed only by the available disk space.
FFFF FFFF = 1111 1111 1111 1111 1111 1111 1111 1111 = 32bits
DWORD(32 bits/4 bytes) and QWORD(64 bits/8 bytes)
Use dd on a 32-bit dump
Use dq on a 64-bit dump

8. 32-bit vs. 64-bit Virtual Memory
Memory Allocation Settings 32 bit versions 64-bit versions
Total amount of virtual
address space 4GB
16 TB (8TB user, 8TB
kernel)
Amount of virtual address
space per 32-bit process
2GB (3GB if the/3G switch is
added to the boot.ini file)
2GB(4GB if using
/LARGEADDRESSAWARE)
Amount of virtual address
space for the 64-bit processes Not applicable 8TB

9. Programs, Processes, and Threads
Program - A Static Sequence of Instructions
Process – Own resources Reserved for the Thread
Thread - Entities which Execute Instructions
Composed of:
 Changing set of registers
 Private storage area
 One used when running in user mode, and one used in kernel mode
 Thread ID

10. Tools

11. Tools
Debugging Tools for Windows (Part of Windows SDK)
 WinDbg
 Cdb
 Kd
 Adplus
DebugDiag
Visual Studio Native Debugger
ProcDump
Other Debuggers

12. Windbg Setup and Configuration

13. Choose the right installer
Search internet for “Debugging Tools for Windows” or “windbg”
For Windows 8 and 8.1
http://msdn.microsoft.com/en-US/windows/desktop/bg162891
For Windows 7
http://www.microsoft.com/en-us/download/details.aspx?id=8279
Download the right package – Both x86 and x64 versions available.

14. Installer screen

15. Symbols

16. What are Symbol Files?
Symbols are files.
They contain the data that map the executable code back to the
source code.
Symbols hold variety of data which may not be necessary for a
program's execution but debugging.

17. How do symbols help in debugging?
WITHOUT SYMBOLS
Call Site
sqlservr+0xd81879
sqlservr+0x31f04f0
WITH SYMBOLS
Call Site
sqlservr!HoBtFactory::DirtyLockResourceLookup+0x9d
sqlservr!GetHoBtLockInternal+0x185
sqlservr!IsRowsetBTree+0xc5
sqlservr!RowsetNewSS::Init+0x158
sqlservr!OpenRowsetSS::OpenRowset+0x105
sqlservr!OpenSystemTableRowset+0x336
sqlservr!CMEDScanBase::Rowset+0x315
 IT CAN CONVERT AND TRANSLATE USEFUL INFORMATION

18. Public Symbol vs Private Symbol
Public Symbol Files
 Global variable names
 Function names and the address of their entry points
 FPO data
Private Symbol Files
 Local variable names
 Source-line numbers
 Type information for variables, structures, etc.

19. Setting the symbol path
Different ways to set the symbol path in WinDbg:
Set the _NT_SYMBOL_PATH environment variable
to point to the root of the directory tree containing
the symbols before starting the debugger.
Use the -y command line option.
Use the .sympath (Set Symbol Path) debugger
meta-command.
Use the “Symbol File Path” command in the File menu

20. Microsoft Symbol Server
Microsoft Public Symbol Server is http://msdl.microsoft.com/download/symbols
Centralized symbol server. Not browse able. Debugger can download symbol on
need basis.
.sympath srv*DownstreamStore*http://msdl.microsoft.com/download/symbols
.symfix+ DownstreamStore
When using the public symbol store, you should always use a downstream
store. Otherwise you will end up downloading the same file several
times!

21. Open a Dump and load symbols (Demo)

22. Demo: Loading a dump file
Open windbg; set symbol path; load the dump, load the symbols
Verify symbols are loaded correctly.
 Use !sym –noisy
 Use symchk.exe
 lmvm

23. Dialects of debugging
 Thread call stack
 Frame
 Registers, variables
 Exception Context

24. Walking thru the stack
Always read from the bottom to top (that’s a stack you know)
Return addresses should always equal the previous stack entry’s
symbolic name. In this stack -
Child-SP RetAddr Call Site
00000000`27609f10 00000000`0105ff95 sqlservr!LatchBase::UnpendEligibleWaiters+0x196
00000000`2760a060 00000000`010e01e6 sqlservr!LatchBase::ReleaseInternal+0xca
00000000`2760a0e0 00000000`010607f0 sqlservr!BPool::ReadPageCompletion+0x236
00000000`2760a3d0 00000000`0106041c sqlservr!FCB::IoCompletion+0x90
00000000`0105ff95 should equal to sqlservr!LatchBase::ReleaseInternal+0xca
00000000`2760a060 is the childEBP of sqlservr!LatchBase::ReleaseInternal+0xca

25. Basic Debugger commands
.sympath [to get the current symbol path]
!sym -noisy [Generate verbose output]
.load [Load a debugger extension]
.unload [Unload a debugger extension]
.reload [Reload symbols]
.reload /f [to reload symbols]
.reload /f sqlservr.exe [to reload symbols for sqlservr]
.cls [to clear debug output window]
.logopen c:output.txt [to redirect the output into a file]
.logappend <logfile name>
.logclose c:output.txt [to stop redirection of output to file]

26. SQL Server dump files

27. SQL Server dump files
SQL Server generates a dump file when…
Dump files location
Not only minidumps
SQLDumper.exe – What it does
Control the way dumps are generated
Be alert when a dump is generated

28. SQL Server generates a dump file when…
 Non-yielding scheduler
 Non-yielding resource monitor
 Non-yielding IOCP listener
 Deadlocked Schedulers
 Access Violation (Exception or Assertion)
 Database Corruption
 Latch Timeout
 .NET Framework runtime exception
See http://support.microsoft.com/kb/2028589 for a list of event IDs
and messages.

29. Dump files location
Default location is LOG folder
For one occurrence, a set of three files are generated:
SQLDumpNNNN.txt  Symtom dump file
SQLDumpNNNN.log  Snippet from ERRORLOG
SQLDumpNNNN.mdmp  The memory dump file.
To change the default location:
◦ Use SQL Server Error and Usage Reporting under Configurations Tools from
Programs Menu.
◦ Alternatively, edit the registry
e.g. for SQL 2012 instance
HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
ServerMSSQL11.<Instance_Name>CPE
Value = "ErrorDumpDir“

30. Not only minidumps
Mini dump
Filtered dump
Full user-mode dump

31. Not only minidumps
Crash dumps : These kinds of dumps are generated in the process crash
scenarios. In SQL server, whenever an Exception occurs, SQL Server generates a
mini dump. Depending upon the nature of Exception, either SQL Server is
terminated, or the particular session is terminated.
Hang dumps : These kinds of dumps are always taken manually (using
adplus etc.). In some scenarios, e.g. SQL agent job takes ages to complete a job
– we can take a hang dump. Even in high CPU scenarios also , these kinds of
dumps are helpful.

32. Exception and Assertion
Exception – Catch me and throw for an error
Assertion – Raise me if I don’t stand true
SQL Server handles exception and assertion in the same way by generating a minidump
The minidump contains current thread's stack into a minidump
Server * BEGIN STACK DUMP:
Server * spid 123
Server * ex_handle_except encountered exception C0000005 – Server terminating
0xC0000005 STATUS_ACCESS_VIOLATION Reading or writing to an inaccessible memory location.
* Exception Address = 0021AC24
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred writing address 67192000
* Input Buffer 48 bytes -
* select * from sysindexes

33. SQLDumper.exe – What it does
SQLDumper.exe is internally called by the SQL Server process to
generate a dump file when the process encounters an exception.
SQL Server passes flags to the Sqldumper.exe utility.
You can use trace flags to change the flags that SQL Server passes to
the utility in the context of an exception or in the context an assertion.
For details on using Sqldumper.exe, refer to this KB article:
How to use the Sqldumper.exe utility to generate a dump file in SQL Server 2005
http://support.microsoft.com/kb/917825

34. Control the way dumps are generated
Using SQLDumper.exe to manipulate the parameters.
http://support.microsoft.com/kb/917825
Take manual dumps using DBCC STACKDUMP
Schedule to generate dumps on certain errors using DBCC DUMPTRIGGER
Use adplus (especially for hang dumps)
Using task manager in Windows 2008 and above

35. Be alert when a dump is generated
Scan ERRORLOG for dump generation messages
Using 'dbghelp.dll' version '4.0.5'
**Dump thread - spid = 0, EC = 0x0000000000000000
***Stack Dump being sent to
X:DataMSSQL10.Instance_NameMSSQLLOGSQLDump0008.txt
*
*******************************************************************
************
*
* BEGIN STACK DUMP:
Custom task to monitor Dump directory for recent dump files

36. Analysing SQL Server dump files
Demo

37. Resources
Debugging Applications for Microsoft® .NET and Microsoft Windows®
Windows® Internals
Windows via C/C++
http://msdn.microsoft.com/en-us/library/cc917684.aspx
http://mssqlwiki.com
http://troubleshootingsql.com
http://sqlactions.com

38. Q n A