Más contenido relacionado
Similar a Dtrace Overview (20)
Dtrace Overview
- 2. AGENDA
1. Context
2. DTrace Framework
3. System inspection and analysis
4. DTrace Toolkit
5. Demo
2
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 4. CONTEXT
We need tools to debug and observe different situations …
Understand the
Analyze the system
system utilization or
performance
saturation
Debug an Debug a
application system/kernel crash
4
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 5. CONTEXT
… in several areas
pkill pargs pflags truss mdb
pgrep pstop
Process Process Process
control statistics debugging
System Kernel Kernel
statistics statistics debugging
vmstat lockstat
iostat cpustat kstat kmdb
Examples of some Solaris commands
5
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 6. CONTEXT
• Some existing utilities are process based
• Some are only inspecting certain parts of the system
• Disks
• Virtual memory
• Network
• Kernel
Need a tool to simplify the analysis of all parts of a system
This tool must not make the system any slower
6
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 8. DTRACE FRAMEWORK
• DTrace : A facility for dynamic instrumentation of production systems
Another way to debug and observe the entire system and understand
the big picture
• Open source (CDDL = Common Development and Distribution
License)
• Supported platforms : Unix-like systems :
8
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 9. DTRACE FRAMEWORK
User level
Operating System DTRACE Programs
Kernel level
• Examine how the system works
• Track down performance problems across many layers of software
• Locate the cause of aberrant behavior
Does not replace or retire other system utilities
9
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 10. DTRACE FRAMEWORK
• A framework for real-time analysis and observation
• Designed for live production systems
A totally safe way to inspect live data on production systems
No need to stop or restart applications
Live measurement and interpretation
• Over 30 000 data monitoring points spread in all system
• Zero effect when disabled, safe and no system slowdown when enabled
10
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 11. DTRACE FRAMEWORK
• Everything about DTrace is virtualized per consumer
There is no limit on the number of concurrent DTrace consumers
• Possibility to create custom programs to dynamically instrument the
system
• Example of problems that can be detected and analyzed:
High CPU utilization
Acute memory pressure
Abnormal I/O activity
Excessive network traffic
11
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 13. SYSTEM INSPECTION AND ANALYSIS
PROBES
• Probes are programmable sensors placed all over the system
• A probe fires when the event happens
Then, certain actions are executed only if the predicate expression
is true
• General form of a probe clause :
probe description
/predicate/
{
actions
}
13
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 14. probe description
SYSTEM INSPECTION AND ANALYSIS
/predicate/
PROBES {
actions
}
• A predicate allow action to only be taken when certain
conditions are met
Example : Look only for a process which has the pid=1203
Match a process which has the name firefox-bin
• An action can be “ record a stack trace, a timestamp, or the argument to
a function ”
• When some probes fire, DTrace gathers the data and reports it back to
you
If no action specified, DTrace will just take note of each time the
probe fires
14
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 15. SYSTEM INSPECTION AND ANALYSIS
PROBES
• Actions are used to record data to a DTrace buffer
• Different types of actions:
Type of actions Example Explanation
trace() Records the result of trace to the buffer
Data recording
printf() Traces a D expression
stack() Records a kernel stack trace
stop() Stops the process which has executed the probe
Destructive
raise() Signal a process at a precise point during execution
panic() Force a crash dump
exit() Stop tracing and exits
Special
copyin() Creates a buffer and returns its address
strlen() Returns the length of a string in bytes
15
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 16. probe description
SYSTEM INSPECTION AND ANALYSIS
/predicate/
PROBES {
actions
}
• Each probe is uniquely identified by a 4-tuple :
< provider, module, function, name > Probe description
The name of the module in which the probe is located
Module Either the name of a kernel module or the name of a user library
If this probe correspond to a specific program location, it’s the name
Function of the program function in which the probe is located.
The final component of the tuple is the string name of the probe that
Name give you some idea of the probe’s semantic meaning.
• Some probes does not have a module and function (e.g. : BEGIN, END)
We can identify a probe only by its name
16
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 17. SYSTEM INSPECTION AND ANALYSIS
PROVIDERS
< provider, module, function, name >
• DTrace probes come from a set of kernel modules called providers
Which correspond to a particular kind of instrumentation
• We can list the probes that a provider can transmit to the DTrace
Framework
• A provider pass the control to DTrace when you decide to enable one of
its probes
17
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 18. SYSTEM INSPECTION AND ANALYSIS
PROVIDERS
• Example of some providers :
SYSCALL PROC SCHED
• Holds the entire • Handle process creation • CPU scheduling: sleeping,
communication from and termination running threads
userland to kernel space • Sending and handling • CPU time: which threads
• Every system call on the signals are run by which CPU and
system for how long
IO MIB PROFILE
• I/O system • Counters for • Time based probing at
• Disk input and output management information specific interval of times
requests bases • To sample anything in
• I/O by device, process, • IP, IPv6, ICMP, IPSec the system every unit time
size, filename
18
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 19. SYSTEM INSPECTION AND ANALYSIS
D LANGUAGE /* D Program */
• A D program consist of one or more probe clauses
probe1 description
/predicate/
{
• D Language is like C language with some actions
constructs similar with awk : }
• Support ANSI C operators probe2 description
/predicate/
• Support Strings {
actions
• D expressions are based on built-in variables : }
pid, execname, timestamps, curthread
probe3 description
/predicate/
• No control-flow constructs : loops, if statements {
actions
• Floating-point arithmetic is not permitted (only }
Integers)
19
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 20. SYSTEM INSPECTION AND ANALYSIS
D LANGUAGE : AGGREGATIONS
@name[keys] = aggfunction(args)
• Used to aggregate data and look for trends
• Simple to generate report about :
• Total system calls by a process or an application
• Total number of read or write by process
• Aggregating functions: • count() • min()
• sum() • max()
• avg() • quantize()
20
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 21. SYSTEM INSPECTION AND ANALYSIS
D LANGUAGE : SECURITY
D program source files
userland
DTrace consumer: dtrace()
• D programs are compiled into a safe
intermediate form that is used for
DTrace safe execution
execution when a probe fires environment
kernel
DTrace providers
• Programming mistakes : DTrace will
report errors and disable instrumentation
It is impossible to construct a bad script that would cause damage to
the system
• DTrace handle run-time errors and report them (dividing by zero,
dereferencing invalid memory, …)
21
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 23. DTRACE TOOLKIT
• Collection of over 100 useful documented scripts built on top of DTrace
Framework
Developed by the OpenSolaris DTrace community
• The toolkit contains :
the scripts
the man pages
the example documentation
the notes files
the tutorials
• The script are sorted by categories
23
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 24. DTRACE TOOLKIT
CATEGORIES
DTrace Toolkit
Extra, User, System
Applications
Processes
Network
Memory
System
Kernel
Disk
CPU
DTrace Framework
24
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 26. DEMO
Command line scripts Chime scripts
tcpstat system calls
System calls counter by process,
TCP bytes received and send
function, module, …
httpdstat php calltime
Measure PHP elapsed times for
Real-time Apache Web Server stats
functions
26
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 28. REFERENCES & ADDITIONAL INFORMATION
• PowerPoint presentations
• Joyent Performance Visualization – Brendan Gregg
• DTrace & DTrace Toolkit – Stefan Parvu
• Links
• DTrace community site:
http://hub.opensolaris.org/bin/view/Community+Group+dtrace/WebHome
• Brendan Gregg site: http://www.brendangregg.com/dtrace.html
• DTrace Blogs: http://dtrace.org/blogs/
• DTrace online guide: http://wikis.sun.com/display/DTrace/Documentation
• DTrace wiki: http://www.solarisinternals.com/wiki/index.php/DTraceToolkit
• DTrace training: http://dtracehol.com
28
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
