It’s important to be able to figure out what’s going on when things go wrong in your Node.js production application. Tools are needed to investigate memory leaks, crashes and other "interesting" events in production. The post-mortem community working group (https://github.com/nodejs/post-mortem) is working on these problems. Come and learn about the key issues being worked, and the progress of the working group so far as illustrated through examples and code.
4. About The Postmortem Workgroup
Howard Hellyer
@hhellyer
David Pacheco
@davepacheco
Julien Gilli
@mistredjules
Michael Dawson
@mhdawson
Chris Bailey
@seabaylea
Daniel Khan
@danielkhan
Joshua Clulow
@jclulow
Yunong Xiao
@yunong
James Bellenger
@jbellenger
Bradley Meck
@bmeck
Luca Maraschi
@lucamaraschi
David Clements
@davidmarkclements
Richard Chamberlain
@rnchamberlain
11. “The method described in this article was designed to
provide a core dump… with a minimal impact on the
spacecraft… as the resumption of data acquisition from
the spacecraft is the highest priority.”
- Chafin, R. "Pioneer F & G Telemetry and Command Processor Core Dump
Program." JPL Technical Report XVI, no. 32-1526 (1971): 174.
12. Core Dumps: Brief History
● Magnetic core memory
● Dump out the contents
of “core” memory for
debugging
● “Core dump” was coined
● Initially printed on paper
● Postmortem debugging
was born
13. Production Constraints
● Uptime is critical
● Not easily reproducible
● Can’t simulate environment
● Resume normal operations ASAP
26. Postmortem WG - Mission
Guide improvements in postmortem
● Interfaces/APIs
● Dump formats
● Tools and Techniques
27. State of key tools today
Heap dump - snapshot of heap
● heapdump module - https://github.com/bnoordhuis/node-heapdump
● Chrome developer tools
● Limitations
● Need to modify application
● Slow to generate (minutes or hours)
● O(N) memory usage
● Limited content
● Output is large
28. State of key tools today
Core dump - memory image
● Creation
○ Crash, signal
○ --abort-on-uncaught-exception
○ Fast (relative to heap dumps)
○ Size matches process memory
● OS debuggers
○ Examination at C/C++ or assembler level
○ No knowledge of Node/v8 structures
● Node core dump inspectors
○ MDB (limited platform support)
○ IDDE (IBM SDK specific)
○ LLNODE (newer, less complete)
29. Example commands
MDB_V8 command LLNODE command IDDE
Print a stack trace jsstack, jsframe v8 bt !stack, !frame
Find objects findjsobjects v8 findjsobjects,
v8 findjsinstances <type>
!jslistobjects
!jsgroupobjects
!jsfindbyproperty
!jsobjectsmatching
Print an object jsprint v8 inspect !jsobject
Print function source jssource v8 source (prints source for a stack
frame)
!jsobject, !string + work
Find constructor for an
object
jsconstructor n/a !jsconstructor
Print elements of a
FixedArray
v8array v8 inspect <instance> !array
Find native memory
backing a buffer
nodebuffer v8 inspect <instance> !nodebuffers
30. How to make this better?
● Improve ease of use
● Common APIs to introspect dumps
● Cross platform support
● Common command set
● Lightweight dump
31. The Postmortem WG is working on...
Common Heap Dump Format
Improved Core Dump Analysis
● Library in C & JS
● Tools: mdb_v8(mdb), llnode(lldb), ...
Node Report
32. Common Heap Dump Format
Enabler for new tools
Generation
● mdb
● llnode
Consumption
● Conversion to existing v8 format - > chrome dev tools
● C/Javascript APIs
33. Core Dump Analysis
Currently working on
● Platform coverage
● Re-use of command implementation
● Common APIs
Soon to be
nodejs/llnode !
https://github.com/nodejs/post-mortem/issues/37
35. Node Report
Lightweight Dump
● Fast
● Small
● Human readable
● Key information to start investigating
● Triggers: exception, fatal error, signal, JavaScript API
36. NodeReport
example - heap out
of memory error
NodeReport content:
● Event summary
● Node.js and OS versions
● JavaScript stack trace
● Native stack trace
● Heap and GC statistics
● Resource usage
● libuv handle summary
● Environment variables
● OS ulimit settings
37. Javascript API
API in Javascript
● More accessible
● Leverages
○ llnode
○ Common Heap Dump (future)
39. Summary
What is postmortem debugging
Example of where it’s helpful
Activities of the working group
● Common heap format
● APIs (C/JS)
● Tools(lldb, mdb_v8, NodeReport)
40. Get Involved !
Great chance to learn
● Low level machine details
● Key debugging techniques
● Different platforms/operating systems
Where
● Most work done through GitHub issues/Pull Requests
● http://github.com/nodejs/post-mortem/
42. Some production problems are otherwise impossible
Save complete process state for debugging later
43. Copyrights and Trademarks
IBM, the IBM logo, ibm.com are trademarks or registered
trademarks of International Business Machines Corp.,
registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml
Node.js is an official trademark of Joyent. IBM SDK for Node.js is not formally related to or endorsed by the official
Joyent Node.js open source or commercial project.
Java, JavaScript and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or
its affiliates.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United
States, other countries, or both.