Joyent is one of the largest deployers of Node.js in production systems. In order to successfully deploy large-scale, distributed systems, we must understand the systems we build! For us, that means having first-class tools for debugging our software, and understanding and improving its performance. Come on a whirlwind tour of the tools and techniques we use at Joyent as we build out large-scale distributed software with Node.js: from mdb for Post-Mortem Debugging, to Flame Graphs for performance analysis; from DTrace for dynamic, production-safe instrumentation and tracing, to JSON-formatted logging with Bunyan.

1. Node.js In Production At Joyent
__
.
.
_| |_
| .-. . . .-. :--. ||_
_|
;|
|| |(.-' | | |
|__|
`--' `-' `;-| `-' ' ' `-'
/ ;
`-'
Node.js at Joyent:
Engineering for Production
Joshua M. Clulow
jmc@joyent.com
Joshua M. Clulow
@jmclulow
⊕ Joyent

2. Node.js In Production At Joyent
node-vasync
Observable Asynchronous Control Flow
* Written by Dave Pacheco (Joyent)
davepacheco/node-vasync on github
* Similar in concept to caolan/async, but with a focus
on observability
* Returns status objects updated as flow progresses that
can be inspected in core files, or exported via web
services or REPLs
* Provides the barrier() primitive: specify some set of
named operations (you can extend the set mid-flight)
and get a drain event when they all complete
Joshua M. Clulow
⊕ Joyent

3. Node.js In Production At Joyent
Kang
Distributed System Observability
* Written by Dave Pacheco (Joyent)
davepacheco/kang on github
* A library for exposing self-describing state from
a set of processes via a web service
* A client library and command-line tool for collecting,
aggregating and inspecting that state
* The exposed state is a set of "objects" arranged by
"type", formatted as a JSON payload
Joshua M. Clulow
⊕ Joyent

4. Node.js In Production At Joyent
Post-mortem Debugging
* Traditionally done for C software on UNIX systems
* A program could call abort(), or some signals
(e.g. SIGSEGV) would end the program and write
a core file
* The program can be restarted immediately so that
service may be restored
* Engineers can inspect this core file later with
a debugger, similar to a black box flight recorder
* Operators may also generate a core file without
interrupting the program using gcore(1)
Joshua M. Clulow
⊕ Joyent

5. Node.js In Production At Joyent
Post-mortem Debugging in Node.js
* From node v0.10.8 onwards, we can abort() on
an uncaught exception by using:
node
--abort-on-uncaught-exception
app.js
* This preserves the stack that caused the exception
in the core file, where we can see it in the
debugger
* Use this flag everywhere!
Joshua M. Clulow
⊕ Joyent

6. Node.js In Production At Joyent
mdb(1)
The Modular Debugger
* The illumos (and SmartOS) debugger
* Supports node core files through mdb_v8.so
::jsstack
-- Javascript stack traces
::findjsobjects
-- inspect all Javascript heap objects
::jsprint
-- print properties of a single object
Joshua M. Clulow
⊕ Joyent

7. Node.js In Production At Joyent
DTrace
Production-safe Dynamic Instrumentation
* Zero disabled overhead; low enabled overhead
* In-kernel aggregation for high quantity events
* First class support for handling JSON-formatted
payloads with the json() subroutine
* Add semantically interesting probe points to
your applications with chrisa/node-dtrace-provider
* Easily measure latency between events and plot the
distribution
Joshua M. Clulow
⊕ Joyent

8. Node.js In Production At Joyent
Bunyan
JSON-formatted Logging Library
* Written by Trent Mick (Joyent)
trentm/node-bunyan on github
* Logs should be formatted for machines,
not for people
* Use a filter to make them human-readable
* Log auditing information at INFO to files
* Log verbosely (off by default) to DEBUG, and
consume those logs via DTrace probes (bunyan -p)
(uses node-dtrace-provider!)
Joshua M. Clulow
⊕ Joyent

9. Node.js In Production At Joyent
Restify
REST framework for Web Service APIs
* Written by Mark Cavage (Joyent)
mcavage/node-restify on github
* Route handler chains, like Express
* Built-in DTrace probes to measure per-route and
per-handler latency
* Easy integration with Bunyan for logging
Joshua M. Clulow
⊕ Joyent

10. Node.js In Production At Joyent
Flamegraphs
Profiling and Call Stack Visualisation
* A node implementation, davepacheco/node-stackvis,
based on Brendan Gregg's original Flamegraph work
* Use DTrace to profile the application, i.e. regularly
sample the current stack
* Collect the like components of each sample and draw them
together to give a picture of where the program
spends its on-CPU time
* Pipe your stacks through c++filt to demangle the C++
method names
Joshua M. Clulow
⊕ Joyent

11. Node.js In Production At Joyent
General Development Tips
* Compile C/C++ code with -fno-omit-frame-pointer,
so that stack traces work in DTrace and mdb(1)
* Prefix your property names, making them easier to
find when debugging core files, e.g.
var iface = { if_id:
0x3423,
if_name:
'interface 0',
if_ipaddr: '192.168.1.1' };
* Name your 'anonymous' functions, making them easier
to identify in stack traces
Joshua M. Clulow
⊕ Joyent

12. Node.js In Production At Joyent
Thanks For Listening!
Read more about our Node.js Engineering Practices:
http://www.joyent.com/developers/node
Joshua M. Clulow
⊕ Joyent