This document summarizes Christian Amor Kvalheim's experience over 4 years developing the Node.js driver for MongoDB. It describes the many challenges faced such as constant changes in Node.js and MongoDB, lack of testing frameworks, and difficulties supporting multiple platforms like Windows. It provides advice on managing open source projects including challenging contributors, giving credit, and ignoring criticism. The document reflects on mistakes made and lessons learned around testing, backwards compatibility, and managing a project through rapid change.
5. IT was crap
• It had a “Hash” and “Ordered Hash” class
• Complete mess with horrible structure
• Used Hacked JSpec for testing at the start
Friday, August 16, 13
6. Node was liquid
• Constant changes in Node.js
• Constant changes in driver to work with
Node.js
• OH yeah no NPM yet
Friday, August 16, 13
8. Types Oh Types
• No int64 support in Javascript
• Oh yeah Numbers is a Double (53 bit int
max)
• Decided to use Google Long closure
library class
Friday, August 16, 13
9. Features
• Started adding features like GridFS in late
jan 2010
• Replicaset and Mongos support in
2010/2011
• Stream support for Cursor and GridFS
• High Availability
Friday, August 16, 13
10. C++
• Profiled and found BSON parser to be
main bottleneck
• Wrote horrible C++ parser
• Later cleaned up by a real C++ developer
from Lucasarts (RIP)
Friday, August 16, 13
11. C++
• Initial version had own implementation of
all BSON types
• C++ Long, Binary etc.
• Hard as hell to maintain
• Bought me nothing in the end asV8
changed
Friday, August 16, 13
12. Making JS BSON faster
Buffers
• Another major refactoring to support
properly
• Calculate size of Document
• Single allocation instead of multiple small
ones
• Using Buffers (optimized by v8)
• Indexed entry a[0] direct memory access
Friday, August 16, 13
13. Pool handling
• No traditional Connection Pool
• No checking OUT and IN Connections
• Connection pinning
• The “Spice” must flow
Friday, August 16, 13
14. Write concerns
• Mongodb wire protocol
• Insert/Update/Remove + GetLastError
• Operations must be pipelined
• Write the op + GetLastError in one
operation
Friday, August 16, 13
15. Reconnecting
• No Blocking == difficult reconnect
• Buffer operations until reconnect
• Still a possible problem waiting for a
solution
Friday, August 16, 13
16. Custom Serialization
off Object to BSON
• Added toBSON support
• Specify on your objects and return the
true representation of the object
• Modeled after toJSON
Friday, August 16, 13
17. Replicaset
• Problem
• Servers come and leave the cluster
• Need to update status of servers
• Need to pick the right server for the
readPreference
Friday, August 16, 13
18. Replicaset
• Single thread SUCKS
• Have to use setTimeout/setInterval
• Has tons of subtle state changes
• Never going to get Webworkers in Node.js
• Processes are to granular
Friday, August 16, 13
20. Windows
• Hello Windows Users
• Node.js goes Windows
• What to do About C++ ?
• Build DLL’s for win32 and win64 in
Windows 7 and package with driver
Friday, August 16, 13
21. Windows Build Hell
• Oh yeah Node extensions only build with
Visual Studio 2010 + Windows 7 64 bit
SDK
• Maybe better now but I doubt it
• Node-gyp will fail to build extensions on
Win causes confusion
Friday, August 16, 13
22. Windows Build Hell
• Confusion about build process on
Windows
• More “Cannot” install module questions
on Win than you can swing a cat at
• Mostly PBKC
• Who the Hell still uses XP (SERIOUSLY)
Friday, August 16, 13
23. ARM
• Build C++ extension on ARM
• Learned about memory alignment
difference between ARM and X86
• Runs on Raspberry PI (HUZZA)
Friday, August 16, 13
25. Regrets
var A = function() {
this.something = "a"
}
A.prototype.some = function() {
return this.something
}
Friday, August 16, 13
26. Should have
var B = function() {
var something = "a"
this.some = function() {
return something
}
}
Friday, August 16, 13
27. Lesson
• If it’s possible to access internal state they
will access internal state
• Only expose state and functions you wish
end user should be able to use
• Examples of My BAD
• Collection class
• Cursor class
Friday, August 16, 13
29. Testing it ALL
• In the beginning
• Simple integration testing with JSpec
• Later moved to JUnit
Friday, August 16, 13
30. 3D Reality hits
Mongo 2.0 Mongo 2.2 Mongo 2.4
Node 0.8
Node 0.10
Node 0.X
with auth
replicaset
sharding
+
2.2 features
+
2.4 features
etc etc etc
etc etc etc
Friday, August 16, 13
31. A New Beginning
• New test framework
• Introduce configurations (replset/etc)
• Introduce runners (in auth mode, etc)
• Run tests against configurations
• Test does not manage configurations
• Still not there
• Need to work on the concept more
Friday, August 16, 13
33. Documentation
• Had to provide tons of usage examples
• Realized I had them in code already
• Built some tools to help me generate
most of the docs
• Spaghetti JS + Python Spinx
Friday, August 16, 13
34. Documentation
/**
* Example of inserting a document containing functions
*
* @_class collection
* @_function insert
* @ignore
*/
exports.test = function(configuration, test) {
var db = configuration.db();
// DOC_LINE var db = new Db('test', new Server('locahost', 27017));
// DOC_START
// Fetch a collection to insert document into
var collection = db.collection("simple_document_insert_with_function_safe");
// Insert a single document
collection.insert({hello:'world'
, func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) {
test.equal(null, err);
// Fetch the document
collection.findOne({hello:'world'}, function(err, item) {
test.equal(null, err);
test.ok("function() {}", item.code);
test.done();
})
});
// DOC_END
}
Friday, August 16, 13
37. On Backwards
Compatibility
• Drivers should be boring
• Change is bad
• Yeah some of the API sucks but to many
depends on it
• This week CoffeeScript next week
LiveScript (REALLY?)
Friday, August 16, 13
38. On Managing The
Project
• Challenge people to help
• Be specific about what you need
• Give credit for pull requests
• Ignore flames or criticism
• Just ignore bad faith issues
• Your code is always going to suck, roll with
it
Friday, August 16, 13
40. Take aways
• No Environment is Static
• Testing Drivers isVery Hard
• Don’t take criticism personally
• Challenge people to help you
• Most people are nice and will help
• Credit people who help you
Friday, August 16, 13
41. In Conclusion
• It’s been A crazy 3 1/2 years of serving
the Node community
• I’ve made tons of mistakes a long the
way
• I’m sure to make a ton of mistakes in
the future
• I would do it all again in a second
Friday, August 16, 13
42. Feel free to Ask
Questions
Friday, August 16, 13