1. javascript & the future

2. Jeff Miccolis
Development Seed

3. Open Atrium
Features
Context
Strongarm

4. Jeff Miccolis
MapBox

6. First, a brief apology.

7. Three “war” stories
about things that are
hard with php & drupal

8. ...and easier in node.js

9. Two lessons I learned

10. ...from Drupal and
node.js work

11. SERVER SIDE
JAVASCRIPT

12. V8

13. CHROME ‘S
JAVASCRIPT ENGINE

14. ON YOUR SERVER

15. 1. Sending lots of email

16. Open
Atrium

18. Single emails
Digests email

19. SMS
XMPP (jabber)
Twitter
etc, etc, etc...

20. channel independent messages
don’t send e-mails to users, send them 'messages'
delivered by mail, IM, SMS, etc...
d.o/project/Messaging

21. Problems

22. Send an email to one user - 50 ms
28 users - 1.4 seconds
600 users - 30 seconds
Problems

23. Run cron every 5 minutes
and you can send a lot more emails
but if one run doesn’t complete...
Problems

24. <?php
$addresses = array(
'jeff@mapbox.com',
'alex@mapbox.com',
'eric@mapbox.com'
);
$message = 'I am writing to inform you of a...';
$count = 0;
foreach($addresses as $sucker) {
if (mail($sucker, 'New opportunity', $message)) {
$count++;
};
}
echo "$count messages sent";

25. var mail = require('mail');
var addresses = [
'jeff@mapbox.com',
'alex@mapbox.com',
'eric@mapbox.com'
];
var message = 'I am writing to inform you of a...';
var count = 0;
addresses.forEach(function(sucker, i) {
mail(sucker, 'New opportunity', message, function(err) {
if (!err) { count++; }
if (i == addresses.length - 1) {
console.log(count +" messages sent");
}
});
});

26. // Attempt to send email.
if (mail($sucker, 'New opportunity', $message)) {
$count++
};
// Once email is sent, get the next address.
echo 'next';
// Attempt to send email.
mail(sucker, 'New opportunity', message, function(err) {
if (!err) { count++; }
});
// Don't wait, get the next address NOW.
console.log('next');

27. Send an email ~ 5 ms processing, 45 ms waiting
28 emails ~ 140 ms to process, done 45 ms later.
600 users ~ 3 seconds to process, done 45 ms later.
Stop waiting around

28. 600 emails w/Php: ~30 seconds
600 emails w/node.js: ~3 seconds
Stop waiting around

29. DEMO
Stop waiting around

30. 2. Handling feeds

32. Fetch RSS feeds
Fetch original article
Tag items (calais)
Tag items (watchlist)
Tag items (wikipedia)
Geocode items
The task

33. Thousands of feeds.
Millions of items.
Gigs and Gigs of data.
The scale

34. cron.php
Problems

35. maggie

36. multi-threaded python daemon
maggied

37. 4 “retriever” workers get batches of 50 items.
they fetch/tag/geocode each item.
maggied

38. retrieve original story: 300ms
tag: 100ms
geocode: 150ms
TOTAL: 550ms
maggied

39. Nearly all of that 550ms is spent idle.
Stop waiting around

40. ...but we could run “packs” of retrievers!

41. Is that really the best
idea?

42. Replace the retrievers
with a single
HYPERACTIVE SQUID!

44. The squid runs up and down as fast as it
can dealing with each item in turn.
It fires off any long running I/O operations
and then moves on to the next item.
When the I/O operation reports progress,
it does a little more work on behalf of the
corresponding item.

45. Event loop
100% async

46. Anything that leaves v8 takes a callback.
100% async

47. filesystem, network, stdio, timers, child processes
100% async

48. var fs = require('fs');
fs.readFile('/etc/passwd', function (err, data) {
if (err) throw err;
console.log(data);
});
var request = require('request');
request('http://example.com', function (err, resp, body) {
if (!err && resp.statusCode == 200) {
console.log(body);
}
});

49. Limiting factors change:
how many open sockets are you allowed?
how much bandwidth can you grab?
how fast can you issue requests?
100% async

50. 3. Big files, long sessions.

51. gigabyte
What’s big?

52. hours
What’s long?

53. HTTP
What’s a session?

54. A category of stories...

55. upload_max_filesize
post_max_size
max_input_time
max_execution_time
Big uploads in php

56. this approach caps out ~500mb
Big uploads in php

57. Opens the door to DOS.
Tolerates application bloat.
Problems in production can get really bad.
Never gonna get gigabyte uploads.
Problems

58. Php makes you look elsewhere.
Problems

59. If only we could stream...

60. If only we could stream...

61. Deal with things bucket by bucket
and you don’t need all that memory.
Streaming

62. write that files to disk as it comes in
...or stream it off to s3!
Streaming

63. var formidable = require('formidable');
var http = require('http');
http.createServer(function(req, res) {
if (req.url == '/upload' && req.method == 'POST') {
var form = new formidable.IncomingForm();
form.parse(req);
form.onPart = function(part) {
part.addListener('data', function(chunk) {
// Do cool stuff, like streaming!
console.log(chunk.toString('utf8'));
});
};
}
}).listen(80);

64. _changes

65. A persistent connection to your database,
which feeds you new data when it has some.
_changes

66. it’s amazing
_changes

68. 1. Map uploads to s3
2. Save a record to CouchDB
3. “Downloader” listens for new maps
MapBox uploads

69. node.js process
very long-lived http connection
to CouchDB
MapBox uploads

70. non-blocking I/O & single event loop
Everything is different

71. 4. Package Management

72. Package management for Drupal
drush make

73. d.o - project namespace
d.o - inclusive project policy
drush make

74. a way to stay sane.
drush make

75. ...and is part of drush proper now!
drush make

76. But I’d been using Drupal for YEARS by then
drush make

77. pear

78. PHP Extension and Application Repository
pear

79. high threshold for new projects
pear

80. wildly inclusive
awesomely useful
awesomely successful
imagine if pear was...

81. wildly inclusive
awesomely useful
awesomely successful
npm

82. node package manager
npm

83. pear: 584
d.o: 15, 296 (~3,600 for D7)
npm: 7,976 (2 years old!)
packages

84. {
"author": "Jeff Miccolis <jeff@miccolis.net>",
"name": "portal",
"description": "Data Catalog",
"version": "0.0.0",
"engines": {
"node": "~v0.6.6"
},
"dependencies": {
"couchapp": "https://github.com/.../attachment_operators",
"underscore": "= 1.2.0",
"request": "= 2.1.1"
}
}
npm - package.json

85. you’ll love it.
npm

86. 5. Nice hammer

87. “Javascript, really?”
Nice hammer...

88. “Clearly he’s overly excited about this async stuff”
Nice hammer...

89. “...and thinks it’ll work for everything.”
Nice hammer...

90. “Do it with Drupal”, eh?

91. computationally heavy tasks
databases
node.js is bad for...

92. interacting with other services.
node.js is awesome for...

93. databases, mail servers, web services, web clients..
services like...

94. http://substack.net/posts/b96642
http://blog.nelhage.com/2012/03/why-node-js-is-cool/
Other people’s words

95. “The primary focus of most node modules is on
using, not extending... A big part of what makes
node modules so great is how they tend to have
really obvious entry points as a consequence of
focusing on usability and limited surface area”
Limited surface area

96. “Instead of the http server being an external
service that we configure to run our code, it
becomes just another tool in our arsenal”
Callback Austerity

97. "The upshot of this pressure is that, since
essentially every node.js library works this way,
you can pick and choose arbitrary node.js
libraries and combine them in the same
program, without even having to think about the
fact that you’re doing so."
Async by default

98. If nothing else you’ll get better at javascript.
Try it!

99. But I bet you’ll like it.
Try it!

100. Thanks!

101. Photo credit due to these wonderful people who offer
their photos on flickr under a creative commons
license:
Spam - http://www.flickr.com/photos/olivabassa/
Cows - http://www.flickr.com/photos/lynndombrowski/
Dogs - http://www.flickr.com/photos/photosightfaces/
Squid - http://www.flickr.com/photos/laughingsquid/
Ants - http://www.flickr.com/photos/fuzzcat/
Stream - http://www.flickr.com/photos/universalpops/
Boxes - http://www.flickr.com/photos/sillydog/
Pear - http://www.flickr.com/photos/reebob/
Hammer - http://www.flickr.com/photos/kefraya

103. What did you think?
Locate this session on the
DrupalCon Denver website
http://denver2012.drupal.org/program
Click the “Take the Survey” link.
Thank You!