Learning node.js: my experiences and helpful resources

Here are my notes from learning node.js over the winter vacation. You can think of this post as asking me what resources did you find helpful in learning node.js? There is a lot of node.js stuff out there, but few posts on what is noteworthy, popular or tricky - I am sick of seeing posts explaining what node.js is and showing me the same five lines of code.

Structuring code

I recommend reading the series on control flow in node.js from Tim Caswell:

Scope and this are different/tricky in JS, read this and this.

Update Feb 2011:

I wrote about the event loop in Node.js, and how it works.
And here is my list of essential patterns in Node.js.
And tomorrow (Feb 3rd), I’ll publish something on the problems you run into with for loops.

Update Jan 2011:

John Resig’s online Javascript tutorial is also great.
You can refer back to this page about what features of ECMAScript 5 are implemented in V8.

Inheritance patterns in Javascript (using a modern style that optimizes well when using the Google Closure compiler).

You’ll probably run into problems with assigning values if you use the class pattern. The problems is that adding variables in the prototype makes them shared among instances. You’ll have to explicitly initialize any per-instance variables in the constructor, or otherwise the object instance will keep accessing the prototype property => property acts like it would be static. Obvious in retrospect, but hindsight makes fools of us all.

Modules

Have a look at the node.js modules page on github for a comprehensive list of modules. Install npm to install packages. Or don’t, just drop the repos in a subdirectory and use require, since require in node.js is flexible.

Debugging

Simply printing out stuff can be done using sys.puts(string) (first “var sys = require(‘sys’)”).

Since you are on V8, you can use JSON.stringify(object) natively. console.log() also works, it automatically prettyprints objects, but only one level deep (no nested objects).

For those rare occasions where JSON.stringify(object) fails (e.g. due to recursion) you can use require(‘util’).inspect(object) as an alternative.

Setting up the server

Update: Here is how I got Node.js up and running on Centos 5 with nginx + monit.

First, you’ll want to decide how you want to build your applications – will you use node.js for the whole stack, or will you mix it with more familiar scripting languages. If you want to mix two technology stacks, I strongly recommend NOT trying to combine node.js with other web app stacks using subdomains, since you’ll have an unlimited amount of trouble due to the same origin policy enforcement that is built into all browsers. You can either solve that problem for all browsers:

IE6/7 has no mechanism,
IE8/9 has XDomainRequest,
Mozilla/Chrome/Opera have XMLHttpRequest Level 2 (note that level 2 is needed for cross-domain support).

Or you can solve the problem on the server:

by only using node.js for all of your stack, running a bare node.js server
by running a mixed stack using Nginx or Apache to proxy node.js requests

I’m not quite ready to start from scratch, given the productivity that my non-node-js stack gives; I did the mixed route with Apache which is adequate for trying things out. If you want to scale a bit more, you’ll want to setup Nginx.

You can proxy requests from Apache to Node.js (second example), or do a rewrite. You’ll want to use a subdirectory for node.js requests so that you don’t have to deal with the same origin policy. Here is what I did:

Source code

<VirtualHost *:80>

    ServerAdmin webmaster@localhost

    DocumentRoot /path/to/www/

    ServerName example.com

    ProxyPass /node/ http://127.0.0.1:8001/ retry=0 timeout=120

    <Proxy *>

        Allow from all

    </Proxy>

</VirtualHost>

The “retry=0″ parameter prevents Apache from waiting 60 seconds if a node.js response fails (e.g. due to server restart).

Comet with node.js

Have a look at socket.io (github page) for streaming. In my testing, I couldn’t get it to work with IE9 (sending stuff) and Chrome seemed to keep dropping connections. Firefox worked solidly and Chrome after I set it to reconnect, but I couldn’t figure out why IE9 was not working (it could receive messages, but sending them did not seem to work..). I am sure that socket.io will get there, reliable real-time transmission in all major browsers just isn’t quite yet a problem that has been solved neatly. There are still issues to resolve, it seems. To be fair, that’s for cross-domain requests.

If you use socket.io, you will probably want to implement your own abstractions over it, since you’ll want to do something with channels. There is the broadcast(message, list_of_excluded_user_ids) method, but you probably want to have more fine-grained control which can be done via additional abstractions.

There are also two other promising comet projects: Push-it and Faye. Push-it (github, stackoverflow) is built on top of socket.io. Faye (homepage, github) also looks interesting but I spotted this to-do on the repo: “Detect failed WebSocket connection and fall back to polling transports” – e.g. it doesn’t seem to work in non-Websocket browsers (IE6/IE7), something I would like to have (in fact right now, Dec 2010 WebSockets is disabled in newer FF 4 builds and Opera 11 due to security concerns).

Connecting to MySQL

The top three MySQL bindings are felixge’s node-mysql, Sannis’ node-mysql-libmysqlclient and sidorare’s nodejs-mysql-native.

node-mysql and nodejs-mysql-native are pure node.js clients, while node-mysql-libmysqlclient uses libmysqlclient.

Sannis publishes benchmarks of the prominent node.js MySQL bindings which point to node-mysql-libmysqlclient being the fastest. Looking at the GitHub stats for the different projects, felixge’s implementation is most popular (most followed and forked). Both Sannis and felixge are actively committing (Sannis has daily commits, felixge approximately weekly). sidorare’s last commit is from August (3 months ago; checked Dec 2010) so the project seems to be less active. Regarding performance, felixge notes (prior to developing node-mysql):

“Performance should be a secondary concern here. Show me a realistic MySql scenario where you perform 170k queries / sec against a single database server. Inserting 13k records / sec doesn’t sound like a good use case for MySql either. The mysql driver is pretty unlikely to become a bottleneck in the real world. Anyway, I think there is plenty of room for improvement.”

So basically it comes down to whether you prefer to have an all-node.js solution or a slightly faster libmysqlclient-dependent solution.

I went with node-mysql, which works really nicely. What I am bit unsure of is how I can make sure that the code I write using the library performs well, I’d love to see a better explanation of what goes on inside node-mysql when I do a connect or a query…

Once you get that done, you’ll want to look into connection pooling. Have a look at node-pool.

Or, you might opt to just have one shared persistent connection to MySQL. Apparently, this is what Felix’s company does or did for quite a while.

Deploying node.js with monitoring

Deploying Node.js with Upstart and Monit — by Tim Smart
Deploying node.js with Upstart — a few updates to the above article — by Caolan McMahon

Performance/maturity discussion

Amit Dalihefendic from Plurk writes about the performance of node.js and notes that while they were able to serve millions of real customer notifications during a 8-month period; however, they decided to go with Java and Netty due to current performance problems. Amit attributes the limitations they ran into to the V8 engine – which makes some assumptions which make sense on a browser, but which are problematic on a server (Igor Sysoev, the author of nginx, summarizes this as “V8 will work well in any program, provided that the program is called Chrome“; via Google Translate). On Hacker News, Amit concludes:

“I think the ultimate perfomance is found in pure java.nio/C/C++ solutions, but I think having a bit slower perfomance and higher abstraction is better since it makes it much easier to maintain and debug the system. My general impression of java.nio is that it’s very low-level and a generally hard to code against and it’s the main reason why we didn’t choose it.

This said, I think node.js offers great usability while perfomance is pretty good. So if I was developing a new comet solution I would give node a go – you can always rewrite to something more low level once you begin to hit limits. IMO going after java.nio directly is a premature optimization and most projects won’t hit limits with node.js.”

I figure this is a fair conclusion.

Testing and debugging

Frameworks

Express.js seems to be the most popular web application framework for node.js. Geddy is another (popular?) alternative. Connect provides middleware for framework development, so it is more bare-bones — but since node.js app development is still in the early stages, I’ve seen it used in many repositories. Express is built on Connect, for example. Check out this review of node.js frameworks from May 2010. Since I opted not to write everything in node.js, I’ll have to get back to you on the experiences with these.

Mixu's tech blog