Using Monit with Node.js and PM2

As I’ve said before, I’m a bit of a whore when it comes to learning new languages and development frameworks. So it comes as no surprise to myself that at some point I’d start looking at Node.js and JavaScript.

I have another confession to make. I hate JavaScript, even more so than PHP. Nothing about the language is appealing to me, whether its the rules about scoping and the use of var or the bizarre mechanism by which classes are declared (not to mention there are several ways). Semicolons are optional, kind of, but not really. I know plenty of developers who enjoy programming in Objective-C, Python, Ruby, etc.; I have never met anyone who says “Me? I love JavaScript!” Well, perhaps a web UI developer whose only other “language” is CSS or HTML. In fact, a lot of people go out of their way to articulate why JavaScript sucks.

So along comes Node.js, which we can all agree is the new hotness. I’m not sure why it is so appealing. JavaScript on the server! Event-driven programming! Everything is asynchronous and nothing blocks! Okay, great. I didn’t really ask for JavaScript on the server, and event-driven programming is not new. When you develop iOS applications you’re developing in an event-driven environment. Python developers have had the Twisted framework for years. The venerable X system is built upon an event loop. Reading the Node.js hype online one would think event-driven callback execution was invented in the 21st century.

Of course, the Node.js community is also reinventing the wheel in other areas as well. What do the following have in common: brew, apt-get, rpm, gem, easy_install, pip. Every last one is a “package manager” of some sort, aimed at making your life easy, automagically downloading and installing software along with all of its various dependencies onto your system. A new framework is nothing without a package manager that it can call its own, thus the Node.js world gives us the Node Package Manager, or npm. That’s fine. I like to think of myself as a “full-stack developer”, so if I need to learn a new package manager and all of its quirks, so be it.

Unfortunately it didn’t stop there. Node.js has its own collection of application “management” utilities; you know, those helper utilities that aim to provide an “environment” in which to run your application. Apparently Forever was popular for some time until it was displaced by PM2, a “Production process manager for Node.js / io.js applications”

I’m not quite sure when it became en vogue to release version 0 software for production environments, but I suppose it’s all arbitrary (hell, Node.js is what, 0.12?) But true to a version 0 software release, PM2 has given me nothing but fits in creating a system that consistently brings up a Node.js application upon reboot. Particularly in machine-to-machine (M2M) applications this is important; there is frequently no opportunity to ssh into a device that’s on the cellular network and installed out in an oil field tank site. The system must be rock-solid and touch free once it’s installed in the field.

To date the most pernicious bug I’ve come across with PM2 is it completely eating the dump.pm2 file that it ostensibly uses to “resurrect” the environment that was operating. A number of people have reported this issue as well. If I can’t rely on PM2 to consistently restart my Node.js application, I need something to watch the watchers. So who watches the watchers? Monit of course.

Because PM2 refused to cooperate I decided to utilize monit to ensure my Node.js application process was up and running, even after a reboot. In this configuration file example I am checking my process pid (located in the /root/.pm2/pids directory) and then using pm2 start and pm2 stop as the start and stop actions.

NB: Monit executes its scripts with a bare bones environment. If you are ever stumped by why your actions “work on the command line but not with monit”, see this this Stack Overflow post. In the case of PM2, it is critical that the PM2_HOME environment variable be set prior to calling pm2.

The first iteration of my monit configuration looked like this:

Only if this were sufficient, but it isn’t.

For some reason PM2 insists on appending a process ID to the pidfile filename (perhaps for clustering where you need a bunch of processes of the same name), so a simple pidfile check won’t suffice. Other folks even went to the Monit lists looking for wildcard pidfile support and quoted PM2 as the reason why they felt they needed it.

So, now our monit configuration takes advantage of the matching directive and looks like this:

Granted, we should not be running as root here. Future iterations will move the applications to a non-privileged user, but for now this gives us a system that successfully restarts our Node.js applications after a reboot. PM2 is a promising tool and definitely takes care of a lot of mundane tasks we need to accomplish to daemonize an application; unfortunately it is a little rough around the edges when it comes to consistent actions surrounding the ability to survive system restarts. Don’t take my word for it: read the Github issues.

Conclusion

A rolling stone gathers no moss. The more things stay the same, the more things change (or is it the other way around?). I have nothing against new frameworks, but there are times when being an early adopter requires one to pull out a tried-and-true applications to get the job done. In this case our old friend Monit helps up fill in the gaps while Node.js and PM2 mature.