Zero downtime deployments with PM2
PM2 is the most popular process management tool for Node/Express apps in production. It allows us to run Node/Express application as background process. Multiple running applications on a single machine can be managed with ease using PM2. If you wish to explore all the offerings from PM2 in detail you can read more here.
Objective
We are going to focus mainly around the process of achieving Zero downtime deployments with PM2. If you are not familiar with the basics of PM2, please read this Quick Start Guide & explore various possibilities with PM2.
Zero Downtime Deployments
What & Why?
Zero downtime deployments allows us to deploy new changes to already running application without impacting the application users. It is most likely that when we are doing deployment, some or many of our users are using the application. Pushing the new changes to a running Node/Express application requires application server restart . Most of the times Node/Express server requires some time after restart to be ready to accept requests from consumers due to various factors like Database Connection(s), Logger setup, Web Socket setup, cron job schedules & many more application specific needs.
In order to ensure minimum business impact (downtime) due to the deployments, we should always ensure Zero downtime deployments.
How?
We can utilise various PM2 parameters to achieve zero downtime deployment. We need to configure our restart command with various options (as per our requirements) & minimal change in our code for this to work properly.
Concepts
Before we start, we need to explore a few terminologies in terms of PM2 to completely understand the whole configuration.
Cluster Mode
PM2 offers to run any Node/Express application in cluster mode.
The cluster mode allows networked Node. js applications (http(s)/tcp/udp server) to be scaled across all CPUs available, without any code modifications. This greatly increases the performance and reliability of your applications, depending on the number of CPUs available. Read More >>
So, basically we can run multiple instances of our application listening on same port & these instances are automatically load balanced by PM2 in round robin fashion.
If we deploy our application in cluster mode & use pm2 reload instead of pm2 restart, PM2 will restart these multiple instances one by one, so that we don’t have a downtime (as other worker processes are already running).
wait-ready
If we run our application with PM2 along with wait-ready configuration, during next redeployment (i.e. pm2 reload), it will wait for one instance to become ready to accept requests before moving to next instance for a restart.
This very common that Node/Express applications takes some time to be ready for serving user requests after a restart due to various factors like Database Connection(s), Logger setup, Web Socket setup, cron job schedules & many more application specific needs.
This configuration requires one line of code in our application to signal that server is ready. This is done by running following line of code after we are done with all initialisation stuff mentioned above (DB setup etc.):
process.send('ready');
listen-timeout
By default, PM2 waits for 3s for an instance to signal “ready” (if we use wait-ready option mentioned above). If our server takes more than 3s, it will move to next instance restart without waiting further.
In order to fix the same, we need to specify listen-timeout with our start configuration. Suppose generally our server takes 5-8 seconds to be ready, we can specify a listen-timeout of 10000 (10s) to be on safer side. Now PM2 will wait for maximum of 10 seconds to get "ready" signal from an instance. If it gets ready signal before 1listen-timeout, it will move to next instance immediately for next instance restart.
Configuring the startup command
The above mentioned configuration can be achieved using following startup command & above specified 1 liner code change.
pm2 start your_startup_file.js --name YOUR_APP_NAME -i 2 --wait-ready --listen-timeout 10000
Replace your_startup_file.js with your actual startup file (app.js, server.js, index.js etc.). Apart from this, you can also change the listen-timeout value & number of instances (-i) as per your need.
Redeploying the changes
- Pull the changes (git/zip download & extract)
- Run command pm2 reload
YOUR_APP_NAME
A better way to manage startup configurations
As we can see, now we have more flags with our startup commands & it is more prone to mistakes while executing the startup command.
We can save this configuration to json/js file & then use that file to execute startup command.
Create a file with a name of your choice (e.g. pm2.json, pm2.stage.json, pm2.prod.json etc.) in your project root (keep cwd in mind) & add following configuration (modify as per your needs).
Special Notes
instance_var : This key is used to specify env var name for instance identifier. here we provided it as APP_INSTANCE_SEQ, but you can specify any string here. inside your code you can get value of instance identifier as process.env.APP_INSTANCE_SEQ (or whatever string you provided with instance_var). Instance identifier is helpful in identifying specific application instance to perform specific task on a single instance instead of all application instances, e.g. Cron job schedules - we don't want to do duplicate task execution while running multi instance setup.restart_delay : If our application instances go down due to some unusual reason, PM2 will try to restart application instances automatically. In such cases, we don't want to bombard our resources (like DBs) continuously for startup connections. We can use this value to specify a delay for automatic restarts in case of failures.cwd: This is the working directory, relative to which PM2 looks for "script" (startup file). If you specify it as "." (or keep unspecified), PM2 will look for startup file relative to directory from where startup command was executed.env: This can be used for injecting environment variables for our application instances. PM2 supports multi environment configuration setup as well, but generally we do environment related stuff using system env vars or a .env files.RECOMMENDATION : PM2 supports multi-app configuration in a single file but it is better to use separate files for managing different configurations for prod, stage, dev & so on - pm2.<YOUR_ENV>.json,
Executing tasks on single instance
As we will be running our application in cluster mode, every instance will be identical & everything will be performed on all instances. But many times we need to do special things that should not be duplicated.
For example we have a cron job schedule, that runs every 12 hours.
If we have 3 instances of our application, every 12 hours our cron job will be executed 3 times but we don’t want to do that !!
In order to avoid such situation, we need to identify primary instance & do those specific actions on that instance only. This can be easily done by APP_INSTANCE_SEQ env variable that we configured earlier.
so, suppose at the application startup, we are registering cron job schedule, we can check following:
I hope this comprehensive guide to setup your PM2 deployment will help you achieve zero downtime deployments for your next/existing project(s).
I will be back with more exciting content from my day to day technical learnings. So, stay tuned &
