Zero-downtime web server restarts with Flying Passenger Enterprise

Relevant selection for this article:

Flying Passenger allows one to decouple Passenger's life time from the web server's life time, so that the web server can be independently restarted from Passenger, and from any of the application processes served by Passenger.

Table of contents

• Loading...

Introduction

Normally, Passenger starts together with the web server, and shuts down together with the web server. The advantages of this default behavior is that it makes Passenger easy to administer: one only has to deal with the web server process and can expect all relevant processes to be cleaned up after a web server shut down. However this also brings about a disadvantage: every time one restarts the web server (e.g. to make a minor configuration change), Passenger and all its application processes also get restarted.

This problem is solved by 'Flying Passenger', which is an advanced mode of operation in Passenger that allows the web server to be independently restarted from Passenger. When this mode is enabled:

• One must start Passenger separately from the web server, namely by starting the Flying Passenger daemon. This daemon must - to an extent - be separately configured and managed from the web server.
• The web server must be configured to forward requests to the Flying Passenger daemon.
• You should beware of the caveats and limitations.

The Flying Passenger mode is off by default. You have to explicitly enable it.

There is also a distinct problem, namely that of restarting web applications without downtime after you have deployed application updates. That problem is solved by the rolling restarts feature, not by Flying Passenger. To achieve full zero-downtime, you will want to enable both rolling restarts and Flying Passenger.

Requirements

• The Flying Passenger feature is only available in Passenger Enterprise. Buy Passenger Enterprise here.
• Passenger version 4.0.6 or higher installed on Nginx.

Basic usage

Start the Flying Passenger daemon by invoking the flying-passenger command. The only required option is --socket-file. Depending on whether you wish to enable user account sandboxing, you have to start flying-passenger with root privileges or not.

$ sudo flying-passenger --socket-file=/var/run/flying-passenger.sock
I, [2013-06-14T09:10:13.095339 #77179]  INFO -- : Welcome to Flying Passenger 5.0.11
I, [2013-06-14T09:10:13.095339 #77179]  INFO -- : Starting PassengerAgent...
I, [2013-06-14T09:10:13.097036 #77179]  INFO -- : PassengerAgent started on PID 77181
...
I, [2013-06-14T09:10:13.129017 #77179]  INFO -- : PassengerAgent initialized properly
I, [2013-06-14T09:10:13.129127 #77179]  INFO -- : Flying Passenger up and listening on /var/run/flying-passenger.sock!

Now configure the web server to make use of the Flying Passenger daemon by setting the passenger_fly_with option to the socket filename:

http {
    ...
    passenger_fly_with /var/run/flying-passenger.sock;
    ...
}

After (re)starting the web server, Flying Passenger is fully operational.

You can test it by adding a virtual host for a web app:

http {
    ...

    server {
        listen 80;
        server_name www.foo.local;
        root /webapps/foo/public;
        passenger_enabled on;
    }
}

Verify that it works by making an HTTP request to it:

$ curl http://www.foo.local/

Now let's verify that restarting the web server does not restart the just-spawned application process. Run passenger-status to obtain the PID of the application process:

$ sudo passenger-status
Version : 5.0.11
Date    : 2015-06-24 12:47:13 +0200
Instance: XE6bilnV (nginx/1.8.0 Phusion_Passenger/5.0.11)

----------- General information -----------
Max pool size : 6
App groups    : 1
Processes     : 1
Requests in top-level queue : 0

----------- Application groups -----------
/webapps/foo/public:
  App root: /webapps/foo
  Requests in queue: 0
  * PID: 77283    Sessions: 0       Processed: 49991   Uptime: 6h 22m 3s
    CPU: 2%      Memory  : 110M    Last used: 0s ago

As you can see, the PID of the application process is 77283. Now let's see what happens if we restart the web server:

$ sudo service nginx restart
$ sudo passenger-status

The application process should remain there, unchanged:

$ sudo passenger-status
Version : 5.0.11
Date    : 2015-06-24 12:47:45 +0200
Instance: XE6bilnV (nginx/1.8.0 Phusion_Passenger/5.0.11)

----------- General information -----------
Max pool size : 6
App groups    : 1
Processes     : 1
Requests in top-level queue : 0

----------- Application groups -----------
/webapps/foo/public:
  App root: /webapps/foo
  Requests in queue: 0
  * PID: 77283    Sessions: 0       Processed: 49992   Uptime: 6h 22m 3s
    CPU: 2%      Memory  : 110M    Last used: 0s ago

Configuring Flying Passenger

Flying Passenger gets some configuration from Nginx, but not all. In particular, most web server directives that are only valid in the http context, e.g. passenger_log_level, have no effect when using Flying Passenger. Instead, you are supposed to pass these configuration directives through command line options to the Flying Passenger daemon. Configuration directives that have no effect on Flying Passenger are documented as such. You can assume that configuration directives that are not documented as such, work fine on Flying Passenger.

For example, to achieve the same effect as setting passenger_log_level to 2, run the Flying Passenger daemon as follows:

$ sudo flying-passenger --socket-file=/var/run/flying-passenger.sock --log-level=2

Currently, not all configuration directives have a Flying Passenger equivalent. Run the following command to see an overview of available options:

$ flying-passenger --help

Managing the Flying Passenger daemon

The Flying Passenger daemon runs in the foreground by default. This is undesirable on server environments. You can make it go into the background by passing --daemonize, --log-file and --pid-file:

$ sudo flying-passenger --socket-file=/var/run/flying-passenger.sock \
    --daemonize --log-file=/var/log/flying-passenger.log \
    --pid-file=/var/run/flying-passenger.pid

You can shut down a Flying Passenger daemon by sending SIGINT or SIGTERM to it:

$ kill `cat /var/run/flying-passenger.pid`

We recommend using daemontools or runit for managing the Flying Passenger daemon. These tools will allow automatically starting the Flying Passenger daemon at boot, and will automatically restart the daemon if it crashes. You can create and enable a daemontools/runit service as follows:

$ sudo mkdir /etc/service/flying-passenger
$ sudo nano /etc/service/flying-passenger/run
#!/bin/sh
exec /path-to/flying-passenger \
    --socket-file=/var/run/flying-passenger.sock \
    --log-file=/var/log/flying-passenger.log \
    --pid-file=/var/run/flying-passenger.pid

Immediately after creating the run file, daemontools/runit automatically runs it to start the daemon. Note that the location (/etc/service) depends on the OS or Linux distros. Sometimes it's /service. Also note that we start the Flying Passenger daemon without --daemonize.

To shut down a daemontools/runit-managed daemon, you need to use svc -d /etc/service/flying-passenger (daemontools) or sv stop /etc/service/flying-passenger (runit) instead of sending a signal to the process.

Using Flying Passenger with MRI 1.8 or JRuby

Using Flying Passenger in combination with MRI Ruby 1.8 or with JRuby requires special attention. This is because the Flying Passenger daemon is written in Ruby, and requires proper Process.spawn support, which neither MRI 1.8 nor JRuby support.

It is however possible to use Flying Passenger with MRI Ruby 1.8 and JRuby. You can't run the Flying Passenger daemon in MRI 1.8 or JRuby, but you can still run the web applications – hosted under Flying Passenger – in MRI 1.8 or JRuby.

First, edit your web server configuration file and specify a Ruby interpreter for your web applications. For example:

# Connect to the Flying Passenger daemon on the following socket
passenger_fly_with /var/run/flying-passenger.sock;
...

server {
    listen 80;
    server_name www.foo.com;
    root /webapps/foo/public;
    passenger_enabled on;
    # Use JRuby for this web application
    passenger_ruby /opt/jruby/bin/jruby;
}

Then you need to install a Ruby 1.9-compatible Ruby interpreter with POSIX spawn support, alongside JRuby/MRI 1.8. Ruby interpreters which can be used for running the Flying Passenger daemon include:

• MRI Ruby ≥ 1.9.
• Rubinius.

The following example demonstrates how you can install MRI Ruby 1.9 in parallel with your MRI Ruby 1.8 or JRuby installation.

Example for Debian/Ubuntu users:

# Install Ruby 1.9
sudo apt-get install -y ruby1.9.3
# Run the Flying Passenger daemon in Ruby 1.9
ruby1.9 -S flying-passenger --socket-file=/var/run/flying-passenger.sock

Example for RVM users:

# Install Ruby 1.9
rvm install 1.9.3
# Run the Flying Passenger daemon in Ruby 1.9
rvm-exec 1.9.3 ruby -S flying-passenger --socket-file=/var/run/flying-passenger.sock

The Flying Passenger daemon will now be run on Ruby 1.9, while the web application www.foo.com will be run on JRuby.

Caveats and limitations

Beware of the following caveats and limitations when using Flying Passenger.

General

The Flying Passenger daemon is written in Ruby. It requires a Ruby interpreter with proper Process#spawn support. At the time of writing, all Ruby interpreters in existance satisfy this requirement, except for MRI Ruby 1.8 and JRuby. See Using Flying Passenger with MRI 1.8 or JRuby for more information.

When you add a new application to the web server configuration, Flying Passenger will automatically pick up the application's settings and spawn this new application upon the first request to it. However it is not capable of automatically starting the new app before a request has been sent to it (i.e. passenger_pre_start-like behavior is not available in this case). As a workaround, you can send an HTTP request to your application after starting the daemon, which forces it to spawn application processes.

Nginx

The Nginx executable must be compiled with the same version of Passenger as the Flying Passenger daemon. Failing to meet this requirement may result in cryptic errors, or may result in certain features not working, until you've fixed the situation. When upgrading Passenger, you must restart both Nginx and the Flying Passenger daemon.

The passenger_root directive has no effect. When using Flying Passenger, you are not supposed to set passenger_root.

When you remove an application from the web server configuration, Flying Passenger will not detect the removal and will not shut down the associated application processes. Killing the application processes will also not help, because Flying Passenger will restart them per the (now-removed, but still in the Flying Passenger daemon's memory) passenger_min_instances setting. At the moment, there are two ways to get rid of those processes:

1. Before removing the application from the web server configuration, explicitly set its passenger_min_instances to 0. Next, send a request to it, which will cause the Flying Passenger daemon to take over the new passenger_min_instances 0 option. You can then proceed with removing the application from the web server configuration, and restarting the web server. Finally, kill the PIDs associated to those application processes and remove the application configuration.
2. Restart the Flying Passenger daemon.