Viewing overall server status report: `passenger-status`
on Passenger + Nginx

At its core, Passenger is an HTTP router and process manager that forwards requests and supervises application processes for you. Many things are taken care of for you, but sometimes you may want to see what Passenger is doing and what is going on, perhaps for troubleshooting reasons. The main tool for doing that is passenger-status.

passenger-status reports:

Which applications Passenger is serving.
Which application processes Passenger is managing, and their status such as CPU and memory usage.
The status of the request queues.
Any currently active requests.
Various internal state.

Based on the information in passenger-status, you can detect and diagnose various problems.

Table of contents

Viewing process and request queue information

Here is a basic passenger-status invocation. If you run it without any further arguments, then passenger-status will report process and request queue information. Please read on for a breakdown of the output.

$ sudo passenger-status
Version : 5.0.13                      1
Date    : 2015-07-06 12:18:38 +0200   2
Instance: bdVuBLEf (nginx/1.8.0 Phusion_Passenger/5.0.13)

----------- General information -----------
Max pool size : 9                 3
App groups    : 2                 4
Processes     : 3
Requests in top-level queue : 0   5

----------- Application groups -----------
/var/www/passenger_status_service/current/public:   6
  App root: /var/www/passenger_status_service/current
  Requests in queue: 0   7
  * PID: 18257   Sessions: 0       Processed: 66179   Uptime: 5h 53m 29s   8
    CPU: 3%      Memory  : 110M    Last used: 0s ago

/var/www/phusion_blog/current/public:
  App root: /var/www/phusion_blog/current
  Requests in queue: 0
  * PID: 18334   Sessions: 0       Processed: 4595    Uptime: 5h 53m 29s
    CPU: 0%      Memory  : 99M     Last used: 4s ago
  * PID: 18339   Sessions: 0       Processed: 2873    Uptime: 5h 53m 26s
    CPU: 0%      Memory  : 96M     Last used: 29s ago

Breakdown:

The version number of the passenger-status tool. This is usually the same as, but not necessarily, the version number of the Passenger instance that is running.
The current date and time when passenger-status was run.
The value of the passenger_max_pool_size configuration option.
The number of application groups that Passenger is serving.
The amount of requests in the top-level request queue. Due to the way Passenger is implemented, this value is supposed to be almost always zero. If it is non-zero for an extended period of time, then there is something very wrong, possibly a Passenger bug.
An application group entry and its processes. The application group name and the path to the application root are shown here.
The number of requests in the application-group-specific request queue. If this number is higher than 0, then it means that all application processes are busy handling requests, and that none of them can currently handle one more request. A request waits in this queue until one of the application processes is done handling a request.
Various statistics about the processes belonging to this application group.
- PID is the operating system process ID of this process.
- Sessions is the number requests that this application process is currently handling.
- Processed is the number of requests this application process has handled so far.
- Uptime is the uptime of this application process.
- CPU is the amount of CPU that this application process is currently using.
- Memory is the memory usage. This value here deserves special explanation.
- Last used is the time when a request was last routed to this application processes.

There are several fields that you should pay attention to to detect common problems.

Requests in queue

Is Requests in queue higher than ~2 for a long time? Then it indicates that the application is either stuck or not fast enough to handle the traffic. To deal with stuck applications, please read Dealing with common request-related problems. To handle traffic more quickly, please read the optimization guide or scale your cluster.

Sessions

Is Sessions non-zero for a long time? Then may mean that the process is stuck on a request. Please read Dealing with common request-related problems.

Memory

Does Memory look much higher than the average memory usage of your other processes in the same application group? Or does it keep increasing over time, with no sign of stopping? Then you may have a memory leak. Here's what you can do about that:

Terminate the process with the command kill <PID>. This will allow the operating system to completely release that process's memory. The memory leak will come back eventually, but it will buy you some time.
Enable the configuration option passenger_memory_limit. This will allow Passenger to automatically terminate a process when its memory usage grows too high. It won't solve your memory leak, but will keep the server stable until you have solved the underlying cause.
Read Debugging memory leaks in Ruby by Sam Saffron.
Read What I Learned About Hunting Memory Leaks in Ruby 2.1 by Peter Wagenet.
Use the Rbkit memory profiler.

Viewing active requests

You can see which HTTP requests Passenger is currently handling by passing --show=requests:

$ sudo passenger-status --show=requests

This will output a JSON representation of the router's state:

{
   "threads" : 2,
   "thread1" : {
      "active_client_count" : 1,
      "active_clients" : {
         "2-2315" : {
            "connected_at" : {
               "local" : "Mon Jul  6 13:19:13 2015",
               "relative" : "0s ago",
               "timestamp" : 1436181553.120679
            },
            "connection_state" : "ACTIVE",
            "current_request" : {
               "app_response_http_state" : "PARSING_HEADERS",
               "app_sink_state" : {
                  "callback_in_progress" : false,
                  "initialized" : true,
                  "io_watcher_active" : false
               },
               "app_source_state" : {
                  "callback_in_progress" : false,
                  "initialized" : true,
                  "io_watcher_active" : true
               },
               "flags" : {
                  "dechunk_response" : true,
                  "https" : true,
                  "request_body_buffering" : false
               },
               "host" : "blog.phusion.nl",
               "http_major" : 1,
               "http_minor" : 1,
               "http_state" : "COMPLETE",
               "method" : "GET",
               "path" : "/",
               "refcount" : 1,
               "request_body_already_read" : 0,
               "request_body_fully_read" : true,
               "request_body_type" : "NO_BODY",
               "response_begun" : false,
               "session" : {
                  "gupid" : "16d1dbc-9VEXQm82is",
                  "pid" : 18334
               },
               "session_checkout_try" : 1,
               "started_at" : {
                  "local" : "Mon Jul  6 13:19:13 2015",
                  "relative" : "0s ago",
                  "timestamp" : 1436181553.121373
               },
               "state" : "WAITING_FOR_APP_OUTPUT",
               "sticky_session" : false,
               "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-2315",
            "number" : 2315,
            "output_channel_state" : {
               "bytes_buffered" : {
                  "bytes" : 0,
                  "human_readable" : "0 bytes"
               },
               "callback_in_progress" : false,
               "mode" : "IN_MEMORY_MODE",
               "nbuffers" : 0,
               "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
         }
      },
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 127,
      "free_request_count" : 3,
      "mbuf_pool" : {
         "active_blocks" : 4,
         "active_memory" : {
            "bytes" : 2048,
            "human_readable" : "2.0 KB"
         },
         "chunk_size" : 512,
         "free_blocks" : 10,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 5120,
            "human_readable" : "5.0 KB"
         }
      },
      "pid" : 5171,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 9209549,
      "total_clients_accepted" : 2315,
      "total_requests_begun" : 2315,
      "turbocaching" : {
         "fetches" : 1,
         "hit_ratio" : 0.0,
         "hits" : 0,
         "store_success_ratio" : null,
         "store_successes" : 0,
         "stores" : 1
      }
   },
   "thread2": {
      ...
   }
}

Here is the breakdown of the most important fields:

threads is the number of threads that the Passenger core process uses. The Passenger core process is the one that routes requests to application processes. The number of threads is usually equal to the number of CPU cores on the system.
threadN contains the state of a particular thread.
host and path allow you to infer the URL of a request.
session -> pid indicates the application process that is currently handling the request.
started_at indicates how long this request has been running.

Pay special attention to the started_at field. If a request has been running for a long time (e.g. longer than 30 seconds) then it may be an indication that the application process is stuck on this request. There are several things you can do in response to this:

(Ruby) Send the SIGQUIT signal to the application process's PID. Everything will keep running as-is, and the application process will log its backtrace to the Passenger log file for you to diagnose the underlying problem.
Terminate the process with the command kill <PID>. Passenger will notice the terminated process, abort the request (HTTP 502), and spawn a new application process if needed.
Setup the feature passenger_max_request_time. This way, Passenger will automatically terminate the process when it gets stuck on a request.

Viewing the internal state; machine-readable output

You can instruct passenger-status to dump an XML representation of the Passenger instance's state. You can do this by passing --show=xml:

$ sudo passenger-status --show=xml

Miscellaneous considerations

To sudo or not to sudo

For security reasons, access to passenger-status is restricted in the following manner:

If you invoke passenger-status without root privileges, then you can only see information about the processes and application groups that are running as the same user that invoked passenger-status. This requires version 5.0.10 or later.
If you invoke passenger-status with root privileges, then you can see everything.

Ruby users sudo notes

Are you using RVM, and did you install Passenger via RubyGems or source tarball? Then be sure to use rvmsudo instead of sudo.

Do you have Passenger as an entry in your Gemfile? Then be sure to prepend bundle exec to the command (but after sudo/rvmsudo).

So for example, suppose that you are using RVM, and you have Passenger as an entry in your Gemfile. Here's how you invoke passenger-status with root privileges:

$ cd /path-to-your-app
$ rvmsudo bundle exec passenger-status

Accurately measuring memory usage

The way Passenger tools report memory usage deserves special attention. You probably have experience with reading memory usage statistics in ps and top. But did you know they are not accurate and may even be misleading? You may have seen that those tools present various memory usage metrics, such as "RSS" and "VMSize". Different metrics represent different things, but which one is the most accurate representation of "real" memory usage? What metric does Passenger use?

This is actually quite a hard question to answer. Intuitively, we think that the question of "how much memory does a process use?" is simple. But the reality is that memory management on modern operating systems is quite complicated, and the concepts behind them are quite complicated too. Because of this, there is not one single right answer to this question, nor is there a short answer.

In the article Accurately measuring memory usage we will explain why a naive reading of ps and top output can be misleading, and how Passenger more accurately reports memory usage than those tools.

Viewing overall server status report: passenger-status on Passenger + Nginx