Deployment error resistance
on Passenger + Apache
Passenger Enterprise provides a feature called "Deployment error resistance". This feature reduces the chance of visitors experiencing any errors in case the deployment of an application update failed.
Suppose that you have upgraded your application and you have issued a command to restart it, but the application update contains an error (e.g. a syntax error or a database configuration error) that prevents Passenger from successfully spawning a process. Passenger would normally display an error message to the visitor in response to this.
By enabling deployment error resistance, Passenger Enterprise would "freeze" the application's process list. Existing application processes (belonging to the previous version) will be kept around to serve requests. The error is logged, but visitors do not see any error messages. Passenger keeps the old processes around until an administrator has taken action. This way, visitors will suffer minimally from deployment errors.
How it works
When deployment error resistance is enabled, Passenger Enterprise would perform the following steps whenever it finds that it is unable to start an application process:
- It ensures that all requests are routed to existing application processes (that belongs to the previous version of the application). Passenger will not display an error message to the visitor.
- It logs the error to the log file.
- It sets an internal flag (Deployment Error Resistance Mode) so that no processes for this application will be spawned (even when the current traffic would normally result in more processes being spawned) and no processes will be idle cleaned. Processes could still be shutdown because of other events, e.g. because their memory limit have been reached.
You can check whether Passenger Enterprise has entered Deployment Error Resistance mode by invoking passenger-status and checking for the message "Resisting deployment error" then the flag is set.
Enabling the feature
Deployment error resistance only works when a rolling restart is performed. So to enable this feature, we recommend that you set PassengerRollingRestarts on and PassengerResistDeploymentErrors on. Enabling deployment error resistance only works if rolling restarts are also enabled.
For example, open your Apache configuration file and set:
PassengerRollingRestarts on PassengerResistDeploymentErrors on
Then restart Apache to activate the change.
When using passenger-config restart-app, be sure to pass --rolling-restart.
Troubleshooting
A common problem is that Passenger Enterprise may appear to erroneously enter deployment error resistance mode. That is, it would enter deployment error resistance mode even when no new application update has been deployed, or when the application hasn't been restarted.
You should know that deployment error resistance mode is entered on any application process spawning error. By default, Passenger Enterprise changes the number of processes dynamically based on traffic conditions. If there is a (temporary) traffic spike, then Passenger Enterprise may decide at that moment to spawn another process in order to handle the traffic. If an error occurs at that time, e.g. Passenger Enterprise is unable to spawn the process because of an out-of-memory condition, then Passenger Enterprise will enter deployment error resistance mode.
To solve this problem, you you should find out the exact cause of the spawn error by checking the log file for error messages. Passenger Enterprise will have logged the error reason. For example, if the error message says that the spawn error is caused by an out-of-memory condition, try increasing the server's RAM or swap.
