Increase Nginx worker open files and connections

Each active connection in Nginx consumes file descriptors (FDs).

The default open file limit, for the user running the nginx prcoess, in Linux based systems is often quite low, like around 1000 is considered low, and varies by distribution.

Default worker_connections limit for Nginx is also set low by default, around 1000 or below, and can easily be exhausted during traffic spikes.

Assuming default open file limit, for the user running the nginx process, is 1024 and connections for each nginx worker is set to 768 by default, we can calculate how many connections nginx is able to proxy.

If nginx is set to allow a maximum of 768 connections * 2 FDs per connection => 1536 file descriptors per worker, which defaults to the number of CPU cores off the system, e.g. 1536 * number of CPU cores.

If the nginx user open file limit is set to 1024, nginx process can theoretically proxy only 1024 / 2 => 512 connections in total acrtoss all workers, before encountering the dreaded "too many open files" error.

It gets worse, nginx also needs to open files to serve them, all the js, css, html, fonts, images etc, which use one file descriptor per file.

Increasing the number of files and connections that Nginx can handle requires that the operating system is set up correctly for it to work.

First check which user is running nginx by running the command:

Example output:

This means:

• The master process is running as root.
• The worker processes are running as www-data.

If Nginx is running as a different user, adjust accordingly (e.g., if it's running as nginx, use nginx in the following commands).

This means Nginx worker processes are running under the www-data user.

The master process running as root is responsible for managing and spawning worker processes, but it doesn’t handle client connections directly.

The master process does not need to have a high open file limit because it primarily handles configuration, signals, and managing worker processes.

Master process usually doesn’t need changes unless running at a very large scale.

Now, check the limits for the worker processes, which run under the www-data or nginx user.

Example output:

This shows the soft and hard limits for open files for the worker process.

To check the limits for the user running Nginx (e.g., www-data or nginx), run:

Check the system-wide limit for open files:

Modify the limits for the Nginx user (e.g., www-data or nginx).

Open the file for editing using nano:

Safe limits for soft open files, you can probably go much higher with some testing.

Add the desired limit for the user running nginx workers by adding the lines below at the end of the file:

Open the file for editing using nano:

Add this line:

Open the file for editing using nano:

Add this line:

Then reload the daemon:

Modify kernel parameters to allow more open files.

Open the file for editing using nano:

Add this line:

Apply changes:

Open the file for editing using nano:

Add this line:

Open the file for editing using nano:

Add this line:

After restarting, check if the new limits are applied.

Check the limits for the Nginx worker processes:

Check the system-wide open file limit:

To handle a large number of connections, you should also adjust the following settings in the Nginx configuration.

The key settings to tweak are:

• worker_rlimit_nofile: Increases the maximum number of open files for worker processes.
• worker_connections: Specifies the maximum number of connections each worker can handle.

Limits on how many files each Nginx worker can open is set using worker_rlimit_nofile in the main Nginx config file, usually found in /etc/nginx/nginx.conf.

This will allow nginx to open more static files (html, js, css, fonts) for serving and to be able to open more connections to servers for proxying.

Set worker_rlimit_nofile in the nginx config, with values to use only half the maximum value set in the system.

Leave half of the allowed limit for other users and processes.

Edit the Nginx configuration file:

Modify the worker_rlimit_nofile setting:

Restart nginx for the settings to take effect.

After enabling each nginx worker to open enough files by setting a high enough value for worker_rlimit_nofile, it's time to raise the number of connections that nginx can open.

Nginx uses 2 file descriptors per connection for proxying, one for receiving and one for sending.

16384 files open / 1 worker / 2 files descriptors per connection => max 8192 connections per worker, set to 4096 to leave 8192 for other files.

This way Nginx should be able to proxy 4096 concurrent connections per CPU and be able to serve files at the same time.

Edit the Nginx configuration file:

Modify the worker_connections setting:

Restart nginx for the settings to take effect.

By following these steps, you've optimized open file limits for Nginx:

• User limit: Increased via /etc/security/limits.conf for the user running Nginx.
• Per-process limit: Increased via systemd configuration for Nginx worker processes.
• System-wide limit: Increased via /etc/sysctl.conf and systemd settings.

This setup ensures Nginx can handle more connections efficiently by allowing more open files, especially for proxying and serving a large number of files.

These limits should be safe, the best is, as always, to test and tweak with your own particular setup.

You might be able to go even higher.