Tuning Linux Kernel and TCP Stack for High-Concurrency Nginx & PHP-FPM Web Servers

Default Linux kernel network parameters are tuned for conservative, general-purpose workloads. Under high traffic spikes, web servers running Nginx and PHP-FPM often encounter subtle bottlenecks: dropped SYN packets, connection timeouts, ephemeral port exhaustion, and socket starvation. In this guide, we dive deep into the Linux TCP/IP stack and configure production-grade sysctl settings for high-throughput environments.

Understanding High-Concurrency Bottlenecks

When handling thousands of requests per second, a web server encounters bottlenecks across four primary layers:

• File Descriptors: In Linux, every active network socket is a file descriptor. Hitting the process or system-wide nofile limit leads to immediate accept4() failed (24: Too many open files) errors in Nginx logs.
• SYN / Listen Backlogs: Incoming TCP connection requests queue up in the kernel's SYN backlog before being accepted by Nginx or PHP-FPM. If the queue overflows, the kernel silently drops packets.
• TIME_WAIT Socket Exhaustion: Fast HTTP/1.1 reverse-proxying from Nginx to PHP-FPM creates short-lived TCP sockets. When closed, sockets linger in TIME_WAIT for up to 60 seconds, exhausting local ports.
• TCP Buffer Sizing: Inadequate receive/transmit buffer memory restricts throughput, while overly large buffers cause bufferbloat and exhaust server RAM.

1. Overcoming File Descriptor Limits

First, raise the system-wide and per-process limits on open file handles.

System-wide File Limits

Check current limits with sysctl fs.file-max and adjust in /etc/sysctl.d/99-high-performance.conf:

fs.file-max = 2097152
fs.nr_open = 2097152

Process-level Security Limits

Update /etc/security/limits.conf to permit your web server users (www-data, nginx) to allocate high descriptors:

* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
www-data soft nofile 1048576
www-data hard nofile 1048576

2. Eliminating Backlog Queue Bottlenecks

The connection handshake traverses two queues in the Linux kernel:

1. SYN Queue (half-open connections): Controlled by net.ipv4.tcp_max_syn_backlog.
2. Accept Queue (fully established connections waiting for accept()): Controlled by net.core.somaxconn and the application's backlog parameter.

# Increase the maximum number of open half-connections
net.ipv4.tcp_max_syn_backlog = 65536

# Maximum queue length of complete sockets waiting to be accepted
net.core.somaxconn = 65536

# Maximum number of packets queued on the input interface before processing
net.core.netdev_max_backlog = 65536

Important: Nginx and PHP-FPM must also specify a matching backlog in their respective configuration files to take advantage of the kernel's increased queue capacity.

3. Ephemeral Port Range & Socket Recycling

When Nginx proxies requests to upstream backend servers (or PHP-FPM listening on TCP), it binds an ephemeral port for each connection. Increase the available range and enable fast socket reuse:

# Expand ephemeral port range
net.ipv4.ip_local_port_range = 10240 65535

# Allow fast recycling of TIME_WAIT sockets for outgoing connections
net.ipv4.tcp_tw_reuse = 1

# Reduce socket linger time in FIN-WAIT-2 state (default is 60s)
net.ipv4.tcp_fin_timeout = 15

4. Enabling Google BBR Congestion Control

Google's BBR (Bottleneck Bandwidth and RTT) congestion control algorithm significantly outperforms traditional loss-based algorithms like CUBIC on high-latency or packet-loss-prone networks.

# Set fair queuing scheduler
net.core.default_qdisc = fq

# Enable BBR congestion control
net.ipv4.tcp_congestion_control = bbr

Verify BBR is active by running: sysctl net.ipv4.tcp_congestion_control.

5. Complete Production Sysctl Configuration

Create the file /etc/sysctl.d/99-high-performance.conf and paste the comprehensive configuration below:

# /etc/sysctl.d/99-high-performance.conf

# File Descriptors
fs.file-max = 2097152
fs.nr_open = 2097152

# Queue and Backlog Limits
net.core.somaxconn = 65536
net.core.netdev_max_backlog = 65536
net.ipv4.tcp_max_syn_backlog = 65536

# Ephemeral Ports & TIME_WAIT Handling
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# Memory & Buffer Management (min, default, max in bytes)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Congestion Control (BBR)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# TCP Keepalive & Resilience
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_syncookies = 1

# Disable Swappiness Aggressiveness
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

Apply changes immediately without rebooting:

sudo sysctl --system

6. Synchronizing Nginx and PHP-FPM

To align with the tuned kernel parameters, update your service configs:

Nginx Configuration (/etc/nginx/nginx.conf)

user www-data;
worker_processes auto;
worker_rlimit_nofile 1048576;

events {
    worker_connections 65535;
    multi_accept on;
    use epoll;
}

http {
    # Server block
    server {
        listen 443 ssl http2 default_server backlog=65535;
        # ...
    }
}

PHP-FPM Pool Configuration (/etc/php/8.4/fpm/pool.d/www.conf)

; Increase backlog queue to match kernel somaxconn
listen.backlog = 65535

; Set process limits
rlimit_files = 1048576

7. Monitoring and Verification

Use modern socket utilities to check active queues and verify no packet drops occur during peak traffic:

# Inspect socket summaries
ss -s

# Check listen queue sizes and backlog overflows (Send-Q vs Recv-Q)
ss -lnt '( sport = :http or sport = :https )'

# Check TCP statistics for dropped packets
nstat -az TcpExtListenOverflows TcpExtListenDrops

Conclusion

By raising connection queues, tuning ephemeral port lifecycles, enabling BBR congestion control, and aligning Nginx and PHP-FPM process limits, your Linux infrastructure can effortlessly sustain massive concurrent traffic with sub-millisecond connection establishment times.