Web Servers

NGINX ADVANCED INSTALLATION, HARDENING, AND MONITORING GUIDE

18 min read3634 words66 code blocks

At a glance#

Purpose: Install, harden, tune and monitor nginx as a production web server and reverse proxy.
Applies to: nginx on Ubuntu and RHEL-family distributions.
Risk: Medium - configuration changes require reloads; hardening steps can break existing sites.
Time: 2-3 hours for a full build.

This documentation provides:

Advanced NGINX installation script for Ubuntu 22.04+
Hardened configurations with load balancing and security
Monitoring setup (stub_status, VTS, Prometheus, Grafana)
Log analytics
Explanations for every part

This is suitable for production systems handling 1–5K requests per second.

Purpose#

This script:

Installs NGINX and essential tools
Applies hardened nginx.conf
Adds security, gzip, and SSL best-practice snippets
Configures a load-balanced site
Provides a fully commented, modifiable setup

File Name#

install_nginx_advanced.sh

Run#

text

sudo bash install_nginx_advanced.sh

Script#

text

#!/usr/bin/env bash
# Advanced NGINX Installation and Configuration Script for Ubuntu 22.04+
# Features:
# - Installs NNGINX
# - Hardened nginx.conf
# - Security, gzip, SSL, proxy configuration
# - PHP/Node upstream ready
# - Load balancer ready
# - Fully commented

set -euo pipefail

if [[ "$EUID" -ne 0 ]]; then
  echo "Error: Must run as root."
  exit 1
fi

if ! command -v apt-get >/dev/null 2>&1; then
  echo "Error: Supported only on Debian/Ubuntu systems."
  exit 1
fi

echo "Updating package index..."
apt-get update -y

echo "Installing NGINX and tools..."
apt-get install -y nginx ufw curl

if command -v ufw >/dev/null 2>&1; then
  ufw allow 'Nginx Full' || true
fi

NGINX_ETC="/etc/nginx"
SITES_AVAILABLE="$NGINX_ETC/sites-available"
SITES_ENABLED="$NGINX_ETC/sites-enabled"
SNIPPETS_DIR="$NGINX_ETC/snippets"

mkdir -p "$SITES_AVAILABLE" "$SITES_ENABLED" "$SNIPPETS_DIR"

echo "Backing up nginx.conf..."
cp "$NGINX_ETC/nginx.conf" "$NGINX_ETC/nginx.conf.bak.$(date +%F-%H%M%S)" || true

cat > "$NGINX_ETC/nginx.conf" << 'EOF'
user www-data;
worker_processes auto;

error_log /var/log/nginx/error.log warn;
pid /run/nginx.pid;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time uct=$upstream_connect_time '
                    'urt=$upstream_response_time uht=$upstream_header_time';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 15;
    types_hash_max_size 2048;

    include /etc/nginx/snippets/security.conf;
    include /etc/nginx/snippets/gzip.conf;

    limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=5r/s;
    limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

    server_tokens off;

    include /etc/nginx/sites-enabled/*;
}
EOF

cat > "$SNIPPETS_DIR/gzip.conf" << 'EOF'
gzip on;
gzip_disable "msie6";

gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_min_length 1024;
gzip_http_version 1.1;

gzip_types
    text/plain
    text/css
    text/javascript
    application/javascript
    application/json
    application/xml
    application/rss+xml
    application/vnd.ms-fontobject
    application/x-font-ttf
    font/opentype
    image/svg+xml;
EOF

cat > "$SNIPPETS_DIR/security.conf" << 'EOF'
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;

location ~ /\.(?!well-known) {
    deny all;
}

if ($request_method !~ ^(GET|HEAD|POST)$) {
    return 405;
}
EOF

cat > "$SNIPPETS_DIR/ssl-params.conf" << 'EOF'
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EECDH+CHACHA20:EECDH+AES256:EECDH+AES128:!aNULL:!MD5:!DSS";

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

ssl_stapling on;
ssl_stapling_verify on;

resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;
EOF

DOMAIN="example.com"
SITE_CONF="$SITES_AVAILABLE/${DOMAIN}.conf"

cat > "$SITE_CONF" << 'EOF'
upstream app_backend {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    server_name example.com www.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name example.com www.example.com;

    ssl_certificate     /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;
    include /etc/nginx/snippets/ssl-params.conf;

    access_log /var/log/nginx/example.com.access.log main;
    error_log  /var/log/nginx/example.com.error.log warn;

    include /etc/nginx/snippets/security.conf;

    limit_req zone=req_per_ip burst=10 nodelay;
    limit_conn conn_per_ip 20;

    root /var/www/example.com/html;
    index index.html index.htm;

    location /static/ {
        alias /var/www/example.com/static/;
        access_log off;
        expires 7d;
    }

    location / {
        proxy_pass http://app_backend;

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout   5s;
        proxy_send_timeout     30s;
        proxy_read_timeout     30s;

        proxy_buffering on;
        proxy_buffers 16 16k;
        proxy_busy_buffers_size 32k;

        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }

    location /health {
        access_log off;
        return 200 "ok\n";
    }
}
EOF

mkdir -p /var/www/example.com/html
echo "<h1>NGINX operational</h1>" > /var/www/example.com/html/index.html

ln -sf "$SITE_CONF" "$SITES_ENABLED/${DOMAIN}.conf"

nginx -t
systemctl reload nginx || systemctl restart nginx

echo "NGINX installation and configuration completed."

End Script.

Key Metrics to Monitor#

Requests per second
Response time
Upstream errors
Upstream timeouts
4xx / 5xx error rate
CPU usage
Memory usage
Open connections
Open file descriptors
Network throughput
Bandwidth per request
Queueing/latency under load

The sections below show how to set up best-practice monitoring tools.

Built-in module for essential real-time metrics.

Create:

/etc/nginx/sites-available/status.conf

text

server {
    listen 8080;
    server_name _;

    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

Enable:

text

ln -s /etc/nginx/sites-available/status.conf /etc/nginx/sites-enabled/
nginx -t
systemctl reload nginx

Test:

text

curl http://localhost:8080/nginx_status

Install module:

text

apt install nginx-module-vts -y

Add to http block in nginx.conf:

text

vhost_traffic_status_zone;

Create dashboard:

text

server {
    listen 8090;

    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
        allow 127.0.0.1;
        deny all;
    }
}

Reload:

text

nginx -t && systemctl reload nginx

Dashboard URL:

http://localhost:8090/status

System-level metrics.

Install:

text

apt install prometheus-node-exporter -y

Metrics:

text

http://localhost:9100/metrics

Download:

text

wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.11.0/nginx-prometheus-exporter_0.11.0_linux_amd64 \
  -O /usr/local/bin/nginx-exporter
chmod +x /usr/local/bin/nginx-exporter

Service:

text

[Unit]
Description=NGINX Prometheus Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/nginx-exporter \
  -nginx.scrape-uri http://localhost:8080/nginx_status
Restart=always

[Install]
WantedBy=multi-user.target

Enable:

text

systemctl enable --now nginx-exporter

Metrics:

text

http://localhost:9113/metrics

Install:

text

apt install prometheus -y

Prometheus config:

text

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['localhost:9113']

  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Restart:

text

systemctl restart prometheus

Install:

text

apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" \
  | tee /etc/apt/sources.list.d/grafana.list

apt update
apt install grafana -y
systemctl enable --now grafana-server

Grafana panel URL:

http://server_ip:3000

Recommended dashboards:

Node Exporter Full
NGINX Prometheus Exporter
VTS Dashboard

Top requested endpoints:

text

awk '{print $7}' access.log | sort | uniq -c | sort -nr | head

Count 4xx/5xx:

text

awk '{print $9}' access.log | sort | uniq -c | sort -nr

Slow requests:

text

awk '$NF > 1' access.log

Find abusive IPs:

text

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

Upstream issues:

text

grep -i upstream /var/log/nginx/error.log

This document provides:

A full advanced NGINX installation and configuration script
Hardened templates for gzip, SSL, and security
Load-balancing configuration
Built-in and advanced monitoring systems
Prometheus and Grafana setup
Log analysis commands

For Ubuntu 22.04+

This guide provides:

Worker process tuning
Network stack tuning
TCP optimizations
File descriptor tuning
Buffering and proxy optimizations
Rate limiting strategies
Caching strategies
Upstream failover and load balancing
System kernel tuning (sysctl)
Testing and benchmarking methodology

This applies to both PHP-FPM and Node.js upstreams.

For 1–5K RPS, recommended minimum:

CPU: 4–8 cores
RAM: 4–16 GB
SSD storage
1 Gbps network

NGINX is CPU-efficient, so high clock speed is more important than high core count.

In /etc/nginx/nginx.conf:

text

worker_processes auto;
worker_rlimit_nofile 200000;

Worker processes should match the number of CPU cores.

Worker connections:

text

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

Explanation:

worker_connections defines maximum concurrent connections per worker
epoll is the fastest event loop on Linux
multi_accept allows accepting multiple simultaneous connections

Linux defaults are too low.

Add to:

/etc/security/limits.conf

text

* soft nofile 200000
* hard nofile 200000

Also:

/etc/systemd/system.conf

text

DefaultLimitNOFILE=200000
DefaultLimitNPROC=200000

Reload:

text

systemctl daemon-reexec

Create file:

/etc/sysctl.d/99-nginx-performance.conf

Add:

text

net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

net.ipv4.ip_local_port_range = 1024 65000

net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_max_tw_buckets = 2000000

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

net.ipv4.tcp_fastopen = 3

net.ipv4.tcp_mtu_probing = 1

vm.swappiness = 10
vm.max_map_count = 262144

Apply:

text

sysctl --system

Explanation:

somaxconn, backlog: Helps handle connection spikes
tw_reuse: Allows reuse of TIME_WAIT sockets
rmem/wmem: Larger buffers for high throughput
fastopen: Faster TLS and TCP handshakes
mtu probing: Helps avoid packet fragmentation

Add to http block:

text

sendfile on;
tcp_nopush on;
tcp_nodelay on;

keepalive_timeout 15;
keepalive_requests 10000;

client_body_timeout 15s;
client_header_timeout 15s;

reset_timedout_connection on;

server_tokens off;

Explanation:

tcp_nodelay helps small responses
keepalive_timeout affects resource usage
reset_timedout_connection prevents long-lived dead connections

Add inside http block:

text

proxy_buffers 32 32k;
proxy_buffer_size 16k;
proxy_busy_buffers_size 64k;

proxy_connect_timeout   5s;
proxy_send_timeout     30s;
proxy_read_timeout     30s;

proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 128;

proxy_http_version 1.1;
proxy_set_header Connection "";

Explanation:

Improves response handling under load
Prevents upstream timeouts
Supports long keepalive connections

For Node.js or PHP-FPM:

text

upstream backend_pool {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    keepalive 64;
}

Keepalive prevents reconnections for every request.

Load balancing methods:

round_robin (default)
least_conn
ip_hash

Example:

text

upstream backend_pool {
    least_conn;
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
}

Prevent small attacks or sudden spikes.

text

limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=10r/s;
limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;

Inside server block:

text

limit_req zone=req_limit_per_ip burst=20 nodelay;
limit_conn conn_limit_per_ip 40;

Purpose:

Prevents abuse
Protects CPU
Smooths sudden traffic spikes

Gzip:

text

gzip on;
gzip_comp_level 5;
gzip_min_length 1024;
gzip_types text/plain text/css application/json application/javascript;

Brotli (if installed):

text

brotli on;
brotli_comp_level 5;
brotli_types text/plain text/css application/json application/javascript;

text

location /static/ {
    alias /var/www/html/static/;
    access_log off;
    expires 7d;
    add_header Cache-Control "public, max-age=604800";
}

Useful for APIs and dynamic sites.

Example 1 second cache:

text

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=microcache:50m max_size=1g inactive=60m;

server {
    location / {
        proxy_cache microcache;
        proxy_cache_valid 200 1s;
        proxy_cache_use_stale error timeout invalid_header updating;
        proxy_pass http://backend_pool;
    }
}

Microcaching can multiply performance by 10x.

text

open_file_cache          max=50000 inactive=20s;
open_file_cache_valid    30s;
open_file_cache_min_uses 2;
open_file_cache_errors   on;

Recommended:

text

client_header_timeout 10s;
client_body_timeout 10s;
send_timeout 10s;
proxy_read_timeout 30s;

Short headers/body to drop bots.

Long upstream timeout for slow backend.

Use modern protocols:

text

ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

Enable OCSP stapling:

text

ssl_stapling on;
ssl_stapling_verify on;

Disable unnecessary services:

text

systemctl disable --now snapd.service
systemctl disable --now multipathd.service

Increase entropy for TLS:

text

apt install haveged -y
systemctl enable --now haveged

Enable GRO/LRO and increase network buffers:

text

ethtool -K eth0 gro on
ethtool -K eth0 lro on
ethtool -G eth0 rx 4096 tx 4096

Edit:

/etc/php/8.1/fpm/pool.d/www.conf

text

pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 20

Disable slow logs unless debugging.

Increase request termination timeout:

text

request_terminate_timeout = 30s

Use PM2:

text

npm install pm2 -g
pm2 start app.js -i max

Enable clustering:

text

pm2 start app.js -i $(nproc)

Install wrk:

text

apt install wrk -y

Test:

text

wrk -t8 -c400 -d30s http://localhost/

Interpreting results:

Latency < 100ms is good
High 4xx/5xx indicates backend issues
High timeout errors means upstream slow
CPU near 80–90% indicates bottleneck

Check NGINX status:

text

curl localhost:8080/nginx_status

Check upstream errors:

text

grep upstream /var/log/nginx/error.log

Find slow requests:

text

awk '$NF > 1' /var/log/nginx/access.log

Check worker saturation:

text

ps -L -o pid,tid,pcpu,stat -p $(pidof nginx)

Check open connections:

text

ss -s

High traffic tuning requires adjustments across:

NGINX worker configuration
OS file limits
Kernel networking parameters
Proxy and buffering configuration
Upstream keepalive
Rate limiting
Caching (microcache + static)
TCP and SSL performance tuning
Backend process management (PHP/Node)
Proper load testing
Real monitoring via Prometheus + Grafana

When applied together, this configuration stack supports 1,000 to 5,000 RPS on a single modern VM without instability.

If you want, I can also generate:

A complete combined script to apply all sysctl + limits + NGINX optimizations automatically
A full Notion-style troubleshooting playbook: "When site is slow, do this"
An auto-installer that includes NGINX + tuning + monitoring stack

Just say what you want next, brother.

Documentation for:#

A combined Linux + NGINX tuning script
A structured troubleshooting playbook
An all-in-one installer for NGINX + tuning + basic monitoring stack

This script:

Applies Linux kernel/sysctl tuning for high traffic
Raises file descriptor limits
Sets NGINX worker and event tuning
Keeps things explicit and commented

It is designed for Ubuntu 22.04+ and assumes you want NGINX tuned for 1–5K RPS range.

1.1. Save Script#

File name:

text

nginx_system_tuning.sh

Contents:

bash

#!/usr/bin/env bash
# NGINX High-Traffic System + NGINX Tuning Script
# Target: Ubuntu 22.04+
# Applies:
# - File descriptor limits
# - sysctl kernel/network tuning
# - NGINX worker and event tuning

set -euo pipefail

if [[ "$EUID" -ne 0 ]]; then
  echo "Error: Must run as root."
  exit 1
fi

if ! command -v nginx >/dev/null 2>&1; then
  echo "Error: nginx is not installed. Install nginx first."
  exit 1
fi

echo "=== 1) Setting file descriptor limits ==="

cat >/etc/security/limits.d/nginx-high-fd.conf << 'EOF'
* soft nofile 200000
* hard nofile 200000
EOF

mkdir -p /etc/systemd/system/nginx.service.d

cat >/etc/systemd/system/nginx.service.d/override.conf << 'EOF'
[Service]
LimitNOFILE=200000
EOF

echo "Reloading systemd daemon..."
systemctl daemon-reload

echo "=== 2) Applying sysctl kernel/network tuning ==="

cat >/etc/sysctl.d/99-nginx-performance.conf << 'EOF'
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

net.ipv4.ip_local_port_range = 1024 65000

net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 2000000

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_mtu_probing = 1

vm.swappiness = 10
vm.max_map_count = 262144
EOF

sysctl --system

echo "=== 3) Tuning nginx.conf (backup and template) ==="

NGINX_ETC="/etc/nginx"
NGINX_CONF="${NGINX_ETC}/nginx.conf"

if [[ -f "$NGINX_CONF" ]]; then
  cp "$NGINX_CONF" "${NGINX_CONF}.bak.$(date +%F-%H%M%S)"
  echo "Backed up existing nginx.conf to nginx.conf.bak.*"
fi

cat >"$NGINX_CONF" << 'EOF'
user www-data;
worker_processes auto;
worker_rlimit_nofile 200000;

error_log /var/log/nginx/error.log warn;
pid /run/nginx.pid;

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time uct=$upstream_connect_time '
                    'urt=$upstream_response_time uht=$upstream_header_time';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 15;
    keepalive_requests 10000;
    types_hash_max_size 2048;

    client_body_timeout 15s;
    client_header_timeout 15s;
    send_timeout 15s;
    reset_timedout_connection on;

    server_tokens off;

 # Zones for rate limiting and connection limiting
    limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

 # Optional: open file cache for static-heavy workloads
    open_file_cache          max=50000 inactive=20s;
    open_file_cache_valid    30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   on;

    include /etc/nginx/snippets/gzip.conf;
    include /etc/nginx/snippets/security.conf;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}
EOF

echo "Testing nginx configuration..."
nginx -t

echo "Restarting nginx..."
systemctl restart nginx

echo "Done. System and nginx tuned for high traffic."
echo "You may need to log out and log back in for ulimit changes to apply to interactive shells."

1.2. Run Script#

bash

chmod +x nginx_system_tuning.sh
sudo ./nginx_system_tuning.sh

After running:

NGINX worker config is tuned
Linux kernel parameters are optimized
File descriptor limits are raised
nginx.conf is backed up and replaced with a tuned template

You can adjust any values later directly in /etc/nginx/nginx.conf or the sysctl file.

This is your mental model and checklist when something is broken or slow.

You can drop this straight into Notion as an "NGINX Incident Playbook" page.

2.1. When Site Is Completely Down#

Symptom: Browser shows connection error, timeout, or cannot connect.

Checklist:

Is NGINX running?

```bash systemctl status nginx ps aux | grep nginx

```

If failed: run nginx -t and check errors.
Fix config, then: systemctl restart nginx.

Is NGINX listening on the expected ports?

```bash ss -tulpn | grep nginx

```

You should see entries like:

:80 for HTTP
:443 for HTTPS

DNS points to the correct server?

From your local machine:

```bash dig yourdomain.com +short

```

Confirm IP matches your server.

Firewall or cloud security group blocking?

```bash sudo ufw status

```

Make sure NGINX is allowed:

```bash ufw allow 'Nginx Full'

```

Config syntax validation:

```bash nginx -t

```

Fix any errors reported (paths, typos, invalid directives).

Check error logs:

```bash tail -n 50 /var/log/nginx/error.log

```

Look for:

bind() to 0.0.0.0:80 failed (port in use)
no such file or directory for SSL keys or web roots
permission denied

Fix the underlying cause, then reload:

bash

systemctl reload nginx

2.2. When You See 502 / 504 Errors (Bad Gateway / Gateway Timeout)#

Symptom: Site returns 502 or 504.

Root cause is usually backend/app, not NGINX itself.

Checklist:

Check upstream block in NGINX config:

Example:

``` upstream app_backend { server 127.0.0.1:3000; }

```

Verify app is running on that host/port.

Test app directly:

On server:

```bash curl -v <http://127.0.0.1:3000/>

```

If this fails, the app is broken or not listening.
Fix app or restart its process (PHP-FPM, Node/PM2, etc).

Check NGINX error log:

```bash grep -i upstream /var/log/nginx/error.log | tail -n 50

```

Common messages:

upstream timed out → app is slow
connect() failed (111: Connection refused) → app not listening
no live upstreams → all upstreams down

Check timeouts in NGINX:

Ensure reasonable settings:

``` proxy_connect_timeout 5s; proxy_read_timeout 30s; proxy_send_timeout 30s;

```

Check app capacity:
- PHP-FPM: pm.max_children too low
- Node: too few workers

Scale app side if necessary.

2.3. When Site Is Very Slow#

Symptom: Pages load but slowly.

Checklist:

Measure latency:

```bash curl -w "Total: %{time_total}s\\n" -o /dev/null -s <https://yourdomain.com/>

```

Check server load:

```bash htop free -h iostat -x 1

```

Look at CPU, memory, IO.

Access log latency:

If log_format includes $request_time, run:

```bash awk '{print $(NF-3)}' /var/log/nginx/access.log | sort -nr | head

```

Or to see slow requests:

```bash awk '$NF > 1 {print $0}' /var/log/nginx/access.log | head

```

Check upstream latency:

If logging upstream times, look for high urt= values.

Check rate limiting:

If you configured limit_req, see if you are hitting it:

```bash grep "limiting requests" /var/log/nginx/error.log

```

Check network:

```bash ss -s iftop

```

2.4. When Error Rate Spikes (Many 4xx/5xx Codes)#

Summarize status codes:

```bash awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

```

If many 404:
- Misconfigured routes
- Missing files
If many 403:
- Permission issues on files, directories
- deny all in config
If many 429:
- Being rate limited by limit_req directives
If many 5xx:
- Check upstream, timeouts, memory, CPU

2.5. When You Suspect an Attack or Abuse#

Top IPs:

```bash awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

```

Block abusive IP (example):

```bash ufw deny from 1.2.3.4

```

Combine with rate limiting:

Use limit_req and limit_conn in the server block to absorb bursts.

2.6. Quick Incident Response Flow#

Check NGINX status and config:

```bash systemctl status nginx nginx -t

```

Check logs:

```bash tail -n 50 /var/log/nginx/error.log tail -n 50 /var/log/nginx/access.log

```

Check upstreams:

```bash curl -v <http://127.0.0.1>:PORT/

```

Check system health:

```bash htop free -h df -h ss -s

```

Fix the root cause, then:

```bash systemctl reload nginx

```

This script is like a lab bootstrap:

Installs NGINX
Applies tuned nginx.conf
Enables stub_status
Installs Node Exporter
Installs NGINX Prometheus Exporter
Installs Prometheus
Installs Grafana

This is ideal for a single server where you want everything in one place.

3.1. Save Script#

File name:

text

nginx_full_stack_install.sh

Contents:

bash

#!/usr/bin/env bash
# NGINX Full Stack Installer
# - NGINX install
# - High-traffic tuning
# - stub_status
# - Prometheus Node Exporter
# - NGINX Prometheus Exporter
# - Prometheus server
# - Grafana

set -euo pipefail

if [[ "$EUID" -ne 0 ]]; then
  echo "Error: run as root."
  exit 1
fi

if ! command -v apt-get >/dev/null 2>&1; then
  echo "Error: Debian/Ubuntu only."
  exit 1
fi

echo "=== Updating system and installing base packages ==="
apt-get update -y
apt-get install -y nginx ufw curl wget apt-transport-https software-properties-common

echo "=== Configuring firewall for NGINX, Prometheus, Grafana (optional, adjust as needed) ==="
if command -v ufw >/dev/null 2>&1; then
  ufw allow 'Nginx Full' || true
  ufw allow 3000/tcp || true    # Grafana
  ufw allow 9090/tcp || true    # Prometheus
fi

echo "=== NGINX basic tuning and stub_status site ==="

NGINX_ETC="/etc/nginx"
SITES_AVAILABLE="${NGINX_ETC}/sites-available"
SITES_ENABLED="${NGINX_ETC}/sites-enabled"
SNIPPETS_DIR="${NGINX_ETC}/snippets"

mkdir -p "$SITES_AVAILABLE" "$SITES_ENABLED" "$SNIPPETS_DIR"

# Backup nginx.conf
if [[ -f "${NGINX_ETC}/nginx.conf" ]]; then
  cp "${NGINX_ETC}/nginx.conf" "${NGINX_ETC}/nginx.conf.bak.$(date +%F-%H%M%S)"
fi

cat > "${NGINX_ETC}/nginx.conf" << 'EOF'
user www-data;
worker_processes auto;
worker_rlimit_nofile 200000;

error_log /var/log/nginx/error.log warn;
pid /run/nginx.pid;

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time uct=$upstream_connect_time '
                    'urt=$upstream_response_time uht=$upstream_header_time';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 15;
    keepalive_requests 10000;

    client_body_timeout 15s;
    client_header_timeout 15s;
    send_timeout 15s;
    reset_timedout_connection on;

    server_tokens off;

    limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

    open_file_cache          max=50000 inactive=20s;
    open_file_cache_valid    30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   on;

    include /etc/nginx/snippets/gzip.conf;
    include /etc/nginx/snippets/security.conf;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}
EOF

cat > "${SNIPPETS_DIR}/gzip.conf" << 'EOF'
gzip on;
gzip_disable "msie6";

gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_min_length 1024;
gzip_http_version 1.1;

gzip_types
    text/plain
    text/css
    text/javascript
    application/javascript
    application/json
    application/xml
    application/rss+xml
    application/vnd.ms-fontobject
    application/x-font-ttf
    font/opentype
    image/svg+xml;
EOF

cat > "${SNIPPETS_DIR}/security.conf" << 'EOF'
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;

location ~ /\\.(?!well-known) {
    deny all;
}

if ($request_method !~ ^(GET|HEAD|POST)$) {
    return 405;
}
EOF

# stub_status site
cat > "${SITES_AVAILABLE}/nginx_status.conf" << 'EOF'
server {
    listen 8080;
    server_name _;

    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}
EOF

ln -sf "${SITES_AVAILABLE}/nginx_status.conf" "${SITES_ENABLED}/nginx_status.conf"

echo "Testing nginx configuration..."
nginx -t
systemctl enable --now nginx

echo "=== Installing Prometheus Node Exporter ==="
apt-get install -y prometheus-node-exporter

systemctl enable --now prometheus-node-exporter

echo "=== Installing NGINX Prometheus Exporter ==="

if ! command -v nginx-exporter >/dev/null 2>&1; then
  wget <https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.11.0/nginx-prometheus-exporter_0.11.0_linux_amd64> \\
    -O /usr/local/bin/nginx-exporter
  chmod +x /usr/local/bin/nginx-exporter
fi

cat >/etc/systemd/system/nginx-exporter.service << 'EOF'
[Unit]
Description=NGINX Prometheus Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/nginx-exporter \\
  -nginx.scrape-uri <http://localhost:8080/nginx_status>
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now nginx-exporter

echo "=== Installing Prometheus ==="
apt-get install -y prometheus

PROM_CONF="/etc/prometheus/prometheus.yml"

cp "$PROM_CONF" "${PROM_CONF}.bak.$(date +%F-%H%M%S)" || true

cat >"$PROM_CONF" << 'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'nginx'
    static_configs:
      - targets: ['localhost:9113']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
EOF

systemctl restart prometheus
systemctl enable prometheus

echo "=== Installing Grafana ==="

if [[ ! -f /etc/apt/keyrings/grafana.gpg ]]; then
  mkdir -p /etc/apt/keyrings
  wget -q -O - <https://packages.grafana.com/gpg.key> | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
  echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] <https://packages.grafana.com/oss/deb> stable main" \\
    >/etc/apt/sources.list.d/grafana.list
  apt-get update -y
fi

apt-get install -y grafana
systemctl enable --now grafana-server

echo "=== Summary ==="
echo "NGINX running:         systemctl status nginx"
echo "stub_status endpoint:  <http://localhost:8080/nginx_status>"
echo "Node Exporter:         <http://localhost:9100/metrics>"
echo "NGINX Exporter:        <http://localhost:9113/metrics>"
echo "Prometheus:            <http://localhost:9090>"
echo "Grafana:               <http://localhost:3000>"
echo "Default Grafana login: admin / admin (change immediately)."

3.2. Run Script#

bash

chmod +x nginx_full_stack_install.sh
sudo ./nginx_full_stack_install.sh

After that:

NGINX is installed and tuned
stub_status is available on port 8080
Node exporter on 9100
NGINX exporter on 9113
Prometheus on 9090
Grafana on 3000

You just need to:

Add Prometheus as a data source in Grafana
Import ready-made dashboards for Node and NGINX