Linux Performance Troubleshooting: CPU, Memory, Disk and Network

At a glance#

• Purpose: Diagnose a slow or unresponsive Linux server and identify which resource is the bottleneck.
• Applies to: Any Linux distribution.
• Risk: None — all commands are read-only.
• Time: 15–60 minutes.

Overview#

"The server is slow" is one symptom with four possible causes: CPU, memory, disk I/O or network. The mistake is guessing. This article gives a fixed order that identifies the constrained resource in a few minutes, then goes deeper on each.

Install the standard toolset first — most of these are not present on a minimal build:

# RHEL family
sudo dnf install -y sysstat procps-ng iotop htop

# Debian / Ubuntu
sudo apt install -y sysstat procps iotop htop

The first 60 seconds#

Run these in order. Together they identify the bottleneck in almost every case.

# 1. load average and uptime
uptime

# 2. CPU, memory and I/O wait over 5 samples
vmstat 1 5

# 3. per-CPU breakdown
mpstat -P ALL 1 3

# 4. top processes
top -b -n1 | head -20

# 5. memory
free -h

# 6. disk I/O
iostat -xz 1 3

# 7. disk space - a full disk causes symptoms that look like anything
df -h

Reading the load average#

load average: 4.52, 3.81, 2.44
              1min  5min  15min

Compare against core count:

nproc

A load equal to the core count means fully busy. Load well above core count means processes are queuing. Rising across the three figures means the problem is getting worse; falling means it is recovering.

Note: On Linux the load average counts processes waiting on disk I/O as well as CPU. A load of 20 with idle CPUs is an I/O problem, not a CPU problem. This is why vmstat matters more than uptime.

Reading vmstat#

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1245680 145280 2458120  0    0    12    45  520 1024 15  3 81  1  0

This single output usually names the bottleneck: high r with low wa is CPU; high b and wa is disk; non-zero si/so is memory pressure.

CPU#

# top CPU consumers
ps aux --sort=-%cpu | head -15

# per-core utilisation
mpstat -P ALL 2 5

# interactive
htop

# what a specific process is doing
sudo strace -c -p <PID>       # syscall summary
sudo perf top -p <PID>        # where CPU time goes, if perf is available

Distinguishing the cause from the us/sy split:

• High us — application code. Profile the application.
• High sy — kernel work; often excessive syscalls, context switching or network interrupts.
• High si/hi — software or hardware interrupts, typically network under load.
• High st (steal) — the hypervisor is not giving this VM the CPU it asked for. The fix is at the virtualisation layer, not in the guest.

Note: Steal time is the giveaway for an over-committed host. If st is consistently above a few percent, check host contention in vCenter rather than tuning the guest.

Memory#

free -h

              total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.1Gi       1.2Gi       0.3Gi       4.4Gi       5.0Gi
Swap:          2.0Gi       0.0Gi       2.0Gi

Warning: Read the available column, not free. Linux deliberately uses spare memory for cache and gives it back on demand. Low free with healthy available is normal and correct — it does not mean the server is out of memory.

Memory is a genuine problem when available is low and swap is active:

# top memory consumers
ps aux --sort=-%mem | head -15

# swap usage per process
for f in /proc/*/status; do
  awk '/^Name|^VmSwap/{printf "%s ", $2} END {print ""}' "$f"
done | sort -k2 -hr | head -10

# has the OOM killer been active?
sudo dmesg -T | grep -i -E "out of memory|killed process"
sudo journalctl -k | grep -i "oom"

An OOM kill in dmesg is definitive — the kernel ran out and terminated something. That is a capacity or leak problem, not a tuning one.

Clear cache only for diagnosis, never as a routine fix:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Disk I/O#

iostat -xz 1 5

Find the process responsible:

sudo iotop -oPa

-o shows only processes actually doing I/O, -a accumulates totals.

Always check free space and inodes at the same time — both produce I/O-shaped symptoms:

df -h
df -i

Warning: A filesystem at 100% capacity, or out of inodes, causes application errors that rarely mention disk space. Databases fail writes, web servers return 500s, and logs stop. Check df -h and df -i early in any investigation.

Find what is consuming space:

sudo du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20
sudo du -h --max-depth=1 /var 2>/dev/null | sort -hr | head -20

# deleted files still held open - space not released
sudo lsof +L1

The lsof +L1 case is worth knowing: a deleted log file held open by a running process keeps consuming space that du cannot see, so df and du disagree. See Understanding lsof.

Network#

# interface errors and drops
ip -s link

# listening sockets and connection counts
ss -tulpn
ss -s

# connections by state
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

# top talkers by connection count
ss -tn | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10

# live bandwidth, if installed
sudo iftop -i eth0

Watch for a large TIME-WAIT count under load, and for non-zero error or drop counters in ip -s link, which indicate a physical or driver problem rather than a capacity one.

Latency and path:

ping -c 10 8.8.8.8
mtr -rw -c 20 8.8.8.8
traceroute 8.8.8.8

Historical data with sar#

vmstat shows now. sar shows what happened at 3am, which is usually when the problem occurred.

# enable collection
sudo systemctl enable --now sysstat

# today's CPU
sar -u

# memory
sar -r

# disk
sar -d -p

# network
sar -n DEV

# a specific past day (the 11th)
sar -u -f /var/log/sa/sa11

Note: Enable sysstat on every server as part of the build. Without historical data, investigating an incident that has already passed is guesswork.

Verification#

After a fix, confirm the metric that was wrong has returned to normal:

uptime
vmstat 1 5
free -h
iostat -xz 1 3
df -h

Compare against the baseline in sar for the same time on a previous day, rather than against an assumption of what "normal" looks like.

Troubleshooting#

Related#

• Understanding lsof: Linux Open Files and Network Connections
• Log Analysis with journalctl and rsyslog
• Linux Storage & Filesystem
• Managing Services with systemd