Linux: troubleshoot a slow server in 5 minutes

Identifying the root cause of a sluggish Linux server is a critical skill for any infrastructure professional. Performance bottlenecks can manifest in…

Identifying the root cause of a sluggish Linux server is a critical skill for any infrastructure professional. Performance bottlenecks can manifest in various ways, from unresponsive applications to slow SSH logins, and pinpointing the culprit quickly requires a systematic approach. This guide outlines a practical, five-minute methodology using standard Linux utilities to diagnose common performance issues across CPU, disk I/O, memory, and network subsystems.

The goal is rapid triage: understand the primary resource under contention, gather initial metrics, and inform the next steps for deeper investigation or remediation. We'll focus on tools available in most Linux distributions (e.g., RHEL 7/8/9, Ubuntu 18.04/20.04/22.04, Debian 10/11).

1. Initial Assessment: The "Top" View (CPU & Memory Overview)

The top command is often the first tool reached for due to its real-time, interactive nature, providing a high-level overview of system resource usage. It's excellent for quickly identifying processes consuming excessive CPU or memory.

top -b -n 1 | head -n 20

This command runs top in batch mode (-b) for a single iteration (-n 1) and pipes the output to head -n 20 to capture the initial snapshot without interactive refresh. Key areas to examine:

CPU Usage (first few lines):
- us (user space): High values indicate user applications consuming CPU.
- sy (kernel space): High values can point to kernel processes, drivers, or system calls bottlenecking.
- id (idle): Low idle values (e.g., <10%) suggest CPU saturation.
- wa (I/O wait): High wa is a strong indicator of disk I/O bottlenecks.
- st (steal time): Relevant in virtualized environments. High st means the hypervisor is oversubscribed or throttling your VM's CPU.
Memory Usage (Mem and Swap lines):
- total, free, used, buff/cache: Check available free memory. A low free value is normal if buff/cache is high (Linux uses available RAM for disk caching).
- Swap: If swap usage is high and actively increasing, the system is likely memory-starved.
Process List: Sort by CPU (P key in interactive mode) or Memory (M key). Identify the top 5-10 processes and their associated users. Are these expected applications or rogue processes?

2. Disk I/O Analysis with iostat

If top shows high I/O wait (wa), or applications are slow during file operations, iostat is your next stop. It provides detailed disk utilization statistics.

iostat -xz 1 5

This command runs iostat with extended statistics (-x), displays statistics for each device (-z), and reports every second for 5 iterations. Focus on these metrics:

%util: The percentage of time the device was busy servicing transfer requests. Values consistently near 100% indicate a bottleneck.
await: The average time (in milliseconds) for I/O requests issued to the device to be served. This includes time spent in the queue and time spent servicing them. High await (e.g., >20-30ms for local storage, >50-100ms for network storage) is a strong sign of I/O contention.
svctm (service time): The average service time (in milliseconds) for I/O requests that were issued to the device. Compare await to svctm; a large difference implies significant queuing.
r/s, w/s: Reads per second, writes per second.
rkB/s, wkB/s: Kilobytes read per second, written per second.

Identify the disk device(s) with high %util and await. For instance, if /dev/sda shows 95% utilization and 150ms await, that device is your bottleneck.

3. Memory & Swapping with vmstat

While top gives an overview, vmstat provides more granular details on memory, paging, and CPU activity, particularly useful for diagnosing memory pressure and excessive swapping.

vmstat 1 5

This command reports statistics every second for 5 iterations. Pay attention to:

r (runqueue length): Number of processes waiting for CPU. High values (e.g., > num_CPUs) suggest CPU contention.
b (blocked processes): Number of processes waiting for I/O. High values here corroborate disk I/O issues.
swpd (swapped): Amount of virtual memory used. Non-zero or increasing values mean swap is being used.
si (swap in), so (swap out): Kilobytes swapped in/out from disk per second. Sustained non-zero values (e.g., >100-200 kB/s) indicate significant memory pressure and can severely degrade performance. This is often the prime indicator of a memory bottleneck.
free (idle memory): Amount of idle memory.
buff, cache: Memory used for buffers and cache.

If si and so are consistently high, the server is actively swapping, meaning it doesn't have enough physical RAM for its workload. This often leads to extremely slow performance.

4. Network Connectivity and Sockets with ss

Network issues can manifest as slow responses, application timeouts, or failed connections. The ss utility (a modern replacement for netstat) provides detailed socket statistics.

ss -s

This command prints summary statistics for all sockets. Look for:

Total/TCP/UDP/RAW sockets: Check for unusually high numbers of open sockets, which might indicate a denial-of-service attack, connection leaks in an application, or excessive outbound connections.
estab (established): Number of actively connected TCP sockets.
syn-recv (SYN-RECEIVED): Sockets in the SYN_RECV state. A high number could indicate a SYN flood attack.
time-wait (TIME-WAIT): Sockets in the TIME_WAIT state. A very high number can exhaust ephemeral ports, but it's often a normal consequence of high connection rates on busy servers.

For more detailed network interface statistics, use ip -s link show eth0 (replace eth0 with your network interface name) or nmon if installed.

# Check for interface errors or drops
ip -s link show eth0

Look for incrementing rx errors, tx errors, dropped, or overruns. These indicate physical layer or driver issues, or network congestion upstream.

5. Kernel Messages for Deeper Clues

The kernel message buffer often holds critical information about hardware failures, driver issues, low-level errors, or resource exhaustion that might not be immediately apparent from higher-level tools.

dmesg -T | tail -n 50

The -T flag (or --ctime on older kernels) translates timestamps into human-readable format. Examine the last 50 lines for:

Hardware errors: ECC memory errors, disk failures (e.g., disk error, SCSI error), or RAID controller issues.
OOM killer: Messages like Out of memory: Kill process X (Y) score Z or sacrifice child indicate the system ran out of memory and had to terminate processes. This is a definitive sign of severe memory pressure.
Network driver issues: Errors related to your Ethernet card.
Filesystem errors: Read-only mounts due to corruption, or I/O errors.
CPU throttling: Messages indicating CPU being throttled due to thermal limits or power capping (especially on physical servers).

If you see relevant errors, investigate further based on their timestamps and nature. For persistent issues, reviewing the full dmesg output (dmesg -T | less) or log files (/var/log/messages, /var/log/syslog) might be necessary.

Common Pitfalls & Troubleshooting

Assuming CPU saturation: High CPU us is obvious, but don't overlook high wa (disk I/O) or high st (hypervisor contention) as primary CPU bottlenecks.
Misinterpreting free memory: Linux uses available RAM for disk caching. A low "free" memory count in top or vmstat is not necessarily bad if "buff/cache" is high. True memory pressure is indicated by active swapping (high si/so in vmstat) or the OOM killer.
Ignoring load average: The load average (first line of top or uptime) indicates the number of processes either running or waiting for CPU/disk I/O. A load average significantly higher than the number of CPU cores often indicates a bottleneck, but doesn't specify which resource. Use the other tools to pinpoint.
Short-term vs. sustained issues: A momentary spike in resource usage might be normal. Observe commands like iostat and vmstat over several iterations to distinguish transient spikes from sustained bottlenecks.
Lack of historical data: These tools provide real-time snapshots. For recurring or intermittent issues, consider using historical monitoring tools (e.g., sar from the sysstat package, Prometheus/Grafana) to analyze trends.
Application-specific issues: Even with system resources appearing healthy, an application might be slow due to inefficient code, database deadlocks, or misconfiguration. After ruling out system-level bottlenecks, application-specific logging and profiling tools are the next step.