A server's load average reads 12 on a box with four cores. The usual reaction is to restart the service or the whole machine, and that throws away the evidence you needed. This guide explains how a Linux server is put together, from boot to PID 1 to your service's processes and the cgroups that contain them. It then works through one service, myapp, on a throwaway VM, breaking it the ways real services break. The aim is that the numbers mean something when it's your server at 3 a.m.

The interview prep guide (opens in a new tab) covers the same ground as questions and answers, including a full "server is slow" procedure. This article is the explainer behind it: how the parts fit, and why the commands show what they show.

Linux in one paragraph

For an operator, a Linux server is a short chain of responsibility. Firmware hands off to a bootloader, and the bootloader starts the kernel. The kernel starts one process, PID 1, which on most servers is systemd. systemd then starts everything else as units: your web server, sshd, the log collector.

Every running program is a process with a state, a parent and a memory footprint. It also has a place in exactly one cgroup, which the kernel docs describe as "a mechanism to organize processes hierarchically and distribute system resources along the hierarchy". Boot, processes, systemd, filesystems, memory and CPU all follow from that chain. The job isn't knowing every command. It's having a method when a box misbehaves, so you rule things out in order instead of rebooting and hoping.

How it works

Boot: from firmware to PID 1

The boot chain matters day to day because each step has its own failure mode and its own logs.

  • Firmware. UEFI or BIOS initialises the hardware and hands off to a bootloader, usually GRUB 2.
  • Bootloader. GRUB loads the kernel and an initramfs, a small temporary root filesystem.
  • Kernel. The kernel uses the initramfs to find and mount the real root filesystem.
  • PID 1. The kernel starts PID 1, normally systemd. systemd brings up default.target, which is usually an alias for multi-user.target on a server or graphical.target on a desktop. It starts units in dependency order.
Mermaid
flowchart TD
    FW["Firmware (UEFI or BIOS)"] --> GRUB["Bootloader (GRUB 2)"]
    GRUB --> K["Kernel + initramfs"]
    K -->|mounts real root filesystem| PID1["PID 1: systemd"]
    PID1 --> T["default.target (alias of multi-user.target)"]
    T --> U1["sshd.service"]
    T --> U2["myapp.service"]
    T --> U3["systemd-journald.service"]
    PID1 --> S["run-*.scope (from systemd-run --scope)"]
    subgraph CG["cgroup of myapp.service"]
        P1["myapp main process"] --> C1["child process"]
        P1 --> C2["child process"]
    end
    U2 --> P1
    P1 -.stdout and stderr.-> U3
    PID1 -.adopts and reaps orphans.-> C2

Each service unit runs its processes in its own cgroup, and the cgroups form a tree. A new process starts in its parent's cgroup, so a child forked by myapp lands in myapp's cgroup without anyone asking. That is how systemd knows which processes belong to which service, and it is where memory limits and per-service pressure figures apply. Limits set nearer the root of the tree can't be overridden further down. A .scope unit, which systemd-run --scope creates, wraps a command you started by hand in a cgroup of its own. The worked example uses one.

journald collects what service processes write to stdout and stderr. If a parent dies, PID 1 adopts its children. That adoption is the mechanism behind zombie cleanup, covered next.

You can read the chain back on a running box:

Bash
systemctl get-default
systemctl list-dependencies multi-user.target
systemd-analyze blame | head -n 10
journalctl -b -p err
  • get-default names the target the box boots into.
  • list-dependencies shows the units that target pulls in.
  • systemd-analyze blame lists the units that took longest to start.
  • journalctl -b -p err shows messages from the current boot at error priority and above. It is the first place to look after a slow or failed boot.

Processes and their states

The STAT column of ps shows a process's state. These are the five you will meet most often. ps documents a few others, so check man ps on your system for the full list.

StateMeaningWhat it tells you
RRunning or runnableUsing CPU, or waiting for a CPU
SInterruptible sleepWaiting for something, such as a socket; normal
DUninterruptible sleepBlocked in the kernel, usually on I/O
ZZombieExited, but the parent hasn't collected its status
TStoppedPaused by a signal or a debugger

To see states alongside the parent-child tree, run:

Bash
ps -eo pid,ppid,stat,cmd --forest

A zombie has already exited, so you can't kill it. It doesn't appear in its cgroup's cgroup.procs either, and it can't be moved to another cgroup. Its parent never called wait() to collect the exit status, and the fix is to fix or restart the parent. Once the parent exits, PID 1 adopts the zombie and reaps it. You can make one in a few seconds. The shell below starts a short sleep in the background, then replaces itself with a long sleep that never waits for its child:

Bash
sh -c 'sleep 1 & exec sleep 30' &
sleep 2
ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'

You should see a sleep entry marked <defunct>, whose PPID is the long-running sleep. When that parent exits after 30 seconds, the zombie is adopted and reaped, and running the ps line again shows nothing.

This matters in containers. If your container's entrypoint spawns children and isn't a real init, nothing reaps them. A small init such as tini does that job.

Signals and shutdown

Stopping a process means sending it a signal.

  • SIGTERM asks the process to stop. A process can catch it, so a well-behaved server finishes in-flight requests and exits.
  • SIGKILL can't be caught. The kernel ends the process immediately.

This loop catches SIGTERM, cleans up and exits with status 0:

Bash
bash -c 'trap "echo got SIGTERM, cleaning up; exit 0" TERM; while true; do sleep 1; done' &
pid=$!
sleep 1
kill -TERM "$pid"
wait "$pid"; echo "exit status: $?"

It prints the trap message and exit status: 0. The trap runs once the current sleep 1 returns, so the reply can take up to a second. Repeat it with kill -KILL "$pid". The trap never runs, and the exit status is 137, which is 128 plus signal 9. That is the same 137 you will see on a container killed for memory.

Kubernetes follows the same sequence for a pod: SIGTERM, a wait of terminationGracePeriodSeconds (30 by default), then SIGKILL. A classic way to drop requests on every deploy is a shell script running as PID 1 that doesn't forward signals. The script receives SIGTERM, the application never does, and the application dies on SIGKILL mid-request.

systemd: units, restarts and logs

systemd manages units, the targets that group them, restart policy, logging through journald, and resource control through cgroups. For an operator, the parts that matter most are how a unit is loaded, how it restarts and where its logs go. Here is the unit the worked example uses:

INI
# /etc/systemd/system/myapp.service
[Unit]
Description=Example application
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=exec
User=myapp
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

What each part does:

  • Type=exec makes systemctl start fail if the binary is missing or the User= doesn't exist. The default when only ExecStart= is set is Type=simple, which reports success in both cases. The systemd docs recommend exec for long-running services.
  • Restart=on-failure with RestartSec brings the service back after a crash.
  • StartLimitIntervalSec and StartLimitBurst stop it from restarting forever. A service that crashes on start would otherwise hammer its database every few seconds.
  • User=myapp drops root. NoNewPrivileges=true means the service and its children can never gain privileges through execve(), for example via setuid binaries or file capabilities.
  • ProtectSystem=strict makes the whole filesystem read-only for the service, apart from /dev, /proc and /sys. Any path it writes to has to be allowed with ReadWritePaths=. The worked example hits exactly this.

Two loading rules cause most "my edit didn't work" moments. First, systemd reads drop-in files from myapp.service.d/*.conf and merges them after the main unit, in alphanumeric order, with /etc/ beating /run/ beating /usr/lib/. Packaged units live under /usr/lib/systemd/system/. Put your changes in /etc rather than editing the packaged file. Second, an unknown or misspelled option only produces a warning in the log. The unit still loads, and the setting you thought you applied silently isn't there.

After editing a unit, reload systemd's view of it, then restart the service and read its logs:

Bash
sudo systemctl daemon-reload
sudo systemctl restart myapp
systemctl status myapp --no-pager
journalctl -u myapp --since "10 min ago"

systemctl status shows the active state, the main PID, the cgroup with its processes, and the last few log lines. Two more commands are worth knowing:

Bash
systemctl cat myapp
systemctl list-units --type=service --state=failed

systemctl cat prints the unit as systemd loaded it, including drop-ins. When an edit "didn't take effect", compare its output with the file you edited, then check whether you ran daemon-reload. list-units --state=failed is the quickest way to see everything broken on a box.

Load average and pressure

Load average is the most misread number on a Linux box. On Linux it is the damped average number of tasks that are either runnable or in uninterruptible sleep (D state), over 1, 5 and 15 minutes.

Bash
cat /proc/loadavg

The first three fields are the averages that uptime also prints. Because they count D-state tasks, high load with idle CPUs usually means tasks are blocked on I/O, not that you need more cores.

Pressure Stall Information (PSI) answers the question load can't: which resource are tasks waiting on? It lives in /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io. Each file has this format, shown here with the zeros from the kernel documentation:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
  • some is the share of time at least one task was stalled on that resource.
  • full is the share of time all non-idle tasks were stalled at once, which is the "nothing is getting done" case.
  • avg10, avg60 and avg300 are averages over 10, 60 and 300 seconds. total is the cumulative stall time in microseconds.

The CPU file's full line is undefined at the system level. Since kernel 5.13 it is reported as zero for compatibility, so don't read it as "no CPU stalls". If the files are missing, the running kernel doesn't have PSI enabled.

PSI also exists per cgroup, as cpu.pressure, memory.pressure and io.pressure in each cgroup's directory. That turns "the box is under memory pressure" into "this service is under memory pressure".

Memory limits and the OOM killer

When memory runs out, the kernel's OOM killer picks a victim and kills it.

  • It scores processes mainly by memory use, adjusted by oom_score_adj, and kills the one with the highest score.
  • In a cgroup with a memory limit, the kill happens inside that cgroup even if the node has plenty of free memory. That surprises people who check free and see gigabytes available.
  • Confirm a kill with journalctl -k.
  • In Kubernetes, the container's last state shows OOMKilled with exit code 137.

Every process exposes its current score and adjustment. For the shell you're typing in:

Bash
cat /proc/$$/oom_score /proc/$$/oom_score_adj

Replace $$ with any PID to check a service's process. The PID is in systemctl status.

An inode holds a file's metadata. A file name is just a link to an inode.

  • A hard link is another name for the same inode.
  • A symlink is a separate file that holds a path.

This takes ten seconds to see:

Bash
cd "$(mktemp -d)"
echo hello > a
ln a b
ln -s a c
ls -li
rm a
cat b
cat c

In ls -li, a and b share an inode number and show a link count of 2. c has its own inode and points at a. After rm a, cat b still prints hello, because the inode still has a link. cat c fails with "No such file or directory", because the path it holds is gone.

The data is freed only when the last link is gone and no process has the file open. That is why deleting a large log file doesn't always free any space.

A worked example

The way to learn this is to break a throwaway VM on purpose and watch the commands report it. Don't run any of this on a server you care about. The VM needs stress-ng, which usually isn't installed by default, and the sysstat package, which provides iostat and pidstat. Every step below uses commands from the sections above, so you can check what you see against the mechanism.

Step 1: take a one-line picture

Before touching anything, define what "slow" means: latency, errors or throughput, and since when. Then take a quick picture of the whole box. This is the sequence I run:

Bash
nproc
uptime
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
vmstat 1 5
free -h
df -h && df -i

How to read each one:

  • Compare the uptime load averages with nproc.
  • PSI shows which resource tasks are stalled on. On an idle VM the some averages should sit near zero.
  • In vmstat, r is the run queue, si/so show swapping, wa is time spent waiting on I/O, and st is CPU time stolen by the hypervisor.
  • df -i catches a filesystem that is out of inodes while df -h still shows free space.

Keep this baseline. Every later step is read against it.

Step 2: saturate the CPU

Start a CPU load with one worker per core:

Bash
stress-ng --cpu "$(nproc)" --timeout 60s

While it runs, repeat uptime, vmstat 1 5 and cat /proc/pressure/cpu in a second terminal. The run queue r climbs past the core count, load follows it, and the CPU some averages rise, avg10 first. The CPU full line stays at zero, for the reason given above. To see which process is responsible, sample per-process CPU and disk use:

Bash
pidstat -u -d 1 5

Step 3: look for I/O-blocked tasks

If load is high but CPU is idle, look for tasks in uninterruptible sleep, along with the kernel function they're waiting in:

Bash
ps -eo pid,stat,wchan:20,cmd | awk '$2 ~ /^D/'
iostat -xz 1 3

iostat reports %util and the average wait time for each device. Current sysstat versions split the wait into r_await and w_await, and older versions show a single await column. One device near full utilisation with long waits, plus a list of D-state processes and a rising /proc/pressure/io, is the I/O case. Load average alone can't tell it apart from a CPU problem.

Step 4: trigger an OOM kill inside a cgroup

Run a memory hog inside a transient scope limited to 200M, and ask it to allocate 400M:

Bash
sudo systemd-run --scope -p MemoryMax=200M stress-ng --vm 1 --vm-bytes 400M --timeout 30s
journalctl -k --since "5 min ago" | grep -i oom

The kernel log should record an OOM kill, even though free -h on the host shows memory to spare. That is the cgroup behaviour described earlier: the limit belongs to the scope's cgroup, not the machine. MemoryMax= is the cgroup v2 setting. If the VM has swap, the scope may swap instead of being killed. In that case, run it on a VM without swap, or also limit swap with -p MemorySwapMax=0. On a real incident, widen the search:

Bash
journalctl -k --since "1 hour ago" | grep -i -E 'oom|killed process'

Step 5: fill a disk that du can't explain

This step reproduces the "df says full, du finds nothing" case. In one terminal, create a 200 MB file and keep it open:

Bash
python3 -c "
f = open('/tmp/big', 'wb')
f.write(b'0' * 200 * 1024 * 1024)
f.flush()
input('holding /tmp/big open; press Enter to release ')
"

In a second terminal, delete it, check the space and find the holder:

Bash
rm /tmp/big
df -h /tmp
sudo lsof +L1 | grep /tmp/big

df still counts the 200 MB because the inode is alive while the Python process holds it open. lsof +L1 lists open files with no links left, and shows the process holding the deleted file. Press Enter in the first terminal and the space comes back.

On a server, the culprit is usually a daemon still writing to a log you rotated by deleting it. Restarting that process, or having it reopen its logs, frees the space.

Step 6: deploy a hardened unit and watch it fail

The last step uses the myapp unit and breaks it the way hardening usually breaks things. First, create a system user, a state directory it owns, and a tiny myapp that writes a heartbeat file every five seconds:

Bash
sudo useradd --system --user-group --no-create-home --shell /usr/sbin/nologin myapp
sudo install -d -o myapp -g myapp /var/lib/myapp
sudo tee /usr/local/bin/myapp >/dev/null <<'EOF'
#!/bin/sh
while true; do
  date > /var/lib/myapp/heartbeat || exit 1
  sleep 5
done
EOF
sudo chmod 755 /usr/local/bin/myapp

Save the unit from the systemd section as /etc/systemd/system/myapp.service. Check it, load it and start it:

Bash
sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl start myapp
sleep 30
systemctl status myapp --no-pager
systemctl show myapp -p NRestarts
journalctl -u myapp --since "5 min ago" --no-pager

systemctl start succeeds: the binary exists and the user exists, so Type=exec has nothing to object to. The failure comes from inside the service. ProtectSystem=strict makes /var/lib/myapp read-only for the service, even though myapp owns it. The write fails, the script exits 1, and Restart=on-failure brings it back every five seconds. After five starts inside 60 seconds, StartLimitBurst stops the loop. systemctl status should show the unit as failed, and the journal should show the write error from the script. Without the start limits, this loop would run forever.

The fix is to allow that one path. Rather than editing the unit, add a drop-in, so the change sits in its own file:

Bash
sudo mkdir -p /etc/systemd/system/myapp.service.d
printf '[Service]\nReadWritePaths=/var/lib/myapp\n' | sudo tee /etc/systemd/system/myapp.service.d/writable.conf

Then reload, clear the failed state, and start again:

Bash
sudo systemctl daemon-reload
sudo systemctl reset-failed myapp
sudo systemctl start myapp
sleep 6
cat /var/lib/myapp/heartbeat
systemctl cat myapp

reset-failed matters here. A unit that hit its start limit stays failed until you clear it, so a plain start can be refused. The heartbeat file should now hold a timestamp. systemctl cat shows the main unit followed by writable.conf, which is how you confirm the drop-in was picked up.

While it runs, check which cgroup the service lives in:

Bash
cat /proc/"$(systemctl show -p MainPID --value myapp)"/cgroup

On a cgroup v2 host this prints a single line starting 0::, followed by the cgroup path, which names myapp.service. That directory is where the per-service memory.pressure file lives. Clean up with sudo systemctl stop myapp, then remove the unit file, the myapp.service.d directory, the script, /var/lib/myapp and the user.

Going deeper when the picture isn't enough

When one process is the problem but the one-line picture doesn't say why, three tools go further:

  • strace -f -tt -p <pid> shows the system call the process is blocked in. Replace <pid> with the PID from pidstat or ps.
  • perf top shows where CPU time goes.
  • tcpdump -i any -nn port 53 captures DNS traffic, for intermittent resolution problems.

For a DNS problem, check /etc/resolv.conf first. On systemd-resolved hosts, also run resolvectl status. Then query the server directly with dig. That separates "the resolver is wrong" from "the DNS server is wrong". For sockets, these two commands cover most questions:

Bash
ss -s
sudo ss -tlnp

ss -s summarises connection counts by state. ss -tlnp lists listening TCP sockets with the process that owns each one. It answers "is anything actually listening on that port?"

Trade-offs and pitfalls

SymptomTempting conclusionWhat it often isCheck with
Load well above core countNeed more CPUsTasks blocked on I/O (D state)vmstat, PSI, ps ... awk '$2 ~ /^D/'
CPU full in PSI always 0No CPU stallsUndefined at system level, reported as zeroCPU some line
Process killed, host has free memoryKernel bugcgroup memory limit hitjournalctl -k, exit code 137
df full, du finds nothingCorrupt filesystemDeleted file held open, or inodes exhaustedlsof +L1, df -i
Unit edit ignoredsystemd is brokenNo daemon-reload, or a misspelled option that only warnedsystemctl cat, the journal
start succeeded, service never ranIt's runningType=simple with a missing binary or userType=exec, systemctl status
Unit won't start after fixing itFix didn't workStart limit still trippedsystemctl reset-failed
Hardened service fails on writePermissions bugProtectSystem=strict made the path read-onlyjournalctl -u, ReadWritePaths=
Dropped requests on every deployLoad balancer issuePID 1 shell script not forwarding SIGTERMEntrypoint, signal handling
Dependency overloaded after a crashDependency too smallUnbounded restart loopStartLimitIntervalSec, StartLimitBurst

The broader pitfall is jumping to a fix. A reboot clears the zombie, the held-open file and the stuck I/O, and leaves you nothing to learn from. The order I follow:

  1. Define "slow".
  2. Take the one-line picture.
  3. Go deeper only on the resource it points at.
  4. Make the fix.
  5. Add an alert so you hear about it before your users do next time.

Each tool has its own limits:

  • sysstat. It isn't on every image, so iostat and pidstat may be missing exactly when you need them. Bake the package in ahead of time.
  • cgroup v1 and v2. The memory lab assumes cgroup v2. Mixing v1 and v2 controllers on one host is possible, but the kernel docs strongly discourage moving controllers between them in production. Test on the distribution and cgroup setup you actually run.
  • Hardening options. ProtectSystem=strict and NoNewPrivileges=true trade convenience for safety. Every path the application writes to has to be listed, and a missed one only shows up at runtime, as in Step 6.
  • Unit typos. Because an unknown option only warns, a misspelled hardening directive leaves the service running without it. Read systemctl cat and the journal after every change.
  • Entrypoint scripts. /bin/sh is not necessarily Bash. It is dash on Debian and Ubuntu, and BusyBox in Alpine. A script that relies on Bash features can behave differently as a container entrypoint. The shell scripting comparison (opens in a new tab) covers the differences.

Further reading

Next, run the six steps above on a throwaway VM running your production distribution. Keep the output from each step, and compare it with what your monitoring showed during the same run. Wherever the dashboard missed what the command showed, that's the alert to add first.