A server's load average reads 12 on a box with four cores. The usual reaction is to restart the service or the whole machine, and that throws away the evidence you needed. This guide explains how a Linux server is put together, from boot to PID 1 to your service's processes and the cgroups that contain them. It then works through one service, myapp, on a throwaway VM, breaking it the ways real services break. The aim is that the numbers mean something when it's your server at 3 a.m.
The interview prep guide (opens in a new tab) covers the same ground as questions and answers, including a full "server is slow" procedure. This article is the explainer behind it: how the parts fit, and why the commands show what they show.
Linux in one paragraph
For an operator, a Linux server is a short chain of responsibility. Firmware hands off to a bootloader, and the bootloader starts the kernel. The kernel starts one process, PID 1, which on most servers is systemd. systemd then starts everything else as units: your web server, sshd, the log collector.
Every running program is a process with a state, a parent and a memory footprint. It also has a place in exactly one cgroup, which the kernel docs describe as "a mechanism to organize processes hierarchically and distribute system resources along the hierarchy". Boot, processes, systemd, filesystems, memory and CPU all follow from that chain. The job isn't knowing every command. It's having a method when a box misbehaves, so you rule things out in order instead of rebooting and hoping.
How it works
Boot: from firmware to PID 1
The boot chain matters day to day because each step has its own failure mode and its own logs.
- Firmware. UEFI or BIOS initialises the hardware and hands off to a bootloader, usually GRUB 2.
- Bootloader. GRUB loads the kernel and an initramfs, a small temporary root filesystem.
- Kernel. The kernel uses the initramfs to find and mount the real root filesystem.
- PID 1. The kernel starts PID 1, normally systemd. systemd brings up
default.target, which is usually an alias formulti-user.targeton a server orgraphical.targeton a desktop. It starts units in dependency order.
flowchart TD
FW["Firmware (UEFI or BIOS)"] --> GRUB["Bootloader (GRUB 2)"]
GRUB --> K["Kernel + initramfs"]
K -->|mounts real root filesystem| PID1["PID 1: systemd"]
PID1 --> T["default.target (alias of multi-user.target)"]
T --> U1["sshd.service"]
T --> U2["myapp.service"]
T --> U3["systemd-journald.service"]
PID1 --> S["run-*.scope (from systemd-run --scope)"]
subgraph CG["cgroup of myapp.service"]
P1["myapp main process"] --> C1["child process"]
P1 --> C2["child process"]
end
U2 --> P1
P1 -.stdout and stderr.-> U3
PID1 -.adopts and reaps orphans.-> C2Each service unit runs its processes in its own cgroup, and the cgroups form a tree. A new process starts in its parent's cgroup, so a child forked by myapp lands in myapp's cgroup without anyone asking. That is how systemd knows which processes belong to which service, and it is where memory limits and per-service pressure figures apply. Limits set nearer the root of the tree can't be overridden further down. A .scope unit, which systemd-run --scope creates, wraps a command you started by hand in a cgroup of its own. The worked example uses one.
journald collects what service processes write to stdout and stderr. If a parent dies, PID 1 adopts its children. That adoption is the mechanism behind zombie cleanup, covered next.
You can read the chain back on a running box:
systemctl get-default
systemctl list-dependencies multi-user.target
systemd-analyze blame | head -n 10
journalctl -b -p errget-defaultnames the target the box boots into.list-dependenciesshows the units that target pulls in.systemd-analyze blamelists the units that took longest to start.journalctl -b -p errshows messages from the current boot at error priority and above. It is the first place to look after a slow or failed boot.
Processes and their states
The STAT column of ps shows a process's state. These are the five you will meet most often. ps documents a few others, so check man ps on your system for the full list.
| State | Meaning | What it tells you |
|---|---|---|
R | Running or runnable | Using CPU, or waiting for a CPU |
S | Interruptible sleep | Waiting for something, such as a socket; normal |
D | Uninterruptible sleep | Blocked in the kernel, usually on I/O |
Z | Zombie | Exited, but the parent hasn't collected its status |
T | Stopped | Paused by a signal or a debugger |
To see states alongside the parent-child tree, run:
ps -eo pid,ppid,stat,cmd --forestA zombie has already exited, so you can't kill it. It doesn't appear in its cgroup's cgroup.procs either, and it can't be moved to another cgroup. Its parent never called wait() to collect the exit status, and the fix is to fix or restart the parent. Once the parent exits, PID 1 adopts the zombie and reaps it. You can make one in a few seconds. The shell below starts a short sleep in the background, then replaces itself with a long sleep that never waits for its child:
sh -c 'sleep 1 & exec sleep 30' &
sleep 2
ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'You should see a sleep entry marked <defunct>, whose PPID is the long-running sleep. When that parent exits after 30 seconds, the zombie is adopted and reaped, and running the ps line again shows nothing.
This matters in containers. If your container's entrypoint spawns children and isn't a real init, nothing reaps them. A small init such as tini does that job.
Signals and shutdown
Stopping a process means sending it a signal.
- SIGTERM asks the process to stop. A process can catch it, so a well-behaved server finishes in-flight requests and exits.
- SIGKILL can't be caught. The kernel ends the process immediately.
This loop catches SIGTERM, cleans up and exits with status 0:
bash -c 'trap "echo got SIGTERM, cleaning up; exit 0" TERM; while true; do sleep 1; done' &
pid=$!
sleep 1
kill -TERM "$pid"
wait "$pid"; echo "exit status: $?"It prints the trap message and exit status: 0. The trap runs once the current sleep 1 returns, so the reply can take up to a second. Repeat it with kill -KILL "$pid". The trap never runs, and the exit status is 137, which is 128 plus signal 9. That is the same 137 you will see on a container killed for memory.
Kubernetes follows the same sequence for a pod: SIGTERM, a wait of terminationGracePeriodSeconds (30 by default), then SIGKILL. A classic way to drop requests on every deploy is a shell script running as PID 1 that doesn't forward signals. The script receives SIGTERM, the application never does, and the application dies on SIGKILL mid-request.
systemd: units, restarts and logs
systemd manages units, the targets that group them, restart policy, logging through journald, and resource control through cgroups. For an operator, the parts that matter most are how a unit is loaded, how it restarts and where its logs go. Here is the unit the worked example uses:
# /etc/systemd/system/myapp.service
[Unit]
Description=Example application
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=exec
User=myapp
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
[Install]
WantedBy=multi-user.targetWhat each part does:
Type=execmakessystemctl startfail if the binary is missing or theUser=doesn't exist. The default when onlyExecStart=is set isType=simple, which reports success in both cases. The systemd docs recommendexecfor long-running services.Restart=on-failurewithRestartSecbrings the service back after a crash.StartLimitIntervalSecandStartLimitBurststop it from restarting forever. A service that crashes on start would otherwise hammer its database every few seconds.User=myappdrops root.NoNewPrivileges=truemeans the service and its children can never gain privileges throughexecve(), for example via setuid binaries or file capabilities.ProtectSystem=strictmakes the whole filesystem read-only for the service, apart from/dev,/procand/sys. Any path it writes to has to be allowed withReadWritePaths=. The worked example hits exactly this.
Two loading rules cause most "my edit didn't work" moments. First, systemd reads drop-in files from myapp.service.d/*.conf and merges them after the main unit, in alphanumeric order, with /etc/ beating /run/ beating /usr/lib/. Packaged units live under /usr/lib/systemd/system/. Put your changes in /etc rather than editing the packaged file. Second, an unknown or misspelled option only produces a warning in the log. The unit still loads, and the setting you thought you applied silently isn't there.
After editing a unit, reload systemd's view of it, then restart the service and read its logs:
sudo systemctl daemon-reload
sudo systemctl restart myapp
systemctl status myapp --no-pager
journalctl -u myapp --since "10 min ago"systemctl status shows the active state, the main PID, the cgroup with its processes, and the last few log lines. Two more commands are worth knowing:
systemctl cat myapp
systemctl list-units --type=service --state=failedsystemctl cat prints the unit as systemd loaded it, including drop-ins. When an edit "didn't take effect", compare its output with the file you edited, then check whether you ran daemon-reload. list-units --state=failed is the quickest way to see everything broken on a box.
Load average and pressure
Load average is the most misread number on a Linux box. On Linux it is the damped average number of tasks that are either runnable or in uninterruptible sleep (D state), over 1, 5 and 15 minutes.
cat /proc/loadavgThe first three fields are the averages that uptime also prints. Because they count D-state tasks, high load with idle CPUs usually means tasks are blocked on I/O, not that you need more cores.
Pressure Stall Information (PSI) answers the question load can't: which resource are tasks waiting on? It lives in /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io. Each file has this format, shown here with the zeros from the kernel documentation:
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0someis the share of time at least one task was stalled on that resource.fullis the share of time all non-idle tasks were stalled at once, which is the "nothing is getting done" case.avg10,avg60andavg300are averages over 10, 60 and 300 seconds.totalis the cumulative stall time in microseconds.
The CPU file's full line is undefined at the system level. Since kernel 5.13 it is reported as zero for compatibility, so don't read it as "no CPU stalls". If the files are missing, the running kernel doesn't have PSI enabled.
PSI also exists per cgroup, as cpu.pressure, memory.pressure and io.pressure in each cgroup's directory. That turns "the box is under memory pressure" into "this service is under memory pressure".
Memory limits and the OOM killer
When memory runs out, the kernel's OOM killer picks a victim and kills it.
- It scores processes mainly by memory use, adjusted by
oom_score_adj, and kills the one with the highest score. - In a cgroup with a memory limit, the kill happens inside that cgroup even if the node has plenty of free memory. That surprises people who check
freeand see gigabytes available. - Confirm a kill with
journalctl -k. - In Kubernetes, the container's last state shows
OOMKilledwith exit code 137.
Every process exposes its current score and adjustment. For the shell you're typing in:
cat /proc/$$/oom_score /proc/$$/oom_score_adjReplace $$ with any PID to check a service's process. The PID is in systemctl status.
Filesystems: inodes and links
An inode holds a file's metadata. A file name is just a link to an inode.
- A hard link is another name for the same inode.
- A symlink is a separate file that holds a path.
This takes ten seconds to see:
cd "$(mktemp -d)"
echo hello > a
ln a b
ln -s a c
ls -li
rm a
cat b
cat cIn ls -li, a and b share an inode number and show a link count of 2. c has its own inode and points at a. After rm a, cat b still prints hello, because the inode still has a link. cat c fails with "No such file or directory", because the path it holds is gone.
The data is freed only when the last link is gone and no process has the file open. That is why deleting a large log file doesn't always free any space.
A worked example
The way to learn this is to break a throwaway VM on purpose and watch the commands report it. Don't run any of this on a server you care about. The VM needs stress-ng, which usually isn't installed by default, and the sysstat package, which provides iostat and pidstat. Every step below uses commands from the sections above, so you can check what you see against the mechanism.
Step 1: take a one-line picture
Before touching anything, define what "slow" means: latency, errors or throughput, and since when. Then take a quick picture of the whole box. This is the sequence I run:
nproc
uptime
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
vmstat 1 5
free -h
df -h && df -iHow to read each one:
- Compare the
uptimeload averages withnproc. - PSI shows which resource tasks are stalled on. On an idle VM the
someaverages should sit near zero. - In
vmstat,ris the run queue,si/soshow swapping,wais time spent waiting on I/O, andstis CPU time stolen by the hypervisor. df -icatches a filesystem that is out of inodes whiledf -hstill shows free space.
Keep this baseline. Every later step is read against it.
Step 2: saturate the CPU
Start a CPU load with one worker per core:
stress-ng --cpu "$(nproc)" --timeout 60sWhile it runs, repeat uptime, vmstat 1 5 and cat /proc/pressure/cpu in a second terminal. The run queue r climbs past the core count, load follows it, and the CPU some averages rise, avg10 first. The CPU full line stays at zero, for the reason given above. To see which process is responsible, sample per-process CPU and disk use:
pidstat -u -d 1 5Step 3: look for I/O-blocked tasks
If load is high but CPU is idle, look for tasks in uninterruptible sleep, along with the kernel function they're waiting in:
ps -eo pid,stat,wchan:20,cmd | awk '$2 ~ /^D/'
iostat -xz 1 3iostat reports %util and the average wait time for each device. Current sysstat versions split the wait into r_await and w_await, and older versions show a single await column. One device near full utilisation with long waits, plus a list of D-state processes and a rising /proc/pressure/io, is the I/O case. Load average alone can't tell it apart from a CPU problem.
Step 4: trigger an OOM kill inside a cgroup
Run a memory hog inside a transient scope limited to 200M, and ask it to allocate 400M:
sudo systemd-run --scope -p MemoryMax=200M stress-ng --vm 1 --vm-bytes 400M --timeout 30s
journalctl -k --since "5 min ago" | grep -i oomThe kernel log should record an OOM kill, even though free -h on the host shows memory to spare. That is the cgroup behaviour described earlier: the limit belongs to the scope's cgroup, not the machine. MemoryMax= is the cgroup v2 setting. If the VM has swap, the scope may swap instead of being killed. In that case, run it on a VM without swap, or also limit swap with -p MemorySwapMax=0. On a real incident, widen the search:
journalctl -k --since "1 hour ago" | grep -i -E 'oom|killed process'Step 5: fill a disk that du can't explain
This step reproduces the "df says full, du finds nothing" case. In one terminal, create a 200 MB file and keep it open:
python3 -c "
f = open('/tmp/big', 'wb')
f.write(b'0' * 200 * 1024 * 1024)
f.flush()
input('holding /tmp/big open; press Enter to release ')
"In a second terminal, delete it, check the space and find the holder:
rm /tmp/big
df -h /tmp
sudo lsof +L1 | grep /tmp/bigdf still counts the 200 MB because the inode is alive while the Python process holds it open. lsof +L1 lists open files with no links left, and shows the process holding the deleted file. Press Enter in the first terminal and the space comes back.
On a server, the culprit is usually a daemon still writing to a log you rotated by deleting it. Restarting that process, or having it reopen its logs, frees the space.
Step 6: deploy a hardened unit and watch it fail
The last step uses the myapp unit and breaks it the way hardening usually breaks things. First, create a system user, a state directory it owns, and a tiny myapp that writes a heartbeat file every five seconds:
sudo useradd --system --user-group --no-create-home --shell /usr/sbin/nologin myapp
sudo install -d -o myapp -g myapp /var/lib/myapp
sudo tee /usr/local/bin/myapp >/dev/null <<'EOF'
#!/bin/sh
while true; do
date > /var/lib/myapp/heartbeat || exit 1
sleep 5
done
EOF
sudo chmod 755 /usr/local/bin/myappSave the unit from the systemd section as /etc/systemd/system/myapp.service. Check it, load it and start it:
sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl start myapp
sleep 30
systemctl status myapp --no-pager
systemctl show myapp -p NRestarts
journalctl -u myapp --since "5 min ago" --no-pagersystemctl start succeeds: the binary exists and the user exists, so Type=exec has nothing to object to. The failure comes from inside the service. ProtectSystem=strict makes /var/lib/myapp read-only for the service, even though myapp owns it. The write fails, the script exits 1, and Restart=on-failure brings it back every five seconds. After five starts inside 60 seconds, StartLimitBurst stops the loop. systemctl status should show the unit as failed, and the journal should show the write error from the script. Without the start limits, this loop would run forever.
The fix is to allow that one path. Rather than editing the unit, add a drop-in, so the change sits in its own file:
sudo mkdir -p /etc/systemd/system/myapp.service.d
printf '[Service]\nReadWritePaths=/var/lib/myapp\n' | sudo tee /etc/systemd/system/myapp.service.d/writable.confThen reload, clear the failed state, and start again:
sudo systemctl daemon-reload
sudo systemctl reset-failed myapp
sudo systemctl start myapp
sleep 6
cat /var/lib/myapp/heartbeat
systemctl cat myappreset-failed matters here. A unit that hit its start limit stays failed until you clear it, so a plain start can be refused. The heartbeat file should now hold a timestamp. systemctl cat shows the main unit followed by writable.conf, which is how you confirm the drop-in was picked up.
While it runs, check which cgroup the service lives in:
cat /proc/"$(systemctl show -p MainPID --value myapp)"/cgroupOn a cgroup v2 host this prints a single line starting 0::, followed by the cgroup path, which names myapp.service. That directory is where the per-service memory.pressure file lives. Clean up with sudo systemctl stop myapp, then remove the unit file, the myapp.service.d directory, the script, /var/lib/myapp and the user.
Going deeper when the picture isn't enough
When one process is the problem but the one-line picture doesn't say why, three tools go further:
strace -f -tt -p <pid>shows the system call the process is blocked in. Replace<pid>with the PID frompidstatorps.perf topshows where CPU time goes.tcpdump -i any -nn port 53captures DNS traffic, for intermittent resolution problems.
For a DNS problem, check /etc/resolv.conf first. On systemd-resolved hosts, also run resolvectl status. Then query the server directly with dig. That separates "the resolver is wrong" from "the DNS server is wrong". For sockets, these two commands cover most questions:
ss -s
sudo ss -tlnpss -s summarises connection counts by state. ss -tlnp lists listening TCP sockets with the process that owns each one. It answers "is anything actually listening on that port?"
Trade-offs and pitfalls
| Symptom | Tempting conclusion | What it often is | Check with |
|---|---|---|---|
| Load well above core count | Need more CPUs | Tasks blocked on I/O (D state) | vmstat, PSI, ps ... awk '$2 ~ /^D/' |
CPU full in PSI always 0 | No CPU stalls | Undefined at system level, reported as zero | CPU some line |
| Process killed, host has free memory | Kernel bug | cgroup memory limit hit | journalctl -k, exit code 137 |
df full, du finds nothing | Corrupt filesystem | Deleted file held open, or inodes exhausted | lsof +L1, df -i |
| Unit edit ignored | systemd is broken | No daemon-reload, or a misspelled option that only warned | systemctl cat, the journal |
start succeeded, service never ran | It's running | Type=simple with a missing binary or user | Type=exec, systemctl status |
| Unit won't start after fixing it | Fix didn't work | Start limit still tripped | systemctl reset-failed |
| Hardened service fails on write | Permissions bug | ProtectSystem=strict made the path read-only | journalctl -u, ReadWritePaths= |
| Dropped requests on every deploy | Load balancer issue | PID 1 shell script not forwarding SIGTERM | Entrypoint, signal handling |
| Dependency overloaded after a crash | Dependency too small | Unbounded restart loop | StartLimitIntervalSec, StartLimitBurst |
The broader pitfall is jumping to a fix. A reboot clears the zombie, the held-open file and the stuck I/O, and leaves you nothing to learn from. The order I follow:
- Define "slow".
- Take the one-line picture.
- Go deeper only on the resource it points at.
- Make the fix.
- Add an alert so you hear about it before your users do next time.
Each tool has its own limits:
- sysstat. It isn't on every image, so
iostatandpidstatmay be missing exactly when you need them. Bake the package in ahead of time. - cgroup v1 and v2. The memory lab assumes cgroup v2. Mixing v1 and v2 controllers on one host is possible, but the kernel docs strongly discourage moving controllers between them in production. Test on the distribution and cgroup setup you actually run.
- Hardening options.
ProtectSystem=strictandNoNewPrivileges=truetrade convenience for safety. Every path the application writes to has to be listed, and a missed one only shows up at runtime, as in Step 6. - Unit typos. Because an unknown option only warns, a misspelled hardening directive leaves the service running without it. Read
systemctl catand the journal after every change. - Entrypoint scripts.
/bin/shis not necessarily Bash. It is dash on Debian and Ubuntu, and BusyBox in Alpine. A script that relies on Bash features can behave differently as a container entrypoint. The shell scripting comparison (opens in a new tab) covers the differences.
Further reading
- The interview prep guide linked above has the Linux concepts checklist, the full "server is slow" scenario and more break-a-box labs.
- systemd.service (opens in a new tab) covers
Type=and its failure modes, and the restart settings. - systemd.unit (opens in a new tab) covers the unit search path, drop-ins and default dependencies.
- systemd.exec (opens in a new tab) covers
ProtectSystem=,ReadWritePaths=andNoNewPrivileges=. - Pressure Stall Information (opens in a new tab) in the kernel docs covers the
/proc/pressurefiles, per-cgroup pressure andpoll()triggers. - Control Group v2 (opens in a new tab) covers the cgroup tree,
cgroup.procsand memory limits.
Next, run the six steps above on a throwaway VM running your production distribution. Keep the output from each step, and compare it with what your monitoring showed during the same run. Wherever the dashboard missed what the command showed, that's the alert to add first.



Discussion comments
No comments yet. Start the discussion.