The Sacrificial Lamb: Kernel OOM Priority Calibration Ward
Calibrate Linux kernel Out-Of-Memory badness scores to make mission-critical databases invulnerable while designating expendable workers as prime sacrificial targets.
protect_pid() { local p="$1"; local adj="${2:--1000}"; if [ -w "/proc/$p/oom_score_adj" ]; then echo "$adj" > "/proc/$p/oom_score_adj" && echo "[PROTECTED] PID $p oom_score_adj set to $adj (Immune to OOM killer)"; else echo "Requires root privilege" >&2; fi; }; protect_pid $(pgrep -o postgres) -1000 The Lineage to the Sacrificial Lamb Architecture
In extreme resource starvation, distributed systems and operating systems must make deterministic life-or-death decisions:
“When physical RAM is completely exhausted, the Linux kernel Out-Of-Memory (OOM) killer calculates a ‘badness’ score for every process. The process with the highest score is abruptly sacrificed to preserve kernel stability. Without explicit prioritization, the kernel often sacrifices the largest database engine instead of the rogue worker that caused the leak.” — Robert Love, Linux Kernel Development
The Linux kernel heuristic for badness evaluates: $$\text{badness} = \text{RAM %} + \text{oom_score_adj}$$
Where oom_score_adj ranges from -1000 (completely immune from OOM sacrifice) to +1000 (first in line to be executed).
The Sacrificial Lamb Rite conducts the ancient art of Necromancy and Protection:
- It grants Absolute Immortality (-1000) to mission-critical, hard-to-recover stateful daemons (PostgreSQL, Redis, etcd).
- It marks expendable, stateless, easily-restarted batch workers as Sacrificial Lambs (+1000).
- When an OOM storm strikes, the kernel predictably slays the designated scapegoat, leaving your primary database completely unscathed.
The Spell Formula
Cast this invocation to grant total OOM immunity to your primary database and designate a worker as the sacrificial target:
protect_pid() {
local target_pid="$1"
local adj_value="${2:--1000}"
[ -z "$target_pid" ] && { echo "Usage: protect_pid <PID> [score_adj (-1000 to 1000)]" >&2; return 1; }
if [ -w "/proc/$target_pid/oom_score_adj" ]; then
echo "$adj_value" > "/proc/$target_pid/oom_score_adj"
echo "[NECROMANTIC WARD] PID $target_pid oom_score_adj calibrated to $adj_value."
else
echo "Error: Cannot write to /proc/$target_pid/oom_score_adj (root required)" >&2
return 2
fi
}
# 1. Grant absolute immunity to PostgreSQL primary
protect_pid $(pgrep -o postgres) -1000
# 2. Designate heavy background worker as prime sacrificial lamb
protect_pid $(pgrep -f "worker_celery") 1000
To scry the entire host and list the top 10 processes most likely to be sacrificed by the kernel:
for p in /proc/[0-9]*; do
pid=$(basename "$p")
if [ -r "$p/oom_score" ]; then
printf "%6d | %-20s | OOM Score: %4d | Adj: %5s\n" \
"$pid" \
"$(cat "$p/comm" 2>/dev/null)" \
"$(cat "$p/oom_score" 2>/dev/null)" \
"$(cat "$p/oom_score_adj" 2>/dev/null)"
fi
done 2>/dev/null | sort -k7 -nr | head -n 10
OOM Priority Spectrum
| OOM Adjustment Value | Kernel Sacrificial Priority | Optimal Target Services |
|---|---|---|
-1000 |
Invulnerable (Never killed) | PostgreSQL, Vault, etcd, SSH daemon |
-500 to -999 |
Highly Protected | Web ingress proxies (Nginx, Envoy) |
0 |
Default Heuristic Badness | Standard user processes and daemons |
+500 to +999 |
Preferred Sacrificial Target | Ephemeral caches, image resizers |
+1000 |
First Sacrificial Lamb | Async queue workers, speculative crawlers |
By taking direct control of kernel badness calculations, the Sacrificial Lamb pattern turns chaotic OOM panics into deterministic, orderly self-preservation.