The High-Pressure 2:00 AM Outage Reality

When PagerDuty triggers a critical severity alert for a customer-facing cluster, you don't have time to browse through fragmented StackOverflow threads. Whether it is an unexpected CrashLoopBackOff storm, sudden OOMKilled termination cascades, or entire worker nodes transitioning to NotReady under Kubelet DiskPressure, systematic forensic triage is essential.

This guide provides an end-to-end operational reference for Linux container runtimes (Docker Engine, containerd, CRI-O) and production Kubernetes (v1.28–v1.31+). It pairs a quick-reference Container Exit Code Rosetta Table with an interactive, searchable CLI command deck and deep-dive root-cause workflows.

💡 Quick Navigation: Use the interactive command search below to instantly filter commands by platform, keyword, or incident type (e.g., oom, drain, dns, crictl, secret). Click any command box to copy it directly to your clipboard.

1. Container Exit Code Forensic Matrix

When a container terminates abnormally, the Linux kernel and container runtime record a numeric exit code. Understanding this code immediately narrows down the root cause to application code, memory constraints, or infrastructure eviction:

Exit Code Status / Name Trigger Mechanism Immediate SRE Remediation
0 Success / Completed Process finished execution naturally (e.g. batch job, init-container, migration). Normal behavior for Jobs/CronJobs. If unexpected for long-running service, verify entrypoint loop or binary arguments.
1 Application Error Uncaught runtime exception, syntax failure, missing config file or environment variable. Inspect kubectl logs <pod> --previous. Check for missing database connection string or misconfigured mounted secret.
137 OOMKilled (128 + 9 SIGKILL) Linux Kernel OOM killer invoked because container exceeded cgroup resources.limits.memory. Check kubectl describe pod | grep OOMKilled. Scale up limits.memory, optimize heap size (e.g. -XX:MaxRAMPercentage), or fix memory leaks.
139 Segmentation Fault (128 + 11 SIGSEGV) Native binary attempted to access unallocated or forbidden memory addresses. Rebuild container image with matching glibc/musl toolchains. Check C-extensions, native drivers, or corrupted shared libraries.
143 Graceful SIGTERM (128 + 15) Kubelet initiated pod termination during rolling deployment, node drain, or scale down. Verify application handles SIGTERM cleanly. Increase terminationGracePeriodSeconds if requests are abruptly severed.
255 Entrypoint Fatal Error Container runtime failed to invoke entrypoint script or executable permission denied. Ensure Dockerfile has executable permissions (chmod +x /entrypoint.sh) and proper shebang (#!/bin/sh).

2. Interactive SRE CLI Command Deck

Filter through field-tested production commands for Docker Engine and Kubernetes incident response:

Find All Crashing & Evicted Pods Kubernetes
List all pods across all namespaces not in Running or Completed states with wide IP/node output.
kubectl get pods -A --field-selector status.phase!=Running | grep -v Completed
Triage OOMKilled & Exit Code Kubernetes
Inspect exact container termination reason, exit status, restart count, and last state timeline.
kubectl describe pod <pod> -n <namespace> | grep -E "State:|Reason:|Exit Code:|Last State:"
Fetch Dead Container Post-Mortem Logs Kubernetes
Extract standard error and stack traces from the previous terminated instance before it rebooted.
kubectl logs <pod> -n <namespace> -c <container> --previous --tail=100
Audit Node Health Conditions Kubernetes
Detect host kernel alerts including DiskPressure, MemoryPressure, NetworkUnavailable, and PID exhaustion.
kubectl describe node <node> | grep -A 6 "Conditions:"
Graceful Node Cordon & Drain Kubernetes
Mark node unschedulable and safely evict running workloads without disrupting DaemonSets.
kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force
containerd Image & Container Garbage Collect containerd
Immediately purge unused container images and dead containers on worker nodes to clear Kubelet DiskPressure.
crictl rmi --prune && crictl rm $(crictl ps -a -q --state Exited)
Docker Emergency Host Space Reclaim Docker
Purge all dangling images, dead containers, orphan build caches and anonymous local volumes.
docker system prune -af --volumes
Check Inode & Root Partition Usage Storage
Identify filesystem full scenarios where disk MB shows free space but all i-nodes are consumed by overlay2 layers.
df -h / /var/lib/kubelet /var/lib/docker && df -i /var/lib/docker
Verify CoreDNS Pods & Outage Logs CNI & DNS
Check CoreDNS replica distribution, endpoints, and upstream DNS query resolution failure logs.
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
Ad-Hoc In-Cluster DNS Probe CNI & DNS
Spawn an ephemeral busybox pod to directly test internal CoreDNS latency and resolution health.
kubectl run dns-test -it --rm --image=busybox:1.28 --restart=Never -- nslookup kubernetes.default
Ephemeral Netshoot Debug Injection Kubernetes
Inject a fully armed network troubleshooting container into a production pod's network and PID namespace.
kubectl debug -it <pod> -n <namespace> --image=nicolaka/netshoot --target=<container>
Instant Secret Base64 Decode Pipeline RBAC & Config
Extract and automatically decode all key-value secrets in a Kubernetes Secret object in a single shell command.
kubectl get secret <secret> -n <ns> -o jsonpath="{.data}" | jq 'map_values(@base64d)'
Audit ServiceAccount RBAC Permissions RBAC & Security
Verify whether a specific ServiceAccount or user has privileges to execute API operations (e.g. create deployments).
kubectl auth can-i create deployments --as=system:serviceaccount:<ns>:<sa> -n <ns>
Audit & Renew Cluster TLS Certificates Control Plane
Inspect all control-plane certificate expiration dates and renew them before hard API-server blackout occurs.
kubeadm certs check-expiration && kubeadm certs renew all && systemctl restart kubelet

3. Deep-Dive: Root Cause Workflows

Why OOMKilled Cascades Happen (cgroups v1 vs v2)

In Kubernetes, container memory limits (resources.limits.memory) are enforced by the Linux kernel using control groups (cgroups). When a container process allocates more resident set size (RSS) memory than its configured ceiling, the kernel does not prompt or swap: it immediately dispatches an uncatchable SIGKILL (Signal 9).

In Java (JVM) or Node.js workloads, this frequently occurs when runtime heap configuration is unaware of container boundaries. For example, a Java application with default heap settings on a 64-core machine might allocate 25% of the *host's* 256GB RAM, promptly triggering an OOMKilled exit code 137 within seconds inside a 2GB container. Ensure you pass -XX:MaxRAMPercentage=75.0 to harmonize the runtime with cgroup quotas.

⚠️ Warning on Memory Limits: Setting resources.limits.memory without adequate headroom above typical peak traffic triggers immediate pod termination. In contrast, CPU limits (resources.limits.cpu) only cause CFS (Completely Fair Scheduler) CPU throttling, slowing the application down without crashing it.

Triage Sequence for Kubelet DiskPressure

When a worker node triggers DiskPressure: True, the Kubelet immediately stops accepting new pods and begins evicting existing pods with BestEffort Quality of Service (QoS). This typically stems from:

  • Log accumulation in /var/log/pods/: High-throughput applications logging unbuffered JSON stdout can exhaust gigabytes per hour.
  • Uncollected dead image layers: Kubelet image garbage collection only kicks in when filesystem usage crosses imageGCHighThresholdPercent (default 85%).
  • Orphaned Docker containers & dangling volumes: Continuous CI/CD test runs generating unpruned ephemeral volumes.
SRE Golden Rule: Always run df -i alongside df -h. File systems with millions of tiny temporary cache files can easily exhaust all available inodes even when 50GB of raw disk space remains available.
☸️ CLOUD-NATIVE SRE STANDARDS 🎓 KNOWLEDGE CENTER

Container Outage Recovery & SRE Field Triage

Access printable A4 runbook cards, live diagnostic scripts, and production Kubernetes recovery procedures engineered for DevOps and SRE teams.

📑 A4 Runbook: Card 15 ↗ ⚡ Universal Error Decoder ↗ 📚 All Knowledge Guides ↗
SpecInfo.org is developed as an independent engineering open standard. If this reference helped your laboratory triage, consider supporting the project.
☕ Support Project ↗