Container Exit 137: Verify Memory Pressure Before Tuning

Exit 137 is a clue that a process was killed, not a complete diagnosis of why it stopped. Correlate the runtime's termination information with cgroup memory events and workload activity before increasing limits or declaring a memory leak. Different causes can produce similar symptoms at the application boundary.

The Linux kernel's cgroup v2 memory documentation describes counters such as oom_kill and distinguishes local from hierarchical events. Those counters provide useful evidence, but they still need the correct cgroup and a meaningful observation interval.

Establish what actually terminated

Record the container or job identifier, start and end times, runtime-reported reason, and whether a supervisor restarted it. Match that instance to the incident. A dashboard showing the newest healthy replacement can hide the termination details of the instance that failed.

Check whether an operator, orchestrator timeout, or shutdown procedure deliberately sent a kill signal. A process that exceeded a graceful termination window needs investigation of its shutdown behavior, even if memory usage happened to be high at the same time.

Preserve the relevant logs before retention or replacement removes them. Collect only the time range and fields needed for the diagnosis. Raw application logs can contain customer data, so use the incident's approved storage and access controls.

Read the right memory boundary

Distinguish host-wide availability from the container's configured cgroup limit. A machine can have free memory while a constrained workload reaches its own boundary. Conversely, a host-level memory incident may affect a container whose individual limit was not the immediate trigger.

Resolve the workload's actual cgroup through the runtime or service configuration instead of guessing a filesystem path. Read its memory limit, current usage, and available event counters. Not every environment exposes the same files or grants the same read access.

Do not write to cgroup control files during diagnosis. Changing a limit alters the experiment and can affect neighboring workloads. Keep the first pass read-only and record which observation belongs to which configuration.

Compare counters across the incident interval

A counter's absolute value can include earlier failures. Capture a baseline, reproduce the issue in an authorized test environment, and examine the change. An increase in a relevant kill counter during the failed job is more informative than a nonzero value observed days later.

Be careful with hierarchy. A parent group's event totals may include child workloads. If you attribute that aggregate to one process, you can blame the wrong job. Review local counters where available and document the scope of each metric.

Use a small timeline containing job phase, memory observations, event-counter changes, and termination state. Our Text Diff can compare redacted counter snapshots, but it cannot identify the correct cgroup or determine the cause of a process kill.

Test workload-shaped memory peaks

Average usage can miss the allocation that matters. Test startup, model loading, batch assembly, decompression, and parallel request handling separately. A workload can fit comfortably during steady operation and fail only when two expensive phases overlap.

For an invented service, one worker might peak at 600 MiB during a particular import. Four concurrent workers do not necessarily stay within a 2 GiB boundary, especially with shared service overhead. These are illustrative planning numbers, not a sizing recommendation for your software.

Vary one condition at a time. Reduce concurrency in a test, then compare the peak and outcome. If the failure disappears, that suggests a concurrency relationship; it does not prove that every allocation is necessary or that the application has no leak.

Distinguish growth from bounded working memory

A leak hypothesis needs evidence that memory remains unnecessarily retained across comparable work. Measure repeated cycles with the same input size and idle intervals. A large but stable cache or allocator behavior requires different interpretation from unbounded growth.

Record process-level and cgroup-level views when available. A single process's heap metric may omit native allocations, child processes, or other charged memory. Do not expect one language runtime graph to explain every byte within the resource boundary.

If profiling is needed, use approved tooling and understand its overhead. Capturing a full heap or core can expose secrets and significantly change memory use. Start with the least intrusive evidence that can separate the leading hypotheses.

Tune only after choosing the failure you address

Possible changes include bounded concurrency, smaller batches, streaming inputs, improved cleanup, or a reviewed limit increase. State which observed condition each change targets. Raising the limit can provide temporary headroom while leaving a growth problem unresolved.

Retest the same workload and a plausible stress case. Watch completion time and neighboring services as well as memory. A successful import that now starves other workloads is not an unqualified improvement.

Conclusion

Diagnose forced termination with instance-specific runtime evidence, correctly scoped memory counters, and workload timing. Exit 137 alone does not prove an OOM event or a leak. Tune the observed bottleneck and verify the whole service after the change.