Linux Cgroups V2 Memory Limits for Agent Processes

Cgroup v2 memory controls let teams right-size agent sandboxes for unpredictable peaks.

· 8 min read
Cover illustration for “Linux Cgroups V2 Memory Limits for Agent Processes”
Kernel and OS Primitives for Agents · October 24, 2026 · 8 min read · 1,847 words

Linux cgroup v2 memory limits make agent sandboxes predictable when protection, throttling, hard limits, and swap controls work together in one planned profile. Across a fleet, snapshot-restore scheduling for idle AI agents adds to that control by keeping stored snapshots in place of warm compute between tasks. Compact memory footprints reduce snapshot size and restore latency, while VM isolation supplies the security boundary that cgroups cannot provide.

What cgroup v2's unified hierarchy actually changed about memory enforcement

Cgroup v2 puts enabled resource controllers in a single hierarchy, so memory and CPU limits apply to the same set of processes. In v1, the CPU, memory, and block I/O controllers each had their own tree. A process could belong to different groups for different resources, leaving accounting inconsistent and limits contradictory. In V2, process migration is atomic across all enabled controllers. Reading memory.current and cpu.stat from the same directory therefore describes the same process group, aligning monitoring with enforcement.

The accounting boundary also includes more than application memory. Kernel memory accounting was opt-in under v1 through kmem.limit_in_bytes, but it is always enabled in v2. Slab allocations, kernel stacks, page tables, and socket buffers are charged to the cgroup. An unchanged heap is not an unchanged budget. A workload with substantial socket memory can need a larger allowance after migration even if its application-level memory use has not grown.

Page cache adds another source of apparent discrepancy because file reads increase memory.current without necessarily changing the heap. That cache is reclaimable, but it still counts toward usage. A container that reads large log files can therefore suffer an OOM kill while its heap is still well below its configured limit. Migration calls for reviewing the whole memory budget.

Before applying that review, teams should establish which interface the host exposes. The command test -f /sys/fs/cgroup/cgroup.controllers identifies v2. File names and semantics differ between versions, including memory.limit_in_bytes versus memory.max and memory.max_usage_in_bytes versus memory.peak. Ubuntu 22.04+, Debian 12+, and Fedora 31+ default to v2, but documentation still needs to match the actual host.

The four memory knobs and the distinct job each one does

memory.min establishes a hard protection floor below which the kernel will not reclaim memory, even under severe host pressure. Its protection is stronger than a preference. That strength also limits the host's ability to recover memory when it urgently needs capacity elsewhere. memory.low, by contrast, provides soft protection during global pressure. The kernel tries not to reclaim below that threshold, giving the cgroup priority over groups above their low line, but it does not reserve or guarantee available memory.

Docker maps --memory-reservation to memory.low in v2, despite the stronger implication of the flag's name. The throttling threshold is memory.high, which triggers aggressive reclaim and slows allocating processes when usage exceeds it.high, which triggers aggressive reclaim and slows allocating processes when usage exceeds it. Crossing memory.high` does not invoke the OOM killer. The consequence is higher latency rather than termination, giving v2 a graceful degradation mechanism without an equivalent in v1.

memory.max sets the hard memory limit, where failed reclaim can lead to the OOM killer terminating processes inside the cgroup. This is the enforcement path associated with OOMKilled and exit code 137. The gap below it has a distinct purpose. Headroom between memory.high and memory.max gives reclaim time to work before termination becomes necessary. Setting the two to the same value removes that buffer and turns the profile back toward an abrupt cutoff.

Kubernetes maps resources.limits.memory to memory.max, while resources.requests.memory has no direct cgroup v2 mapping by default. A request is a scheduler hint. The filesystem exposes the enforcement state through memory.current, memory.max, and the low, high, max, oom, and oom_kill counters in memory.events. Child groups also need explicit configuration because creating one does not copy the parent's knob values. An unlimited child setting does not create a separate allowance beyond the parent's aggregate constraint.

Diagram: The Four Memory Knobs: From Protection Floor to Hard Limit. Visualizes: Visualize the four cgroup v2 memory thresholds as a vertical memory scale with labeled bands: memory.min (hard protection floor — kernel will not reclaim below this)…

The swap trap that silently blows past the limit you think you set

memory.max limits memory separately from swap, so setting it alone does not establish a combined memory budget. A process can consume its full RAM allowance and additional swap. V2 exposes swap usage through memory.swap.current and constrains it through memory.swap.max, unlike v1's combined memory.memsw accounting. Setting memory.swap.max to 0 disables swap for that cgroup. With swap disabled, memory.max sets the total memory bound that the sandbox profile calls for.

Docker expresses this through a counterintuitive pair of flags. Setting --memory-swap equal to --memory disables swap, though the flags can suggest that they allow an equal amount of swap. The resulting v2 setting is memory.swap.max equal to 0. Monitoring must follow the separate kernel accounting. Because memory.current excludes swap, dashboards must also read memory.swap.current to reveal heavy swapping and the latency degradation it can conceal.

Why agent processes break the assumptions these knobs were designed around

Diagram: Agent Memory vs. Serverless and Microservices: Peak-to-Average Ratio. Visualizes: Show the stark contrast in peak-to-average memory ratios across three workload types, as reported in AgentCgroup (arXiv 2602.09345): AI coding agents reach…

AgentCgroup, the academic paper identified as arXiv 2602.09345, characterizes AI coding agents as having a peak-to-average memory ratio of up to 15.4×. It compares that with roughly 1.5× for serverless workloads and 2–3× for microservices. Agents can idle in the hundreds of megabytes and then reach multiple gigabytes during tool calls. Those peaks also arrive at unpredictable times, making a stable average a poor guide to the required hard limit.

Budgeting every agent for peak demand wastes capacity during the much longer periods when that demand is absent. Setting memory.max near the average instead exposes tool bursts to OOM termination. The useful distinction is between protecting the persistent baseline and allowing temporary work to consume headroom under reclaim and throttling.

An agent also has more to lose from termination than a web process serving independent requests. A web server can restart and serve the next request, while an agent can lose accumulated LLM context, tool results, and intermediate state. The cost is starting over. The same paper finds that OS execution, including tool calls and container or agent initialization, accounts for a majority of latency across the workloads studied. Memory pressure during those operations therefore affects the critical path of an agent turn, rather than remaining background resource bookkeeping.

A practical knob configuration strategy for agent sandboxes

A layered configuration profile should be built from observed idle and burst footprints. Set memory.low around the stable idle footprint to give in-memory context priority during pressure from other cgroups. That baseline can include loaded model state, session memory, and open file handles. The protection remains soft. Reserve memory.min for the most critical agent tier instead of making hard protection the default across every sandbox.

Set memory.high above the average working set expected for the agent but below the burst limit expected for it. This gives ordinary work room to proceed while directing larger tool-call allocations into reclaim and throttling. A slower turn is preferable to lost state. Set memory.max far enough above memory.high to give reclaim time to work. The gap must reflect observed bursts because a nominal separation that provides no useful headroom cannot deliver the intended graceful response.

Disable swap with memory.swap.max set to 0 so swap latency cannot accumulate on the agent's execution path. Then use memory.events to check how the profile works in production. A high and rising high count signals frequent throttling and a threshold that may be too low for the actual working set. The oom_kill counter records the more costly failure. These signals distinguish a profile that slows bursts from one that still terminates them.

Set memory.max on each ephemeral child rather than assuming that the parent's value is copied into it. AgentCgroup describes a transparent bash wrapper that creates a tool_<pid>_<ts>/ child for each tool invocation, records memory.peak and duration, and removes the group on exit. No agent framework modification is required for that pattern. Per-call peaks provide the observations needed to tune burst allowances without confusing temporary tool demand with the agent's persistent baseline.

What cgroups cannot do about multi-tenant agent isolation

Cgroup v2 governs resource consumption, but it does not establish a security boundary between tenants. Namespaces restrict what processes see, while cgroups restrict what they cost. Neither isolates tenants from a shared-kernel exploit. That distinction is especially important when agents execute untrusted tool calls, browse the web, run code, and interact with external services. A hostile input or compromised tool can probe a boundary that memory throttling was never designed to defend.

Micro-VMs add an isolation layer alongside resource governance. Firecracker uses KVM hardware virtualization to give each workload its own kernel. A compromise of that guest kernel is therefore separated from the host and other workloads by the virtualization boundary, rather than merely by membership in a different cgroup. The two layers serve distinct purposes. Cgroups inside the VM can govern agent and tool resources, while cgroup accounting outside it adds defense in depth because a compromised guest kernel can bypass guest-only limits.

The memory profile must also fit the VM's allocation model. Firecracker's memory hotplugging allows a guest to grow into needed memory during a session without restarting. That lets the guest handle the same unpredictable bursts that make it hard to give agents fixed budgets. It reduces the need to provision every agent for its heaviest possible task at startup. Cgroup pressure controls and VM-level growth therefore address resource behavior, while the VM boundary addresses tenant isolation.

How fleet-level memory governance stops paying for compute while agents sit idle

Across a fleet, each tenant's agent can have its own cgroup v2 child with memory.max and pids.max set explicitly. This puts a per-agent bound on how much memory each agent can use as it grows. But containment alone does not eliminate idle cost. Agents run tool calls in short bursts and can sit idle while the model processes. Keeping their working sets warm still consumes RAM and compute capacity, even when CPU activity is low.

Snapshot-restore scheduling changes what remains allocated during those gaps. An idle agent is snapshotted instead of held warm, replacing compute cost with storage cost until work resumes. A tighter cgroup-governed working set produces a smaller memory image to serialize, and a smaller image restores faster.

The restore mechanism also sets practical limits on the scheduling model. On a QEMU/libvirt q35 stack, an idle warm base restores meaningfully faster than a lightly loaded agent VM with several gigabytes of live memory. Sub-second restore needs primitives built for that job. A snapshot feature alone is not enough. Platform criteria must include the memory footprint being restored, restore latency, persistent state across sleep, and whether idle billing actually switches from compute to storage.

Per-agent controls also need to address duplication. A one-process-per-tenant design with a fixed large RAM allocation makes cluster memory grow linearly with users, while duplicated prompt embeddings add avoidable per-tenant demand. Per-container limits and shared caching address different parts of that problem. The limits constrain each agent's working set, while shared caching avoids repeated allocations that isolation alone cannot remove.

The resulting per-user model is an isolated VM with a compact, cgroup-governed footprint, persistent disk for state that survives sleep, and a scheduler that snapshots idle agents. Predictable memory enforcement thus supports both scheduling economics and protection against OOM kills.

More in Kernel and OS Primitives for Agents