Scheduler Architectures That Sustain Hundreds of Concurrent Agents Under Per-VM Isolation Constraints
Scheduler design, not raw capacity, sustains hundreds of isolated AI agents.

Firecracker micro-VM isolation for AI agents gives each agent its own kernel through hardware virtualization. Sustaining hundreds of these isolated agents requires a scheduler that separates agent lifecycle from compute allocation. Snapshot-restore, coordinated access to shared resources, and speculative prewarming let a fleet preserve agent state while reserving active VMs for work that can proceed.
Why sustaining many concurrent agents is a scheduling problem, not a capacity problem
HiveMind’s April 2026 incident began with 11 concurrent agents sharing one API key. Three died from connection resets and HTTP errors, even though the API had enough aggregate capacity to serve all 11 sequentially. The bottleneck was coordination. HiveMind identified a 5-second stagger between agents as sufficient to have saved all 11. Adding hardware would not have supplied that missing coordination.
Agent workloads make this distinction especially important. They hold state for minutes to hours, burst when invoking tools, and then wait on a model, a human, or an external service. A one-to-one mapping between agents and running VMs therefore reserves resources throughout periods when those resources cannot advance the task. The scheduler’s job is to distinguish work that is ready from work that is merely present, then allocate compute and shared API access accordingly.

How can each AI agent be given its own isolated kernel without containers?
Firecracker, Kata Containers, and libkrun provide the microVM isolation tier described here, where each workload gets its own kernel through hardware virtualization. Containers instead share a host kernel. That distinction is central when untrusted agents run in a multi-tenant environment, because the guest kernel is separated from the host and other guests. Agents in separate VMs can still compete for the same model quota or API key.
Per-VM isolation also places three concrete demands on the scheduler. First, a fresh microVM takes time to reach a usable state, which makes naive on-demand launches a poor fit for latency-sensitive bursts. Second, each active VM reserves memory and vCPU resources even when its agent is waiting. Idle time has a real resource cost. Third, state must survive suspension and resumption, or reclaiming compute destroys the environment the agent needs to continue.
Model-Native Computing Architecture frames the agent runtime as an operating system whose responsibilities include scheduling, permissions, and resource management. This is a useful division of responsibility. The VM supplies an isolation boundary, while the runtime decides when work can run and what resources it may consume. A scheduler must therefore manage both the cost of maintaining that boundary and the contention that persists outside it. Boot latency, idle allocation, and state persistence are the first constraints to resolve.
Snapshot-restore as the answer to idle cost and boot latency
Snapshot-restore scheduling allows an agent to retain its state without occupying a running VM. The platform boots a microVM fully once, snapshots it, and restores it when work resumes. This gives agents warm-pool-class latency without keeping a pool of idle, fully booted instances. Separating agent lifetime from VM running time is the structural change that makes long-lived agents compatible with intermittent compute use.
The underlying mechanism uses MAP_PRIVATE copy-on-write. A restore pays in time and RAM for the pages the guest touches between resume and ready. The relevant quantity is the working set. A large guest that touches only a fraction of its address space during resumption therefore incurs the restore cost associated with that fraction. Image size alone does not describe the resources needed to make the agent usable again.
Human-in-the-loop workflows make the benefit concrete. When an agent hands control back for a decision, the VM can be snapshotted and paused with its working directory, installed dependencies, running processes, and warm page cache frozen to disk. Nothing runs while the human deliberates. The idle period incurs storage cost rather than continuously reserved compute, allowing the environment to persist without treating a human response time as VM execution time. Snapshot-restore resolves idle allocation and boot latency, but it cannot decide how several restored agents should share a rate-limited service.
OS-inspired scheduling primitives for agents that are simultaneously active
HiveMind uses classical scheduling primitives to handle that contention among active agents through a transparent HTTP proxy that needs no changes to existing agent code. Its design treats connection storms, starvation, and blocked work as coordination problems. Separate kernels do not prevent these failures. The scheduling layer must govern the resources agents share after they have started, including provider access, context consumption, and execution priority.
Admission control uses condition variables to keep new agents at the gate until there is room for them, which prevents connection storms. Provider-aware rate-limit tracking checks how much agents use against the actual limits of the upstream provider. AIMD backpressure with circuit breaking supplies another control on contention. Per-agent token budgets prevent an agent from consuming disproportionate context and starving its peers. Priority queuing with dependency DAGs lets high-priority agents run before lower-priority workers and lets work that depends on other work wait without busy-polling.
These controls address different parts of the same failure. HiveMind reports failure rates of 0–18% with scheduling primitives active and finds the primitives most effective in combination. That result argues against treating any single control as a complete scheduler, but it does not establish failure-free operation.
The architecture paper supplies the broader analogy. It maps the LLM inference core to a CPU and the agent runtime to an operating system, with context windows among the finite resources that need management. At the coordination layer, model inference quota is another shared constraint. More running agents cannot create more of either resource. Scheduling instead determines how the available resources are divided among work that competes for them.
Speculative prewarming: eliminating cold-start latency before it happens
Snapshot-restore addresses the wait between sessions, while contention management governs agents that are already active. Neither alone removes the intra-session gap between a tool invocation and the sandbox becoming ready to execute it. Speculative prewarming overlaps environment preparation with the model’s ongoing execution rather than waiting for the agent to finish deciding what to do.
SpecBox uses intent-aware sandbox prewarming to make that overlap possible. Its July 2026 research comes from Beihang University, the University of Leeds, and the University of Sydney. It infers execution intent from streaming token outputs through keyword matching and streaming semantic embedding.
SpecBox also uses stochastic sandbox prefetching to forecast the next environment. A first-order Markov model operates on a sandbox dependency graph built from step-to-step execution transitions, producing probabilistic forecasts. This cuts cold starts even for tool sequences that are new. SpecBox reports that it uses far less memory at its peak than sandbox deployments kept permanently reserved, which supports preparing only the environments likely to be needed.
These techniques complete a layered scheduling design. Snapshot-restore preserves idle agents without reserving running VMs, contention management allocates shared resources among active agents, and speculative prewarming reduces tool-environment waits within an active session. Each addresses a different interval of inactivity. Their combination gives a fleet scheduler control over when compute is released, when work is admitted, and when the next environment is prepared.

Hierarchical coordinator-worker patterns for fleets of hundreds
A hierarchical coordinator-worker model brings these scheduling layers together across the fleet. The coordinator owns the full conversation context and task decomposition, while workers execute scoped tasks in isolated VMs and return compressed results. Idle workers can then be suspended through snapshot-restore. Contention management governs their shared API access, while prewarming prepares environments for anticipated tool use.
Context discipline is essential to this arrangement. Each worker receives a subset of the problem and returns a summary. The coordinator’s context window therefore need not grow linearly with worker count. Parallelism would otherwise shift the bottleneck into context capacity. The OS analogy places the coordinator in the role of a kernel scheduler, the workers as user processes, and the scheduling layer as the resource manager.
The architecture paper formalizes the limit through its Agent Speedup heuristic. Parallel execution yields returns only while agents remain sufficiently independent, because coordination overhead dominates when tasks share too much state. Independence is therefore a governance constraint. The dependency DAG records which workers must finish before the coordinator can proceed. With priority queuing, it gives active VM slots to work on the critical path, keeping work that cannot yet move the overall task forward from filling the fleet.
What the scheduler must track to govern a fleet correctly
CPU and memory measurements alone cannot tell a scheduler whether an agent can make progress. It also needs to know which agents are idle, which are on the critical path, which have exhausted their token budgets, and which are blocked on dependencies. Low utilization is not a sufficient diagnosis. The allocation decision depends on the agent’s state and the resources or results it needs next.
The scheduler needs to track the context budget per agent. An agent that consumes its finite context window inefficiently can block itself while wasting active VM time. Fleet-wide token spend rate is a separate dimension because agents sharing an API key also share its rate limits. Local measurements cannot reveal the whole constraint. Admission control needs to know how much the agents use in total to prevent the contention seen in HiveMind’s incident.
VM working-set size supplies the third dimension. Because restore cost depends on the memory an agent touches during resumption, the scheduler needs that information to estimate the resources required to bring it back. Dependency graph state supplies the fourth. A worker waiting for a prerequisite, or a coordinator waiting for worker results, should be suspended even though its task is unfinished, freeing the active VM while it waits.
These signals make the earlier primitives operationally useful. Admission control needs them to set appropriate thresholds, priority queuing needs them to identify the critical path, and speculative prewarming needs accurate execution traces. Visibility is part of the control mechanism. It also provides the basis for cost allocation that reflects what an agent consumed, rather than treating every long-lived agent as an equal claim on running infrastructure.
How scheduler design determines the economics of agent deployment at scale
Idle-suspend policy directly determines how much compute is wasted between turns. A permissive timeout leaves many idle VMs running simultaneously, even when none can advance its task. HiveMind reports that coordinated scheduling eliminates 48–100% of wasted compute in its measurements. The idle interval is therefore an economic decision. Snapshot-restore infrastructure implements that decision by preserving the environment while releasing its active allocation.
Holding slots for low-priority agents while high-priority work queues increases both latency and cost. Priority queues and dependency DAGs direct those slots toward work that advances the critical path. Prewarming accuracy introduces another tradeoff. Over-prewarming wastes memory on environments that are not needed, while under-prewarming introduces cold starts that extend active VM time, so the stochastic prefetcher uses recorded transitions to calibrate its forecasts.
These allocation choices constrain application-layer pricing. Per-tool-invocation, per-seat, and outcome-based models each assume a different way to spread infrastructure costs across the work agents do. A price attached to a user or an outcome does not remove the underlying cost of idle VMs, blocked workers, or unnecessary warm sandboxes. The scheduler sets which costs the platform has to bear. Its design shapes both which pricing models can cover their costs and how well an existing model uses resources.
Each user can retain a stateful environment without reserving a continuously running machine, paying for storage during idle periods instead. When activity resumes, the scheduling layers restore the environment and govern its access to finite resources. Thousands of agents can consequently share a compute footprint sized for the active minority. That is the economic basis for sustaining per-VM isolation at scale.


