Snapshot-Restore Latency Across Firecracker, QEMU, and Cloud VMs

Firecracker microVMs restore in milliseconds while QEMU and cloud VMs lag behind.

· 9 min read
Cover illustration for “Snapshot-Restore Latency Across Firecracker, QEMU, and Cloud VMs”
Sandbox & Runtime · October 15, 2026 · 9 min read · 2,068 words

Snapshot-restore latency decides whether an agent feels responsive or sluggish to a user inside a chat interface, and that is architectural. Persistent AI agent infrastructure and serverless functions diverge precisely here: serverless optimized for request handling, while agents need a per-turn wake that does not tax the interaction. The gap between a 28ms restore and a 300ms restore is not a benchmark curiosity but a per-turn cost that compounds across millions of agent turns.

The three infrastructure categories, Firecracker microVMs, QEMU-based VMs, and cloud VM provisioning, each occupy a different position on the latency axis, and their ordering is not arbitrary. Every millisecond of wake latency is therefore paid on every turn, while idle time dominates billing. A 1-second cold boot is workable for long-running batch jobs. It is not workable when an agent needs to run a short code snippet and return a result before a user's attention drifts.

Maritime's entire infrastructure model is built on this insight. Operators of agent platforms spend most session time idle between LLM turns, so restore latency is a tax you pay on every interaction. By hosting agents in Firecracker microVMs with sub-second wake times, the platform ensures that per-turn latency stays imperceptible across millions of agent interactions.

What a Firecracker snapshot actually contains and why restore skips the boot sequence

Restore is not a boot. Firecracker maps the memory file, loads the CPU and device state, and unpauses the vCPUs. A snapshot-restore has no boot path, and a cold start does. The guest was already initialized when the snapshot was captured. Restore simply resumes it.

The design choice that makes restore fast regardless of guest RAM size is MAP_PRIVATE. The memory file is mapped lazily and copy-on-write, so no bytes are read at load time. The kernel pages in only the 4 KiB pages the guest actually touches after resume. A read fault maps the page straight from the file. A write fault triggers a private copy of just that one page. All other VMs restoring the same snapshot share the unmodified pages in the page cache. A guest with 2 GiB of RAM that touches only a few hundred megabytes to reach a usable state pays only for those pages.

The two snapshot artifacts are inseparable. vm.state without vm.mem is meaningless, and the reverse is equally true: registers point into RAM, and RAM is uninterpretable without the CPU and device context. Serialization captures a moment of execution, not a disk image. The speed of restore is therefore a direct consequence of what is and is not serialized, and that distinction sets up the contrast with QEMU's far larger device surface.

Why Firecracker's minimal device model is the primary source of its restore advantage over QEMU

The latency difference between Firecracker and QEMU on snapshot-restore follows directly from the size of the device model each VMM must serialize and deserialize. Firecracker emulates exactly six devices: virtio-net, virtio-balloon, virtio-block, virtio-vsock, a serial console, and a minimal keyboard controller. There is no USB, no GPU, no sound, no PCI bus, no BIOS, no UEFI. Because the device state is minimal, the vm.state file is tiny, and loading it is not the bottleneck on the restore path.

QEMU's snapshot and migration machinery is mature and capable. It serializes the state of a far larger device surface, and when configured with SeaBIOS or OVMF, the firmware initialization phase alone adds substantial latency on cold paths. That gap is any order of magnitude and a direct product of the two designs' differing scope.

Firecracker's narrowness is a deliberate trade. It loses live migration, TCG emulation, QMP, and legacy device support in exchange for a device surface small enough that serialization is never the bottleneck. The narrowness reflects the same design philosophy that shapes agent-focused infrastructure: every feature, every device, and every millisecond of latency is a tax paid on every agent wake, so the right platform cuts relentlessly to eliminate bottlenecks that do not serve the core workload.

What the latency numbers actually show across Firecracker restore, QEMU golden-snapshot boot, and cloud VM provisioning

Diagram: Three Infrastructure Tiers: Where Restore Latency Actually Falls. Visualizes: Show three infrastructure categories on a single latency axis, ordered from fastest to slowest, with their concrete measured values.

The three categories are not the same operation, and conflating their numbers produces misleading comparisons. Firecracker and QEMU measurements describe snapshot-restore of a pre-initialized guest; cloud VM provisioning describes a different operation entirely. Firecracker restore runs fastest of the three. AWS researchers measured Firecracker snapshot restore at as little as 4ms in their own experiments. A practitioner building a Go-based sandbox orchestrator measured restore at approximately 28ms in an end-to-end pipeline, with the breakdown running roughly as 5ms for Firecracker process startup, 8ms for the mmap of the memory snapshot, 10ms for CPU and device state restore, and 5ms for vsock reconnection and the ready signal. The full create pipeline, which includes network slot allocation, TAP patching, rootfs reflink, VMM fork-exec, snapshot load, vCPU resume, and a guest reachability probe, lands at a 179ms p50 and roughly 203ms p99 for on-demand restore with no warm pool.

QEMU golden-snapshot boot occupies the middle of the range. The practical conclusion is that QEMU's restore path is slower than Firecracker's, but a tuned golden-snapshot workflow closes much of the gap that cold-boot comparisons suggest. The 28ms versus 300ms difference matters for high-frequency agent turns, but both are effectively instant for occasional operations.

Cloud VM provisioning is a categorically different operation. It involves allocating hardware, configuring networking, and booting an OS image from scratch, not restoring a pre-initialized state. That places cloud VM provisioning latency in the range of tens of seconds to minutes, useful for long-running workloads but structurally incompatible with per-turn agent wake cycles. AWS Lambda SnapStart, which takes a snapshot of an initialized execution environment and restores it on invocation, brings managed cloud wake times into the sub-second range for supported workloads. It represents the cloud's answer to the same problem Firecracker solves at the microVM layer.

How memory-mapping strategy determines whether restore latency scales with guest RAM in a multi-host fleet

If the memory file is not already on local disk, MAP_PRIVATE's lazy paging cannot help you. The naive path downloads the entire multi-gigabyte vm.mem file to local disk before restore can begin. That is pure dead time, and the majority of it covers pages the guest will never touch on a particular boot.

Firecracker's userfaultfd, or UFFD, backend is an alternative to the local File backend. Guest memory is backed by a userspace fault handler that fetches pages on demand from object storage, so the guest can begin executing before the full RAM image has arrived. UFFD streaming wins clearly for fleets that restore the same templates repeatedly on the same hosts, where the working set is warm in local cache. Forking adds a further dimension: a same-host fork runs in roughly 400 to 750ms because the parent's memory is already resident and the rootfs reflinks locally, while a cross-host fork costs meaningfully longer because artifacts must move over the network first.

The MAP_PRIVATE copy-on-write memory-mapping strategy that allows Firecracker to restore agents cheaply regardless of total allocated RAM is foundational to making persistent agent hosting economically viable. Because restore cost does not scale with memory, idle agents can be billed on storage rather than compute. That pricing insight enables platforms like Maritime to offer flat-rate per-agent deployment.

Operational correctness problems that restore latency numbers do not capture

VMM-layer restore timing and application readiness are not the same measurement, and the gap between them can be large enough to break production agent behavior. A filed issue in the opensandbox-group/fast-sandbox repository, dated October 8, 2026, documents one such pathology. Restoring a Firecracker checkpoint into a new network slot preserves the guest's cached gateway MAC address, but the new TAP has a different MAC. The guest sends SYN-ACKs to the old gateway MAC, those packets are dropped until the guest re-resolves its gateway via ARP, and TCP exponential backoff delays the first successful HTTP request by approximately 64 seconds. The 64-second delay is entirely invisible to VMM-layer metrics.

Guest clock freeze produces a second class of failure. From inside the guest, zero time elapsed between snapshot and restore: the clock was frozen at capture time and resumes from there. TLS handshakes fail when the gap between snapshot and restore is large, because the guest believes the server's certificate notBefore is in the future, so validation rejects it. This is an operational correctness problem, not just a performance one, on any platform where templates are baked once and restored many times days or weeks later.

Uniqueness restoration after cloning is the third challenge. Two interfaces, MADV_WIPEONSUSPEND and VmGenId, are presented and compared to alternative solutions for this problem in the serverless cold-start context. Each of these failure modes is architectural, which means each is fixable at the architecture level rather than by application code patching around it.

What density and idle-cost economics look like once restore latency is under 200ms

If create latency is low enough, you do not need a warm pool of idle VMs. An operator can create on demand and destroy on completion, so idle cost approaches zero because there are no idle VMs. The idle-time problem is the dominant billing variable for agent workloads. If an agent spends the large majority of session time waiting for an LLM to stream tokens, every second of idle compute is waste, and the billing model, not the headline rate, determines what that waste costs.

Firecracker's memory overhead of under 5 MiB per microVM, versus roughly 20 to 30 MiB for a comparable QEMU configuration, enables higher co-location density on the same host hardware. Firecracker's documented creation rate also means provisioning is not the throughput bottleneck at fleet scale. The snapshot-and-restore scheduler pattern, bake the snapshot once on cold boot, then restore on demand in under 200ms, means the slow path is paid exactly once per template, not once per agent turn. A pool-mode alternative exists: run fewer VMs than active users and multiplex, accepting directory-level isolation instead of kernel-level isolation.

Choosing the right restore architecture for agent workloads: a decision framework by threat model and access pattern

The right restore architecture depends on three axes: isolation requirement, meaning who is running the code and how trusted they are. Access pattern, meaning how often a given template is restored and on which hosts. And acceptable latency tier, meaning whether the agent turn is user-facing or background.

For customer-facing agents executing untrusted or arbitrary code, Firecracker microVMs provide hardware-level kernel isolation. A guest exploit cannot reach the host or other tenants because KVM is the boundary. For high-density internal fleets where code is trusted, gVisor or Kata offer a middle tier with lower per-VM overhead but without full hardware VM isolation. Pool mode runs fewer VMs than users on a shared kernel, so you get directory-level isolation only. It is appropriate for fully trusted, single-tenant workloads, not for multi-tenant code execution.

Memory backend selection follows the access pattern. The File backend with MAP_PRIVATE on local disk gives the lowest restore latency when the memory file is already in the page cache, so it is the right choice when templates are restored repeatedly on the same host. The UFFD streaming backend is the right choice for large multi-host fleets where templates must restore on any host without pre-downloading the full RAM image. It incurs network latency for cold-host working-set faults. Sub-200ms end-to-end create with Firecracker on-demand is appropriate for user-facing agent turns where wake latency is part of the interaction, and it enables eliminate-the-pool economics where idle cost is storage rather than compute. Sub-500ms golden-snapshot QEMU is appropriate for near-interactive workloads where QEMU's broader device model or live migration capability is needed and the slightly higher restore cost is acceptable. Cloud VM provisioning in the tens of seconds is appropriate for long-running agent processes where the VM lives for hours and provisioning latency is amortized, not for per-turn or per-session wake cycles. Operational correctness requirements, clock correction on resume, ARP cache invalidation on network slot reassignment, and uniqueness restoration after cloning, apply regardless of which tier is chosen and must be solved at the platform level, not left to application code. Maritime's architecture, each agent in its own Firecracker microVM, waking from a snapshot in under a second, paying for storage during sleep rather than idle compute, is built directly on the mechanisms this article describes: MAP_PRIVATE lazy paging, minimal device serialization, and a snapshot-restore scheduler that makes on-demand creation economically viable at per-user scale.

Sources

  1. Restoring Uniqueness in MicroVM Snapshots - ar5iv - arXiv

More in Sandbox & Runtime