Kernel Idle Detection Hooks That Trigger Agent Snapshot Evacuation

Kernel hooks detect idle agent windows and trigger efficient snapshot-restore cycles.

· 8 min read
Cover illustration for “Kernel Idle Detection Hooks That Trigger Agent Snapshot Evacuation”
Kernel and OS Primitives for Agents · November 3, 2026 · 8 min read · 1,765 words

Snapshot-restore scheduling for idle AI agents replaces resident, waiting VMs with saved state that restores on demand. Kernel idle detection hooks make that transition safe by finding quiet periods and starting snapshots at a between-turns boundary. With a fast micro-VM restore path, the result is storage rather than idle compute cost, without sacrificing wake latency.

Why agents spend most of their time waiting, and what that costs naively

Agents spend most of their time waiting between turns. Yet a naive deployment keeps a full VM resident for each agent, using compute the whole time it waits. In a multi-tenant fleet, the overwhelming majority of agents are idle at any given moment, so this waiting dominates the bill. PandaStack implements the alternative in production, running no warm pool of idle VMs and restoring every agent from a baked Firecracker snapshot. Making that model viable requires both a reliable idle signal and a restore path fast enough that shutdown is not a one-way door.

Why "idle" is harder to detect than it sounds at the kernel level

Silence blindness is the first failure mode. A hook that fires when the work queue drains can detect an explicit transition, but an agent that simply goes quiet may produce no such event. The scheduler is left without a callback trigger precisely when the agent has stopped generating traffic.

Concurrency creates a separate problem at the turn boundary. hermes-agent PR #709, dated Oct 25, 2026, records an idle sweep racing with an in-progress turn. The sweep can pop an agent after an inactivity timeout or between running-slot release and turn completion, even though the turn has not drained. The result can be a missed snapshot or a double-commit.

ANOLISA PR #6154, dated Oct 6–7, 2026, addresses a one-shot “pending snapshot row exists” marker left on an evicted connection. A new connection reusing the same kernel key inherits that marker and is never snapshotted. Interruption and crash paths also miss the abandoned call, so the same bug affects them as well as the idle sweep.

How a kernel clock and idle tracker replace the broken on_idle hook

Making time an explicit kernel input closes the silence-blindness gap. corvidae PR #174, merged on Oct 29, 2026, introduced a clock in corvidae/clock.py, registered as clock. It provides wall time, monotonic time, interruptible sleep, and one-shot timers. Silence can therefore become observable through elapsed time, without requiring another queue event.

The idle tracker turns that time input into activity history. Registered as idle in corvidae/idle.py, it receives activity and drain signals from the agent. Plugins then register per-channel or system watches, each with its own period. A watch fires after its configured quiet period and repeats at most once per period while quiet continues. Traffic cannot starve retention, although a frequent heartbeat can delay it up to memory.retention.max_delay, whose default is 3600 s.

Startup needs an idle history as well. PersistencePlugin seeds each channel as quiet since its last logged message, allowing a restart with no traffic to consolidate correctly. The change removed on_idle, Agent._maybe_fire_idle, and IdleMonitorPlugin. It also made the already-dead daemon.idle_cooldown_seconds and daemon.idle_poll_interval configuration errors.

This registration boundary separates detection from the actions it triggers. Memory consolidation, snapshot scheduling, and retention can register independent watches. The clock supplies the time, the tracker records activity, and watches set how long each consumer needs things to be quiet. That division makes the signal composable without making every plugin an idle detector.

How the idle signal gets wired to snapshot initiation safely

An idle signal authorizes a check. Snapshot initiation still needs a between-turns boundary, never a point inside a model call. With a commit-once lock, either the sweep or turn completion can commit the eviction, and only one can do so. Separately, one-shot markers must not survive eviction and contaminate a reused connection key.

The agent-pro idle package, published Sep 12, 2026, has an exit-on-idle state machine with set bounds that checks whether the snapshot is stable and uses an occupy probe. Its FirstDelay is 30 s for timeouts at or above that threshold, while shorter timeouts sample immediately. Each idle-exit cycle performs at most SamplesPerCycle (3) resting-plus-occupy checks. After soft exit, a DefaultGrace of 5 s gives it time before forced shutdown.

Reset logging makes rejected exits distinguishable from successful ones. The package records snapshot_err, changed, not_ready, occupied, queue, and final_occupy explicitly. spore-host/spawn PR #969 shows why the snapshot-initiation contract must also set who may change the sampled baseline, apart from turn completion or connection markers.

What the VM substrate needs to make snapshot-restore worth triggering

Diagram: The Suspend-to-Restore Pipeline: Five Stages, Milliseconds Each. Visualizes: Show the sequential stages of the Firecracker micro-VM restore path, contrasting cold boot (~1 second) with snapshot restore as a single end-to-end flow.

The hook engineering pays off only if eviction is reversible in milliseconds. NumaVM’s production measurements put first-spawn cold boot at roughly 1 second, with subsequent restores delivering a multiple-of-six improvement. Restore skips rootfs copy, API configuration, kernel boot, and init entirely.

The measured path includes ~5 ms for Firecracker process startup and ~8 ms to mmap the memory snapshot file. Restoring CPU and device state takes a slightly longer step, followed by ~5 ms for vsock reconnection and the ready signal. These measurements cover separate stages of the restore pipeline, each accounting for only part of total wake time.

With MAP_PRIVATE, the kernel establishes virtual-memory bookkeeping for a file-backed mapping without eagerly reading its contents. Mapping a 2 GiB snapshot and a 200 GiB snapshot therefore costs about the same at that stage. This does not mean every later memory access has the same cost.

Density also depends on what remains shared after restore. Firecracker’s own figures put per-VM overhead at less than 5 MB. Hundreds of sandboxes restored from the same golden image can read the same underlying physical pages for memory they have not dirtied. Together, low overhead and lazy mapping make restoring isolated VMs practical without keeping every agent resident.

The two-phase suspend protocol that hooks into the idle signal

With a single threshold, every quiet interval gets the same treatment. A two-phase protocol instead uses a cheap freeze for short idle periods and full snapshot eviction for long ones. The first phase preserves sub-second wake for the common case, while the second releases compute during deeper inactivity. The kernel’s idle signal drives the move from one phase to the next as well as the on/off decision.

Each hook has a distinct correctness job. /freeze stops CPU consumption before capture begins, preventing dirty state from accumulating during the snapshot. /fsfreeze flushes dirty pages and closes the sync-to-pause race window. /collapse reduces the number of distinct pages that must be faulted in on resume, improving responsiveness after restore.

Resume needs its own repair step as well. /init re-synchronizes the wall clock, which drifted during suspension, and restores environment state. Without that distinction, a protocol can preserve the captured state yet overlook the passage of time outside the VM.

Sampling races and the idle-baseline integrity problem

The idle baseline must have exactly one writer. Even a clock-driven tracker can misclassify activity when independent paths advance the fields used to measure change. The spawn race involved a Prometheus HTTP scrape goroutine and a monitor loop updating the same idle-sample fields without a lock. A scrape moved the baseline, so traffic looked idle when measured against the new baseline.

The fix assigned the whole decision to one owner. evaluateIdle samples, decides, and publishes an idleSnapshot under a dedicated idleMu. Nothing else calls the sampler. For implementations with metrics collectors, health checks, or debug endpoints, the lesson is to expose that published snapshot to readers.

chenhz01/idle-agent-scheduler uses a coarser heartbeat-age proxy. Its tick() serializes an agent silent longer than idle_timeout_s to one atomic JSON file in state_dir, writing a temporary file before os.replace, and then releases the slot. Long silent computation must heartbeat or accept being frozen. That trade-off suits single-machine schedulers where false-idle evictions are recoverable, but not cases where snapshot creation is expensive or state loss is unacceptable.

The idle package protects its baseline at the probe boundary. Watchdog.Tick() alone captures the resting snapshot, and ProbeOccupied uses that snapshot as its before-state. The post-probe state is never written back as the new baseline. Otherwise, an ephemeral placeholder could become apparent session activity on the next tick.

How persistent VM isolation makes idle detection safe to act on

Correct detection is useful only if eviction preserves the agent. It must be possible to recover files, credentials, running processes, and installed packages after compute is released. With Firecracker micro-VMs, each agent has its own kernel, persistent disk, and memory space. The snapshot captures CPU registers, memory pages, device state, and vsock connection state, allowing execution to resume where it paused.

That boundary makes suspension different from reinitialization. The agent needs no dependency reinstall or re-authentication after restore. Phase-one freeze also preserves IP, hostname, and PVCs because the VM is the isolation unit.

Shared container namespaces and serverless functions have different structural limits. In a shared namespace, snapshotting captures namespace-level state that can include more than one tenant. A serverless function has no persistent disk or kernel state to snapshot, so restoring the agent means cold reinitialization.

Hook reliability also depends on identifying the correct process. ciresnave/agentlife v0.30.3, dated Nov 1, 2026, hardened SessionStart diagnosis by retrying process-table capture up to three times with 25 ms pauses until the hook PID appears. If the final snapshot still lacks that process, it returns a specific claude_pid error.

What the full idle-hook-to-restore chain looks like end to end

The kernel clock supplies wall and monotonic time, while the scheduler sleeps on the clock. Agent activity and drain signals update the tracker, and registered watches set how long each channel or the system must be quiet. When a watch fires, the scheduler checks quiescence at a between-turns boundary and uses the commit-once lock to coordinate eviction.

The suspend path then turns permission into preserved state. /freeze stops consumption through frozen cgroups, /fsfreeze flushes dirty pages, and /collapse compacts the heap. SnapshotProvider serializes memory and device state before the compute slot is released. On user return, the VM is recreated, /init re-synchronizes the clock and environment variables, and vsock reconnects so the agent can resume in tens of milliseconds.

The economic distinction follows the agent’s lifecycle. A short idle can retain a frozen VM with near-zero CPU consumption, while a fully evicted agent incurs storage cost alone. A managed implementation can expose this chain as a persistent, isolated agent endpoint without requiring developers to operate the snapshot scheduler. For fleets where most agents are idle most of the time, snapshot-on-idle and restore-on-demand is the only scheduling model here that scales billing with active concurrency rather than total agent count, making per-user deployment economically viable.

Sources

  1. idle package - github.com/xhd2015/agent-pro/pkgs/tty/detection/idle - Go Packages

More in Kernel and OS Primitives for Agents