Fork removes the cost of starting an episode. Most of what is left in a Kubernetes episode is waiting — readiness probes, CrashLoopBackOff timers, controller reconcile ticks. A 3-node Ubuntu cluster is all-idle ~70% of the time and only 7–10% of its vCPUs are awake.
k7d already owns the guest clock end to end (kvm-clock, no RTC, no NTP client, KVM_SET_CLOCK on every restore), so the same pause that makes a fork correct can also move every member’s clock by the same delta. Two mechanisms, one rule: forward only, lockstep only, whole-cluster pause or nothing — a declined warp is always correct.
Time warp is a native-engine feature. backend: firecracker / runtimeClassName: k7-fc VMs reject dilation > 1 and warp requests with a named error — see Backends.
The numbers#
An 8.7-minute guest-time episode on a 3-node Ubuntu k3s cluster:
| Mechanism | What it does | Wall time | Fidelity cost |
|---|---|---|---|
| Stock clock | — | 522 s | — |
Lockstep jump (auto_warp jump, no kernel patch) | When every vCPU is idle, pause, add the same δ to kvmclock + every TSC offset up to the next armed timer, resume. ~0.2 ms pause, ~20 Hz | 286 s (1.83×) | zero — same probe failures, restarts, leases, splats as stock |
| Continuous dilation ×N (carried ~200-line KVM patch) | Guest clocks run N× wall; every clock the guest can read agrees. Set per VM, changed live on a paused cluster in ~0.4 ms, inherited or overridden on fork | 79 s (8.00×) at N=8; exactly N through 16, knee at N=12 | real, bounded: work does not dilate, so guest-visible fsync 2.3 → 14 ms, pod RTT 0.69 → 0.92 ms, a guest-paced churn loop does 37 rounds instead of 86 |
| Both | — | 76 s (8.11×) | the two do not stack: dilation consumes the idle windows jumps need |
Lockstep jumps — warp_clock and auto_warp#
warp_clock moves every live VM on one cluster bridge forward by delta_ms:
The whole cluster is paused for the write and resumed after it. A uniform jump is what etcd heartbeats, lease TTLs, and TLS skew checks tolerate; a skewed one is not. A warp that fails on one member rolls that member back and errors — all or none.
Instead of telling the daemon “+90 s”, let it find the idle time itself:
| Mode | Behaviour |
|---|---|
jump | Probe the cluster; when every vCPU is provably idle, jump to just before the earliest armed timer. Zero fidelity cost — timers fire exactly when they would have |
micro | Warp by a fixed 50 ms quantum every quantum regardless of idleness. Fidelity cost: a timer can fire late by at most one quantum |
off | Stop the loop. The reply carries the finished run’s counters |
Every reason a cycle declines to warp is a correct outcome; declining costs a sample, guessing costs correctness.
Caller contract: no guest action may be in flight during a warp. Daemon ops run under the VM registry lock the policy loop also takes, so they cannot overlap a jump. A client driving a member from outside the daemon — an agent exec straight over vsock — is invisible to that lock and must take a hold first:
Otherwise that guest sees its clock jump mid-syscall. Releasing a hold nobody took is an error, not a no-op.
Continuous dilation — dilation and set_dilation#
Dilation makes every clock the guest can read run N× faster than wall time, for the whole life of the VM. It needs a small carried KVM patch on the host (per-VM continuous time dilation). The patch and the install/revert loop ship in the repo at utils/kvm-dilation/:
install.sh stops k7d, refuses the swap if another VM still holds /dev/kvm, and puts stock modules back if the patched pair fails to load. Ubuntu x86, root, and the running kernel’s linux-source / headers (pulled from Launchpad). A host without the patch fails VM creation for dilation > 1 instead of silently running at 1×. Lockstep jumps need no patch.
Set it at creation — every member of a cluster is created from one VmConfig, so a cluster is dilated in lockstep by construction:
Change it live on a running cluster (the cluster is paused for the write; the kernel refuses the change while any vCPU is runnable):
Lowering the factor is not a rewind — the guests keep every nanosecond they have already been given; they only stop accumulating them so fast. Raising is the same write in the other direction.
Forks and dilation#
A fork lands on the source’s clock and inherits the source’s factor unless told otherwise:
So one ×1 source tree serves every mode: fork some children at ×8 for cheap exploration and keep others at ×1 for a faithful control. tree_nodes reports each live node’s factor in live_dilation; fork responses carry dilation.
Why N=8 and not 16#
Compression is exactly N through 16 — but the fidelity cost is a curve, not a cliff. Extra liveness-probe failures across N ∈ 16 went 0 / 0 / 0 / 3 / 19 / 32, and in-guest churn rounds collapsed 110 → 7. The host-side reading that moved with the damage is busy-phase mean awake vCPUs: at 6 of 6 there is no idle left to sell. That is the signal an adaptive policy should watch — not probe failures, which arrive through the guest’s API server, the exact thing dilation is starving.
Adaptive dilation: per-tree policy loop#
In 0.8.0, the daemon can watch that signal itself. Set dilation_policy on tree_create, tree_create_cluster, or tree_adopt_cluster, and a per-tree loop samples every vCPU’s KVM_GET_MP_STATE once per cadence. It changes the factor through the same pause → set → resume path as set_dilation:
| Mode | Behaviour | Measured on one scripted 240 s guest-time episode |
|---|---|---|
fixed (default) | No loop. The create-time factor stays | ×8: 32.0 s (8.00×); ×4: 60.9 s (4.00×) |
down | When the cluster is saturated, drop one rung down the ladder 8 → 4 → 2 → 1. Never raises | 128.4 s (1.91×) |
track | Same drop, and raise one rung when the cluster is quiet again | 73.9 s (3.37×) |
“Saturated” means at least 80% of the cluster’s vCPUs are awake. “Quiet” means 45% or fewer. Each decision rests on 3 samples, and the gap between the two thresholds keeps the factor from oscillating. All of these can be set per tree with dilation_config (window, saturated, quiet, cadence_ms, ladder); see the API reference. The create-time factor has to be one of the ladder’s rungs.
down exists because a rate that follows host load makes every guest-time measurement in an episode (a probe period, a backoff, a reward) incomparable with the same measurement in another episode. Dropping is monotone, happens at most once per rung, and is reported. track recovers more compression when a busy phase ends, and pays for it with a clock rate that depends on load. Which trade is right is the caller’s decision.
Forks inherit the source node’s mode and knobs unless the fork sets its own. That includes tree_fork / tree_fork_batch, which accept dilation_policy and dilation_config too, but not cadence_ms: a tree has one loop and one sleep, so the root owns the cadence. tree_nodes reports each node’s dilation_policy and its dilation_trajectory, the list of factors the loop has run it at. The loop needs the native engine and the KVM patch. It is not persisted, so a tree reopened with tree_open comes back fixed.
A window of 1 can spend a rung on the sample taken as the loop starts, before any fork exists, because the vCPUs are still awake from the snapshot resume that created the tree. Keep the window at 3 unless you are testing the loop itself.
Intel hosts: rebuild the patch#
The carried KVM patch is at ABI generation 3 in 0.8.0. On Intel hosts that use the VMX preemption timer (kvm_intel preemption_timer=Y), generation 2 left a LAPIC timer armed before a live set_dilation running at the old rate. A guest sleep 0.25 in flight across a 4 → 8 raise overran by up to 249 ms of guest time. Generation 3 re-arms in-flight timers after the ratio change, and the worst overrun across 20 raise/drop pairs is 36 ms. AMD hosts were not affected. If you installed the patch from 0.7.0, run make dilation-install again from a 0.8.0 tree.
The same timer detail affects lockstep jumps on Intel. A jumped LAPIC one-shot on the preemption timer fires at the next pause rather than at its host deadline, so auto_warp jump compresses a crash-loop backoff somewhat less on those hosts: 163 s against a 200 s ceiling, versus about 132 s on AMD.
What is and is not compressible#
- Compressible: guest-side waiting — timers, backoffs, probe periods, lease TTLs, reconcile intervals.
- Not compressible: compute. A dilated guest still takes the same wall time to
fsyncor serve a request, so from the guest’s point of view those operations look N× slower. That is the fidelity cost above, and it is why the honest default is a jump policy at zero cost, with dilation opt-in per tree. - Not touched: the source’s clock when you fork. Warp and dilation apply to the cluster you name; siblings on other bridges are unaffected.
Related pages#
- Daemon API — clock operations
- Cluster mode — the cluster these ops act on
- Benchmarks — enforced budget for the lockstep warp