Skip to main content

Katakate is a thin orchestration layer over a curated stack of battle-tested infrastructure. This page explains what the moving parts are and how they talk to each other.

High-level diagram#

Components#

CLI — k7#

A Typer-based CLI distributed as a Debian package (apt install k7). Sandbox-management commands (create, list, fork, …) talk to the K7 API by default, with an API key or an OIDC login (k7 login). Node-local commands (install, shell, top, tenant, nodes, and the key commands unless --api is passed) still run on the cluster with kubectl. Hidden --core bypasses the API and calls K7Core in-process. Interactive k7 shell shells out to kubectl exec.

API — k7-api#

A FastAPI service that exposes a REST surface for sandbox lifecycle, exec, snapshots, identity (keys, role bindings), tenants and usage. The API runs as a K3s Deployment in kube-system — 2 replicas spread across nodes with a PodDisruptionBudget, uid 1000, all capabilities dropped — behind a NodePort Service (31007), with an in-cluster ServiceAccount and a scoped ClusterRole. It is stateless: API keys live in the kube-system/k7-api-keys Secret (hashes only, snapshot refreshed every 5 s), role bindings in ConfigMap k7-role-bindings, tenants as Tenant custom resources, and OIDC tokens are validated against the issuer’s JWKS with no sessions. The replica holding the k7-tenant-status / k7-metering-leader Leases reconciles Tenant.status.usage and samples storage usage. The image is ghcr.io/katakate/k7-api deployed by digest from the release’s images.lock.yaml (cosign-signed); --api-image local builds it from a checkout instead.

The pod carries a Caddy TLS sidecar: FastAPI listens on :8000 inside the pod, Caddy terminates HTTPS on the NodePort with a cluster CA the installer writes to /etc/k7/tls/ca.crt (or a Let’s Encrypt / operator-supplied certificate; the Let’s Encrypt overlay pins back to one replica on the first master). On Cilium clusters k7 install --api-allow-cidr adds a CiliumNetworkPolicy (k7-api-ingress) restricting who can reach the port. See API transport. Temporarily stop it with k7 api disable (scale to 0) and bring it back with k7 api enable.

The entire K7Core is async (kubernetes_asyncio + httpx), so the API can serve concurrent requests without thread-pool workarounds. Both k7-api and k7-agent serve Prometheus metrics on :9464 and log JSON lines with a request id that follows each request from the API into the agent (Observability).

Per-node agent — k7-agent#

A DaemonSet in kube-system (same image as k7-api, running k7.api.agent:app) that exposes node-local operations the central API can’t do remotely: k7d VM ops (pause / resume / fork / lookup, which need the node’s /run/k7d/k7d.sock and containerd socket), storage-pool utilization (GET /agent/v1/storage), and the usage sampler that meters every sandbox pod on the node into a local spool. The API always goes through the agent for k7d ops (K7_K7D_REMOTE_ONLY=1), authenticated with a shared token provisioned by the installer (/etc/k7/agent_token) and locked down by a CiliumNetworkPolicy so only the k7-api pod can reach the agents. If no agent is Ready on the target node, the operation fails loudly — never a silent no-op.

The agent is not privileged: uid 0 plus the k7d group (gid 770 on every node, so the DaemonSet carries one static gid), all capabilities dropped, read-only root filesystem, RuntimeDefault seccomp, no ServiceAccount token. Its only host mounts are the containerd socket directory, /run/k7d, /var/lib/k7d (ro), /run/k7 (ro) and the usage spool.

Python SDK — k7-sdk#

A small requests/httpx wrapper published to PyPI as k7-sdk (from k7_sdk import Client). Both Client (sync) and AsyncClient (async, pip install "k7-sdk[async]") speak the same JSON envelope as the API, authenticate with an api_key or a token_provider callable (OIDC), and cover keys, tenants, role bindings, node dedication, usage and remote install as well as the sandbox lifecycle.

K3s#

The CNCF-certified Kubernetes distribution we install on every node (pinned v1.36.5+k3s1; --k3s-version overrides before the first install). K3s is single-binary, supports embedded etcd for HA, and ships its own containerd. Katakate does not touch your existing Docker or containerd — K3s installs its own runtime so installation is non-disruptive, and since 0.5.0 the installer needs no Docker at all.

Already running Kubernetes? The Helm chart (deploy/helm/k7) installs k7-api, k7-agent, the k7-node-installer, RuntimeClasses, snapshot GC and the Cilium policies onto RKE2, kubeadm or EKS, and never installs Kubernetes, the CNI or the CSI driver. Node preparation on those clusters is done by the k7-node-installer image (the playbook’s node-prep role, k7 install --node-prep-only). See Multi-node clusters.

Tenants#

A Tenant custom resource (tenants.k7.katakate.org/v1alpha1, cluster-scoped) groups namespaces and carries dedicated nodes, a default sandbox size, quotas and per-VM limits. k7-api resolves the namespace’s tenant on every request, enforces quotas under a per-tenant Lease, and stamps every object it creates with the tenant and the creating key id. The ValidatingAdmissionPolicy k7-runtime-annotations makes sure only k7-api can set the k7d.katakate.org/* and io.katacontainers.* pod annotations the runtimes act on. See Tenants, roles and OIDC.

Cilium#

Cilium is the default CNI plugin. It enables FQDN-based egress (e.g. allow api.openai.com instead of a CIDR) via CiliumNetworkPolicy. Cilium runs with kubeProxyReplacement=true for a leaner stack. Pass --cni flannel at install time to fall back to K3s’ default CNI (CIDR-only egress).

Kata Containers#

Kata is the OCI runtime that turns each pod into a lightweight VM. We install both runtime classes:

  • kata — Firecracker microVMs (used by kata-firecracker-devmapper)
  • kata-qemu — QEMU-based VMs (used by kata-qemu-longhorn)

Kata guest seccomp is enforced (disable_guest_seccomp = false) on both classes.

The other two runtime classes are not Kata. k7 and k7-fc are both served by the k7d containerd shim (containerd-shim-k7-v1) talking to the node-local k7d daemon; k7-fc differs only by a per-runtime ConfigPath (/etc/k7d/shim-k7-fc.toml, backend = "firecracker") that tells k7d to drive stock Firecracker under the stock jailer instead of its own VMM. See k7d backends.

Backends#

A backend is a tuple of (runtime class, storage driver). There are four:

BackendAliasRuntimeStorageBest for
kata-firecracker-devmapperkfdKata + Firecracker + jailer (kata)LVM thin-pool snapshotterLightest footprint, fastest cold boot, ephemeral workloads
kata-qemu-longhornkqlKata + QEMULonghorn CSI (replicated PVCs)Persistent volumes, disk-level snapshot/pause/resume/fork
k7d—k7d Rust VMM (runtimeClassName: k7)erofs on virtio-blk + host-local state~5 ms warm fork, snapshot trees, RL / agent tree search
k7d-fck7-fck7d daemon driving stock Firecracker + jailer (runtimeClassName: k7-fc)erofs on virtio-blk + host-local stateFirecracker isolation (jailer, seccomp) with k7d fork/tree/pool semantics; multi-tenant --docker

A node can support one or more backends; on a multi-node cluster, different nodes can specialize. See Backends for a deeper comparison.

Storage — Longhorn (kata-qemu-longhorn only)#

Longhorn provides cluster-wide replicated block storage via a Kubernetes CSI driver. Each kata-qemu-longhorn sandbox gets its own root PVC (<sandbox>-root-lh), backed by numberOfReplicas replicas spread across nodes. The default StorageClass uses WaitForFirstConsumer binding and dataLocality: best-effort so the volume is created on (and one replica stays on) the node where the pod is scheduled. VolumeSnapshots power pause/resume/fork.

Storage — devmapper thin-pool (kata-firecracker-devmapper only)#

A raw disk is provisioned as an LVM thin-pool. Containerd’s devmapper snapshotter creates per-sandbox logical volumes from this pool, giving each microVM its own block device while sharing the underlying physical storage efficiently. --docker sandboxes on this backend get an ephemeral graph LV through the k7-docker-lvm StorageClass (OpenEBS LVM LocalPV).

Storage — Docker graph disks (--docker)#

A sandbox created with --docker gets a dedicated block device for Docker’s overlay2 graph: a Longhorn PVC <name>-docker-lh on kql, an LVM LocalPV on kfd, or a per-VM scratch disk on k7d / k7d-fc that is reflinked on fork. See Docker in a sandbox.

Request lifecycle: k7 create#

  1. Client → API: POST /api/v1/sandboxes with a SandboxConfig JSON body. The API authenticates the API key (SHA-256 hash lookup in the Secret snapshot) or OIDC JWT, resolves the role, checks that the route allows it and that the namespace belongs to the caller’s tenant, applies the key’s node scope, checks the image registry host against the public allowlist (SSRF guard), fills limits from the tenant’s defaultSize, refuses limits the backend cannot enforce, takes the tenant’s quota Lease and checks the quota, then forwards to K7Core.create_sandbox. One audit line is written whatever the outcome.
  2. K7Core: builds a Kubernetes Deployment (1 replica, labelled with the tenant, key id and a fresh sandbox-uid), an optional Secret from env_file, an optional PVC (<name>-root-lh) on kata-qemu-longhorn, an optional Docker graph disk, an ingress NetworkPolicy (deny-all, or opened for ingress_ports / ingress_from), an optional NodePort Service for expose_ports, and either a CIDR NetworkPolicy or a CiliumNetworkPolicy for egress (whose DNS rule carries the allowlist’s own matchers). The always-on CiliumClusterwideNetworkPolicy k7-sandbox-platform-deny (installed once per cluster) keeps every sandbox off nodes, the apiserver and platform namespaces regardless of these per-sandbox objects.
  3. K3s scheduler picks a node based on nodeSelector (k7.katakate.org/backend-<backend>=true, set by the installer per host), the tenant’s dedicated nodes (selector + toleration for the k7.katakate.org/tenant taint), and an explicit node_name pin if any.
  4. Kata or k7d (via the appropriate containerd shim) launches a VM, mounts the rootfs (devmapper or Longhorn PVC), and starts the user container inside it.
  5. before_script runs via kubectl exec while egress is still open. Once it completes, the egress NetworkPolicy is applied and the pod is marked Ready.
  6. The API returns 201 Created with a Location header.

Source code map#