Multi-node clusters

Katakate scales horizontally via multi-node K3s clusters, configured with an Ansible-style inventory. This guide explains the inventory format, the HA story, and how to mix backends across nodes.

Why multi-node?#

Horizontal scale — more nodes, more sandboxes.
Replicated storage — Longhorn keeps multiple replicas of every volume across nodes (numberOfReplicas configurable). Data survives a node failure.
Workload locality — pods land on nodes whose backend matches the requested --backend.
High availability of the control plane — embedded etcd across 3+ K3s servers.

Topology at a glance#

A typical 3-server + 1-agent cluster with mixed backends (including all-backend nodes running k7d) and Longhorn replication:

What this shows:

Any server can serve the API NodePort — clients can hit any of node1..3 on :31007 over HTTPS (one cluster CA, /etc/k7/tls/ca.crt on the first master, covers every node IP). If you installed with --api-allow-cidr, the Service switches to externalTrafficPolicy: Local, so only the node actually running the k7-api pod answers — point clients at that node (or run a load balancer in front that health-checks :31007).
etcd quorum spans the three servers. One server can fail without losing the control plane.
Backend labels decide where a sandbox lands. --backend kfd is restricted to node1 and node4; --backend kql to node1, node2, node3; --backend k7d (and k7d-fc, which shares the daemon and adds the k7-fc RuntimeClass) to node1 and node2.
k7d nodes run the node-local k7d daemon (systemd unit, owns every runtimeClassName: k7 and k7-fc microVM on that node) plus its XFS disk pool at /var/lib/k7d/disks. k7d-fc nodes additionally carry the pinned Firecracker + jailer binaries (/usr/local/bin/firecracker, jailer) and /etc/k7d/shim-k7-fc.toml.
The k7-agent DaemonSet runs on every node. The k7-api forwards k7d pause / resume / fork to the agent on the sandbox’s node (shared-token auth, ingress restricted to the k7-api pod by a CiliumNetworkPolicy), so k7d VM ops work through the API regardless of which node hosts the sandbox. It also serves per-node storage-pool stats for k7 nodes storage.
Longhorn replicas of each kata-qemu-longhorn PVC are spread across kql-capable nodes (longhorn_replicas=2 keeps two copies; dataLocality: best-effort keeps one local to the pod).
Devmapper thin-pools and k7d disk pools are per-node and not replicated — kfd sandboxes are ephemeral by design, and k7d fork/snapshot trees are host-local (a fork lands on the source’s node; cross-node fork is on the k7d roadmap).

Inventory format#

k7 install -i inventory.ini reads a standard Ansible INI inventory. Each host declares which backend(s) it supports via k7_backends (comma-separated).

A complete example (src/k7/deploy/inventory.ini.example):

[k7_servers]node1 ansible_host=203.0.113.10 k7_backends=kata-qemu-longhorn,kata-firecracker-devmapper,k7d k7_disk_source=device:/dev/nvme1n1node2 ansible_host=203.0.113.11 k7_backends=kata-qemu-longhorn,k7dnode3 ansible_host=203.0.113.12 k7_backends=kata-qemu-longhorn[k7_agents]node4 ansible_host=203.0.113.13 k7_backends=kata-firecracker-devmapper k7_disk_source=device:/dev/nvme1n1[k7_cluster:children]k7_serversk7_agents[k7_cluster:vars]ansible_user=rootansible_ssh_private_key_file=~/.ssh/id_ed25519longhorn_replicas=2

For a 3-node HA cluster with all backends on every node (kfd, kql, k7d, k7d-fc), one command from the first master brings everything up. Install the 0.5.0 CLI from the PPA or .deb on that master and run k7 install from any directory — the signed k7-api image is pulled from GHCR by digest, so no checkout and no Docker are needed. Fresh nodes have no SSH host keys yet — either ssh-keyscan the peers into known_hosts or set ansible_ssh_common_args as below:

[k7_servers]k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fck7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fck7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d,k7d-fc[k7_cluster:children]k7_servers[k7_cluster:vars]ansible_user=rootansible_ssh_private_key_file=/root/.ssh/id_ed25519ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/nulllonghorn_replicas=3

k7 install -i inventory.ini --ha --api-mode development --k7d-version 0.7.0

--api-mode is required (production or development; see Production and development modes). --k7d-version 0.7.0 is the playbook default and downloads the public GitHub release; another release needs its checksum (k7d_sha256=<hex>), and a tarball you built from k7d source goes through --k7d-artifact /root/k7d-v0.9.0-x86_64-linux.tar.gz. k7 0.4.0+ copies Firecracker v1.16.2 for k7d-fc.

Two-node (one server + one agent, no --ha — embedded etcd needs 3+ servers). After each box has Ubuntu on one NVMe and a raw spare (TWO_DISK):

[k7_servers]k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc[k7_agents]k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc[k7_cluster:children]k7_serversk7_agents[k7_cluster:vars]ansible_user=rootansible_ssh_private_key_file=/root/.ssh/id_ed25519ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/nulllonghorn_replicas=2

k7 install -i inventory.ini --api-mode development --k7d-version 0.7.0

Servers that must not run sandboxes (typical masters in a cluster that has agents) declare k7_backends=none — omitting k7_backends is an error, not an implicit empty set. Tenant isolation comes later with k7 nodes dedicate NAME --tenant ID (see Tenants and roles); do not dedicate the first master.

Don’t pass --backend alongside -i: the per-host k7_backends in the inventory is authoritative (an explicit --backend overrides it).

Note

On dual-NVMe boxes where the OS lives on one disk and the other is a raw spare, omit k7_disk_source (the default spare-disk): NVMe enumeration (nvme0n1 vs nvme1n1) is not stable across reboots, so a hardcoded device can point at the OS disk after a reboot. The playbook auto-detects the empty non-root whole disk. Set it only when a node has several spare disks and you must pick a specific one.

Per-host variables#

Variable	Required	Description
`ansible_host`	yes	SSH target IP or hostname
`ansible_user`	yes	SSH user (commonly `root` on bare-metal)
`k7_backends`	yes	Comma-separated list of backends supported by this node. Values: `kata-qemu-longhorn` (`kql`), `kata-firecracker-devmapper` (`kfd`), `k7d`, `k7d-fc` (`k7-fc`)
`k7_disk_source`	optional	Where the kfd thin-pool and k7d disk image live: `spare-disk` (default — auto-detect the empty non-root whole disk), `device:<path>` (e.g. `device:/dev/nvme1n1`; wipe with `utils/wipe-disk.sh` first), or `loopback` (XFS image only — no kfd thin-pool, so `kfd` is rejected). The legacy `k7_devmapper_disk=<path>` still means `device:<path>`. On dual-NVMe (`TWO_DISK`) boxes leave it unset so a swapped `nvme0n1`/`nvme1n1` cannot point at the OS disk.
`longhorn_extra_disk`	optional, `kata-qemu-longhorn` only	Existing mount path (e.g. `/mnt/longhorn-extra`) to register as an extra Longhorn disk. Not a `/dev/...` device — format and mount it yourself first. Without this, Longhorn uses `longhorn_data_path` (default `/var/lib/longhorn`) on the root filesystem.
`kata_thinpool_pv_size`	optional, `kata-firecracker-devmapper` only	LVM PV size for the kfd thin-pool (default `100G`)
`k7d_disks_image_size`	optional, `k7d` only	Sparse XFS image size for the k7d volume pool at `/var/lib/k7d/disks` (default `32G`)

Group variables (`[k7_cluster:vars]`)#

Variable	Default	Description
`ansible_user`	`root`	SSH user for all hosts
`ansible_ssh_private_key_file`	`~/.ssh/id_rsa`	Private key used for SSH
`longhorn_replicas`	`min(3, node count)`	Longhorn replica count
`longhorn_data_path`	`/var/lib/longhorn`	Longhorn data directory
`k3s_version`	`v1.36.5+k3s1` (playbook pin)	K3s to install on fresh nodes; `k7 install --k3s-version` overrides. k7 never upgrades a running K3s.
`k7_dns_upstreams`	the host’s resolvers	CoreDNS upstream resolvers, 1–3 IPv4 comma-separated (e.g. `1.1.1.1,9.9.9.9`); same as `k7 install --dns-upstream`. See CoreDNS upstreams.
`k7_distro`	set by `k7 install` (`k3s`)	Selects the node-prep adapter (`roles/k7_node_prep/vars/<distro>.yml`); the Helm chart’s installer uses `rke2`, `kubeadm` or `eks-al2023`.

Roles: servers vs agents#

[k7_servers] — K3s control-plane nodes. The first host listed becomes the cluster-init server; subsequent servers join it (HA via embedded etcd).
[k7_agents] — K3s worker nodes. They join the first server with the K3s join token.

Servers can also schedule sandboxes (they aren’t tainted).

Single-master vs HA#

1 server: the simplest setup; no HA. Failure of the server takes the cluster down.
2 servers: avoid — embedded etcd has no fault tolerance with even counts. Better: 1 server + 1 agent.
3+ servers (odd numbers): full HA. Quorum tolerates (N-1)/2 server failures.

Mixed backends per node#

A node can advertise both backends in k7_backends:

node1 ansible_host=... k7_backends=kata-qemu-longhorn,kata-firecracker-devmapper k7_disk_source=device:/dev/nvme1n1

The playbook installs both Kata runtime classes (kata and kata-qemu), merges containerd configs, and labels the node with both:

k7.katakate.org/backend-kata-qemu-longhorn=truek7.katakate.org/backend-kata-firecracker-devmapper=true

When you k7 create --backend kql ..., the scheduler uses nodeSelector: { k7.katakate.org/backend-kata-qemu-longhorn: "true" } to land the pod on a compatible node. If no compatible node exists, k7 detects the FailedScheduling event and returns a clear error listing which backends are actually available.

Longhorn topology and replication#

Longhorn is configured to be topology-aware:

volumeBindingMode: WaitForFirstConsumer — the PVC binds when the pod is scheduled, ensuring the volume is created on the node where the pod will run.
dataLocality: best-effort — Longhorn keeps one replica on the same node as the pod (reduces network reads).
replica-auto-balance: best-effort — replicas auto-spread across nodes.

Practical effect: writing to /mnt/state from inside the sandbox stays on the local node when possible, and the data has redundant replicas elsewhere in the cluster. If the pod’s node fails, the Deployment reschedules onto another node and Longhorn re-attaches the volume from a surviving replica.

Joining nodes incrementally#

You don’t have to install everything in one shot. To add an agent to a running cluster:

# On the new node, after `apt install k7`:k7 install \  --role agent \  --join https://master.example.com:6443 \  --join-token "$(ssh master 'sudo cat /var/lib/rancher/k3s/server/node-token')" \  --backend kql

To add a new server (HA):

k7 install \  --role server \  --join https://master.example.com:6443 \  --join-token "..." \  --ha

For a fresh 3-server HA cluster, prefer the inventory-based install — it handles token distribution and join ordering automatically.

CNI choice across the cluster#

Cilium (the default) is cluster-wide and applies its eBPF datapath uniformly across nodes. FQDN egress works regardless of which node the sandbox lands on. If you pass --cni flannel, every node uses Flannel and FQDN egress is unavailable cluster-wide.

Verification#

After k7 install -i inventory.ini:

# Cluster nodessudo k3s kubectl get nodes -o wide# Per-backend labelssudo k3s kubectl get nodes --show-labels | grep katakate# Cilium healthsudo k3s kubectl -n kube-system get pods -l k8s-app=cilium# Longhornsudo k3s kubectl -n longhorn-system get nodes,replicas# k7-api pod + per-node k7-agent DaemonSetsudo k3s kubectl -n kube-system get deploy/k7-api ds/k7-agent -o widek7 api status# Per-node storage-pool utilization (thin-pool + k7d disk pool, via the agents)k7 nodes storage

A clean cluster will show all nodes Ready, Cilium pods Running, the k7-api Deployment with 2/2 ready (two replicas behind a PodDisruptionBudget since 0.5.0), and Longhorn replicas distributed across nodes. k7 nodes list shows every node with its backends and tenant dedication.

Helm chart for existing clusters#

Since 0.5.0, deploy/helm/k7 installs K7 on a Kubernetes cluster you already run — validated on RKE2 v1.36.5 + Cilium 1.19.2, with kubeadm and eks-al2023 node-prep adapters. The chart deploys k7-api, k7-agent, the k7-node-installer DaemonSet, the RuntimeClasses, snapshot GC and the Cilium policies; it does not install the CNI, Longhorn, the CSI snapshot controller or OpenEBS LVM LocalPV (kfd --docker) — those are the cluster’s own. values.schema.json rejects unknown keys, and several values are required with no default:

Value	Choices	Notes
`networkPolicy.provider`	`cilium` / `kubernetes` / `none`	`cilium` gives FQDN + CIDR egress and the sandbox→platform deny (pre-install hook requires the DNS proxy with `dnsRejectResponseCode=nameError`). `kubernetes` is CIDR-only (domain egress → 400) and needs `acknowledgeNoPlatformDeny: true`; `none` also needs `acknowledgeNoEgressPolicy: true`.
`tls.mode`	`certManager` / `secret` / `insecureHTTP`	`insecureHTTP` is refused with `api.mode=production`.
`api.mode`	`production` / `development`	Same semantics as `k7 install --api-mode`.
`nodeInstaller.distro`	`k3s` / `rke2` / `kubeadm` / `eks-al2023`	Node-prep adapter for nodes labelled `k7.katakate.org/node-pool=true`.
`nodeInstaller.diskSource`	`spare-disk` / `loopback` / `device:/dev/<disk>`	Same meaning as the inventory’s `k7_disk_source`.
`images.*.digest`	`sha256:<64 hex>`	The release pipeline fills them (`images.lock.yaml`); the chart is not published to an OCI registry yet — install from the checkout.
`backends.*.enabled`	at least one	`kql` also needs `storageClass` and `volumeSnapshotClass`.
`observability.scraperSelector`	namespace + pod labels	Required while `observability.metrics` is on (default); only those pods may reach port 9464.
`metering.sampleInterval`	5–60 s	Required with `api.mode=production`.

Other notable values: api.replicas (2), api.service.type (ClusterIP / NodePort / LoadBalancer), api.remoteInstall (development mode only), nodeInstaller.allowServerRestart (the installer restarts containerd, which restarts the apiserver on server nodes — refused unless true), nodeInstaller.wipeDisks / forceCleanup for uninstall, observability.serviceMonitor / prometheusRule / grafanaDashboards, auth.oidc.* and metering.* (same documents as the k7 install flags). The release namespace runs the privileged installer, so label it pod-security.kubernetes.io/enforce=privileged. Hooks refuse a cluster that cannot enforce the values, create the agent-token and API-key Secrets, and undo node prep on uninstall; helm test creates, execs and forks one sandbox per backend. Pass helm a generous --timeout (e.g. 35m) — node prep is bounded by hookTimeoutSeconds.

kubectl label ns k7-system pod-security.kubernetes.io/enforce=privilegedkubectl label node worker-1 k7.katakate.org/node-pool=truehelm install k7 deploy/helm/k7 -n k7-system --create-namespace --timeout 35m \  --set networkPolicy.provider=cilium --set tls.mode=certManager \  --set tls.certManager.issuerRef.name=letsencrypt --set 'tls.certManager.dnsNames={k7.example.com}' \  --set api.mode=production --set metering.sampleInterval=15 \  --set nodeInstaller.distro=rke2 --set nodeInstaller.diskSource=spare-disk \  --set backends.kfd.enabled=true --set backends.k7d.enabled=true \  --set observability.scraperSelector.namespace=monitoring \  --set 'observability.scraperSelector.podLabels.app\.kubernetes\.io/name=prometheus'helm test k7 -n k7-system --timeout 35m

The full RKE2 walk-through (host packages, RKE2 without kube-proxy, Cilium, Longhorn, OpenEBS, TLS, chart, upgrade, uninstall) is deploy/helm/k7/ci/rke2-hetzner.md; the values file it uses is ci/rke2-cilium-values.yaml.

Remote node install through the API#

POST /api/v1/install lets a cluster-wide admin add nodes without SSH from the controller. It is off by default (404) and development mode only — k7 install --api-mode production --api-remote-install (and the chart with api.mode=production + api.remoteInstall=true) refuse the combination before deploying.

k7 install --backend kfd,k7d --api-mode development --api-remote-install \  [--node-installer-image local|ghcr.io/katakate/k7-node-installer@sha256:…] \  [--node-installer-allow-server-restart]

The API never runs Ansible (its image has none): the flag deploys the k7-node-installer DaemonSet in k7-system, configured from the first master’s distro, backends and disk settings, with its image pinned at deploy time (default from images.lock.yaml). A request is exactly {"nodes": ["worker-3"]} — any other key (playbook, image, inventory, config) is a 400, an unknown node 404, a K3s server node without --node-installer-allow-server-restart 409, a drifted DaemonSet image 500. The API labels the nodes k7.katakate.org/node-pool=true and returns 202 with an operation id; GET /api/v1/install/{id} reports each node pending / running / succeeded / failed. In the SDK: Client.install(nodes) and Client.install_status(id) (the pre-0.5.0 playbook-upload form is gone). Running k7 install again without --api-remote-install converges: pool labels, installer DaemonSet, RBAC, admission policies and the operation records are removed (unless Helm owns them). Endpoint details: Tenants, nodes and install.

Reference#

Example inventory: src/k7/deploy/inventory.ini.example
Ansible playbook: src/k7/deploy/k7-install-node.yaml