Networking

Katakate gives every sandbox its own VM-level network namespace and applies opinionated network policies on top. This page explains the model.

CNI plugin: Cilium by default#

By default, K3s is installed with Flannel disabled (--flannel-backend=none --disable-network-policy) and Cilium as the CNI. Cilium runs with kubeProxyReplacement=true (no kube-proxy), and its eBPF datapath attaches to the host-side veth — including for Kata VMs, whose virtio NICs are bridged to a host veth.

# Cilium is the defaultk7 install --backend k7d --api-mode development                  # Cilium# Opt out (CIDR-only egress, no FQDN policies, no platform deny)k7 install --backend k7d --api-mode development --cni flannel

The Helm chart, which installs onto a cluster you already run, exposes the same choice as networkPolicy.provider: cilium (FQDN + CIDR egress, platform deny, agent ingress policy; the pre-install hook requires Cilium’s DNS proxy with dnsProxy.dnsRejectResponseCode=nameError), kubernetes (v1 NetworkPolicy, CIDR egress only — a domain entry is 400), or none (no egress policy; any egress field is 400). The weaker providers need explicit acknowledge* values.

When the cluster is on Cilium, k7 uses CiliumNetworkPolicy for FQDN egress. On Flannel it falls back to the standard Kubernetes NetworkPolicy with CIDR-only ipBlock rules.

Sandboxes never reach the platform (always on, Cilium)#

A cluster-wide, deny-only CiliumClusterwideNetworkPolicy named k7-sandbox-platform-deny is installed by k7 install. It stops every sandbox — in every egress mode, including --egress-open — from reaching:

• the node it runs on and the other nodes (SSH, kubelet, k3s ports),
• the Kubernetes API server (10.43.0.1:443),
• link-local / cloud metadata (169.254.0.0/16),
• the kube-system and longhorn-system pods — k7-api, k7-agent, Longhorn — except CoreDNS, which stays reachable so FQDN egress keeps working.

It is deny-only (enableDefaultDeny: false), so it never narrows what a whitelist allows beyond those targets, and it cannot turn an open sandbox into a default-deny one. Cilium-only: --cni flannel clusters do not get this isolation.

Ingress: denied by default, opt-in per sandbox#

Every sandbox gets a deny-all {name}-deny-ingress NetworkPolicy. Practical implications:

• Sandbox-to-sandbox networking is denied — even within the same namespace — until you open a port.
• kubectl exec and k7 shell still work — they go through the Kubernetes API, not pod networking.
• kube-system traffic is allowed — needed for cluster services (DNS, metrics, etc.).

To accept connections, open TCP ports and say who may connect:

Sources:

Anything else is a hard error at create time. --ingress-from without --ingress-port is an error too: sources alone open nothing. --ingress-port with no --ingress-from means “other sandboxes in this namespace”, never the world — reaching a sandbox from anywhere requires --ingress-from cidr:0.0.0.0/0, typed out in full. These are stock v1 NetworkPolicy rules, so ingress works on --cni flannel as well.

A sandbox is a bare Deployment with no Service, so ingress rules only cover in-cluster traffic. To reach a sandbox from outside the cluster, expose it.

Exposing a sandbox outside the cluster#

--expose-port <port> (repeatable; expose_ports in YAML / API) creates a NodePort Service named {name}-expose in the sandbox’s namespace. k7 create prints the resolved http://<node-ip>:<nodeport> for each port, and k7 list shows the allocated NodePorts.

k7 create web python:3.12-slim \  --backend k7d --egress-open \  --ingress-port 8000 --ingress-from cidr:0.0.0.0/0 --expose-port 8000 \  --before-script 'nohup python3 -m http.server 8000 >/tmp/http.log 2>&1 &'# 🌐 Port 8000 exposed at http://<node-ip>:31234# That URL is the node *running* the pod. python:3.12-slim does not listen# on 8000 by itself. On k7d/kfd the container pid 1 is `sleep 365d`, so a# server started from --before-script must be `nohup … &` or it dies with# the script's shell. `--cmd` only replaces pid 1 on kql.

• Every --expose-port needs a matching --ingress-port. k7 refuses otherwise: a public NodePort in front of a deny-all policy is a confusing no-op, and one in front of an accidentally-open policy is a breach.
• The Service uses externalTrafficPolicy: Local, so the client’s real source IP reaches the pod and --ingress-from cidr:… rules mean something. Consequence: the NodePort only answers on the node running the sandbox (k7 list shows which one).
• A cidr: allowlist does not constrain clients on the cluster’s own nodes; verify it from a genuinely external client — curl from a node passes no matter how the rule is written.
• On a bare-metal host the node IP is public, so --expose-port with --ingress-from cidr:0.0.0.0/0 publishes an untrusted sandbox to the internet. k7 warns on stderr on every such create.
• k7 delete removes the Service. A k7 fork inherits the source’s ingress and egress policy but not its Service — a fork is not reachable from outside until you expose it.

Egress control#

A sandbox has exactly one of three egress modes:

--egress-open and --egress are mutually exclusive. Defaults are asymmetric (kept for compatibility): the CLI without flags is block-all; an API/YAML request that omits egress_whitelist is open.

How FQDN egress works under Cilium#

When the whitelist contains FQDNs, k7 creates a CiliumNetworkPolicy like:

apiVersion: cilium.io/v2kind: CiliumNetworkPolicymetadata:  name: <sandbox>-egressspec:  endpointSelector:    matchLabels:      katakate.org/sandbox: <sandbox>  egress:    # Cluster DNS, with rule that observes lookups for FQDN matching    - toEndpoints:        - matchLabels:            "k8s:io.kubernetes.pod.namespace": kube-system            "k8s:k8s-app": kube-dns      toPorts:        - ports: [{ port: "53", protocol: UDP }, { port: "53", protocol: TCP }]          rules:            dns:                      # the same matchers as toFQDNs — nothing else resolves              - matchName: "api.openai.com"              - matchPattern: "**.huggingface.co"    # FQDN allowlist    - toFQDNs:        - matchName: "api.openai.com"        - matchPattern: "**.huggingface.co"    # Optional CIDR additions    - toCIDR:        - "10.0.0.5/32"

Wildcards (*.example.com) are supported via Cilium’s matchPattern. Bare exact names use matchName. A leading *. covers subdomains at any depth (registry.docker.com and production.cloudfront.docker.com) but not the apex docker.com — add the apex as its own entry. Under the hood k7 maps *. to Cilium’s multi-label **. matchPattern so CDN-backed registries work. The install also sets Cilium dnsProxy.minTtl=3600 so clients that cache a resolved IP longer than a short CDN TTL keep the FQDN→IP allowance for an hour.

No DNS exfiltration (0.5.0). The DNS rule to kube-dns carries the allowlist’s own matchers, not matchPattern: "*" as in 0.4.0. Cilium’s DNS proxy forwards only allowlisted names and answers every other query itself with NXDOMAIN — the query never reaches CoreDNS or its upstream, so a sandbox cannot tunnel data out through lookups for names it may not reach. A program that resolves a forbidden name now fails at the lookup instead of at connect; in-cluster names (*.svc.cluster.local) resolve only if allowlisted. The install sets dnsProxy.dnsRejectResponseCode=nameError (the default REFUSED makes musl abandon its ndots:5 search list) and k7 install applies it to existing clusters through cilium upgrade; k7-api rewrites older sandbox policies at startup.

When DNS is blocked#

If the whitelist is [] or contains only CIDRs, DNS resolution is denied by the policy. To work around this:

• Use an FQDN allowlist on Cilium — names you list resolve, nothing else does.
• Pre-resolve names in before_script (which runs before the egress policy is applied).
• Embed your dependencies into the image rather than fetching them at runtime.
• Whitelist your own internal DNS server / egress proxy and resolve there.

Bandwidth caps#

network_limits (--egress-bps / --ingress-bps, bits per second) and a tenant’s limits.network cap a sandbox’s bandwidth per VM at the hypervisor — Firecracker token buckets on kfd, tc on the tap for kql — so they work whatever the CNI and in both directions. A request may lower a tenant cap, never raise it. Firecracker’s bucket starts full, so the first second can burst to twice the cap; the 30-second average stays within 10 %. k7d and k7d-fc have no limiter yet: a cap configured for them is refused with 400 LimitNotEnforceable rather than ignored. See Tenants.

CoreDNS upstreams#

By default CoreDNS forwards to the node’s own resolvers (k3s’s --resolv-conf points at systemd-resolved’s upstream list). If the provider’s resolvers degrade, every pod’s lookups fail with them — image pulls, FQDN egress, and POST /api/v1/sandboxes, which then returns 503 “Cannot resolve registry host”. Pin them instead:

k7 install ... --dns-upstream 1.1.1.1 --dns-upstream 9.9.9.9    # inventory: k7_dns_upstreams under [k7_cluster:vars]

1 to 3 IPv4 addresses (the kubelet passes at most 3 and silently drops the rest); IPv6, 0.x and 127.x are refused before any host is touched. The playbook writes /etc/k7/resolv.conf, sets K3S_RESOLV_CONF, and converges an existing cluster on re-run; removing the variable restores the host’s resolvers. Side effect: every dnsPolicy: Default pod gets the same resolvers. The Helm chart does not manage CoreDNS — configure upstreams in your distribution (on RKE2, the rke2-coredns HelmChartConfig).

Network policies created per sandbox#

For each sandbox, k7 creates:

1. A {name}-deny-ingress NetworkPolicy (always; --ingress-port adds allow rules to this same object).
2. Either:
   • A NetworkPolicy with CIDR-only ipBlock egress rules (when only CIDRs are listed, or the cluster runs Flannel), or
   • A CiliumNetworkPolicy for FQDN+CIDR egress (when FQDNs are present and Cilium is installed).
3. A {name}-expose NodePort Service, only with --expose-port.

On top of these, the cluster-wide k7-sandbox-platform-deny policy applies to every sandbox, and the k7-metrics policy keeps port 9464 of k7-api / k7-agent to the scraper you named (Observability). All per-sandbox objects are deleted with the sandbox. A k7 fork inherits the source’s ingress and egress configuration and is rolled back if its policies cannot be created — a fork of a locked-down sandbox never comes up open. On k7d 0.9.0+, the forked VM also keeps a working NIC (its own bridge, NAT’d into the fork pod’s netns), so the inherited policy actually has traffic to govern.

Only k7-api may set the runtime annotations these policies rely on: the ValidatingAdmissionPolicy k7-runtime-annotations rejects k7d.katakate.org/* and io.katacontainers.* annotation changes by anyone else, so a namespace admin cannot move a pod’s VM onto a shared k7d bridge and bypass Cilium egress.

Administrative access#

kubectl exec, k7 shell, the API’s /exec endpoint, and k7 logs all bypass network policies because they ride the Kubernetes API. Pod networking restrictions never affect operator access.

Restricting who can reach k7-api#

k7-api is a NodePort Service on 31007, served over HTTPS by default (see Security model). On a dedicated box the node IP is public. k7 install --api-allow-cidr 203.0.113.4/32 --api-allow-cidr 198.51.100.0/24 (repeatable, Cilium only — --cni flannel fails at the CLI) renders a CiliumNetworkPolicy k7-api-ingress on the API pod and flips the Service to externalTrafficPolicy: Local so the pod sees the real client IP. Both are skipped entirely when the flag is unset, which is the default.

This is defence in depth for operators who know their client CIDRs; it stacks with TLS and does not replace it. Things worth knowing before you debug it:

• With Local, the NodePort answers only on the node running the k7-api pod — the first master, which is the address the playbook writes into /etc/k7/api_endpoint.
• A curl from a node is not a test of the allowlist. Node-local traffic matches Cilium’s host entity and is SNAT’d to the pod network, so it succeeds however wrong the CIDR list is. Test from off-cluster.
• A wrong CIDR only costs API access: the policy selects the k7-api pod endpoint, never the host, so SSH, kubectl, and on-node k7 --core keep working.
• Rollback: re-run k7 install without the flag, or delete ciliumnetworkpolicy/k7-api-ingress in kube-system and patch the Service back to externalTrafficPolicy: Cluster.

Debugging policy drops (Hubble)#

Hubble is opt-in observability, not enforcement — off by default, not a security control. Enable it with k7 install --hubble (Cilium only; --hubble --cni flannel fails loudly). It installs Hubble Relay and a pinned hubble CLI; the Hubble UI is not installed (no authentication, public node IP). Rollback: cilium hubble disable.

export KUBECONFIG=/etc/rancher/k3s/k3s.yamlcilium hubble port-forward &     # kube-system/hubble-relay → 127.0.0.1:4245hubble statushubble observe --verdict DROPPED --last 50hubble observe --namespace <ns> --verdict DROPPED --last 50cilium fqdn cache list           # what Cilium learned for FQDN rulescilium ip list                   # which IPs carry an fqdn: identity

When a sandbox with an FQDN whitelist cannot reach a host, the symptom is always a hang — policy drop, DNS, MTU, and a dead upstream look the same from inside the VM. A Policy denied DROPPED verdict with the destination still labelled world means the FQDN→identity mapping missed (wildcard too narrow, or the learned IP expired).

Multi-node considerations#

• Cilium routes pod traffic across nodes natively (no Flannel VXLAN).
• CiliumNetworkPolicy applies cluster-wide regardless of which node the sandbox runs on.
• Longhorn replication traffic between nodes is not affected — it goes through the host network (Cilium’s host-level rules), not the sandbox’s policy.