Tenants, roles and OIDC

K7 0.5.0 turns the API into a multi-tenant control plane. Three things changed from 0.4.0: every credential now has a role; namespaces are grouped into tenants that carry quotas, defaults and dedicated nodes; and humans can sign in with OIDC instead of holding a key. Keys moved from /etc/k7/api_keys.json on a node to the kube-system/k7-api-keys Secret, so every API replica sees the same store.

Roles#

One table (PERMISSIONS in src/k7/api/authz.py) maps every route to its roles; a route missing from it answers 500 and fails CI. Checks compose — route role, then tenant, then the key’s namespace/node narrowing — and the first failure is a 403 that names the missing role or scope. k7 whoami / GET /api/v1/auth/whoami shows what a credential resolves to.

Tenants#

A Tenant (tenants.k7.katakate.org/v1alpha1, cluster-scoped) claims namespaces and owns everything a tenant key may do:

k7 tenant create acme -n acme-dev -n acme-prod \  --default-size cpu=1,memory=2Gi,disk=10Gi \  --quota sandboxes=20,vcpu=16,memory=32Gi,snapshotStorage=200Gi \  --max-size cpu=8,memory=32Gi \  --network egressBps=100M,ingressBps=100Mk7 nodes dedicate k7-node-01 --tenant acmek7 generate-api-key acme-ci --role user --tenant acmek7 tenant list        # quotas and reconciled usage

• Namespaces are created if missing and labelled k7.katakate.org/tenant. A tenant key reaches every namespace of its tenant (intersected with its own -n list), read from the Tenant on every request — a namespace added later works without reissuing the key, and a key whose tenant is deleted reaches nothing. A namespace claimed by two Tenants (only possible by writing CRs by hand) fails every request there with 500 TenantConflict.
• Dedicated nodes (spec.nodes, or k7 nodes dedicate): the tenant’s sandboxes land only there (node selector + toleration), and the NoSchedule taint keeps everyone else off. This is what gives tenant isolation on k7d, whose daemon is shared per node — a key’s --node pin alone restricts that key, not the others.
• defaultSize (cpu, memory, disk) fills creates without limits. Production mode requires CPU and memory on every sandbox, so set it or make callers pass limits.
• Quotas (sandboxes, vcpu, memory, disk, snapshotStorage) are checked at K7’s two choke points — the sandbox Deployment create and the VolumeSnapshot create — so create, fork, restore, pause-with-snapshot and named snapshots are all covered. The check and the create run under a per-tenant Lease (k7-quota-<tenant>), so concurrent creates on any number of API replicas cannot overshoot; a Lease not acquired in 5 s is 503 and creates nothing. Over-quota is 403 QuotaExceeded with the dimension, limit, usage and request, and is audit-logged. Usage is reserved, not used: a paused sandbox keeps its vCPU, memory and disk until deleted. The leader replica writes Tenant.status.usage every 60 s.
• Per-VM limits : maxSandboxSize caps a request (400 above it). network (egressBps / ingressBps, bits/s) is enforced by the hypervisor — Firecracker token buckets on kfd, tc on the tap for kql — so it works whatever the CNI; a request may lower a cap, never raise it. A backend that cannot enforce a configured limit refuses the create with 400 LimitNotEnforceable instead of running unlimited: diskIO on every backend (no block rate limiter in Kata 3.24 or k7d yet), network on k7d / k7d-fc, and a disk quota or maxSandboxSize.disk on k7d / k7d-fc (K7 cannot read a k7d VM’s writable-disk size).
• Enforcement is at the K7 API. kubectl apply by a cluster admin bypasses it. For a single-namespace tenant a Kubernetes ResourceQuota (k7-tenant-quota) is the backstop; multi-namespace tenants have none, since a ResourceQuota cannot span namespaces.
• Migrating from 0.4.0 namespace-scoped keys: k7 tenant adopt --from-keys proposes one tenant per connected group of key namespaces (dry run), --apply creates them and binds the keys.

API keys#

k7 generate-api-key NAME --role read-only|user|operator|admin [--tenant T] [-n NS]... [--node NODE]... [--expires-days 365]k7 list-api-keys          # role, tenant, scopes; warns on unscoped_legacyk7 revoke-api-key NAME

Tokens are k7_<id>_<secret>; the store holds sha256(secret) (the secret is 256 random bits, so a slow KDF adds nothing) in the kube-system/k7-api-keys Secret. The API never writes the store on authenticate: it serves a snapshot refreshed every 5 s and answers 503 rather than trust one older than 60 s, so revocation reaches every replica within one refresh. Key commands edit the Secret through kubectl (run them on a server node), or go through k7-api with --api using an admin credential (POST /api/v1/keys). Protect etcd (--secrets-encryption on K3s, KMS on managed clusters): anyone with get secrets in kube-system reads the hashes.

Upgrading from 0.4.0: k7 install runs k7 api keys migrate, which copies /etc/k7/api_keys.json into the Secret and gives every pre-role key a role: admin: true → admin, namespace-scoped → user (still narrowed), unscoped → admin flagged unscoped_legacy. Such keys keep working in development mode; re-issue them with a real role before switching to production.

OIDC login#

Install with all five settings (or none; https:// issuers only):

k7 install ... \  --oidc-issuer https://idp.example.com/realms/k7 \  --oidc-audience k7-api \  --oidc-client-id k7-cli \  --oidc-groups-claim groups \  --oidc-username-claim email

Then k7 login (device flow; --browser for Authorization Code + PKCE on a loopback redirect) caches the tokens in ~/.config/k7/credentials.json (0600) and refreshes them itself; k7 logout deletes the cache and revokes at the IdP. Clients send Authorization: Bearer <JWT>; k7-api validates statelessly — signature against the issuer’s JWKS (cached --oidc-jwks-ttl seconds, default 3600; a failed refresh is 503, never a stale allow), alg RS256/ES256 only, iss, aud, exp (60 s leeway) and sub required, email_verified: true when the username claim is email. k7-api stores no sessions and never sees refresh tokens. Register k7-cli as a public client with the device grant enabled. Per-IdP claim settings (Keycloak, Entra ID, Okta, Auth0, Dex) are in docs/HARDENING.md.

Role bindings#

Groups, usernames or subjects map to roles through ConfigMap k7-role-bindings, read on every request:

# k7 role-bindings apply -f bindings.yamlbindings:  - {group: k7-admins, role: admin}  - {group: sre, role: operator}  - {group: acme-devs, role: user, tenant: acme}  - {group: acme-viewers, role: read-only, tenant: acme}deny_subjects: []   # OIDC `sub` or username; blocks on the next request

A subject in several groups gets the highest role per tenant plus the highest cluster role. An unbound subject gets a 403 that echoes its username and groups, to debug claim shapes. A write that leaves no admin binding while no unexpired admin key exists is refused. A JWT is valid until exp — configure 1 h or shorter tokens at the IdP; deny_subjects is the immediate stop. Keep one break-glass admin key offline: keys keep working during an IdP outage, and k7 generate-api-key with the node kubeconfig always works.

Production and development modes#

--api-mode is chosen at install and never defaulted (chart: api.mode). Every response carries X-K7-Mode; /health and k7_api_info{mode} report it.

Switch a development cluster to production by re-running k7 install --api-mode production … once every key has been re-scoped (k7 list-api-keys shows none flagged unscoped_legacy).

Checklist#

1. Migrate keys (k7 api keys migrate runs on install) and re-issue every unscoped_legacy key with the narrowest real role.
2. Give machines tenant keys with --expires-days; give humans OIDC.
3. Create tenants with defaultSize and quotas; dedicate nodes for tenants that must not share a k7d daemon.
4. Bind groups, not people; keep a break-glass admin key offline.
5. Ship the audit log (k7-api stdout, logger k7.audit).
6. Install with --api-mode production.

Reference: SECURITY.md, docs/HARDENING.md, API: Identity, Tenants.