> ## Documentation index
> Fetch the complete documentation index at: https://docs.katakate.org/llms.txt
> Ask it with GET https://docs.katakate.org/ask?q=<keywords or question>. Short keywords rank best.
> Use these to discover all available pages before exploring further.

# Observability and metering

> Prometheus metrics, JSON logs with request ids, dashboards and alerts, the opt-in monitoring stack, and per-tenant usage metering with showback

Since 0.5.0, `k7-api` and `k7-agent` export Prometheus metrics, log JSON lines, and measure what every tenant and sandbox uses. K7 ships Grafana dashboards, alert rules with a runbook per alert, and `k7 install --monitoring` can install a monitoring stack for a single-box install. Nothing is priced: rating and invoices belong to whatever consumes the usage events.

## Metrics

Both processes serve `/metrics` (and nothing else) as plain HTTP on port **9464**. Only the pods you name may reach that port:

| Install | Setting |
|---|---|
| `k7 install` | `--metrics-scraper <namespace>:<label>=<value>[,…]` — without it nobody reaches 9464 |
| `k7 install --monitoring` | the stack's Prometheus pods, set for you |
| Helm chart | `observability.scraperSelector` (`namespace`, `podLabels`), required while `observability.metrics: true` |

On Cilium this is the CiliumNetworkPolicy `k7-metrics` (host and other nodes are refused too); other CNIs get `NetworkPolicy` objects `k7-api-metrics` / `k7-agent-metrics`.

k7-api, each replica: `k7_http_requests_total` / `k7_http_request_duration_seconds` (`route` template, `method`, `status`), `k7_operation_duration_seconds` / `k7_operation_errors_total` (`op`, `backend`, `outcome` / `error_code`), `k7_auth_failures_total` (`reason`: `missing`, `invalid`, `expired`, `forbidden`, `oidc_invalid`), `k7_key_store_last_sync_timestamp_seconds`, `k7_kube_api_errors_total`, `k7_quota_rejections_total` (`tenant`, `dimension`), `k7_sandboxes` (`tenant`, `backend`, `state`; leader only, so `sum()` is the cluster count), `k7_api_info` (`version`, `mode`), `k7_tls_cert_expiry_timestamp_seconds`.

k7-agent, each node: `k7d_rpc_duration_seconds` / `k7d_rpc_errors_total` (`op`), `k7d_daemon_up`, `k7_storage_pool_bytes` (`pool`, `kind`), `k7_storage_report_age_seconds`, `k7_vm_count` (`backend`, `state`), plus the usage counters below.

No series carries a sandbox name, uid or pod (a unit test enforces this): 2 replicas, 20 nodes and 500 tenants come to at most ~17k series. Per-sandbox detail is in `k7 top` and the usage events.

## Logs and request ids

k7-api, k7-agent and the snapshot-gc Job write one JSON object per line on stderr (`K7_LOG_FORMAT` / `K7_LOG_LEVEL`, set by the manifests; chart `observability.logLevel`): `ts`, `level`, `logger`, `msg`, plus `request_id`, `tenant`, `namespace`, `sandbox`, `backend`, `op`, `duration_ms`, `error_code`. Tokens and JWTs are replaced with `[REDACTED]`; no level logs request bodies or exec output.

k7-api accepts a well-formed `X-Request-ID` (8–64 chars of `[A-Za-z0-9-]`) or mints a UUIDv7, returns it in the response and sends it to the k7-agent that runs the VM op, so one id follows a request end to end. Per request you get one access line, an audit line for mutating requests (`k7.audit`, see [Security & networking](/k7/api/security#audit-log)) and a `k7.op` summary per lifecycle op. A line is 400–600 bytes; K3s keeps 10 MiB × 5 files per container, so ship them with your collector or raise the kubelet log limits.

## Dashboards, alerts, runbooks

Sources live in `src/k7/deploy/observability/`: dashboards *API overview*, *Sandbox lifecycle*, *Nodes and storage*, *Usage by tenant*; alert rules in `rules/k7.rules.yaml` with thresholds as `k7:threshold:*` recording rules (`promtool test rules` covers one firing and one quiet case per alert). Alerts: `K7APIDown`, `K7APIHighErrorRate`, `K7OperationSlow`, `K7KeyStoreStale`, `K7AgentDown`, `K7dDaemonDown`, `K7StoragePoolFilling`, `K7StorageReportStale`, `K7SnapshotGCFailing`, `K7UsageSpoolOverflow`, `K7UsageSinkStopped`, `K7TLSCertExpiringSoon`. Every alert links to `docs/runbooks/<alert>.md`, which starts with the first three commands to run.

Chart values: `observability.serviceMonitor` (Service `k7-api-metrics`, ServiceMonitor and PodMonitor), `observability.prometheusRule`, `observability.grafanaDashboards`, `observability.monitorLabels`, `observability.thresholds.*`.

### `k7 install --monitoring`

Opt-in and meant for single-box installs (existing clusters keep their own monitoring and use the chart values). It installs on the first master the pinned, checksummed kube-prometheus-stack 92.3.0 in namespace `monitoring` with 7 days of retention, the K7 ServiceMonitor/PodMonitor, the PrometheusRule `k7`, the dashboard ConfigMaps, a `/metrics` policy admitting that Prometheus, and sets `metering.prometheusUrl` so `k7 usage` works. `--metrics-scraper` cannot be combined with it; `--no-api` is refused.

```bash
kubectl -n monitoring get secret kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Re-running `k7 install` without `--monitoring` leaves the stack but reverts the `/metrics` policy to `--metrics-scraper` (by default nobody), so its K7 targets go down. Removal steps are in [docs/OBSERVABILITY.md](https://github.com/Katakate/k7/blob/main/docs/OBSERVABILITY.md#k7-install---monitoring).

## Metering

| Meter | Unit | Measured by | Dimensions |
|---|---|---|---|
| `sandbox_seconds` | s | node sampler (k7-agent) | tenant, namespace, sandbox, sandbox_uid, backend, state, size class |
| `sandbox_vcpu_seconds` | reserved vCPU·s | node sampler | same |
| `sandbox_memory_gib_seconds` | reserved GiB·s | node sampler | same |
| `sandbox_disk_gib_seconds` | provisioned GiB·s | storage sampler (k7-api leader) | tenant, namespace, sandbox, backend, state |
| `snapshot_gib_seconds` | GiB·s | storage sampler | tenant, namespace, snapshot, kind, location |
| `sandbox_egress_bytes` | bytes | node sampler | tenant, namespace, sandbox, backend, scope |

- **Reserved, not used.** vCPU and memory are the container limits stamped on the Deployment at create / fork / restore. Actual CPU and memory stay in `k7 top`. Size class is `<cpu>c-<memGi>g`, or `unsized` for a sandbox without both limits.
- **States**: `running` accrues everything; `paused` (a k7d VM frozen in place) accrues seconds, GiB·s and disk but **0 vCPU·s**; `stopped` (kql/kfd scaled to 0) accrues disk only.
- **Node sampler**: every `sampleInterval` seconds (5–60; 15 in development, explicit in production) k7-agent lists the node's ready sandbox pods with `crictl` and their eth0 `txBytes`. Windows are closed on the monotonic clock; k7d pause/resume transitions are recorded and split windows exactly; a pod deleted between samples is charged to the midpoint and flagged `approximate`. Time no sampler saw becomes a **gap event** (`metering-start`, `agent-restart`, `leader-change`) — nothing is extrapolated, and no window is emitted twice.
- **Storage sampler**: the k7-api replica holding Lease `k7-metering-leader` runs every `storageInterval` seconds (default 60) over PVCs, ephemeral claims, the 10 GB kfd devmapper root, and every VolumeSnapshot. Not metered yet because their sources are not in K7 yet: k7d writable disks (reported in `/health` as `disk_unavailable`, never as 0) and k7d-native snapshots.
- **Events** are CloudEvents 1.0 JSON (`io.katakate.k7.sandbox.usage.v1`, `.sandbox.usage.gap.v1`, `.egress.usage.v1`, `.disk.usage.v1`, `.snapshot.usage.v1`) with a deterministic `id = sha256(cluster-id, uid, meter, window_start)`, so a replay after a crash deduplicates downstream. The JSON Schemas (`src/k7/metering/schemas/`, `v1` frozen) are the contract. `cluster-id` lives in ConfigMap `k7-cluster-id`, written once at install — keep it in your backups.

### Spool and delivery

Events are appended to `/var/lib/k7/usage-spool/` on each node as fsynced JSON-lines segments; each sink keeps its own offset, so delivery is **at least once**. The spool is capped (`--metering-spool-size`, 1 GiB): past it the oldest segment is dropped, `k7_usage_events_dropped_total` grows, k7-agent `/health` reports `degraded` and `K7UsageSpoolOverflow` fires. `k7 nodes storage` shows each node's spool fill. Sandboxes are never blocked by metering unless you ask: `--metering-require-delivery` makes k7-api steer creates off nodes whose spool is above `--metering-spool-high-water` (80 %) and refuse them with 503 `UsageSpoolFull` when no node qualifies.

### Sinks

**Prometheus** is always on: `k7_usage_sandbox_seconds_total`, `k7_usage_vcpu_seconds_total`, `k7_usage_memory_gib_seconds_total`, `k7_usage_egress_bytes_total` (tenant × namespace × backend × state) from k7-agent, and the gauges `k7_usage_disk_bytes` / `k7_usage_snapshot_bytes` from the k7-api leader. Opt-in sinks are a YAML list (`--metering-sinks-file`, chart `metering.sinks`):

```yaml
- name: openmeter
  type: http
  url: http://openmeter.openmeter.svc:8888
  preset: openmeter          # POST <url>/api/v1/events, drops null data keys
  mode: batch                # application/cloudevents-batch+json
  batchSize: 100
  headersSecretRef: {name: k7-metering-headers, key: openmeter}
- name: logs
  type: stdout               # one event per line on the k7.usage logger
```

`http` backs off on transport errors, 429 and 5xx while the spool keeps the events; any other 4xx **stops that sink** (`k7_usage_sink_stopped`, alert `K7UsageSinkStopped`) until you fix the cause and `rollout restart ds/k7-agent`. Auth headers come from Secret `k7-metering-headers` in the system namespace, read on every request.

### Configuration

`k7 install` and the chart render the same document into ConfigMap `k7-metering`; every key is validated at startup and a change rolls both pods.

| Key | `k7 install` flag | Default |
|---|---|---|
| `sampleInterval` | `--metering-sample-interval` | none in production; 15 in development |
| `storageInterval` | `--metering-storage-interval` | 60 |
| `spool.maxBytes` | `--metering-spool-size` | `1Gi` |
| `spool.highWaterPercent` | `--metering-spool-high-water` | 80 |
| `requireDelivery` | `--metering-require-delivery` | false |
| `prometheusUrl` | `--metering-prometheus-url` (`--monitoring` sets its own) | unset |
| `sinks` | `--metering-sinks-file` | `[]` |

### Showback

```bash
k7 usage --from 7d --group-by tenant,state
k7 usage --tenant acme --from 2026-01-01T00:00:00Z --to 2026-02-01T00:00:00Z --json
```

`k7 usage` and `GET /api/v1/usage` read Prometheus at `metering.prometheusUrl` (400 naming the key when unset). K7 keeps no usage database. Counters are summed exactly from raw samples, so the result matches the events to within one scrape interval per series inside Prometheus retention (7 days with `--monitoring`); disk and snapshot GiB·s integrate the gauges at scrape resolution. A tenant key sees only its own tenant. For anything billable, consume the events.

Reference: [docs/OBSERVABILITY.md](https://github.com/Katakate/k7/blob/main/docs/OBSERVABILITY.md), [docs/METERING.md](https://github.com/Katakate/k7/blob/main/docs/METERING.md). Not yet: tracing (request ids are the correlation mechanism), rate limiting (`k7_rate_limited_total` stays at zero).