Ops health, alerts, and capacity#

The recorder and proxy produce evidence; operations needs to know when that production is degrading. bitexact health (and the Settings ▸ Health panel) aggregate the per-run integrity signals into one status a monitor can gate on.

Status and alerts#

bitexact health scans the store and reports a status — healthy, degraded, or critical — with typed alerts:

The report also carries every alert the coverage scorecard raises over the whole deployment (source-quiet, source-dark, source-silent, source-never-seen, expectation-removed, collector-backlog, collector-unrecorded; see Sources and silence), so a declared source that stops writing is raised however busy the rest of the store is. The inspector's Settings ▸ Health and Overview raise the same alerts, beside auth-off-prod and no-recent-evidence, and name a host only to a role that may open Settings.

The command exits 2 on a critical status, so it drops straight into a monitor:

bitexact health            # human-readable
bitexact health --json     # the raw report for a scraper; exit 2 on critical

BitExact surfaces the health and the non-zero exit; your monitoring (cron, a liveness probe, a scrape of /api/health) turns that into an alert on verify-fail / seal-fail / anchor-rollback. Alerting is deliberately your channel — BitExact does not send mail or hit a pager itself.

What expires on a calendar#

Three things in a deployment expire, and each is a predictable outage if nobody is told: host and recorder certificates (bitexact pki issue, 365 days by default), the revocation list (bitexact pki crl, 365 days - past it the recorder refuses every host) and the license (past it and its grace, recording stops; traffic does not). bitexact health reports each one:

bitexact health --cert /pki/recorder.crt --cert /pki/host-1.crt \
                --crl /pki/ca.crl --warn-days 30

The same standing is in bitexact health --json as standing (state, reasons, records), in the viewer's deployment settings, and as the standing field of every run's run_meta, so a sealed run carries the standing it was recorded under.

Renew a certificate by issuing a new one under the same CA with a new name

• bitexact pki issue never overwrites one (bitexact pki issue host-1-2027 --dir pki --spiffe spiffe://acme/agents/host-1 --days <days>, the same identity as before) - and pointing the component at the new files; the recorder reads its certificate at start, so restart it, and a collector presents its new one on its next connection. Renew the CRL with bitexact pki crl --dir pki; the recorder re-reads it as it changes.

Expiry alerts as events#

With event delivery configured (Event delivery), the same certificate and CRL alerts leave as alert.opened and alert.resolved events, for every certificate a BitExact process serves with or presents, wherever it runs:

Each publisher writes one expiry fact per file to the store (the events/expiry-facts documents: the service, host, path, not-after or next update, subject and SHA-256 fingerprint, or why the file does not read) at start and every hour, reading the file afresh each time. The alert round evaluates every published fact, whichever process holds the alerts lease, so an alert is never resolved because that process cannot see the file. An alert resolves when its subject is renewed - the next fact published carries a date past --warn-days - or when nothing has published it for 24 hours: its process stopped, its file is no longer named, or the peer no longer connects. Until then a fact keeps raising its alert from the date it last published. Once a retired fact's alert has resolved, the process holding the alerts lease deletes the fact, so the store keeps only facts that can still raise one. Alerts are operational signals, never evidence, and a store that refuses a fact is logged (expiry_publish_failed) without stopping the process; the next hourly refresh writes it again. bitexact health reports the same published facts beside the files its --cert and --crl name, so it can be run on any host that reaches the store.

The recorder's health port#

bitexact recorder --health-port 7480 serves plain HTTP beside the mTLS port, for orchestrators and Prometheus:

A lapsed license, a red standing or an expiring certificate is a metric and an alert, never a reason to be unready: taking the recorder out of service only makes every collector spool. The Kubernetes and compose files in deploy/ probe it.

Recording health#

The recording proxy (bitexact serve) answers GET /health/recording, unauthenticated like /healthz and /metricsz beside it, with whether the calls arriving now are being recorded:

{"status": "recording", "store_reachable": true, "disk_free_bytes": 52613349376,
 "breaker": {"state": "closed", "trips_total": 0, "consecutive_failures": 0},
 "license": "ok", "last_write": "ok",
 "counts": {"degraded_total": {}, "unrecorded_calls_total": {},
            "refusals_unwritten_total": 0}}

It answers 200 while recording and 503 while degraded. The counts are history and never make it 503 on their own. It is a recording probe, not a traffic one: calls keep passing through while it is 503, so alert on it and keep the orchestrator's liveness probe on /healthz.

Sources and heartbeats#

Each collector tells the recorder it is up as soon as it connects and then every BITEXACT_HEARTBEAT_S seconds (default 60): its spool counts, its dropped ranges, and each run and SDK label it took events for since its last kept beat (the frame is in the protocol). The recorder keeps the latest in the store's capture_sources table, under the peer the collector's certificate names, with last_heartbeat_at taken from the recorder's own clock and the collector's interval as heartbeat_s. The table holds at most 100,000 sources; past that the least recently written is evicted, and every eviction is counted.

A served proxy and a Python SDK process that write into the store themselves (no BITEXACT_COLLECTOR) keep a writer row of their own in the same table, one per capture path they record through (proxy, mcp-proxy, sdk-python or an adapter's: otel, langchain, openai-agents, claude-agent-sdk, adk, auditant, llamaindex, vapi, or the governance-decision API's governance), keyed by path, host name and process id. The MCP proxy keeps its own mcp-proxy row, so its liveness is never read off the model proxy's, and a process recording policy and human decisions keeps a governance row, so decision recording is never read off the SDK's other traffic. A background thread upserts it as soon as the path starts and then every BITEXACT_HEARTBEAT_S seconds: started_at, the last_call_at of the path's latest call, last_heartbeat_at and the interval as heartbeat_s. A value that is not a positive, finite number of seconds falls back to 60. This row is what lets the scorecard read an idle proxy or SDK process as quiet rather than silent, and as dark once three beats are missed. A restarted process starts a new row; the rows of processes that stopped beating age out under the same 100,000-source bound, each eviction counted.

A heartbeat is operations state, never evidence. A beat the store cannot take is refused and sent again with the next one, and the session goes on recording; a liveness row the store will not take is logged (liveness_failed) and counted in liveness_failures_total, which the served proxy reports at /metricsz, and the agent's call goes on.

From the coverage scorecard, /metrics serves three gauges, so a "no data" monitor in your own Datadog or Grafana can sit on them:

Alert on an age with a threshold, such as bitexact_source_last_evidence_seconds > 3600; it is not a Unix time to subtract from time(). Only declared expectations have series: one inferred from history never alerts, so it serves none.

Every age in one scrape is measured from one instant, the scorecard's own at. The scorecard is computed once per /metrics scrape and never for /readyz. When it cannot be computed, none of these series is served; the scrape itself never fails on it, and readiness never depends on it.

A current heartbeat says the collector is up, never that capture works. The scorecard raises these whatever the heartbeat says:

Capacity#

The report also carries capacity: total bytes on disk and the largest runs, so GB-scale runs are visible before they are a problem. A large run's bundle exports as streaming JSONL (bitexact export --format jsonl, a header line followed by one entry per line) so neither the recorder nor a verifier ever has to hold a multi-gigabyte bundle whole in memory. Redaction, verification, and the fork/replay lab all operate over the streamed form.

Recorder memory. For each run it is writing, the distributed recorder holds the digest and the id of every event it has taken, under 80 bytes an event (the suite holds each part under 40), so it can tell a resend from an equivocation and a reused id from a new one exactly. That memory is released when the run seals or sits idle past the sweep, and the run then lives only on disk. It grows with the events of the runs open at once, not with the store: a recorder with ten runs of a million events open needs about 800 MB for them.