Distributed recording#

Record agents running on many servers into one recorder on its own server, so recording costs the agents almost nothing and the evidence never lives on the hosts it describes.

 agent host (any language)                       recorder server(s)
 ┌─────────────────────────────┐                ┌─────────────────────────┐
 │ agent ─ SDK queue ─┐        │  bitexact      │ bitexact recorder       │
 │ agent ─ proxy ─────┼─► bitexact collector ──►│  mTLS, run tokens,      │
 │ curl  ─ HTTP ──────┘  encrypted spool │ mTLS │  chains, signing key    │
 └─────────────────────────────┘                └─────────────────────────┘

• The agent never waits. The SDKs hand each event to an in-process queue and return; the network and the recorder are never on the agent's path. With the collector stalled, the suite holds a Python send under 100 µs at p99, and a TypeScript or Go send under 5 ms at p99 on a loaded test machine - a send that waited would take seconds. A full queue drops the newest event and declares the loss in the run.
• The collector (one per host) takes events over loopback HTTP from any process holding the host secret. It stamps each event with its source and sequence, writes it to an encrypted spool, and forwards it.
• The recorder is the only process that holds the signing key and writes chains. The inspector's acts - a legal hold, a release, a redaction decision - are handed to it to write and sign. It acknowledges an event only once it is durably written, and the collector erases a spool segment only once every event in it is acknowledged.

The wire protocol is bitexact. The threat model and every attack against it are tested in the suite: see What stops whom below.

Setting it up#

1. Certificates#

bitexact pki init  --dir /etc/bitexact-ca --name "Acme BitExact CA"
bitexact pki issue --dir /etc/bitexact-ca recorder --dns recorder.internal
bitexact pki issue --dir /etc/bitexact-ca host-12 --spiffe spiffe://acme/agents/host-12
bitexact pki crl   --dir /etc/bitexact-ca

• The recorder: a server certificate for its DNS names and IP addresses.
• Each agent host: a certificate naming its workload with a SPIFFE ID. That name is the peer every event from the host is recorded under. It comes from the certificate and never from anything the host sends.

Keys are written readable only by their owner, and nothing is overwritten. The CA's directory holds ca.key, with which anyone can issue themselves any identity: run these as an administrator, on the machine kept for issuing, never as the user a BitExact service runs as. Give each service only its own certificate and key, the CA's certificate and, for the recorder, the revocation list - in a directory of its own, readable by its user alone:

# On the recorder's server:
install -d -o bitexact -g bitexact -m 0700 /etc/bitexact/pki
install -o bitexact -g bitexact -m 0400 /etc/bitexact-ca/recorder.key /etc/bitexact/pki/recorder.key
install -o bitexact -g bitexact -m 0444 /etc/bitexact-ca/recorder.crt \
  /etc/bitexact-ca/ca.crt /etc/bitexact-ca/ca.crl /etc/bitexact/pki/
# On agent host 12, from host-12.crt, host-12.key and ca.crt copied there:
install -d -o bitexact -g bitexact -m 0700 /etc/bitexact/pki
install -o bitexact -g bitexact -m 0400 host-12.key /etc/bitexact/pki/host.key
install -o bitexact -g bitexact -m 0444 host-12.crt /etc/bitexact/pki/host.crt
install -o bitexact -g bitexact -m 0444 ca.crt /etc/bitexact/pki/ca.crt

An organization that already runs a CA, or SPIFFE/SPIRE, can use its own certificates of the same shape instead.

deploy/systemd/ has service units for the recorder and the collector. deploy/compose/docker-compose.yml runs both from the product image, with health checks.

2. The issuance policy#

The recorder issues run tokens only as a policy file allows:

{"rules": [
  {"peer": "spiffe://acme/agents/*", "runs": "lending-*",
   "kinds": ["tool_call", "http_call", "nondet", "identity", "context",
             "marker", "policy_check", "model"]},
  {"peer": "spiffe://acme/reviewers/*", "runs": "lending-*",
   "kinds": ["human_decision", "decision_record"]},
  {"peer": "spiffe://acme/services/inspector", "runs": "*", "kinds": [],
   "acts": ["hold", "release-hold", "redaction-request",
            "redaction-approve", "redaction-reject", "custody", "fork",
            "ledger:cases", "ledger:capture-expectations", "ledger:refusals",
            "log-checkpoint", "ledger-heads", "redact", "access",
            "events-retry"]},
  {"peer": "spiffe://acme/operators/cli", "runs": "*", "kinds": [],
   "acts": ["hold", "release-hold", "redaction-request",
            "redaction-approve", "redaction-reject", "custody", "fork", "ledger",
            "log-checkpoint", "log-backfill", "purge-record", "anchor",
            "ledger-heads", "seal", "redact", "access", "events-retry"]}
]}

• Give the inspector and the command line separate certificates. The operator's (the CLI's) is granted what only an operator does - role definitions (ledger, or ledger:roles), purge records, seals, anchors, the log's backfill - and is never mounted where the web-facing inspector runs; the inspector's rule grants none of those. ledger:<name> grants one ledger (cases, capture-expectations, workflows, roles, refusals); ledger grants them all.

• acts lists the custody acts a peer may hand the recorder to write and sign: the inspector's holds, releases and redaction decisions, and the command line's; custody covers their custody records of an access (an export, a pack, a fork, a hash-only test), and fork lets the inspector hand the recorder a fork's branch, which the recorder checks against the source run and writes as one sealed chain (the rule's runs must cover the new fork's id too). anchor has the recorder anchor its own head of a run (bitexact anchor export with no --key), and purge-record has it sign a purge record for a run the purge rule lets go at the command's cutoff. The deployment's own records need a rule whose runs is exactly * (as written: ? or [*] grants none of them): ledger (roles, capture expectations, cases, workflows, each checked and signed by the recorder as on the recorder host; refusals kept unsigned, naming the certificate that handed them over), log-checkpoint and log-backfill (the transparency log), and ledger-heads (the signed heads of a pack's ledgers, which the recorder signs only when its store holds those ledgers and runs exactly as the pack carries them), access (an access-log event - a view, a listing or a refusal - built and chained by the recorder) and events-retry (a dead-letter requeue, each written to the access log as its row moves). A process that names a recorder and holds no key writes no chain itself, and one that names a recorder writes no access log at all. A rule with no acts grants none, so a collector can never submit one. The recorder takes the submitting peer from its certificate and records it beside the principal the peer names, and it writes an act id once.

• Who a person is. Without single sign-on configured at the recorder, the principal an act names (by) is the inspector's or the command line's statement: the recorder writes it as asserted, beside the certificate that made it, and cannot check it - a compromised inspector could name anyone, or name two people who are one. Set BITEXACT_SSO_ISSUER, BITEXACT_SSO_AUDIENCE and one of BITEXACT_SSO_JWKS_FILE (a pinned key set, for an offline recorder) or BITEXACT_SSO_JWKS_URL on the recorder, and it verifies the person itself: every act that names one - a hold, a release, a redaction request, approval or rejection, a custody or access record, a dead-letter retry, a ledger event - must carry the person's OpenID Connect ID token, and the recorder checks its signature against the issuer's keys (RS256/384/512, PS256/384/512, ES256/384/512 or EdDSA; none and HMAC are refused), the issuer, the audience, expiry and not-before (with BITEXACT_SSO_LEEWAY_S of skew, default 60 s), and takes the principal from sub, or email with BITEXACT_SSO_PRINCIPAL_CLAIM=email. An act whose by is not that principal, or that carries no token, is refused, and its id keeps the reason. What it records carries a verified_identity - issuer, subject, claim, principal - never the token; a redaction's approver must be a different verified person from its requester, whatever names the inspector gives them; and a role grant that opens the two-person flow needs no stated posture. The inspector forwards the token its SSO proxy hands it (BITEXACT_SSO_ID_TOKEN_HEADER, default Authorization), and the command line sends the one in BITEXACT_SSO_ID_TOKEN_FILE; see identity and access. The recorder refuses to start with these settings set only in part.

• During a license lapse the recorder writes only holds, releases and erasure, records of reads (custody accesses other than fork, access), what exports and packs carry (log-checkpoint, ledger-heads), events-retry, and the roles, capture-expectations and refusals ledgers; every other act is refused before its id is claimed, so the same act is written once the license is renewed.

• A host whose certificate matches no rule gets no token, and writes nothing.

• A token grants only the kinds both requested and allowed. A policy that lists no policy_check refuses agents' safety checks, and counts them unrecorded: list it where agents run them.

• The file is read on every request, so a change applies at once.

3. The recorder#

export BITEXACT_SIGNING_KEY=/secure/recorder.key        # or the HSM / KMS settings
export BITEXACT_LICENSE=/secure/bitexact.license
export BITEXACT_RECORDER_POLICY=/etc/bitexact/policy.json
bitexact recorder --host 0.0.0.0 --port 7470 \
  --tls-cert recorder.crt --tls-key recorder.key --tls-client-ca agents-ca.crt \
  --tls-crl agents-ca.crl

--tls-crl names the CA's revocation list.

• bitexact pki revoke --dir /etc/bitexact-ca NAME, where the CA is kept, adds a host's certificate to it; install the new ca.crl over the recorder's /etc/bitexact/pki/ca.crl as in step 1.
• The recorder re-reads the list when it changes. It refuses a revoked host at the handshake and on its very next batch, even in a session already open.
• The list expires after a year. Renew it with bitexact pki crl --dir /etc/bitexact-ca, and install it the same way.
• A list that is missing, unreadable, not signed by the CA, or expired refuses every host rather than readmitting revoked ones.

The recorder refuses to start in any of these cases:

• without TLS 1.3;
• without a client CA;
• without its signing key.

Run it on the embedded store, or set BITEXACT_DATABASE_URL for PostgreSQL. With PostgreSQL, several recorders behind a TCP load balancer share the work; a run's writes are serialized by its database lock.

4. Each agent host#

Agents authenticate to their collector with a host secret. They usually run as other users than the collector, so provision the secret where they are granted read access (a group-readable file, or a Kubernetes Secret mounted into both containers). Create it owner-only - under the default umask it would be readable by every user until the chmod - then open it to the agents' group alone:

(umask 077; head -c 32 /dev/urandom | base64 > /etc/bitexact/intake.secret)
chown bitexact:bitexact-agents /etc/bitexact/intake.secret
chmod 0640 /etc/bitexact/intake.secret

bitexact collector --spool /var/lib/bitexact/spool \
  --recorder recorder.internal:7470 \
  --tls-cert host.crt --tls-key host.key --tls-ca recorder-ca.crt \
  --intake-secret-file /etc/bitexact/intake.secret

Without --intake-secret-file the collector creates a secret in its spool, readable by its own user alone: that suits agents running as the same user.

The spool directory holds the host key that encrypts the spool, the collector's id and its restart epoch. Keep it on local disk, readable only by the collector's user. --max-spool-bytes bounds it (1 GiB by default). On SIGTERM the collector keeps forwarding until the spool is empty or --drain-seconds (30) pass; give it that long to stop.

Point the agents at it:

export BITEXACT_COLLECTOR=http://127.0.0.1:7378
export BITEXACT_COLLECTOR_SECRET_FILE=/etc/bitexact/intake.secret
export BITEXACT_ACTOR=planner            # optional: this agent's name

This works for each kind of agent:

• Python. bitexact.run, @record_tool, the wrapped clients and bitexact serve record through the collector without code changes.
• TypeScript. Use sdk-ts (run, recordTool, wrapFetch, wrapOpenAI, wrapAnthropic, delegate, adoptToken).
• Go. Use sdk-go (Run, RecordTool, WrapTransport, Delegate, AdoptToken).
• Java. Use sdk-java (run, recordTool, sendJson, carry, delegate, adoptToken).
• Anything else. POST /events with Authorization: Bearer <secret>.

The agent hosts need no license. The recorder holds it.

When an event cannot be delivered#

Every SDK behaves the same way, and none of these failures reaches the agent:

• A run id the recorder would refuse. A run id is 1-64 characters from A-Z a-z 0-9 . _ -, starting with a letter or digit. Each SDK checks the id before sending anything. With a bad id the agent's function still runs, outside any run, and the SDK logs the refusal with the id, every time: capture gap: run_id_rejected from the TypeScript, Go and Java SDKs, and a capture_gap event with cause: run_id_rejected and run_id from the Python SDK, whose log lines are JSON.
• A run's end the collector does not take. A full spool drains, so that end is retried with backoff. An end refused for its run id is logged as run_id_rejected.
• Events still queued when the process ends. The SDK delivers for up to two seconds, then logs one capture gap: undelivered line per run. The line gives the number of events, whether the run's end was among them, and any losses not yet declared. These events never reached a collector, so no chain can carry them; the log is the record. In Go, call defer bitexact.Shutdown(5 * time.Second) in main, because a Go program has no exit hook. Java and Python deliver at exit by themselves, and TypeScript makes one bounded try.
• A queue that overflows. The newest event is dropped and the loss is declared in its run, by count, in the next batch that gets through.

Runs written by many agents#

• An agent hands work to another with bitexact.causal_context(), a short string to pass along with the work. The receiving agent wraps its part in bitexact.caused_by(context). Its events then record what caused them, and its clock moves past the sender's.
• An agent that takes part in a run another agent opened uses bitexact.run(run_id, join=True). Leaving the run ends only its own part.
• A sub-agent on another host records into its parent's run under a token the parent delegates: token = bitexact.delegate(peer, kinds, ttl_s) inside the parent's run (peer is the sub-agent host's certificate identity); the sub-agent calls bitexact.adopt_token(run_id, token), then bitexact.run(run_id, join=True). The TypeScript, Go and Java SDKs have the same pair (delegate/adoptToken, Go Delegate/AdoptToken), each waiting on the collector at most a second. Over HTTP: POST /runs/{run_id}/delegate {"peer", "kinds", "ttl_s"} and POST /runs/{run_id}/token {"token"} on the collector intake. A delegation can only narrow (fewer kinds, an earlier expiry, one host), revoking the parent revokes it, and one that fails is a declared token_rejected gap; the agent never sees an exception.
• The recorder seals a run once every agent that wrote to it has finished and everything each said it sent has arrived. If that hasn't happened BITEXACT_SEAL_GRACE_S (30 s) after the opener asked for the end, it seals anyway, with a declared gap for each agent still open.
• bitexact show RUN --causal prints the run in the order things happened.
• bitexact.run(RUN, mode="replay", actor="planner") replays one agent against its own calls.

What stops whom#

Each row is a test that failed before the defense existed (tests/test_recorder_attacks.py, tests/test_spool.py, tests/test_collector.py, tests/test_recorder_causal.py, tests/test_recorder_server.py, tests/test_coverage_expectations.py, tests/test_coverage_sources.py, tests/test_scorecard_alerts.py).

What no design here can stop. A host whose agent is compromised can lie about what that agent did. The content, the actor and the causes are that host's assertion, as they are with the in-process SDK. The chain proves which authenticated host sent each event, under which token, and that nothing changed after it arrived. Calls through the proxy are the ones BitExact observes itself. So a host can keep a dead agent's source live on the coverage scorecard, and the scorecard says only what it knows: which authenticated host sent the evidence, and that the workflow, environment, agent and SDK label are that host's assertion.

Under faults#

Each of these is a test, judged by the same checker: every event once, every source's sequence whole, every loss declared, causes resolved, and the agent unaffected.

• The recorder is killed and restarted. Agents never notice. The spools grow and then drain, with no gap.
• A collector is killed outright. What it acknowledged was already on disk. What it had not acknowledged, the SDK sends again, and nothing is doubled.
• The link is slow, partitioned, refusing, or drops replies after the recorder acted. No loss, no duplicates.
• The database fails over mid-run. What could not be written is retried, with no loss and no duplicates.
• The spool fills, or the disk under it does. The newest events are dropped and their exact range is declared. A batch the disk refuses partway through is taken back whole, so no torn record is left behind.
• The links are degraded, cross-host. The recorder and its database each run on a host of their own, with Toxiproxy on both links. Latency, an 8 KB/s bandwidth cap, packets cut into slices, refused connections, hung connections and resets on either link all still deliver every event once. A slow database delays delivery and never the agent.
• Clocks disagree. Certificates are valid from an hour before issue, so minutes of skew break no TLS. Within BITEXACT_HLC_MAX_SKEW_MS (5 minutes) the hosts' clocks are accepted. Past it, every event is still recorded, and the clock doubt is declared beside it.
• The recorder's store cannot be used. The recorder refuses to start and names why, rather than accepting connections it cannot record.

What it costs, measured#

These numbers come from python -m bench.run and python -m bench.drain. The setup was one workstation (Windows 11, 16 cores) with signing on and a file store that syncs each entry. The load was 50 small model calls a second, open loop, for 30 s × 3 trials (4,500 samples a shape). Latency is what the agent saw, in milliseconds. The fake provider answers in about 1 ms, so these figures are BitExact's own cost. Measure your own hardware the same way before you size a deployment.

† Not a measurement of BitExact's cost, by the bench's own rule: a shape is valid only with no errors and no more than 0.1% of calls missing their slot in the schedule. Here 8 of 4,500 PostgreSQL calls, and 12 of 4,500 local proxy calls, missed their slot, and the local proxy also failed 2 calls. These rows show that the shape could not keep up with 50 calls a second on this machine, where each call pays its own durable write (an fsync per entry, or a database round trip) on the agent's path; the tail is the queue behind them.

The local proxy on Linux. The same bench in a Linux container with 4 CPUs and its file store on the container's own disk (20 s × 3 trials a rate), where the proxy writes concurrent calls to a run together:

So without a collector the local proxy holds 160 calls a second there and saturates between 160 and 240. The Windows row above is that machine's fsync, not the proxy's ceiling.

The recorder "in a Linux container" ran on the same workstation, reached over the container network - a separate kernel and TCP stack, not a separate machine. On the three collector rows every one of the 5,250 calls, warm-up included, was on the recorder's chain. The recorder's distance does not reach the agent: the collector acknowledges once the call is spooled on the agent's own host.

A recorder outage. At 50 calls a second the recorder was stopped for 60 s. The agent's latency did not move: p50 2.0, p99 2.8, p99.9 13.3 during the outage, against 2.1, 3.0 and 6.7 before it. The collector spooled 3,008 events. Once the recorder was back, it drained them in 15.4 s while the load went on, about 195 events a second ahead of the new traffic. All 10,500 calls were recorded, and every run verified.

Recording without a collector costs more per call, because each call pays its own durable write - the bottom three rows. Use a collector wherever calls are frequent, and measure the shape you will run with python -m bench.run on the hardware you will run it on.

Operating it#

• Collector status. GET /status on the intake (with the host secret) reports pending events, spool bytes against the bound, dropped ranges, and events the recorder will never accept. collector_unrecorded log lines name the latter, for example a run the policy does not cover. spool_retrying counts spool segments that could not be read just now (a lock, a passing I/O fault). The collector keeps trying them for 60 s before declaring their events lost; spool_unreadable counts the ones it gave up on or that failed authentication.
• Heartbeat. The collector sends the same status to the recorder as soon as it connects and then every BITEXACT_HEARTBEAT_S seconds (default 60), with each run and SDK label it took events for. The recorder keeps it in the store's capture_sources table under the collector's certificate peer, with last_heartbeat_at on its own clock and the interval as heartbeat_s. A heartbeat is never evidence and never in recording's way.
• Metrics and alerts. The recorder's /metrics serves bitexact_source_last_evidence_seconds{expectation=...}, bitexact_source_state{expectation=...,state=...} and bitexact_collector_last_heartbeat_seconds{peer=...}. A collector whose spool passes 80% of its limit or that reports dropped ranges raises collector-backlog; one that reports events the recorder refused for good raises collector-unrecorded, both however recent its heartbeat. See Ops health.
• Verification. bitexact verify RUN exits 3 when a run declares any gap. --allow-gaps CAUSES accepts the causes you choose.
• Recorder key rotation. Rotating the recorder key invalidates outstanding run tokens. Collectors fetch new ones automatically.