Deploying BitExact#

BitExact runs entirely inside your VPC. Nothing in it can phone home; license validation is offline. This document covers the listener, identity, storage, and lifecycle decisions an operator owns.

Prerequisite: a license bound to this deployment's recorder key#

BitExact is licensed use only. There is no evaluation mode. Between expires and the end of grace_days (14 when the license names none) the deployment keeps recording and logs a license_grace warning at most once an hour; renew then. Past expires plus grace_days the deployment is read-only: the recorder records nothing - checked on each request, bitexact serve forwards every call to the provider unrecorded, marked x-bitexact-recording: off, and an SDK run executes its body unrecorded (with BITEXACT_FAILURE_MODE=strict the proxy refuses with a 402 instead) - replay, fork, regression, anchoring, purge and every other producing command refuse, and the viewer refuses the same acts (/api/whoami lists them as license_refuses before anyone clicks); listing, showing, verifying, exporting and packing the evidence already recorded keep working, and so do the statutory acts - placing and releasing a legal hold, and redaction - which a customer owes to a court or a data subject whatever the state of its renewal. A proxy restarted after a lapse must not cut agents off from their provider.

With no license, or one that fails its check, the viewer serves nothing but its sign-in page, the stylesheet and script its pages are drawn with, and its liveness probe. A browser gets a page saying what is wrong - from the license check itself - and where this deployment's license file goes; an API client gets a 402 with the reason in detail. /healthz answers 200 in every license state, with the state in its body ({"status": "ok", "license": "missing", "standing": "red"}), so an orchestrator never restarts a viewer over a license.

Without a valid license only bitexact capture keygen, bitexact keygen, bitexact license, bitexact pki, bitexact recorder-key, bitexact version and bitexact witness answer. bitexact collector, bitexact mcp-proxy, bitexact recorder and bitexact serve start in any license state and check the license on each call: they pass the agents' traffic through, or declare a gap, rather than record.

Generate the recorder key on the host that will sign, keep the private half there, and send the public half for the license to name:

bitexact keygen -o /secure/recorder.key
bitexact capture keygen -o /secure/capture.key
export BITEXACT_SIGNING_KEY=/secure/recorder.key
export BITEXACT_CAPTURE_KEY=/secure/capture.key
export BITEXACT_LICENSE=/secure/bitexact.license
bitexact license

keygen writes two files: the Ed25519 key and its post-quantum (ML-DSA) half, recorder.pq.key, beside it, where every process that signs reads it (BITEXACT_SIGNING_PQ_KEY names it anywhere else). A signer given the key without its half signs Ed25519 alone, and bitexact health says signing_classical_only. keygen prints both public halves - send both for the license - and the trust-file entry that pins them; give that entry to every party that verifies your evidence, so a verifier pinned by it refuses evidence the post-quantum half did not sign. bitexact recorder-key and bitexact recorder-key --pq print the two halves again.

The capture key keys hash-only fields and the digests of content gaps withheld (capture-controls.md). Every process that records or applies the capture policy - the proxy, the viewer, the recorder and each collector - reads it from BITEXACT_CAPTURE_KEY, and all of them hold the same key.

Both keys are secrets and live with the license, owner-only, never in the data directory or its volume: whoever holds a backup of the evidence must not hold the keys that sign it or open its hash-only fields. Each shipped deployment delivers them so:

The recording proxy (bitexact serve) holds a third secret, the host key: 32 random bytes it seals each streamed response's write-ahead journal under (capture-controls.md, What else the policy reaches). It reads the file BITEXACT_HOST_KEY names or, unset, <data directory>.host.key beside the data directory, and makes it owner-only on first use when it is missing. It is never inside the data directory - a path there is refused - so a backup of the store never carries the key that opens a journal left behind, and only a process holding it opens one (bitexact capture journal FILE). In a container the data directory is a volume such as /data, and its default place (/data.host.key) is on a root the image's user cannot write: name a writable file, or one made ahead, outside the data volume. The pilot's keys step makes secrets/journal-host.key and setup installs it with the other secrets, as BITEXACT_HOST_KEY for the proxy and the cli service. The Compose, Kubernetes and systemd deployments run no proxy and need none; give a proxy added to one the key as the capture key is given.

One deployment per key. A license records only when the deployment signs with a recorder key it names, and turning signing off does not slip the binding; a license that names none records nothing. A second stack needs its own key and its own license.

Copying both the license and the key to a second host cannot be refused by an offline product, but it is not invisible: every sealed run commits a per-deployment fingerprint, so one license_id emitting bundles under two fingerprints is evident in the evidence itself.

Trusted platform#

Every 90 days the license is renewed by file (Licensing), and the renewal request carries a TPM 2.0 attestation of the recorder host. It is taken only when you run bitexact license request, never while recording, so a TPM problem can delay a renewal but never stops a call or a recording. The host needs:

• A TPM 2.0 the product can reach. On Linux, /dev/tpmrm0, readable by the user that runs the request (the tss group, or root); in a container, pass it through (--device /dev/tpmrm0; on Kubernetes, a TPM device plugin). On Windows, the TPM Base Services, which every Windows host has.
• The endorsement hierarchy with no password, which is the default. Nothing is stored in the TPM: its endorsement and attestation keys are derived from the standard TCG templates each time.
• The TPM's endorsement-key certificate, in the TPM's NV storage, where discrete TPMs keep it. Firmware TPMs (Intel PTT, AMD fTPM) often do not: obtain it from the manufacturer's certificate service and pass the file to the request.
• On Linux, integrity measurement (IMA): kernel 5.8 or later, booted with ima_template=ima-ng ima_hash=sha256, securityfs mounted at /sys/kernel/security, and a policy that measures the product's files when they are loaded (measure func=FILE_MMAP mask=MAY_EXEC for the compiled modules, and measure func=FILE_CHECK for anything read as a file). The request reads the measurement log, so run it as root. A host whose policy measured none of the product's files is refused renewal: the attestation is how BitExact learns the files it renews for are the ones it shipped.
• On Windows, the TPM measures boot, not files. Whether renewals from a Windows recorder host are accepted is stated in your Order Form; a Linux recorder, with Windows agents sending to it, attests files.

The listener#

bitexact serve binds loopback by default. For anything beyond one host, terminate TLS on the listener itself:

bitexact serve --host 0.0.0.0 --port 8787 \
    --tls-cert server.crt --tls-key server.key

Add mutual TLS — only workloads holding a client certificate signed by your CA can record:

bitexact serve ... --tls-client-ca clients.pem

The viewer takes the same pair, so bearer tokens and evidence never cross the network in the clear:

bitexact view --host 0.0.0.0 --port 8788 \
    --tls-cert viewer.crt --tls-key viewer.key

A network bind needs an auth mode as well; the viewer refuses one without (see The viewer below).

Behind a reverse proxy (nginx, Envoy, an SSO-terminating gateway), keep the recorder on loopback or a private interface and let the proxy own the public listener. The recorder does not implement SSO deliberately: identity termination belongs to the infrastructure your security team already audits. Forward X-BitExact-Run-Id unchanged.

Give the proxy the client-facing deadlines too. The recorder applies no request-read timeout of its own: a client that sends its request a byte at a time holds its connection open for as long as it likes, and enough such clients exhaust connections before they exhaust anything else. Set the request-read and header-read deadlines and a per-client connection cap on whatever fronts it (client_header_timeout, client_body_timeout and limit_conn in nginx; the equivalents elsewhere). On a loopback-only recorder with no public listener there is nothing here to set.

Reserved local routes — never proxied upstream, never recorded:

Point orchestrator liveness checks at /healthz: it stays 200 while calls pass through, whatever the state of recording. Alert on /health/recording, and set BITEXACT_DISK_MIN_BYTES so it reports degraded before the disk actually fills.

Ending a proxy run over HTTP#

POST /bitexact/runs/<run-id>/seal appends a run's seal - evidence of where it ended - so it is closed until you open it. Make a secret, owner-only, and name it on the recorder:

(umask 077; head -c 32 /dev/urandom | base64 > /etc/bitexact/seal.secret)
export BITEXACT_SEAL_SECRET_FILE=/etc/bitexact/seal.secret

The clients that end runs send it as Authorization: Bearer <secret>:

curl -X POST -H "Authorization: Bearer $(cat seal.secret)" \
    http://localhost:8787/bitexact/runs/loan-42-decision/seal

The route seals only runs the proxy recorded: one it opened, or - through a host collector - one this process sent events for since it started. An SDK run seals itself; the recorder seals what its sources end; anything else is bitexact seal RUN on the recorder's host. The secret lets a client end any run the proxy recorded, so give it to the clients that end runs, not to everything that can reach the proxy. The file is read on every request, so replacing it rotates the secret at once.

The viewer#

bitexact view never edits or deletes a recorded entry. It does act, and every act is itself evidence: placing or releasing a legal hold and requesting or approving a redaction append audit entries to the run's chain (an approved redaction destroys the named values in place and leaves their commitments); a fork, a fork matrix, a regress and a suite run record new runs or reports; a case pins runs on its own ledger. Every such route requires the X-BitExact-Action header and a role that may perform it, and the full route list is docs/api-reference.md. With no authentication configured every caller is admin, so bitexact view refuses any non-loopback --host until BITEXACT_VIEWER_TOKENS or BITEXACT_TRUST_PROXY_AUTH is set. Set BITEXACT_VIEWER_TOKENS to require a token. An API client sends it as Authorization: Bearer <token>. A person opening the inspector in a browser is sent to its sign-in page, which exchanges the token for an HttpOnly, SameSite=Strict session cookie lasting BITEXACT_VIEWER_SESSION_HOURS (default 8). The cookie is signed with the token, so removing a token from the configuration ends every session it opened. Sign out is in the inspector's main menu. Repeated failed sign-ins from one address are throttled. Every request emits a structured access-log line, and evidence-touching actions are additionally recorded on the run's custody chain. For team access, put it behind the same SSO proxy as everything else, and name that proxy in BITEXACT_TRUSTED_PROXY_IPS (default: loopback) so forwarded identity headers are believed from it alone - see identity-access.md. Every request body - an upload to the in-app verify endpoints included - is capped at BITEXACT_MAX_UPLOAD_BYTES (default 64 MiB, 413 above it), and the cap applies before a route authorizes; evidence larger than that is for the offline verifier. A forwarded identity header sent twice is refused.

The agent entrypoint#

Four things in the viewer execute the customer's agent: forking a decision, sweeping one N times, regressing a corpus against the code running now, and running a saved suite as a gate. All four need the callable that runs that agent, and none of them will take it from a request - that would be arbitrary import over HTTP. It is the operator's, named in an environment variable on the viewer process:

BITEXACT_AGENT_ENTRY=myapp.agent:run_once  bitexact view

The entrypoint may take the run as its argument or take none and reach for bitexact.current(); both conventions work.

Unset, each of those four reports that nothing is configured and does nothing; they are off by default. Each is gated by its RBAC action (fork, regress) and the matching license entitlement.

These jobs re-execute the agent sequentially against one rate limit, so one runs at a time; a second is refused with how far along the first is. A sweep is capped at 1000 forks from the browser - a larger one is a batch job for bitexact fork-matrix, which has no cap.

An agent that uses the explicit-capture hooks (record_decision, record_human_decision, record_context_*, record_marker) needs no changes to be forked, replayed or regressed: those hooks are inert outside a recording run, so the same entrypoint serves every mode. The request carries only data - which step to branch at, and optionally a replacement model response - and the branch is gated by the fork entitlement and the fork RBAC action like any other write.

The entrypoint must re-execute the recorded path up to the fork point. If it does not, the replay diverges and the API returns 409 naming the exact diverging step; nothing is sealed. If the agent itself raises while continuing, that is also a 409, naming the agent's own exception - a counterfactual that never ran is never presented as one. A branch is named after its source (<run>-fork-<id>) and, if it did not complete, stays unsealed.

To try it against the seeded sample, point the variable at the sample's own agent:

bitexact seed
BITEXACT_AGENT_ENTRY=bitexact.sample:sample_agent  bitexact view

Then, in the inspector: open loan-decline-sample, press ⑂ Fork, and set Replace a model response to the run's one http_call. The box fills with what the model actually returned - a DECLINE citing 46% DTI and a 604 score. Change that text to an approval and press Create fork. The branch records is_adverse_action: false with no ECOA reason codes and the reviewer's decision as approve, because the sample's agent derives its decision from the model's answer and that same code re-ran against the new one. The replaced step is marked synthetic.

Replace the response rather than forking before it. A fork point earlier than the model call sends the continuation to the real provider, which without an upstream credential returns 401. The branch is still recorded honestly - the 401 is in the chain - but the agent may not check the status, so it can seal having tested nothing. The dialog reports any such refused call rather than showing a bare success, and POST /api/forks returns them in failed_calls.

Regressing and suites from the viewer#

Replay and fork → Replay diff replays a sealed run against the code running now and reports the first divergence. Regression suites on the same screen saves a corpus and a divergence threshold as a named suite, and runs it as the gate it exists to be; the result is written back and shown there.

Comparing two implementations on the same sealed roots needs two callables at once, so a deployment may name several entrypoints:

BITEXACT_AGENT_ENTRIES=baseline=myapp.v1:main,candidate=myapp.v2:main

The default BITEXACT_AGENT_ENTRY appears in that set as default. A request picks by NAME; a module path in a request body is never imported, so offering more entrypoints does not widen what a request can cause to run - the set stays entirely the operator's. A malformed pair is dropped rather than taken as a name. With fewer than two configured, the comparison panel says so and shows the variable to set.

Agents on many servers#

To record agents on many servers into a recorder on its own server - so recording costs the agents almost nothing and the evidence never lives on the hosts it describes - run a collector on each agent host and the recorder service on its own. See distributed recording.

Never in the way#

A failure in BitExact's recording never becomes a failure in your pipeline. Where a model call passes through a BitExact endpoint, what happens when that endpoint cannot be reached depends on whether BitExact code runs on the agent's side of the call:

• Wrapped clients (wrap_openai, wrap_anthropic) send through the run's own recording endpoint, a server inside the agent's process. If that endpoint cannot be reached - its thread has stopped, or the client is used after its run ended - the call fails over. It goes where the proxy would have sent it, by the same BITEXACT_UPSTREAM* settings, and never anywhere else: a call whose provider has no host configured (Azure, Bedrock) is not sent, and the agent gets the connection error. Each failed-over call is a declared gap: recorder_unavailable on the run, or run_ended noted beside a run that has ended. On a BITEXACT_COLLECTOR host, which keeps no store, that note goes to the host collector and the recorder writes it beside the run. verify then exits 3. Only a connection that never opened fails over, so no call is sent twice. Each call tries the endpoint first, and no failover lasts past the call it was made for. The endpoint is on loopback, so its connect is held to 100 ms: a stopped endpoint adds at most that to a call, not the operating system's connect retries (about 2 s on Windows). The upstream call keeps the client's own timeout. Replay, regression and fork runs never fail over. Strict mode raises the connection error instead.
• The recording proxy (bitexact serve), which a client reaches by URL, and bitexact mcp-proxy, over stdio or HTTP. No BitExact code runs on the agent's side of these connections, so nothing of ours can fail the call over. bitexact.mcp.wrap_session runs in the agent, but it wraps a session already connected through the proxy, and an MCP session cannot move to another server partway. If the proxy's process is not running, calls pointed at it fail as calls to any stopped service would. Over stdio the MCP server itself is the proxy's child, so it stops with the proxy. Where a proxy outage must not reach the agent, use the SDK or the collector. Where you run a proxy, run it beside the agent it serves, under a restart policy, with a liveness probe on /healthz (see The listener).
• The SDK's other capture (record_tool, the framework adapters) and the collector, which takes events from an in-process queue, put no process of ours in the call path.

When the recorder cannot do its job - an expired license, a full disk, an unreachable database, a body over BITEXACT_MAX_BODY_BYTES, an argument it cannot serialize, a run id it refuses - the agent's call goes through exactly as it would without BitExact, and the evidence says what it missed:

• The response says so. Through the proxy, a call recorded with a gap carries x-bitexact-recording: degraded; a call not recorded at all carries off. Through the SDK, run.recording_state is recording, degraded or off, and run.refusal holds the error that turned the run off.
• The run says so. The run records a declared capture gap: a signed marker naming the cause (license_lapsed, signing_key_missing, recorder_key_unlicensed, build_modified (this build's files are not the release's), payload_oversize, storage_unavailable, serialization, invalid_argument, recorder_unavailable, run_id_rejected, run_not_owned, journal_failed, run_ended, run_context_lost, spool_overflow, queue_overflow, intake_rejected, source_gap, source_conflict, late_after_gap, causal_inconsistency, checkpoint_failed, unclassified; docs/bundle-spec.md says what each means), where it happened, and the size of any body it withheld with its SHA-256 - keyed with the capture key, or dropped with no key, under a capture policy. A field the capture policy could not scrub in time (scrub_failed), or keeps hash-only with no capture key (capture_key_unavailable), is declared in the entry itself and counted with the gaps (docs/capture-controls.md). A capture policy that cannot be used opens no run at all (capture_policy_invalid, on the refusal ledger): the agent runs on, unrecorded. A gap that cannot join the chain (the run was already sealed) is noted beside the run; one with no run at all, or one neither the chain nor the note beside the run will take, goes to the log as capture_gap_unrecorded, every time. So does run_not_owned: a call through the proxy that names a run another writer opened - an SDK run, a run the recorder writes for its sources - is forwarded, answered x-bitexact-recording: off, and written nowhere in that run: no entry, no gap, no note beside it. It is counted and logged with the run id; strict mode refuses it with a 403.
• Verification says so. bitexact verify exits 3 on a run that declares gaps, listing each cause, until --allow-gaps accepts them; the viewer marks the run "incomplete capture"; the standalone verifier prints the count beside its OK. An intact chain of an incomplete capture is never presented as a complete record.
• Operations sees it. The proxy's /metricsz counts degraded_total and unrecorded_calls_total by cause, and each change of state is logged at warning: bitexact_degraded when a cause starts degrading the process, bitexact_recovered when an entry next commits with no gap, and bitexact_degraded again if the cause recurs. The proxy's /health/recording answers 503 while recording is degraded, and /healthz stays 200 for traffic.
• A dead store costs nothing per call. After 5 consecutive store faults (BITEXACT_BREAKER_FAILURES) the store circuit breaker opens: for 60 seconds (BITEXACT_BREAKER_OPEN_S) the store is not asked anything - no write, and no read either, such as the lookups the proxy makes to see whose run a call names - so no call waits on a store that will not answer, and each call is a declared storage_unavailable gap as any failed write is. A write that fails is a fault; a read is one only when the disk or database failed, not when it found a damaged run. Then one trial goes through: a write that commits closes the breaker, a fault opens it for another full period. It stands in front of every backend. Releasing a run's lock and the /healthz store check are never held back. Each change of state is logged at warning as store_breaker; /metricsz reports store_breaker_state (closed, open, half_open) and store_breaker_trips_total. A value that is not a positive number leaves the default.

BITEXACT_FAILURE_MODE=strict restores fail-closed recording - the caller gets an error and no unrecorded response is delivered - for development and conformance testing. Replay, regression and fork runs always raise: they serve answers from the record, so a missing record is their failure.

Storage and the disk-full policy#

The data directory must be a local filesystem (advisory locks are not reliable on NFS/CIFS). When a write or fsync fails — disk full included — the call is still delivered and the run declares a storage_unavailable gap (Never in the way); nothing already recorded is corrupted. Watch /health/recording, set BITEXACT_DISK_MIN_BYTES, and size the volume for your retention window. BITEXACT_MAX_BODY_BYTES (default 32 MiB) caps any single recorded body, and how much of one the recorder holds in memory: a larger one is delivered as it arrives and declared as a payload_oversize gap carrying its size and its SHA-256, keyed with the capture key, or dropped with no key.

Retention#

On the embedded file store:

bitexact purge --older-than 90 --archive /evidence-archive
bitexact purge --older-than 365 --delete --yes   # destroys evidence

Archived runs keep every file (manifest, sidecars, quarantines, reports) and verify from the archive unchanged. Regress reports accumulate beside manifests and are covered by the same purge.

On the PostgreSQL backend purge refuses, by design: it reclaims the file store by scanning runs/ on disk, and there is no such directory. Use bitexact archive instead, which moves a whole month out of the hot tables and leaves a catalog row behind. See PostgreSQL.

HA stance, stated plainly#

One recorder writes one run. The per-run OS lock makes a second writer fail loudly instead of corrupting a chain — contention is an error you see, not a mode you configure. Run one recorder per agent workload (sidecar or per-host); scale by partitioning runs across recorders, not by clustering writers.

Backup and restore#

PostgreSQL (BITEXACT_DATABASE_URL set): the evidence is in the database, and file tooling over the data directory copies almost nothing. Use continuous archiving for point-in-time recovery - see PostgreSQL.

The embedded store is the whole data directory (BITEXACT_DATA_DIR): runs, their checkpoints and custody, the transparency log, case and workflow ledgers, suites and packs. Back up all of it, together.

• Best: an atomic snapshot of the volume it lives on (a ZFS, LVM or cloud disk snapshot, or a Kubernetes VolumeSnapshot). It needs no pause.
• Otherwise copy it while nothing writes - stop the recorder, the proxy and any SDK process recording into it - or accept that a run being written as the copy runs can restore with a torn last line. The store quarantines such a tail (<run>.jsonl.torn) and bitexact health reports it as a seal-failure; the entries before it verify.

The restore drill - run it on a scratch host, on a schedule, not only when you need it:

1. Restore into an empty data directory, or a fresh database.
2. bitexact reindex (the embedded store's index is derived; rebuild it).
3. bitexact health - every run is re-verified; exit 2 on any critical finding, such as a run that fails verification. A quarantined torn tail is a seal-failure warning: status degraded, exit 0, so read the report.
4. Export one recent run per workflow (bitexact export <run> -o r.json) and verify it with the standalone verifier, pinned to your recorder key: that is the check an auditor will make.
5. If runs are anchored, bitexact health has compared each head against its anchor: a restore that rolled history back below an anchor fails there as anchor-rollback.

Image supply chain#

The Dockerfile pins its base image by digest and builds nothing: it installs one target's release bundle - the compiled product wheel, the open verifier wheel, that target's hash-locked requirements.lock and every dependency wheel it names - with --no-index --require-hashes, then pip check. It carries OCI labels, runs as a non-root user, and health-checks /healthz. Images exist for x86_64 and ARM64 under one multi-architecture tag.

The vendor-side release pipeline is scripts/release.sh, run on the release workstation (releasing.md): after the full checks pass on the tagged commit it compiles the product for every target in scripts/targets.py, with the vendor keys baked in before compiling; writes and signs each build's file manifest; checks every build in a clean runtime of its own kind; generates a CycloneDX SBOM per target (scripts/gen_sbom.py); pushes the images, signs their digest and attests the SBOMs with cosign (customers pin cosign.pub); and attaches the bundles, SBOMs, image tarballs, SDKs and deployment files to a draft release with a cosign-signed SHA256SUMS. deploy/compose and deploy/kubernetes name the image by tag for readability; pin the digest the image line of SHA256SUMS gives (ghcr.io/bitexact/bitexact@sha256:...) in production. deploy/pilot takes the image you loaded from the release tarball through BITEXACT_IMAGE, which the quick start pins to that image's digest.

Every release build carries the vendor keys, compiled in: the build refuses to start without them, and a license is checked against those keys and no other. There is no build in which a customer can sign their own license.

Environment reference#

Every variable the product reads, generated from the code by tools/gen_reference.py (a test fails when one is missing).