bitexact — the collector protocol#

How a host collector hands recorded events to the BitExact recorder. An agent never speaks this protocol itself: SDKs in any language post events to the collector on their own host, and the collector forwards them in batches over one connection to the recorder.

The protocol has one form, and no frame carries a version. Conformance vectors that every implementation must agree with are in protocol-vectors.json.

Frames#

A frame is a 4-byte big-endian unsigned length followed by exactly that many bytes of UTF-8 JSON. The JSON is one object with a string type.

A reader refuses, as a protocol error that closes the connection:

• a frame longer than the agreed max_frame;
• a body that is not UTF-8 JSON, repeats a key, carries NaN or Infinity, or nests deeper than evidence may;
• a body that is not an object, or has no string type;
• a connection that closes partway through a frame.

A connection closed between frames is a normal end.

Hashes are always computed over the RFC 8785 canonical form of a value, never over the bytes on the wire.

Unknown fields and messages#

• A reader ignores any field or capability it does not know.
• A message whose type the recorder does not define is answered with {"type": "unsupported", "of": "<type>"}, and the connection stays open.

Handshake#

The collector speaks first:

{"type": "hello", "protocol": "bitexact",
 "capabilities": ["tokens", "causal", "heartbeat"], "max_frame": 4194304}

The recorder answers:

{"type": "welcome", "protocol": "bitexact",
 "capabilities": ["tokens", "causal", "heartbeat"], "max_frame": 4194304,
 "key_id": "<recorder signing key id>"}

• capabilities in the welcome are those both sides offered. With heartbeat agreed, the collector sends heartbeats (see Heartbeats); a recorder that did not agree it is sent none.
• max_frame is the smaller of the two, and at least 65536.
• A refused hello is answered with {"type": "error", "reason": "..."}, and the connection closes.

Events#

The collector sends {"type": "batch", "events": [...]}. Each event is:

Custody hints#

An agent may say how a field must be kept: stricter than the capture policy (scrub, hash-only, never looser), or cleaned before BitExact saw it (scrubbed-upstream, recorded as the agent's claim; scrubbed-upstream:TOOL names the tool, letters, digits and ._-, up to 64 characters, starting with a letter or digit). Any other hint refuses the event. A tool wrapped with custody sends its event as:

{"run_id": "run-1", "source_id": "c-abc:1", "seq": 1, "kind": "tool_call",
 "custody": {"args": "scrubbed-upstream:presidio", "result": "hash-only"},
 "data": {"tool": "kyc_lookup", "args": ["[PERSON], born [DATE]"],
          "status": "ok", "result": "match"}}

The collector spends the hints when it applies its host's policy, and the entry declares args as {"mode": "scrubbed-upstream", "by": "presidio"} and result as {"mode": "hash-only"}: committed, its value not kept.

Credentials#

A query parameter, object key or name=value text named in the credential table (credential_keys in protocol-vectors.json, compared without case) carries a credential. A wrapped client records such a query parameter with its value replaced by [redacted], URL-encoded, and every other parameter as sent: https://api.example.com/v1/files?token=t0k&sig=abc&page=2 is recorded as https://api.example.com/v1/files?token=%5Bredacted%5D&sig=%5Bredacted%5D&page=2. A parameter's name is matched once URL-decoded, ignoring ASCII case only: api%5Fkey is a credential, a name that does not decode is compared as sent, and a look-alike letter (%E2%84%AAey, a Kelvin sign) names another parameter. Every other byte of the URL is kept exactly as sent: each other parameter's encoding, an empty parameter and the fragment. A user name or password in the URL is recorded as [redacted] too. The recording proxy is sent the query alone and records it by the same rule. The capture policy's credentials detector replaces the value with [CREDENTIAL]. The vectors' recorded_urls and scrubbed rows are the encoded output every implementation must give.

What the recorder writes#

The recorder records each accepted event as an ordinary chain entry. The entry format, its hashing, its signing and the open verifier are all unchanged. The source coordinates go into the entry's data under aer_source, which only the recorder writes:

"aer_source": {"source_id": "c-7f3a91:3", "seq": 812, "actor": "planner",
              "id": "7f3a91c2b0e4d5a6", "hlc": [1730000000123, 4],
              "caused_by": ["1b2c4d5e6f708192"],
              "peer": "spiffe://acme/agents/h12", "token": "<jti>"}

An event that names its sdk label is written with the entry field via set to relay: and the label (relay:sdk-ts), inside the entry hash; an event that names none is written without via. An event whose label is outside the set is written without via too, and its source block declares the label it refused by the length of the label's JSON encoding (keys sorted, compact, ASCII) and a digest of that encoding keyed with the capture key, "sdk_refused": {"bytes": ..., "keyed": ..., "nonce": ..., "key_id": ...}, which a holder of that key tests a candidate label against with capture_policy.open_withheld. With no capture key the digest is dropped and said so, "sdk_refused": {"bytes": ..., "digest": "dropped: no capture key"}, and a capture policy that cannot be read drops it as "dropped: capture key unavailable", declared capture_key_unavailable. The label is the sender's free text, so its value is never written, and an unkeyed digest of a short label would let anyone test each guess at it. The source block's actor is the sender's free text too, and passes through the capture policy's detectors, at the collector and again at the recorder. The label is what the sender asserts: any process holding the host secret could send any label. The peer beside it is what the recorder's mTLS authenticated.

The recorder checks every causal claim and declares what does not hold, as a causal_inconsistency gap beside the event:

• a source's clock that does not move strictly forward;
• a clock more than BITEXACT_HLC_MAX_SKEW_MS ahead of the recorder's;
• an event id used twice in a run;
• and, at the seal, any caused_by naming an event the run never received.

Identity events#

An identity event's data is checked field by field by the one identity rule every write door applies (the in-process SDK and the recorder alike; the cases are identity_fields and identity in protocol-vectors.json):

• scope is a list of strings; every other known field is a string, and on_behalf_of and authority_ref are not empty;
• on_behalf_of_type is one of human, organization, service or agent, and interaction_mode one of interactive or autonomous;
• on_behalf_of_type and authority_ref come with on_behalf_of in the same event;
• workflow_digest is never accepted: the recorder computes it from the composition fields when workflow is present;
• a field set to null is left out, and a key the protocol does not name is kept as sent;
• an event that states no field at all ({}, or every field null) is refused whole, with a gap whose fields is empty: no identity entry lands that a reader could take for a bound one.

A field the rule refuses is never written in a wrong shape. The recorder writes the rest of the event, and beside it a gap marker:

"gap": {"cause": "invalid_argument", "where": "recorder",
        "fields": {"interaction_mode": "interaction_mode must be one of interactive, autonomous"},
        "withheld": {"length": 33, "keyed": "<hex>", "nonce": "<hex>", "key_id": "<hex>"},
        "seqs": [812, 812], "source_id": "c-7f3a91:3", "peer": "...", "policy": "open"}

fields names each refused field and why; withheld is the length of the canonical JSON of the refused fields as sent and its SHA-256 keyed with the capture key, so a capture-key holder proves what was refused (verify --open-withheld) without it being stored. With no capture key, or no capture policy, the digest is dropped and said so ("digest": "dropped: no capture key"): a refused principal or party is short enough to guess, and an unkeyed digest would let anyone holding the bundle test each guess. When the rule keeps nothing, the marker takes the event's place and carries its aer_source; the result is gap_recorded with cause invalid_argument. The entry's aer_source names the event as the source sent it, so a resend of the same seq is still a duplicate and never mistaken for equivocation: as sent, an unkeyed digest of the whole event, when the record holds all it hashed; as sent_keyed, keyed with the capture key, when the identity rule refused a field or the capture policy changed what is stored; and not at all when that happens with no capture key. A redaction of any field of the entry redacts sent with it (the redaction claim names it): left in place, it would let anyone holding the bundle test a guess at the withheld value. A resend of an entry whose sent was redacted or dropped is taken as a duplicate.

bitexact show --causal prints a run in causal order: each source's own sequence and the caused_by links, ties broken by clock and then by chain position. bitexact.run(..., mode="replay", actor="planner") replays one agent of a multi-agent run against its own entries only.

Ending a run#

{"type": "source.end", "run_id", "source_id", "last_seq", "run_end", "token"} says a source has sent everything it will send to a run, up to last_seq. With run_end, it also asks for the run to end. The recorder seals the run once every source that wrote to it has ended and everything each said it sent has landed. If that has not happened once BITEXACT_SEAL_GRACE_S has passed since the request, it seals anyway, with a source_gap declared for each source still open. A source that ends speaks for the earlier epochs of the same collector (<collector>:<epoch>).

peer is the identity of the authenticated mTLS peer the event arrived from. It is taken from the connection and never from the payload. token is the id (jti) of the run token that authorized the write.

Results#

The recorder answers each batch with {"type": "acks", "id": ..., "results": [...]}, one result per event, in order:

A source that lost events before they reached the recorder declares the hole with {"type": "gap", "run_id", "source_id", "first", "last", "cause", "token"}, where cause is spool_overflow or source_gap. The run records it as a capture gap, and the source resumes at last + 1.

Heartbeats#

With the heartbeat capability agreed, the collector sends a heartbeat as soon as it connects and then every BITEXACT_HEARTBEAT_S seconds (default 60):

{"type": "heartbeat", "source_id": "c-7f3a91:3",
 "pending_events": 0, "spool_bytes": 0, "spool_retrying": 0,
 "spool_unreadable": 0, "spool_max_bytes": 1073741824, "unrecorded": 0,
 "dropped_ranges": {"run-42": [[13, 15]]}, "heartbeat_s": 60,
 "seen": [{"run_id": "run-42", "sdk": "sdk-ts",
           "last_accepted_at": "2026-10-30T13:59:10+00:00"}]}

A heartbeat whose seen would not fit one frame is split across frames. The recorder answers each with {"type": "ok"} once it has kept it - with results, one per hwm entry, shaped as a batch result plus run_id, when it carried any - or {"type": "refused", "reason": "..."}; whatever was not kept is sent again with the next heartbeat. The recorder keeps it as operational state under the peer its certificate names and the source id, resolving each run to its workflow, environment and agent: never evidence; only a settled hwm entry writes to a chain, as the source's declared hole, and never opens or extends a sealed run. It is never in recording's way. It tells a quiet agent from a dark host; it never stands in for evidence that a run was recorded.

Authorization#

Nothing is written without it.

• Transport. The recorder serves TLS 1.3 and requires a client certificate from the configured CA. It refuses to start otherwise. Plaintext exists only as an explicit development switch that binds loopback.
• Identity. The peer is the SPIFFE ID (URI SAN) in the client certificate, else its common name.
• Run tokens. Every event carries one. A token is canonical JSON claims (format: aer-run-token, jti, run_id, peer, kinds, exp, and for a delegated token delegated_from) signed with the recorder's own key, as base64url(claims).base64url(signature). It authorizes exactly one peer to write the listed kinds into one run until exp, by recorder time. A collector asks for one with {"type": "token.request", "run_id", "kinds"} and is answered {"type": "token", "token"} with the kinds the policy grants, or {"type": "refused", "reason"}. A token is refused if its signature does not verify, it has expired, it names another run or another peer, it does not grant the kind, or its jti, or that of any token it was delegated from, has been revoked. A token can never grant a kind a source may not send. A token whose claims do not name that format is refused, so no other document the recorder key signs reads as one.
• Delegation. {"type": "token.attenuate", "run_id", "token", "peer", "kinds", "ttl_s"}, from the parent token's holder, is answered {"type": "token", "token"} or refused. The child grants a subset of the parent's kinds, expires ttl_s from now and never after the parent, is bound to the sub-agent host's peer, and names in delegated_from the {jti, peer} of every token above it, root first, at most 8 deep: it can only narrow. Every entry a delegated token authorizes carries the chain in aer_source.delegated_from.
• Refused writes. Every write, hole, source end or attenuation a token does not authorize is answered rejected (or refused). In a run that exists it is recorded on the run's custody chain as write-refused, naming the reason, sources, seqs, kinds and the token by digest, with identity: observed and submitted_by the authenticated peer; a refused write never opens a run. When no fresh token can cure the refusal (the kind is not granted, or the token is delegated and has expired or been revoked), the answer is gap_recorded with cause token_rejected: a gap takes the event's place, carrying its aer_source, so the stream continues and a resend is a duplicate. An expired or revoked root token is answered rejected, and the collector asks for a new one.
• One write per seq. A seq resent unchanged is a duplicate. Resent with different content, it is refused as equivocation: the original stands, and the run declares a source_conflict gap carrying the hash of the rejected content.
• A declared loss stands. Content for a seq its source declared lost is refused as declared lost, and the run declares a late_after_gap gap carrying its hash. An honest collector can send it, when a spool segment it could not read becomes readable, so it is not called equivocation.
• Liveness. A connection that does not say hello within 10 s, or is silent for 300 s, is closed.

What this cannot prove: a compromised agent host can still lie about what its own agent did. The content, actor, hlc and caused_by are what that host asserts, as with the in-process SDK. The chain proves which authenticated peer sent each event, under which token, and that nothing changed after it arrived.