Event delivery#

BitExact can tell the systems you already watch - a SOC webhook, Splunk, Datadog - when something worth knowing happens to the evidence: a run is sealed, a legal hold is placed, a redaction is approved, a run is refused, an integrity alert opens or clears.

An event is a signed notification, not evidence. Each one is derived from a fact BitExact has already written - a chain entry, a custody event, a ledger event, a purge record or a health alert - and never says more than that fact. run.sealed says a run_end entry was written; it never says the run verifies. A receiver that needs a verdict runs the verifier on the run the event names. Events are not in bundles or evidence packs, and the verifier never reads them.

Events carry metadata only: ids, counts, hashes, causes and roles. They never carry recorded content, and never a free-text reason (a redaction's reason can name the data subject). The operator an event names - by, and requested_by on run.redacted - and the party a run acted for - on_behalf_of and authority_ref on run.sealed - reach only a destination that sets include_principal = true; every other destination is sent the event without those fields.

Turning it on#

Delivery is opt-in. With BITEXACT_EVENTS_CONFIG unset, nothing is enqueued, no thread starts and nothing is sent. Set it to a TOML file:

[defaults]
max_pending = 100000          # rows held at most; past it, a declared drop
health_every_s = 300          # how often the alert round runs
schedule = "standard"         # the retry schedule, or "short" for test environments

[[destination]]
name = "soc-webhook"
format = "standard-webhooks"
url = "https://hooks.internal.example/bitexact"
secret_files = ["/etc/bitexact/hook-2026-10.secret"]
types = ["alert.*", "run.hold_*", "redaction.*", "refusal.recorded"]

[[destination]]
name = "splunk"
format = "splunk-hec"
url = "https://splunk.internal:8088/services/collector/event"
token_file = "/etc/bitexact/hec.token"
index = "bitexact"
ack = true

[[destination]]
name = "datadog"
format = "datadog-logs"
site = "datadoghq.eu"
api_key_file = "/etc/bitexact/dd.key"

Check it with bitexact events check, which prints each destination and the fingerprint of each secret (the first 8 hex characters of the SHA-256 of the key bytes, and the file's date), never the secret, and exits 1 on an error. It also resolves each destination's host and judges the addresses as a send does (T3): a host that resolves to a cloud metadata, link-local, unspecified or multicast address, or to loopback without allow_loopback, is that destination's error, naming the address. A host that does not resolve now is a warning, since each send resolves it again. The dispatcher vets every destination the same way when it starts, raising events-destination-refused for each refused one before any event waits on it. A file that does not load turns delivery off and raises the critical events-config-invalid alert, naming what is wrong. Events are still kept meanwhile, with no destination yet (bitexact events status counts them as unrouted, and they count toward max_pending); the first dispatch round after the file loads sends each one to every destination configured then whose types accept it, once.

Events are written to the store's outbox by whichever process records the fact: the SDK, the proxy, the recorder service, the inspector or the CLI. Writing one is one store write after the fact is committed; it never touches the network, never raises, and never fails what it notifies about, in every failure mode including BITEXACT_FAILURE_MODE=strict. Delivery happens on a dispatcher thread of its own, which no recording path and no process exit waits for. The recorder service and bitexact serve run one; so does bitexact events run in the foreground, and bitexact events deliver --once runs one round, for cron. Several dispatchers share one outbox: on PostgreSQL each claims its own rows, and on the embedded store one at a time holds the dispatch lease.

Configuration reference#

[defaults], every key optional:

Each [[destination]]:

A secret is only ever named by its file. A secret written into the TOML - secret, token, api_key, password and the like - is refused at load, naming the key. Every file is read at send time, so replacing a file rotates its secret. An unknown key is refused, never ignored.

Events#

Every type carries the prefix ai.bitexact. (ai.bitexact.run.sealed); types, Splunk's bitexact_type and Datadog's tags use the name without it. subject is the run id, or the entity a ledger event is about. An event derived from a chain entry - each run.* event below, and a capture.gap on the chain - also carries entry_hash in its data, the hash of the entry it came from, so a receiver can tie the notification to evidence the verifier checks.

alert.opened covers every alert bitexact health raises - a run that fails verification, an anchor rollback, a transparency-log finding, an unsealed run, a scrub failure, an expiring certificate or license, the event alerts below - and every kind added later. The alert round runs every health_every_s on the one dispatcher holding the alert lease, so an alert is opened once while it stays open, and resolved once when it clears. An alert is told apart by its kind, run_id, path, expectation, peer, source_id, destination, source, service and host: two expiring certificates, the same certificate on two hosts, two silent workflows, two collectors forwarding unscrubbed values, or two destinations refusing their credentials, open and resolve as two alerts. A certificate or CRL alert is raised from the expiry facts every serving process publishes, whichever process runs the round (Ops health).

The envelope#

Every event is a CloudEvents 1.0 structured event, built once as canonical JSON (RFC 8785) when it is enqueued. Its bytes are what every attempt sends.

{"data":{"gaps":{},"head_hash":"...","run_id":"loan-7731","steps":42},
 "datacontenttype":"application/json",
 "id":"evt_3q2f0c9b1a7e4d58b6c2a1f0e9d8c7b6",
 "source":"urn:bitexact:deployment:<deployment id>",
 "specversion":"1.0","subject":"loan-7731",
 "time":"2026-10-30T14:02:11.204118+00:00",
 "type":"ai.bitexact.run.sealed"}

• id is derived from the fact (BLAKE2b-128 over the kind of record, the entity and the fact's hash; for capture.gap, over the run and the cause), so the same fact is always the same event, and a resend is a duplicate you can drop.
• source names the deployment by a random id written once to the store, never a host name.
• time is the fact's own time, not the time of sending.
• A body over 16 KiB is refused when it is enqueued, counted under events_oversize_total (never as a store failure) and logged.

Standard Webhooks delivery#

A standard-webhooks destination is sent one POST per event, signed as Standard Webhooks specifies:

POST <url>
content-type: application/cloudevents+json; charset=utf-8
user-agent: bitexact-events
webhook-id: evt_3q2f0c9b1a7e4d58b6c2a1f0e9d8c7b6
webhook-timestamp: 1793541731
webhook-signature: v1,<base64 HMAC-SHA256> v1,<base64 HMAC-SHA256 under the old secret>

• The signed content is <webhook-id>.<webhook-timestamp>.<body>. webhook-id is the event id and never changes across retries; webhook-timestamp and the signatures are made afresh for each attempt.
• A secret is whsec_ followed by base64 of 24 to 64 random bytes. bitexact events secret --out FILE writes a new one, owner-only, and prints its fingerprint.
• Rotation. Each file in secret_files adds one v1, signature. Add the new file, deploy the receiver with both secrets, then remove the old file.
• A public key instead of a shared secret. signing_key_file names an ed25519 key (whsk_ followed by base64 of its 32-byte seed) and adds a v1a, signature, which a receiver checks with the whpk_ public key and no secret. It is a key for events only, never the recorder key: the recorder key signs evidence, and an event is not evidence.
• A webhook destination with neither is unsigned: it is allowed, and raises events-destination-insecure, since its receiver cannot tell a forged event.

Verifying a delivery#

Verify the signature over the raw body bytes before parsing them, refuse a timestamp far from your clock, and drop an id you have already processed. Any Standard Webhooks library does this. So does bitexact.events.verify_delivery(headers, body, secrets, tolerance_s=300), which also checks v1a with a whpk_ key, and bitexact events verify-delivery --secret-file F --headers H --body B checks a delivery you captured.

A receiver that runs on the Python standard library alone, to copy:

import base64, hashlib, hmac, json, ssl, time
from http.server import BaseHTTPRequestHandler, HTTPServer

class Rejected(Exception):
    """The delivery does not verify: do not act on it."""

def verify(headers, body, secrets, tolerance=300, now=None):
    """Check one BitExact delivery; return its webhook-id. `body` is the raw
    bytes as received; `secrets` every whsec_ secret you hold (two during a
    rotation). `now` is for tests; leave it None."""
    msg_id = headers.get("webhook-id") or ""
    stamp = headers.get("webhook-timestamp") or ""
    offered = (headers.get("webhook-signature") or "").split()
    if not (msg_id and stamp.isascii() and stamp.isdigit() and offered):
        raise Rejected("missing webhook-id, -timestamp or -signature")
    if abs((time.time() if now is None else now) - int(stamp)) > tolerance:
        raise Rejected(f"webhook-timestamp {stamp} is too far from this clock")
    signed = f"{msg_id}.{stamp}.".encode() + body
    for secret in secrets:
        key = base64.b64decode(secret.strip().removeprefix("whsec_"))
        expected = base64.b64encode(hmac.digest(key, signed, hashlib.sha256))
        for scheme, _, sig in (item.partition(",") for item in offered):
            if scheme == "v1" and hmac.compare_digest(sig.encode(), expected):
                return msg_id
    raise Rejected("no signature matches a secret held")

class Receiver(BaseHTTPRequestHandler):
    secrets, seen = [], set()     # keep `seen` somewhere durable

    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("content-length", 0)))
        try:
            msg_id = verify(self.headers, body, self.secrets)
        except Rejected as exc:
            self.send_response(401)
            self.end_headers()
            self.wfile.write(str(exc).encode())
            return
        if msg_id not in self.seen:      # a resend of one already handled
            event = json.loads(body)     # act on event["type"], event["data"]
            self.seen.add(msg_id)
        self.send_response(204)
        self.end_headers()

if __name__ == "__main__":
    Receiver.secrets = [open("/etc/receiver/bitexact.secret").read()]
    server = HTTPServer(("", 8443), Receiver)
    tls = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    tls.load_cert_chain("/etc/receiver/cert.pem", "/etc/receiver/key.pem")
    server.socket = tls.wrap_socket(server.socket, server_side=True)
    server.serve_forever()

Answer 2xx to an id you have already processed: a delivery is sent until it is accepted, so refusing a duplicate only has it sent again. A refusal's body is kept in the dead letter's error, which, status included, is cut to 500 characters, so a receiver whose clock is off says so where the operator looks.

Splunk#

A splunk-hec destination sends to the HTTP Event Collector, batched up to 100 events or 512 KiB per request:

POST https://splunk.internal:8088/services/collector/event
Authorization: Splunk <the token in token_file>
X-Splunk-Request-Channel: <a UUID, when ack = true>

{"time": 1793541731.204, "host": "<deployment id>", "source": "bitexact",
 "sourcetype": "bitexact:event", "index": "bitexact",
 "fields": {"bitexact_type": "run.sealed", "bitexact_id": "evt_...",
            "severity": "info", "run_id": "loan-7731"},
 "event": { ...the CloudEvent... }}

1. In Splunk, create an HTTP Event Collector token (Settings, Data inputs, HTTP Event Collector). Allow it the index you name in index; with no index, events go to the token's default index.
2. For ack = true, enable indexer acknowledgment on the token. An event then stays awaiting_ack until /services/collector/ack reports it indexed; with no acknowledgment within 5 minutes it is sent again with the same bitexact_id. Indexer acknowledgment is Splunk Enterprise only (Splunk Cloud acknowledges only Amazon Data Firehose), so ack = true is refused for a *.splunkcloud.com URL. Without ack, an HTTP 200 counts as delivered: accepted by HEC, not yet indexed.
3. Write the token to token_file, owner-only, and run bitexact events test --destination splunk.

time is the fact's time, so an event delivered late after an outage lands where it happened. fields are indexed, and delivery is at least once, so dedupe on the event id:

index=bitexact sourcetype="bitexact:event" | dedup bitexact_id
index=bitexact bitexact_type::alert.opened severity::critical | dedup bitexact_id

Datadog#

A datadog-logs destination sends to the Logs intake of its site (https://http-intake.logs.<site>/api/v2/logs), gzip-compressed, batched up to 1000 entries and 4 MiB uncompressed per request:

[{"ddsource": "bitexact", "service": "bitexact", "hostname": "<deployment id>",
  "ddtags": "bitexact_type:run.sealed,severity:info,env:prod",
  "message": "<the CloudEvent, as a JSON string>",
  "bitexact": {"id": "evt_...", "type": "run.sealed", "run_id": "loan-7731",
               "time": "2026-10-30T14:02:11.204118+00:00"}}]

1. Create an API key (Organization Settings, API Keys) and write it to api_key_file, owner-only.
2. Set site to your organization's site; any other value is refused.
3. Add facets on @bitexact.type, @bitexact.run_id and @bitexact.id, and dedupe on @bitexact.id.

No timestamp attribute is sent, so Datadog stamps each entry at intake and a late delivery is not refused for its age; the fact's own time is @bitexact.time.

OCSF#

schema = "ocsf" on a splunk-hec or datadog-logs destination sends the integrity alerts - alert.opened and alert.resolved of verify-fail, anchor-rollback, log-run-gone, log-refusal-gone and log-unreadable - as OCSF 1.9.0 Detection Findings (class 2004; Create on opening, Close on resolution; finding_info.uid is the alert.opened event's id on both, the Close taking it from opened_id, so the two are one finding). Every other event is sent as the CloudEvent itself, unmapped: a capture gap or a legal hold is no finding, and mapping one onto a class would misstate it.

Delivery guarantees#

• At least once. An event is sent until a destination accepts it or its attempts run out. A dispatcher that dies between sending and recording the answer leaves the event to be sent again, with the same id. Dedupe on webhook-id, bitexact_id or @bitexact.id.
• No ordering. Events of one run can arrive in any order, and an alert.resolved can arrive before its alert.opened. Order by time, and match a resolution to its opening by opened_id.
• A crash loses no fact. A process that dies after a fact is committed and before its event is written loses only the notification. Each dispatcher re-derives the last 24 hours of events from the record when it starts - and each bitexact events deliver --once is a start, so cron alone closes the window - and bitexact events reconcile --since <ISO instant> does it for any window: chain entries, custody acts, failures-sidecar gaps, ledger acts and purge records. Custody acts and failure notes are found through the store's index of their times, so a window costs the runs that wrote in it, not every run. The ids are deterministic, so an event already in the outbox is kept once.

Retry schedule#

standard is ten attempts over about 3 days 3 hours, the Standard Webhooks schedule, so an integrity alert outlives a weekend outage; short is seven over about 9 hours. Every delay after the first moves by up to 20% either way, so the events of one outage do not all come due together. A Retry-After on a 429 or 503 is honored, up to the next slot's delay.

At most 64 KiB of an answer is read, within timeout_s.

Dead letters#

An event that will not be delivered is a dead letter, kept 30 days; delivered rows are kept 7 days. Dead letters raise events-dead-letter until they are retried or pruned.

bitexact events status                     # rows by state, oldest pending, dead letters, last delivered
bitexact events list --state dead          # each dead letter, its status and error
bitexact events retry EVENT_ID             # requeue one event's dead letters
bitexact events retry --all-dead --destination splunk

Each retry is written to the access log with the principal (--by, default the OS user) and its role: operator from the command line, the signed-in role from the inspector. A dead letter's error is the status and the answer, cut together to 500 characters, with every configured secret, token and key, and the values of the Authorization, DD-API-KEY and webhook-signature headers, removed first. An event for a destination no longer configured is a dead letter at once.

In the inspector#

Settings, Events in the inspector shows event delivery to an administrator: it needs the settings action, and the section is hidden from every other role. It reads GET /api/events and shows two tables:

• Event destinations. Each destination the configuration names, with its format, its host (never the URL's path or query), each secret's fingerprint and file date, when it last delivered (kept by the store as each delivery settles, so pruning delivered rows does not clear it), its pending and dead counts and, for Splunk, whether it waits for indexer acknowledgment. A configuration that does not load is shown with what is wrong.
• Dead letters. Each event a destination did not accept, newest first: its type, subject, destination, attempts, and the last status and error.

GET /api/events also answers each destination's types, the public half of its signing key, what makes it insecure, its load error and its awaiting_ack count, and each dead letter's created_at and settled_at. It never answers a secret or a secret file's path.

Destinations are read-only here. They are named in the file BITEXACT_EVENTS_CONFIG points to, on the host, and no screen or API writes one (T3). Two actions post, with the X-BitExact-Action header:

• Retry and Retry all (POST /api/events/retry, with destination and either event_id or "all": true) requeue dead letters, as bitexact events retry does. Each requeue is written to the access log with the signed-in principal and role.
• Send test event (POST /api/events/test, with destination) enqueues one events.test naming the signed-in principal as by, whatever the destination's types. The next delivery round sends it. It is refused while delivery is off and while the license is expired; a retry is not.

After the license expires, delivery reaches only the destinations the license covered. A destination added to the file after expiry is refused: its events settle as dead letters, raising events-dead-letter, and a retry after renewal sends them (licensing).

Health and metrics#

bitexact health and the inspector's health view raise:

These are alerts like any other, so they also leave as alert.opened to the destinations that still work. The recorder's /metrics serves bitexact_events_pending, bitexact_events_dead, bitexact_events_dropped_total, bitexact_events_enqueue_failed_total, bitexact_events_oversize_total, bitexact_events_delivered_total (by destination) and bitexact_events_oldest_pending_seconds. Every _total is a deployment-wide counter kept in the store: every process on every host raises it, and no prune lowers it, so a scraper never sees a false counter reset.

An enqueue failure or an oversize body is counted in the store as well as in the process, so bitexact health run as its own process sees the recorder's. When the store refuses the count too, the process that saw it still reports it. bitexact events status exits 2 on events-dropped or a configuration that does not load.

Clocks#

Every host that dispatches events must keep its clock synchronized by NTP (or chrony, or your cloud's time service), and so must every receiver. webhook-timestamp is the sending host's clock, and a receiver refuses a delivery more than 5 minutes from its own: a skewed clock makes every delivery fail, retry and die.

Threat model#