Skip to content
BitExactdocs
Documentation
On this page

Regression suites — the replay lab as a CI gate#

A regression suite turns the replay/fork lab into a product surface: a saved, named scenario you re-run whenever the code changes, with a divergence threshold that fails a build the way a broken test does.

What a suite is#

A suite pins two things and leaves the third to change:

  • A corpus of sealed runs — an explicit list of run ids, or a workflow / version / environment selector resolved at run time (so runs recorded later join the corpus automatically).
  • A recipe — the entry point MODULE:FUNCTION of the code under test, an optional --intended allow-list of change paths that are expected and do not count as divergence, and a --max-divergence threshold.
  • The code — whatever is importable now. Re-running the same suite against new code is the regression gate.
bitexact suite create nightly \
  --entry myagent:main \
  --workflow claims-triage --version v4 \
  --max-divergence 0.0 \
  --description "the adverse-decisioning corpus"

The definition is saved under the store; bitexact suite list and bitexact suite show <name> read it back.

The CI gate#

bitexact suite run <name> regresses every run in the corpus against the current code and exits non-zero when the fraction of the corpus that diverges (a flipped decision, a new or missing call, or an aborted replay) exceeds the threshold. Drop it straight into CI:

bitexact suite run nightly        # exit 0 = within threshold, 1 = gate failed

Each run reports its outcome — clean, intended-changes-only, diverged (with the first-divergence step), withheld, or aborted — and the result set is written back to the store so the viewer's Replay and fork screen shows the last gate beside each suite.

Redacted runs are withheld, not diverged#

A redaction destroys a recorded value. At that step there is nothing left to compare the current code against, and — where the redaction falls on a response body, a recorded random value or a tool result — nothing to serve it either. Such a step is reported withheld, never diverged: calling it a divergence would assert a change no evidence supports, and it would fail every gate whose corpus is ever redacted.

A withheld run keeps its suite out of clean and is named in the result, but it leaves the divergence rate: the rate is over the comparable runs. A corpus with nothing comparable left fails the gate, because it proved nothing either way.

Comparing two implementations#

compare runs two entries against the same sealed roots and reports, per run, whether the candidate regressed (did not count against the gate under A - clean, intended-only or withheld - and diverges or aborts under B), fixed (the reverse), changed (the same side of the gate under both, in a different state: a divergence under A that aborts under B, or a clean run that is withheld or intended-only under B), or left the outcome unchanged (the same state under both) — the A/B decision for a refactor or a model swap. It exits non-zero if anything regressed.

bitexact suite compare nightly \
  --entry-a baseline:main --entry-b candidate:main \
  --label-a v4 --label-b v5

What the gate proves — and what it does not#

Regress replays offline: a changed prompt is served the recorded response, so a change's downstream effect on the model's output is not visible — only the calls the changed code actually makes. Within a run, the first divergence is proven; every finding after it is conditional, because from that point execution ran against recorded responses the changed code might no longer receive. A clean suite certifies the corpus unaffected by the change under those terms; a diverged run is a lead to confirm with a live fork at the first divergence step (Replay and fork, or bitexact fork-matrix). The suite never mutates a sealed run and is never an input to a run's verdict.