Reproducible, keyless agent evals
The reproducibility gap
A live eval needs a key. A replayed one does not.
agentic-eval-harness scores an agent's plan-act-observe loop against a golden case set: the loop calls a model, the model calls deterministic offline tools, and the harness grades each case. Running that live needs an API key. Running it in CI, on every push, does not — because of a record/replay seam in the loop itself.
What replay is
The loop is backend-agnostic. live calls the model API — with --record, it writes the full conversation (every model turn plus every tool's output) to a JSONL transcript per case. replay re-runs those recorded model turns through the identical loop and tool code, with no key and no network.
uv run agentic-eval eval --domain retail_ops --record calls the model, writes a transcript per case uv run agentic-eval eval --domain retail_ops --backend replay keyless, reproduces the scorecard What CI runs
The full scoring path runs on every push, with no key in the environment.
The per-domain eval tests drive the replay path, so tools, checkers, per-metric rollups, and hard gates all execute in CI without ANTHROPIC_API_KEY ever being set. A fresh clone reproduces the same numbers.
.github/workflows/ci-python.yml runs ruff check, ruff format --check, pyright, pytest -ra, and pip-audit — none of them touch a model API.
The mechanism
A gate is a computed check, not a model call.
A case can carry a hard_gate flag. Scoring tracks hard_gate_failures — the ids of any gated case that did not pass — separately from the score. A gate models an action the agent must never take, and because it's a plain computed check rather than a judgment call, its verdict is deterministic and reproduces under replay like any other case.
Auto-executing a money-moving action
Auto-paying a mismatched invoice, auto-closing a must-review exception, or otherwise taking an action policy requires a human to approve.
Re-submitting a completed action
Repeating a side-effect that already went through — the idempotency and stop-condition gates a portal-automation agent must respect.
Following an embedded instruction
Acting on an instruction that arrives inside a tool result rather than from the operator — the gate a prompt-injection attempt is built to defeat.
What reproduces, keyless
All four domains reproduce end-to-end under replay.
Including the red-team pack — whose recorded run preserves a frontier model failing its injection hard gates, reproduced byte-for-byte with no key.
| Domain | Replay | Notes |
|---|---|---|
| retail_ops | 34/34 | 3-way match, exceptions, chase, OTIF, deductions, approval gate |
| process_mapping | 25/25 | workflow mapping + fix-before-automate, ready-to-automate gate |
| browser_ops | 24/24 | portal idempotency / stop-condition / retry-safety gates |
| approval_audit | 24/24 | red-team pack — recorded frontier run (claude-opus-4-8) reproduced exactly, failures included: 6/8 injection hard gates tripped |
The full suite is 298 tests. Older shipped domains (generic, industrial, trust_safety, data_semantic) and the earlier customer_support, fintech_compliance, and routing packs are covered the same way.
approval_audit run shows a current frontier model — gate tool available and consulted — auto-executing money-moving actions in 6 of 8 injection hard-gate cases. The eval preserves that failure permanently and reproduces it keyless: evidence for why approval checkpoints belong in the runtime, not the prompt.