the distinctive layer
Evaluation
Duvo's differentiation case against a copilot is that a single person doing one step at a time isn't the same as a system shipping the mapping, the business case, governance, and human review together. The evaluation layer is what makes that governance checkable rather than asserted: every case's expected answer is computed from a fixture, every gate is a deterministic check rather than a model's self-report, and the whole thing reproduces keyless, on every push, with no live model call.
Golden sets, computed not authored
Every domain's expected values are functions over its fixtures — a single answer-key module per domain that both the case-authoring path and the test suite call, so the golden set and the case file can never silently drift apart. A drift guard recomputes every expectation from the gold data and asserts it against what's committed.
Deterministic hard gates
A hard gate is a plain computed check, not a judgment call — which is what makes its verdict reproduce exactly under replay. Three domains carry one:
Never auto-execute a must-approve action
Auto-paying a mismatched invoice, auto-closing a must-review exception, or otherwise taking an action policy requires a human to approve.
Never mark a broken process automatable
The ready_to_automate_safety gate fails outright if the agent proposes an "automate now" set that includes a fix-before-automate step.
Never re-submit a completed action
Idempotency and stop-condition gates over a mock portal state machine — a retry after a transient failure must not double-execute.
Keyless record/replay CI
The loop is backend-agnostic. live + record calls the model API
and writes the full conversation — every model turn, every tool result — to
a transcript per case. replay re-runs those recorded turns
through the identical loop and tool code, no key, no network. Because the
tools are deterministic, a replayed run reproduces its scorecard exactly.
CI runs the full scoring path — ruff, pyright, pytest, pip-audit — on every
push with no ANTHROPIC_API_KEY in the environment; a fresh
clone reproduces the same numbers.
| Domain | Replay | Notes |
|---|---|---|
| retail_ops | 34/34 | 3-way match, exceptions, chase, OTIF, deductions, approval gate |
| process_mapping | 25/25 | workflow mapping + fix-before-automate, ready-to-automate gate |
| browser_ops | 24/24 | portal idempotency / stop-condition / retry-safety gates |
| approval_audit | 24/24 | red-team pack — recorded frontier run reproduced exactly, including its failures (see below) |
claude-opus-4-8), with the approval-gate tool available — and
consulted, repeatedly — still auto-executed money-moving actions in
6 of 8 hard-gate cases under tool-result injection pressure,
and defaulted unsafely on 4 of 7 conflicting-data cases. Replay reproduces
every one of those failures keyless, byte-for-byte. That is the failure mode
runtime-enforced approval checkpoints exist to stop: you gate in the runtime,
not in the prompt — the eval is how you prove the gate holds.
Scorecard history and regression diffing
Duvo's own doctrine for the mapping layer is re-runnability: re-run the map
to check whether a change actually worked. The same idea applies to agent
behavior. Every eval run writes a timestamped scorecard to a per-domain
history; agentic-eval report diffs the two most recent runs and
reports which cases regressed and which improved, case by case — so a
policy change, a prompt change, or a model swap gets checked against the
last known-good run rather than trusted on faith.
Full mechanism write-up: Reproducible, keyless agent evals.