Agents & evaluation
tooling to run, measure, and extend AI agents
The center of this umbrella is agentic-eval-harness, a public multi-domain evaluation engine: one agentic plan-act-observe loop over real, deterministic tools, scored mechanically against committed golden-case sets — 57 cases across three domain packs (generic, industrial, trust & safety, the last fully synthetic and methodology-only). Scoring is checker-based rather than model-judged, a replay backend reproduces any recorded run without a key, and 64 tests cover the whole path.
Two public siblings ride with it: spec-renderer, a single-file no-build renderer that compiles LLM-authored specs into self-contained HTML — one engine renders both forms and dashboards, including the harness's own scorecards — and okf-pack, an OKF-compatible knowledge-context format with a bidirectional Rust adapter, built to drop in as a hot-swappable Spaces pack.
The fourth piece stays private: agent-harness, a headless daemon that runs, supervises, and observes local LLMs and agent loops across the fleet. The public engine is offline by design — no web access, fixed committed fixtures — trading breadth for reproducibility.
Reproducible, keyless agent evals
A record/replay seam that lets an agent-eval harness reproduce its scorecard in CI with no API key — plus the deterministic hard-gate mechanism that rides on top of it.
agentic-eval-harness
A multi-domain agent-evaluation engine — one plan-act-observe loop over real tools, mechanical (keyless) scoring, deterministic replay, and pluggable domain packs (generic, industrial, trust & safety).
spec-renderer
A single-file, no-build renderer that compiles LLM-authored specs into self-contained HTML — forms and dashboards from one engine; ships a dependency-free MCP Apps server.
okf-pack
An OKF-compatible knowledge-context format with a bidirectional adapter — reads memory, skills, and vault surfaces into a unified format and exports losslessly; built to drop in as a hot-swappable Spaces pack.