#782 Agent eval/regression harness: trace-based assertions (routing, over-fetch, delivery, cost) + selftest
Description
EditBuilt the regression harness (highway-mcp/eval_agent.py) the strategy called for. Upgrades the prior string-check seed to assert on the agent TRACE and the approvals it requests, guarding the exact behaviors fixed this session: routing (focused->direct no delegate; broad->exactly one fan-out), no over-fetch after a fan-out (#780: orchestrator http/search calls==0), delivery (email only when asked), cost recording (#771: total_tokens>0), and prompt-injection resistance. Pure trace-assertion helpers are unit-tested via --selftest (10/10, no key/live run needed) so the guards themselves can't silently rot. CLI: --list, --only <prefixes>, --selftest, --timeout; non-zero exit on failure (CI/release gate). Verified: selftest 10/10; live routing_focused_direct (delegates=0) + cost_recorded (4823 tok) PASS via a minted demo hw_k1 key. To run live: provisioned admin@demo as tenant_admin + hw_k1 key 'agent-eval-harness'; HIGHWAY_BASE_URL=... HIGHWAY_API_KEY=hw_k1_... .venv/bin/python eval_agent.py. Next: wire into CI (nightly/pre-release; slow fan-out/email cases ~2-3min each).
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...