#1062 llm_routes health probe is single-turn: a provider that fails multi-turn tool calls stays green

open high bug Created 2026-10-24 00:16 · Updated 2026-10-24 00:16

Description

Edit
EARS SPEC: - The llm_routes health probe shall, for each candidate, send a two-turn tool-call exchange (user; assistant with one tool_call and the provider's reasoning field echoed per provider config; tool result) and mark the candidate dead when the second turn fails. - The probe shall cost at most 2 requests per candidate per probe interval and keep max_tokens <= 64. EVIDENCE: #1060. On 2026-10-23 23:00-23:07Z health llm_routes reported prism/deepseek-v4-flash live (single-turn probe, HTTP 200, 1062 ms) while every agent tool loop on it failed its second turn. SYNTHESIS: domain = synthetic monitoring; concept = probe the transaction the product depends on (multi-turn tool call), not a ping. Split from #1060 because it is independently solvable (api/utils/llm_route_probe.py) and #1060's fix must not wait on it.

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...