Edit Issue - .issue.db

Title *

Description

ROOT CAUSE: the per-tool rate-limit decorator ran check_and_consume on the agent task own DB connection (ctx.conn, no self-commit). Under concurrent same-tenant load the per-tenant/tool/window counter row in highway.tool_rate_limit_usage was a lock hot-spot; a contended counter query aborted the whole agent transaction (InFailedSqlTransaction, no savepoint isolation), and the existing fail-open then ran the real tool on the aborted connection so the agent_loop failed, triggering durable retry, claim-expiry and queuing. Concurrency looked capped at 2 with transient failures. FIX (docker/highway-core): rate_limit.py wraps check_and_consume in conn.transaction (nested SAVEPOINT) so any error rolls back to the savepoint leaving the agent tx clean then fails open; a denied request rolls back its partial consumes. execution_helpers.py check_rate_limit now fails OPEN on tenant-limiter infra errors. docker-compose.yml worker 2 to 6, activity-worker 1 to 3. VALIDATED 8-wide burst with rate-limiting ON and generous 10k/min and fail_open: aborts 0, PEAK concurrent 6 (was 2), rate-limiter still counting (llm.call 14). Submodule 24f8527 parent 0c676a2.

Due Date (YYYY-MM-DD)

Tags (comma separated)

Related Issues (IDs)

Enter IDs of issues related to this one. They will be linked as 'related'.

Edit Issue #767