#830 Sweeper fail-path can block up to DB-wide statement_timeout (2min) on a live wrapper's jumper-row lock
Description
Editdocker-compose.yml:159 sets statement_timeout=120000 DB-wide (SHOW confirms 2min). The wrapper holds its jumper r_<queue> row lock for the whole run (mark_run_running inside the atomic transaction — workflow_run lock was fixed in #826 but the jumper row remains). TimeoutService/fail_single_run + failure_handler open separate connections and UPDATE that row: against a hung-but-alive worker they queue up to 2min, then die with 'canceling statement due to statement timeout' (seen in worker logs), retried only next 60s cycle. Fix: SET LOCAL lock_timeout (~3s) on sweeper/failure-path writes + treat LockNotAvailable as retry-next-cycle; consider FOR UPDATE SKIP LOCKED claim-style batching. Note heartbeat renews claim_expires_at (FOR UPDATE SKIP LOCKED) so live workers rarely reach the reaper — this is a hung-worker edge.
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...