#825 Cancel endpoint cannot cancel claimed in-flight workflow runs (409); no cooperative boundary cancellation
Description
EditPOST /workflows/<id>/cancel only calls jumper cancel_task on the wrapper (tools.workflow.execute) run. A worker claims that wrapper within ~1-3s of submission and holds the claim for the workflow's ENTIRE life, so cancel_task refuses ('Could not cancel task run') and the endpoint 409s. There is NO fallback: nothing marks the run for cooperative cancellation, so the workflow always runs to completion.
Evidence (2026-07-19, prod):
- 15 consecutive cancel attempts over ~60s (~6 step boundaries of a 30x sleep-10 chain, run 83464b28) -> all HTTP 409, run continued.
- Runs 6d2f34ac + 9efa7727: dashboard Cancel click -> 409 -> both ran to completion.
- Cancel only ever succeeds pre-claim (run b529db84: cancelled 2s after creation, wrapper unclaimed).
- Pause is COOPERATIVE and works while claimed: endpoint flips workflow_run.status='paused'; engine.orchestrator.execution_helpers.check_workflow_paused (called per DSL task, orchestrator/core.py:155) re-queues tasks. No equivalent exists for cancel.
Proposed fix (mirror pause):
1. api cancel_workflow: when jumper cancel_task fails on a claimed run, set workflow_run status='cancelling' (or cancel_requested_at) and return 202 instead of 409.
2. engine: extend the per-task boundary check to abort on 'cancelling' -> finalize run as 'cancelled' (in-flight DSL task completes, nothing new starts; recovery/on_failure must NOT fire).
3. Sweeper reconciliation for orphaned 'running' rows whose wrapper run is gone (see run 812a91a8: 'running' 50+ min for a ~2min chain; cancel 409s 'may have already completed').
Dashboard note: highway-dashboard e2e/action-bar.mjs asserts in-flight cancel surfaces the 409 conflict; flip that assertion to real cancellation when this lands. Historical 'cancel works at step boundaries' belief was a contention fluke (wrapper unclaimed for 60s+ under load).
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...