#671 Add OrphanDetectionService for stuck workflows

closed high Created 2025-12-24 03:30 · Updated 2026-01-02 06:26

Description

Edit
Need a background service that periodically scans for orphaned workflows where: - workflow_run.status IN ('running', 'sleeping') - BUT queue entry is completed/failed/cancelled/missing These orphaned workflows will never complete without manual intervention. The service should: 1. Run every 5 minutes 2. Find orphaned workflows older than 5 minutes (to avoid false positives during normal processing) 3. Mark them as 'failed' with appropriate error message 4. Log the cleanup for audit trail SQL query for detection: SELECT wr.workflow_run_id, wr.workflow_name, wr.status, r.state FROM workflow_run wr LEFT JOIN absurd.r_highway_default r ON r.run_id = wr.absurd_run_id WHERE wr.status IN ('running', 'sleeping') AND (r.state IS NULL OR r.state IN ('completed', 'failed', 'cancelled')) AND NOW() - wr.created_at > INTERVAL '5 minutes';

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...