#852 Circuit-breaker storage split-brain: engine writes to highway_db_v2, API reads resilient_circuit_db, migration covers only one

open high data-integrity engine resilience Created 2026-08-22 01:54 · Updated 2026-08-22 01:54

Description

Edit
EARS SPEC: - The Highway engine and the /api/v1/circuit-breakers endpoint shall read and write circuit-breaker state in the SAME database and table. - When circuit-breaker state storage is provisioned or migrated, every database Highway actually uses for that state shall be covered. - If the two ever diverge, startup shall fail loudly rather than serve a permanently empty endpoint. EVIDENCE (2026-08-22, verified live during the #850 upgrade): Engine write path - engine/config_circuit_breaker.py:39-46 _get_circuit_breaker_storage(): db_name = config.get('database', 'name') -> highway_db_v2 PostgresStorage(connection_string, namespace='default') Every engine breaker path funnels through this one function: engine/config_circuit_breaker.py:118 (create_circuit_breaker) engine/services/activity_worker/circuit_breaker.py:58,114 enterprise/services/activity_worker.py:96,173 and create_circuit_breaker is what engine/tools/decorators/circuit_breaker.py:109 uses. API read path - api/blueprints/v1/circuit_breakers.py:103: db_name = config.get('circuit_breaker_database','name') -> resilient_circuit_db authenticating as the least-privilege rc_app role. Provisioning/migration path - docker/migrations-entrypoint.sh runs 'resilient-circuit-cli pg-setup --yes' and scripts/provision_circuit_breaker_role.py, BOTH against resilient_circuit_db only (RC_DB_NAME=resilient_circuit_db). OBSERVED STATE: resilient_circuit_db.rc_circuit_breakers -> EXISTS, modern schema (TIMESTAMPTZ open_until, composite PK (resource_key,namespace), namespace + execution_log columns), 0 rows. highway_db_v2.rc_circuit_breakers -> DOES NOT EXIST ('relation "rc_circuit_breakers" does not exist'). WHY IT IS DORMANT, NOT CURRENTLY BROKEN: docker/config.ini [features] circuit_breaker_enabled = false (verified in-container: is_circuit_breaker_enabled() -> False), so _get_circuit_breaker_storage() never runs and the highway_db_v2 table is never created. This is a LANDMINE, not an active outage. IMPACT WHEN THE FLAG IS FLIPPED: resilient-circuit's runtime migrator auto-creates rc_circuit_breakers in highway_db_v2 on first use. The engine then writes real breaker state there, while /api/v1/circuit-breakers keeps reading resilient_circuit_db and reports an empty, permanently-green breaker list - an operator would see 'no circuits open' during an actual cascading failure. The rc_app least-privilege grants and every future schema migration also apply to the wrong database. NOTE: engine/config_circuit_breaker.py builds PostgresStorage directly and never calls resilient_circuit.create_storage(), so the RC_DB_* env vars present in every container (RC_DB_NAME=resilient_circuit_db) influence ONLY the CLI - they do NOT redirect the engine. That mismatch is what makes the split easy to miss. SUGGESTED FIX: make [circuit_breaker_database] the single source of truth for both sides - have _get_circuit_breaker_storage() read that section (falling back to [database] only if absent), and assert at startup that the engine's target matches the API's.

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...