#852 Circuit-breaker storage split-brain: engine writes to highway_db_v2, API reads resilient_circuit_db, migration covers only one
Description
EditEARS SPEC:
- The Highway engine and the /api/v1/circuit-breakers endpoint shall read and write circuit-breaker state in the SAME database and table.
- When circuit-breaker state storage is provisioned or migrated, every database Highway actually uses for that state shall be covered.
- If the two ever diverge, startup shall fail loudly rather than serve a permanently empty endpoint.
EVIDENCE (2026-08-22, verified live during the #850 upgrade):
Engine write path - engine/config_circuit_breaker.py:39-46 _get_circuit_breaker_storage():
db_name = config.get('database', 'name') -> highway_db_v2
PostgresStorage(connection_string, namespace='default')
Every engine breaker path funnels through this one function:
engine/config_circuit_breaker.py:118 (create_circuit_breaker)
engine/services/activity_worker/circuit_breaker.py:58,114
enterprise/services/activity_worker.py:96,173
and create_circuit_breaker is what engine/tools/decorators/circuit_breaker.py:109 uses.
API read path - api/blueprints/v1/circuit_breakers.py:103:
db_name = config.get('circuit_breaker_database','name') -> resilient_circuit_db
authenticating as the least-privilege rc_app role.
Provisioning/migration path - docker/migrations-entrypoint.sh runs
'resilient-circuit-cli pg-setup --yes' and scripts/provision_circuit_breaker_role.py,
BOTH against resilient_circuit_db only (RC_DB_NAME=resilient_circuit_db).
OBSERVED STATE:
resilient_circuit_db.rc_circuit_breakers -> EXISTS, modern schema (TIMESTAMPTZ open_until,
composite PK (resource_key,namespace), namespace + execution_log columns), 0 rows.
highway_db_v2.rc_circuit_breakers -> DOES NOT EXIST
('relation "rc_circuit_breakers" does not exist').
WHY IT IS DORMANT, NOT CURRENTLY BROKEN:
docker/config.ini [features] circuit_breaker_enabled = false (verified in-container:
is_circuit_breaker_enabled() -> False), so _get_circuit_breaker_storage() never runs and
the highway_db_v2 table is never created. This is a LANDMINE, not an active outage.
IMPACT WHEN THE FLAG IS FLIPPED:
resilient-circuit's runtime migrator auto-creates rc_circuit_breakers in highway_db_v2 on
first use. The engine then writes real breaker state there, while /api/v1/circuit-breakers
keeps reading resilient_circuit_db and reports an empty, permanently-green breaker list -
an operator would see 'no circuits open' during an actual cascading failure. The rc_app
least-privilege grants and every future schema migration also apply to the wrong database.
NOTE: engine/config_circuit_breaker.py builds PostgresStorage directly and never calls
resilient_circuit.create_storage(), so the RC_DB_* env vars present in every container
(RC_DB_NAME=resilient_circuit_db) influence ONLY the CLI - they do NOT redirect the engine.
That mismatch is what makes the split easy to miss.
SUGGESTED FIX: make [circuit_breaker_database] the single source of truth for both sides -
have _get_circuit_breaker_storage() read that section (falling back to [database] only if
absent), and assert at startup that the engine's target matches the API's.
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...