#844 circuit-breakers 503: Vault password for resilient_circuit_db never applied to the postgres role

closed critical bug ops security Created 2026-08-03 16:44 · Updated 2026-08-03 18:57

Description

Edit
EARS SPEC: - When GET /api/v1/circuit-breakers is called by an authorized caller, the Highway API shall return HTTP 200 with the circuit-breaker list. - The Highway circuit-breaker database credential shall be verified against the live database at deploy time; if the stored credential cannot authenticate, then deployment shall fail loudly rather than leaving the endpoint dead. - If the circuit-breaker database secret is absent, then the Highway API shall fail closed with an explicit configuration error and shall NOT fall back to a hardcoded default password (api/blueprints/v1/circuit_breakers.py:54 currently does `or "postgres"`). - While the circuit-breaker database is unreachable or unauthenticated, the Highway API shall report that condition as a degraded check in GET /api/v1/health, not only on first use of GET /api/v1/circuit-breakers. - The Highway platform shall hold exactly one source of truth for a given database credential; the main database and the circuit-breaker database shall not diverge silently. EVIDENCE (reproduced 2026-08-03): - GET /api/v1/circuit-breakers -> 503 DATABASE_UNAVAILABLE: connection to server at "172.25.0.2" port 5432 failed: FATAL: password authentication failed for user "postgres". - 172.25.0.2 IS highway-postgres (the docker DB). The DB is up and healthy; only this one code path fails. - Vault highway/shared/database -> {"value": "postgres"} (correct, matches the role) - Vault highway/shared/circuit_breaker_database -> {"password": "<32-char random>"} (never applied to the role) - docker/.env RC_DB_PASSWORD=postgres and POSTGRES_PASSWORD=postgres agree; only the Vault secret disagrees, and circuit_breakers.py reads Vault, not RC_DB_*. - Reproduced byte-identical from inside highway-api: PGPASSWORD=<vault value> psql -h postgres -U postgres -d resilient_circuit_db -> same FATAL. PGPASSWORD=postgres -> connects. - No code in either repo WRITES this secret, so it was seeded out of band and never reconciled. - Related: #836 (services all connect as superuser postgres; a dedicated least-privilege rc role is the durable fix).

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...