Edit Issue - .issue.db

Title *

Description

EARS SPEC:
- When the Highway API cannot authenticate to Vault, the /api/v1/health circuit_breaker_database check shall report 'secret store authentication failed', NOT 'has no password key' - the two have different fixes and the current message sends the operator to the wrong one.
- While Vault authentication is failing, /api/v1/health shall NOT report overall status 'healthy'.
- The Highway stack shall hold a Vault token that is valid, and its expiry shall be alerted on before it lapses.

EVIDENCE (2026-08-22, live against highway.rodmena.app):
1. vault.rodmena.app was SEALED (sys/seal-status sealed=true, shamir 3/5, progress 0) since the container started 2026-08-16T04:20:26 - no unseal entry in its log. Operator unsealed it during this session.
2. AFTER unseal, Vault auth still fails. VAULT_TOKEN_ADMIN is 28 chars (a real hvs. service token is ~95+), and
   GET $VAULT_ADDR/v1/auth/token/lookup-self -> HTTP 403 {"errors":["permission denied","invalid token"]}
   So the token is truncated/stale, not merely under-privileged.

CUSTOMER-VISIBLE IMPACT (verified through the product's own interface, not the DB):
- GET /api/v1/auth/oidc/login -> 302 to
  /login/oidc/failure/#auth_error=oidc_unavailable&error_description=RODMENA%20ID%20sign-in%20is%20not%20available%20right%20now.
  API log: hvac.exceptions.VaultError: Vault authentication failed for tenant 'shared'
  at api/oauth_handlers/oidc.py:95 _load -> enterprise/security/secret_providers.py:190.
  RODMENA ID / OIDC SSO login is DOWN.
- /api/v1/health circuit_breaker_database -> status 'misconfigured'; /api/v1/circuit-breakers unusable.

WHY THE STACK STILL MOSTLY WORKS: engine/config.py get_secret resolves env -> ini -> Vault.
Everything with a SECRET_* env or a [secrets] config.ini entry (e.g. jwt_secret_key) keeps working
from the fallback, which is why the outage looked like 'a couple of functions' rather than total.
Only Vault-ONLY secrets (OIDC client creds, circuit_breaker_database, per-tenant tokens) are down.

DEFECT 1 (misleading diagnosis): engine/config.py _get_vault_client() returns None when
is_authenticated() is False, and _get_secret_from_vault() swallows every exception and returns None.
An invalid token is therefore indistinguishable from a missing key, and
api/blueprints/v1/circuit_breakers.py reports 'secret store is reachable but
highway/shared/circuit_breaker_database has no password key'. That is FALSE - the key was never read,
because the token was rejected. #844 added a probe to tell 'unreachable' from 'misconfigured' but
there is no third state for 'reachable but unauthenticated', which is the state we are actually in.

DEFECT 2 (health lies): /api/v1/health returns data.status = 'healthy' and HTTP 200 while
checks.circuit_breaker_database is failed. A load balancer or uptime probe reading the top-level
status would never page. This masked a 6-day Vault outage.

FIX NEEDED (operator, holds the Vault credentials): issue a fresh Vault token with the highway KV
policy, set VAULT_TOKEN_ADMIN, recreate the stack, then re-run
scripts/provision_circuit_breaker_role.py so the rc_app password is reconciled.

Priority

Status

Due Date (YYYY-MM-DD)

Tags (comma separated)

Related Issues (IDs)

Enter IDs of issues related to this one. They will be linked as 'related'.

Cancel

Edit Issue #851