#850 Upgrade bulkman 2.0.1->2.0.3 and resilient-circuit 0.4.7->0.7.0 (multi-process breaker race, thread-safety, CLI rename)

in-progress high dependencies engine resilience Created 2026-08-22 01:44 · Updated 2026-08-22 01:44

Description

Edit
EARS SPEC: - The Highway stack shall depend on bulkman >=2.0.3,<3 and resilient-circuit >=0.7.0,<0.8, so that the multi-process circuit-breaker fixes shipped in resilient-circuit 0.5.0/0.6.0 are actually running. - While the circuit-breaker PostgreSQL storage is shared by the API, workers and activity workers, the engine shall not lose a stored OPEN protection signal to a stale writer in another process. - When the deploy entrypoint provisions the circuit-breaker table, it shall invoke the console script name that the installed resilient-circuit actually ships, and shall fail loudly if that command is absent. - When resilient-circuit migrates rc_circuit_breakers from naive TIMESTAMP to TIMESTAMPTZ, existing stored rows shall be preserved. WHY (upstream evidence, from each project's CHANGELOG): - We run bulkman 2.0.1, which pins resilient-circuit[postgres]>=0.4.6,<0.5. That cap is the ONLY reason we are on 0.4.7. The two must be upgraded together; bumping resilient-circuit alone is unsatisfiable. - resilient-circuit 0.5.0 fixes a cross-process write race in PostgresStorage: load->decide->save was last-writer-wins, so a process holding a stale local CLOSED could clobber a stored OPEN and erase the protection signal. Highway runs 2 worker replicas + 2 internal-worker replicas + 1 activity-worker + API against ONE shared store, so this is our exact topology. - resilient-circuit 0.6.0 fixes: CircuitProtectorPolicy thread-safety (interleaved mark_success/mark_failure could execute a protected call after the circuit tripped); a ValueError crash out of the protected call on malformed stored state; unbounded InMemoryStorage growth; ExponentialDelay overflow; naive-local TIMESTAMP -> TIMESTAMPTZ (cross-timezone peers disagreed on cooldown expiry). - resilient-circuit 0.7.0 adds RC_DB_DSN + RC_DB_SSL* (TLS to the breaker store). - bulkman 2.0.2/2.0.3 are pin-widening only (<0.6 then <0.8); bulkman's own behaviour is unchanged across 2.0.1..2.0.3. BREAKING CHANGE WE MUST HANDLE: - resilient-circuit 0.6.0 renamed the console script `resilient-circuit` -> `resilient-circuit-cli`. docker/migrations-entrypoint.sh:71 runs `resilient-circuit pg-setup --yes 2>&1 || true`. After the upgrade that is a command-not-found, and `|| true` SWALLOWS it, so circuit-breaker table setup would silently stop running. Must rename AND drop the `|| true`. BEHAVIOUR CHANGE (accepted): - admission_refresh_interval now defaults to 1s (was refresh-on-every-call in 0.5.0). Cuts a connect+SELECT per protected call and per rejected call while OPEN; costs up to 1s propagation delay for a peer's OPEN. BLAST RADIUS (direct API surface used by Highway - all stable across 0.4.7..0.7.0): CircuitProtectorPolicy, PostgresStorage, ProtectedCallError, BulkheadConfig, BulkheadThreading, ExecutionResult. No direct get_state/set_state calls in Highway, so the 0.5.0 set_state->bool signature change does not reach us. ALSO: docker/highway-core/pyproject.toml pins `resilient-circuit>=0.4.3` with NO upper bound. For a bank-grade dependency that is wrong - bulkman itself bounds it <0.8. Bound it here too.

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...