#850 Upgrade bulkman 2.0.1->2.0.3 and resilient-circuit 0.4.7->0.7.0 (multi-process breaker race, thread-safety, CLI rename)
Description
EditEARS SPEC:
- The Highway stack shall depend on bulkman >=2.0.3,<3 and resilient-circuit >=0.7.0,<0.8, so that the multi-process circuit-breaker fixes shipped in resilient-circuit 0.5.0/0.6.0 are actually running.
- While the circuit-breaker PostgreSQL storage is shared by the API, workers and activity workers, the engine shall not lose a stored OPEN protection signal to a stale writer in another process.
- When the deploy entrypoint provisions the circuit-breaker table, it shall invoke the console script name that the installed resilient-circuit actually ships, and shall fail loudly if that command is absent.
- When resilient-circuit migrates rc_circuit_breakers from naive TIMESTAMP to TIMESTAMPTZ, existing stored rows shall be preserved.
WHY (upstream evidence, from each project's CHANGELOG):
- We run bulkman 2.0.1, which pins resilient-circuit[postgres]>=0.4.6,<0.5. That cap is the ONLY reason we are on 0.4.7. The two must be upgraded together; bumping resilient-circuit alone is unsatisfiable.
- resilient-circuit 0.5.0 fixes a cross-process write race in PostgresStorage: load->decide->save was last-writer-wins, so a process holding a stale local CLOSED could clobber a stored OPEN and erase the protection signal. Highway runs 2 worker replicas + 2 internal-worker replicas + 1 activity-worker + API against ONE shared store, so this is our exact topology.
- resilient-circuit 0.6.0 fixes: CircuitProtectorPolicy thread-safety (interleaved mark_success/mark_failure could execute a protected call after the circuit tripped); a ValueError crash out of the protected call on malformed stored state; unbounded InMemoryStorage growth; ExponentialDelay overflow; naive-local TIMESTAMP -> TIMESTAMPTZ (cross-timezone peers disagreed on cooldown expiry).
- resilient-circuit 0.7.0 adds RC_DB_DSN + RC_DB_SSL* (TLS to the breaker store).
- bulkman 2.0.2/2.0.3 are pin-widening only (<0.6 then <0.8); bulkman's own behaviour is unchanged across 2.0.1..2.0.3.
BREAKING CHANGE WE MUST HANDLE:
- resilient-circuit 0.6.0 renamed the console script `resilient-circuit` -> `resilient-circuit-cli`.
docker/migrations-entrypoint.sh:71 runs `resilient-circuit pg-setup --yes 2>&1 || true`.
After the upgrade that is a command-not-found, and `|| true` SWALLOWS it, so circuit-breaker
table setup would silently stop running. Must rename AND drop the `|| true`.
BEHAVIOUR CHANGE (accepted):
- admission_refresh_interval now defaults to 1s (was refresh-on-every-call in 0.5.0). Cuts a
connect+SELECT per protected call and per rejected call while OPEN; costs up to 1s propagation
delay for a peer's OPEN.
BLAST RADIUS (direct API surface used by Highway - all stable across 0.4.7..0.7.0):
CircuitProtectorPolicy, PostgresStorage, ProtectedCallError, BulkheadConfig,
BulkheadThreading, ExecutionResult. No direct get_state/set_state calls in Highway,
so the 0.5.0 set_state->bool signature change does not reach us.
ALSO: docker/highway-core/pyproject.toml pins `resilient-circuit>=0.4.3` with NO upper bound.
For a bank-grade dependency that is wrong - bulkman itself bounds it <0.8. Bound it here too.
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...