Lessons Learned
Knowledge base from resolved issues
Lessons
| Category | Lesson | Issue | Date | Actions |
|---|---|---|---|---|
| architecture | get_jumper_client was a process singleton whose queue_name only counted on the first call, so every submit_workflow(queue=highway_internal) from highway-api silently ran on highway_default for months. It surfaced only when a step needed a secret mounted on the internal workers alone (TokenGate admin key). When a factory takes a parameter that changes identity (queue, tenant, region), a cache keyed on nothing is a bug: key the cache on that parameter, and test the second call with a different value. | #1069 | 2026-10-25 | |
| deployment | A rollback image that exists only as a local docker tag is not a rollback: on 2026-10-23/24 the rollback-e58c57d images vanished from vm-1 within two hours with no known actor. Before a deploy, docker save the rollback images to /var/tmp/hw-images/<tag>.tar.gz and gzip -t it; re-check the rollback tag before each restart step. | #1056 | 2026-10-24 | |
| llm | Prism 503 model_unavailable ('provider out of quota or unreachable') can be an upstream request rejection, not unavailability: deepseek-v4-flash via prism answered 503 3/3 to tool_calls with object arguments and 200 to the same request with string arguments (2026-10-24). Before blaming quota, replay the exact failing body against prism with one field varied. Adapters must send canonical OpenAI wire shape (arguments as JSON string) whatever shape the agent history stores; DeepSeek thinking mode also requires the reasoning_content key (empty string accepted) on assistant tool_calls messages. | #1060 | 2026-10-24 | |
| security | Token-authenticated public endpoints must derive every storage key server-side. complete_upload copied and then deleted whatever s3_key the client sent: on S3 that was cross-tenant object theft plus deletion, on local storage a ../ path meant arbitrary file read plus unlink. A capability token proves who may act on ONE upload, never which objects; bind keys to the upload_id and reject the rest before any provider call. | #1042 | 2026-10-22 | |
| testing | Running tests inside the live api container shares its 1.5 GiB limit and its DB: an e2e plus two -n 8 suites OOM-killed highway-api (2026-10-20 07:58:29Z, back 07:58:38). Tests get their own throwaway container with --network none (scripts/run_tests_isolated.sh); network isolation also makes accidental DB writes impossible. | - | 2026-10-20 | |
| testing | highway-api has a 1.5 GiB memory limit (docker inspect HostConfig.Memory=1610612736) and is the production API. 2026-10-20 07:58Z it was OOM-killed (State.OOMKilled=true, RestartCount 1) while #1036 ran a scratch e2e (2 hwe workers + create_app) and two unit suites with the pyproject default -n 8 concurrently inside it; the restart policy brought it back after ~30 s, and the e2e and both suites died. Run unit suites inside highway-api with -n 2 (or -p no:xdist), one suite at a time, never alongside a scratch e2e; other agents share the container. The superproject suite takes ~45 s at -n 2. | #1036 | 2026-10-20 | |
| database | pg-nano TLS (#925): (1) with clientcert=verify-full and scram, a client cert whose CN is another role is reported to the client as 'password authentication failed for user X'; only the server log says 'provided user name (X) and authenticated user name (Y) do not match'. Prove the cert is the cause with a control that changes only the cert. (2) migretti interpolates every ${VAR} in mg.yaml, including unused envs blocks, so 'Environment variable DB_HOST referenced in mg.yaml is not set' fires even when MG_DATABASE_URL is set; the entrypoint writes 'database: {}' in owner mode. (3) The auth library (3.1.0 and 3.3.0) hardcodes its SQLAlchemy pool size; it cannot be capped by env. (4) The Quart test client returns a body for HEAD; only a served hypercorn proves HEAD has an empty body. | #925 | 2026-10-20 | |
| testing | Superproject pytest: never pass --noconftest (or remove tests/conftest.py) for a suite run. Since #1017 the db_write guard (skip unless HIGHWAY_DB_TESTS=1, refuse highway_db_v2) lives in tests/conftest.py pytest_collection_modifyitems, and sessionfinish is gated by the same check. --noconftest disables the guard, so tests/unit/api/test_rate_limit_api.py, test_gateway.py and tests/unit/contracts/test_pydantic_contracts.py would run against highway_db_v2. The memory entry tests_conftest_writes_live_db (use --noconftest) predates #1017 and is now unsafe for full suites. 2026-10-20 (#908): a full-suite run with --noconftest was killed by PID after ~166 tests; collection order puts the first db_write module (test_gateway.py) at position 187, so no db_write test ran. | #908 | 2026-10-20 | |
| architecture | Activity work does not run in the api's PID namespace, and on this deployment ([sandbox] unsandboxed_tenants empty) every tools.shell.run activity runs in a sandbox container that carries no run/tenant labels, so label-based run cleanup (#916) never sees it. Cancelling an activity must be done by the activity worker that claimed it (worker_id) through engine.tools.activity_processes: processes via activity_processes.popen/run_shell (own session, killpg), containers via register_stopper. Any new tool that spawns processes or containers in an activity must go through that registry or it survives a cancel. | #1024 | 2026-10-20 | |
| api | spectree validates every status a route declares in resp=Response(...). A 4xx/5xx body that does not match ErrorResponse (including the bodies of require_permission, which sits under @spec.validate on ~142 routes) used to become a 500 pydantic list. Since #1020 api/openapi/plugin.py serves error statuses unvalidated (2xx still validated) and logs a warning for a dict body mismatch. Do not remove backend=ErrorPassthroughQuartPlugin from the SpecTree in api/openapi/__init__.py. | #1020 | 2026-10-20 | |
| security | nginx here sets X-Forwarded-For with $proxy_add_x_forwarded_for, which APPENDS the client address: the FIRST entry is whatever the client sent. Never read request.headers['X-Forwarded-For'].split(',')[0]; use api.utils.client_ip.client_ip(), which trusts the header only from [api] trusted_proxies (172.25.0.1/32, the bridge gateway docker-proxy uses) and takes the rightmost untrusted hop. This affected approver audit IPs and API-key ip_whitelist checks (#1019). | #1019 | 2026-10-20 | |
| security | Prefer rewording a false-positive literal over allowlisting it. A global gitleaks regex allowlist for the internal URL http://api:7822/api/v1/health exempted it in every file; the CI self-test 'the exemption ... leaks to other paths' caught it (pipeline 3570). Gitleaks allowlists must be scoped to rule+path+regex together. | - | 2026-10-20 | |
| security | Content-addressed dedup in a multi-tenant table must include the tenant in the unique key. workflow_definition had a global UNIQUE(definition_hash) and _find_existing_definition(hash) ignored the tenant, so two tenants submitting identical JSON shared one row: tenant B's runs pointed at tenant A's definition_id (37 runs demo<->test), and name lookups (get_latest_definition) picked whichever tenant had the highest version, which made every tenant's durable run_goal leaves execute demo's agent_leaf. Rule: every query function on a tenant table takes a required keyword-only tenant_id; never default-open (#1014). | #1014 | 2026-10-20 | |
| api | spectree response validation is live: a route that declares HTTP_4xx/5xx=ErrorResponse but returns jsonify({'error': str}) answers 500 with a pydantic error list. Return error_response() on such routes; undeclared status codes pass through unvalidated. | #1012 | 2026-10-20 | |
| testing | tests/unit/api is NOT DB-free: test_rate_limit_api.py and test_gateway.py open real connections to highway_db_v2 and write (tool_rate_limits, approval_tokens). tests/conftest.py pytest_sessionfinish runs UPDATEs via psql, and psql exists in highway-api. Before running any existing test file, grep it for create_direct_connection/psycopg.connect/get_db_manager; run named files only, from a tree copy without tests/conftest.py. | #1017 | 2026-10-20 | |
| testing | Running a test suite 'in the api container' means running it against PRODUCTION config: tests/unit/db wrote to highway_db_v2 and called a cleanup with a 60 s TTL (2026-10-20). Rules: DB tests are opt-in (HIGHWAY_DB_TESTS=1) and refuse the production DB name; agents run unit suites with --ignore=tests/unit/db unless on a scratch DB. | - | 2026-10-20 | |
| testing | The core repo's docker/highway-core/tests/unit/db/ tests connect to the database named in /etc/highway/config.ini and WRITE: test_cleanup_events inserts into jumper.e_highway_default / w_highway_default and calls jumper.cleanup_events('highway_default', 60, 1000), which deletes up to 1000 waiter-less events older than 60 s (production TTL is 24 h). --noconftest does not stop it (the init_db fixture lives in the test module). Running 'pytest tests/unit' for the core inside highway-api hit highway_db_v2 on 2026-10-20 ~02:29Z (security-batch agent). Always run core unit tests with --ignore=tests/unit/db (and stress), never the whole tests/unit. | #994 | 2026-10-20 | |
| ci | ci-python images: the job runs as root but cannot read /proc/<pid>/exe of the postgres server (no ptrace); find the server's bin dir with SELECT setting FROM pg_config WHERE name = 'BINDIR'. Steps share the container but not shell state: export variables from a file sourced in each step. 'test -z "$(git status --porcelain)"' passes when git itself fails; assign to a variable first so set -e catches the failure. | #918 | 2026-10-20 | |
| security | highway-engine depended on 'jumper' with no [tool.uv.sources]; PyPI has an unrelated package named jumper (0.3.90, 'SDK for using Jumper's emulator'). A standalone install of the engine would have pulled it. Fixed in #918 by jumper = { path = "lib/jumper" } in the engine pyproject and an engine uv.lock; the engine CI job asserts jumper 0.1.0 installed from lib/jumper. | #918 | 2026-10-20 | |
| security | gitleaks 8.30.1: a top-level [[allowlists]] entry with paths skips the whole file during 'gitleaks dir', even with condition = "AND" and a regex. A real key planted in the allowlisted file is not reported ('scanned ~0 bytes'). Scope false positives with [[rules]] id = "<default rule id>" plus [[rules.allowlists]] (paths AND regexes); that keeps the default rule and only drops the matching secret. Prove every allowlist by planting a key inside the allowlisted file. | #918 | 2026-10-20 | |
| architecture | A parallel branch runs its own branch workflow: a handler named by a branch task must be defined inside the branch lambda. WorkflowBuilder only recognises handlers referenced from the same builder, so a parent-level handler for a branch task is auto-chained after the previous parent task and runs unconditionally (golden v2/disaster carried this for months; the ALL_OF join looked satisfied because the auto-chained handler was in executed_tasks). Since #966 the engine rejects that shape at submit and executor init, and branch/loop executors run handler ids only as callbacks. | #966 | 2026-10-20 | |
| testing | Template {{var}} in a tools.shell.run command is substituted as raw text before the shell parses it (#972): {{x}} is the stripped stdout, {{x.stdout}} keeps the trailing newline and ends the command line. Wrap values in single quotes. A demo/validator that checks only final run status cannot see lost or wrong data: the #961 rewrite of the demo matrix to value checks via /steps/logs exposed an empty-foreach body execution (#973) and condition-branch ordering (#974) that the status-only matrix had passed for months. On a failed run /events and /steps only show workflow_failed (#976), so diagnose failures from GET /workflows/{id} error. | #961 | 2026-10-20 | |
| architecture | Branch task outcomes are NOT in the parent ctx: a failed branch task is in neither executed_tasks nor failed_tasks of the parent run; only highway.parallel_branch_results (join id parallel_<fork>_<workflow_run_id>_join) knows. Any parent-side logic over branch tasks (joins, trigger rules) must read that table, and in-pass vs resume readiness must share one helper (#955: join returned join_satisfied=false as success; #956: in-pass readiness ignored trigger_rule after a handled failure while resume honoured it). | #955 | 2026-10-17 | |
| testing | Sandboxed shell (#915): every tools.shell.run attempt is a fresh container, so files, flags and counters do not survive between tasks or retries. Pass state through result_key variables; for fail-once retry demos use a deadline variable from an earlier task. Step events truncate results to 200 chars (Python repr); validators must read full output from GET /workflows/{id}/steps/logs?step_name=X. | #944 | 2026-10-17 | |
| integration | TokenGate refund returns QUOTA units only; it never returns a token to a rate_limit bucket, and 'a subject with no assignment is uncapped' (results: []). So: (1) 'refund on rollback' cannot apply to per-call rate_limit policies - use idempotency keys per attempt instead; (2) always verify the consume result names the policies you expect, or an unprovisioned profile silently passes; (3) mint runtime keys with require_assignment so an unassigned subject fails 422 instead of passing. | #928 | 2026-10-17 | |
| security | mail-api is ONE mailbox (highway@mail.rodmena.co.uk) shared by every Highway tenant. Any per-tenant inbound routing (plus-tags) must be scoped server-side from ctx.tenant_id (tools.email uses h+32hex sha256(tenant_id, tag)) and the consumer must re-check recipient_tag client-side, never trust the query filter alone. Plus-address local parts are capped at 64 chars (RFC 5321): mail-api 422 invalid_recipient for highway+<60-char tag>. | #950 | 2026-10-17 | |
| engine | Postgres JSONB does not preserve object key order (keys sort by length, then bytewise). Any engine logic that relies on dict order of a definition loaded from JSONB (e.g. 'last task', 'first task') is wrong; derive order from the dependency graph. Found in #938. | #938 | 2026-10-17 | |
| security | Run isolation/privilege tests as the same uid, groups and capabilities as the production process. 'docker exec' runs as root, but Highway workers run as uid 1000 via gosu with zero caps; #945 tests passed 12/12 under docker exec and the deployed feature failed on every call. Verify the identity of the real process (/proc/<pid>/status Uid/Groups/CapEff) before designing a privilege boundary. | #945 | 2026-10-16 | |
| reliability | A decorator that wraps a side-effecting call must never call it inside the try whose except branch also calls it. with_rate_limit called func() inside try and again in its fail-open 'except Exception', so every failing http/email/shell/llm call ran twice in production for months (#942). Guard evaluation goes in the try; the guarded call goes after it, exactly once. Test: a tool that raises must be invoked exactly once. | #942 | 2026-10-16 | |
| debugging | When verifying a peer's claim from logs, size the search window to the EVENT time (their audit timestamp / mail sent time), not to when you read the message. Poll-driven mail means hours can pass between a peer's action and your read; a too-small --since window makes genuine deliveries look like non-deliveries and nearly caused a false accusation on #842. Pair every absence claim with a known-positive probe (garbage logout_token -> 400 + WARNING line) before asserting non-delivery. | #842 | 2026-08-01 | |
| database | SQLAlchemy create_all never ALTERs: an embedded library's schema (auth_rbac) created by an old version keeps narrow varchar columns after upgrades even when current models declare Text. Encrypted cells make this a time bomb - existing short emails fit varchar(64), the first longer email fails INSERT. Reconcile by diffing live information_schema against current model metadata at migration time (varchar->TEXT is metadata-only in postgres). Found because test probes used longer emails than any existing user - test data diversity catches what production data hides. | #838 | 2026-07-31 | |
| testing | auth.database.create_tables() swallows DDL failures by default and Highway logged unconditional success after it - a checkmark that could never go red. Boot-path schema init must use create_tables(raise_on_error=True) AND verify the effect (required tables present in information_schema) rather than the return. | #834 | 2026-07-30 | |
| dependencies | Never build-time patch third-party libs (sed in Dockerfile). The auth PGSSLMODE sed patch (a) silently no-ops if upstream reshapes the line - build stays green, RBAC dies at runtime; (b) masked upstream behavior so diffing installed-vs-pristine wheels misattributed a long-standing hardcode as a new regression. Correct approach: fix the environment (TLS on postgres via docker/init-ssl.sh) or get upstream fixed; never fork silently. | #834 | 2026-07-30 | |
| testing | Test a feature's actual backend path, not just the UI that links to it. The #824 consolidation surfaced Simulate prominently (ActionBar link + 'no side effects' hint) and the plan/memory claimed it was a safe dry-run — but the E2Es only checked that the button navigated; nobody ever ran an actual simulation. It turned out simulate re-executed REAL tool side effects (live shell/http/email) and 500'd on any failure. A single real simulate call in a test would have caught both. Rule: if you advertise a behavior ('no side effects'), assert THAT behavior end-to-end, not the affordance that triggers it. | #833 | 2026-07-19 | |
| architecture | Row-lock scope + TOCTOU in lifecycle endpoints: never write a status row inside a long-running transaction at its START — the lock is held until commit, every concurrent lifecycle UPDATE queues for the whole duration, and their post-lock writes land on stale premises (proven: pause overwrote terminal 'completed' after a 90s lock wait, minting permanent zombies). Status transitions must be single atomic guarded UPDATEs (condition in the WHERE, not a prior SELECT) plus SET LOCAL lock_timeout so residual contention fails fast. Also: a repair sweeper needs an invoker — zombie_detection sat registered-but-never-called with 3 latent bugs (unbindable INTERVAL placeholder, f-string JSON injection, hardcoded queue) while zombies accumulated. | #826 | 2026-07-19 | |
| dashboard-a11y | Don't paper over a11y-lint with a wrong role. A modal backdrop with onclick was given role=button+tabindex=-1 just to silence a11y_no_static_element_interactions — but that makes the scrim announce as a button and nests role=dialog inside role=button (invalid ARIA). Correct fix: the stretched-button pattern — a real <button aria-label='Close dialog'> absolutely positioned behind the dialog (tabindex=-1 so it's out of the tab order; keyboard closes via Escape/✕), the backdrop stays a plain presentational scrim, and keydown lives on the dialog. Native <button> satisfies the click-events rule with no suppression. Same pattern also solved the storage-ref-inside-open-button card (button-in-button is likewise invalid). | #819 | 2026-07-13 | |
| dashboard-ui | Shared component refactors: unify only what's genuinely the same pattern. The app had ~10 tab bars; only 5 (underline: label/count/icon/href-or-button) were a clean fit for one Tabs component. Forcing in the specialized ones (per-tab icon-component switches, disabled+group logic, pill styling, or intentional filter/aria-pressed semantics like approvals FilterTabs) would bloat the component and risk regressions for zero user gain — leave them and document why. Bake cross-cutting fixes into the shared component: Tabs sets flex-shrink:0 so the scroll-container-min-height:0 collapse bug (that hit admin+activities) can never recur on any adopter. Two a11y modes matter: URL-driven tabs are navigation (<nav>+aria-current), in-page tabs are a widget (role=tablist + roving tabindex + arrow keys) — don't give links the tab role. | #819 | 2026-07-13 | |
| dashboard-apps | Marketplace install-status: GET /api/v1/apps/marketplace returns a GLOBAL catalog with installation_count across ALL tenants but NO per-tenant 'installed' flag. To show Installed vs Install per tenant, correlate client-side against GET /api/v1/tenant/apps by app_id (present on both responses) — the app-detail page already does exactly this (installations.find(i => i.app_id === appId)). Chose client cross-reference over adding a backend EXISTS-subquery flag because the latter needs a core-engine change + Docker rebuild + golden-demo run for a pure display concern. | #819 | 2026-07-13 | |
| dashboard-rbac | Dashboard admin-nav gating: the correct 'is this user an admin' signal is the manage_users permission (the string every Admin endpoint enforces via require_permission), NOT the tenant_admin role name (tenants can define custom roles) and NOT isPlatformAdmin (that would lock out tenant admins). CRITICAL gotcha: API-key auth BYPASSES tenant RBAC roles entirely — require_permission reads g.token_scopes (the api_keys.permissions column) where null/empty = full access, so /rbac/my-permissions returns [] for an API key that actually has full admin access. To mirror a permission check client-side you MUST branch on auth type: JWT -> GET /rbac/my-permissions (role-based); API key -> POST /auth/api-key/verify .permissions (scope-based, null/empty = all). | #819 | 2026-07-13 | |
| durability | Exactly-once for external side effects is IMPOSSIBLE at the framework level (durable steps commit only at the task-txn boundary; a crash between a non-transactional SMTP/Telegram effect and that commit re-runs it on replay). Two correct patterns: (1) delegate to the service via a deterministic Idempotency-Key (http.request: key=run_id+step+url); (2) LOCAL at-most-once - claim a deterministic key (run_id:turn:i) in a dedup table on a SEPARATE autocommit connection BEFORE the effect so the claim survives the rolled-back task txn; a crash-replay sees the claim and skips. Best-effort (guard DB down -> run anyway); drop the claim on JumperSleepError so resume re-attempts. See _run_side_effect_once + public.agent_tool_effects. | #750 | 2026-07-05 | |
| llm | Native structured-output enforcement (ollama format) is NOT sufficient alone: cloud thinking models (glm-5) return empty content under a bare schema constraint they never saw in text. Always pair out-of-band enforcement with the schema spelled out in the prompt, and make validation-retry messages include the schema - the model can't fix what it can't see. | #798 | 2026-07-01 | |
| engine | Features nobody exercises rot silently: send_signal had a jsonb=json type error (psycopg Json vs jsonb column) meaning it NEVER worked end-to-end; the first real smoke test found it. Every new primitive needs at least one live harness/smoke case - unit imports are not enough. | #797 | 2026-07-01 | |
| api | Response-model enums must track DB CHECK constraints: pause was half-built (DB allowed 'paused', spectree WorkflowStatus enum did not) so GET /workflows/<id> 500'd on any paused run - a valid state turned into a server error by response validation. Same for 'orphaned'. When adding a status value, grep BOTH migrations and api/openapi/models. | #796 | 2026-07-01 | |
| agents | run_goal re-executes the ENTIRE turn on every durable resume (JumperSleepError replays the task from the top). Any pre-approval/pre-dispatch GUARD that counts persisted state (e.g. delegate rounds via parallel_join_state) will re-run on resume and must EXCLUDE the current in-flight call's own row, or it will reject the call's own resume. Identify the current call via its framework-injected idempotency/orchestration key. Symptom seen: with max_delegate_rounds=1 the first delegate's resume was rejected as 'over cap', the orchestrator lost the sub-agent results, and it over-fetched. JumperSleepError is commit-and-suspend, so state seeded before the suspend (the join_state) IS present on resume. | #780 | 2026-06-29 | |
| architecture | ActivityContext must satisfy the DurableContext attribute interface that tools/app-executor read, or app-action-as-activity paths fail. Known required attrs so far: conn(None)/get_connection, jumper_client, jumper_task_id, jumper_run_id, workflow_run_id, tenant_id, worker_id, current_step_name(None). Prefer adding the attr to ActivityContext over getattr-guards in callers, so it stays a true drop-in. _managed_connection in executor already hasattr-guards db_connection/conn for ActivityContext. | #776 | 2026-06-29 | |
| logging | Per-activity log capture (BufferingActivityLogHandler) attaches to the ROOT logger, so under concurrent activity execution (bulkhead threads) every handler receives every thread's records -> cross-talk. Capture must be thread-scoped: bind owner thread on attach, filter record.thread in emit. Also refcount any root-logger level change shared across concurrent handlers so interleaved attach/detach doesn't strand the level. Trade-off: logs from sub-threads an activity itself spawns are not captured (acceptable vs cross-talk). | #774 | 2026-06-29 | |
| architecture | Tools that may run BOTH inline (DurableContext, ctx.conn held) AND as an activity (ActivityContext, ctx.conn=None) must acquire DB connections via engine.context_db.context_connection(ctx), never ctx.conn directly. It yields the held atomic conn for a durable task (orchestrator commits) or a short-lived auto-commit pooled conn for an activity (so a long LLM/HTTP call never pins a connection). Activity-routed tools: a task is sent to the activity worker when it is non-waiting AND timeout_policy > activity_threshold_seconds(30s). durable_leaves=true makes agent_leaf's execute_leaf_agent run this way. ActivityContext exposes a lazy jumper_client (singleton; methods take conn explicitly). Also: activity worker completion/failure events MUST be emitted with the workflow tenant_id (await_event resolves tenant-scoped since H-7) or non-default-tenant runs hang. Activity logs persist to DataShard via enterprise/tools/datashard_logger (activity_logs table, mirrors task_logs); the engine.tools.activity_logger is a python-logger-only stub. | #771 | 2026-06-29 | |
| testing | Parallel test cleanup must be scoped: Never use broad LIKE patterns in test cleanup fixtures. With xdist parallel runners, broad DELETE patterns destroy rows belonging to other concurrent tests. Track specific IDs and only delete those. | #748 | 2026-02-09 | |
| architecture | IPC isolation: (1) JumperSleepError MUST propagate through IPC server to task executor. (2) Functions from dynamic modules cannot be pickled across IPC - checkpoint() must execute locally. (3) Use GatewayTokenService not ApprovalService for approval token generation in IPC handlers. | #741 | 2026-02-09 | |
| general | Static event names in workflow DSL templates cause cross-run deadlocks. The jumper event table (e_highway_default) stores events globally by event_name without run scoping. When a workflow runs twice, the second run finds the stale event immediately, causing await_event to NOT sleep. This holds the e_ row FOR UPDATE lock while also needing rate_limit_usage lock, creating ABBA deadlock with emit_event. Fix: Always scope intra-workflow event names to {{workflow_run_id}}. | - | 2026-02-08 | |
| general | Bootstrap app registration must scan ALL app directories (pre_installed/ AND examples/), not just one. The preinstalled_apps.yaml must list every app that should be auto-installed per tenant. After any new app is created in enterprise/apps/, it must be added to preinstalled_apps.yaml or it won't survive a clean deploy. | - | 2026-02-08 | |
| testing | DSL template batch test: 27/57 pass. Failures are app/environment level not engine level. Fix: install missing apps for tenant, fix app code bugs (IPC mode, datetime serialization, wrong kwargs), address SMTP timeout. | - | 2026-02-08 | |
| general | File split pattern for backward compat: When splitting a module into a package (e.g. db.py -> db/), use __init__.py re-exports. When splitting into sibling files (e.g. encryption.py + encryption_keys.py), add re-exports at the BOTTOM of the original file after all names are defined. Import from the new file using bottom-of-file imports to avoid circular imports. Docker COPY requires rebuild after adding new files. | - | 2026-02-07 | |
| general | Maturity review pattern: When reviewing concurrency in a DB-backed engine, check (1) state transition guards with WHERE clause + rowcount check, (2) mark-before-execute ordering for idempotent callbacks, (3) DAG cycle detection at both submission and runtime, (4) savepoint/cache rollback alignment. Agent-reported race conditions should be verified against source - many were already mitigated by atomic SQL functions and idempotency keys. | - | 2026-02-07 | |
| general | Signal handler pattern for blocking waits: When signal handler sets a shutdown flag, also set any Event objects that threads may be waiting on. Otherwise threads blocked on Event.wait() won't wake until timeout. | - | 2025-12-28 | |
| general | Docker signal handling: When using shell expansion in docker-compose command, add 'exec' before the actual command to make the process PID 1 and receive signals directly. Without exec, the shell is PID 1 and absorbs SIGTERM. | - | 2025-12-28 | |
| best-practice | Use sync httpx.Client not async httpx.AsyncClient in workers - asyncio can cause zombie workers where container shows healthy but all threads are dead. httpx.Client is equally capable and avoids asyncio concurrency bugs. ThreadPoolExecutor workarounds for asyncio are fragile. | #721 | 2025-12-27 | |
| architecture | Docker-in-Docker temp files must use shared mount path: When spawning containers from inside containers (DinD with Docker socket), temp files created in /tmp exist only inside the parent container. Volume mounts use HOST paths. Use a shared mount path (same absolute path on host and container) for files that need to be mounted into child containers. | #720 | 2025-12-27 | |
| architecture | Parallel IDs must use workflow_run_id not absurd_run_id: When generating identifiers that need to persist across retries (like parallel branch join_event_name), ALWAYS use workflow_run_id (stable across retries) not absurd_run_id (changes on each retry). absurd_run_id represents one execution attempt, workflow_run_id represents the entire workflow lifecycle. | #719 | 2025-12-27 | |
| architecture | ActivityOperator is queue-only: When creating internal operators (like ReflexiveOperator creating ActivityOperator), execute_activity_operator only QUEUES the activity and returns immediately. Internal operators must add their own await_event() call to wait for completion. See engine/interpreters/inline_executor.py _execute_reflexive_operator | #713 | 2025-12-25 | |
| security | Python DSL must execute in isolated container: No DB libs (psycopg), no network libs (requests), no secrets. Use separate Docker network (internal:true). API calls http://dsl-compiler:8080/compile. | #678 | 2025-12-24 | |
| api/visibility | Terminal workflow states (completed, compensated, failed) should always show progress_percentage=100. Switch/conditional branches that aren't executed shouldn't reduce progress for finished workflows. | #677 | 2025-12-24 | |
| api/visibility | Progress calculation requires unlimited query: (1) Fork/join workflows have branch step_succeeded events not in task_ids - use set intersection. (2) LIMIT 50 drops early events - split into limited query for UI and unlimited query for progress count. See workflows.py:1424-1471. | #675 | 2025-12-24 | |
| PostgreSQL | JSONB column data extraction: When querying PostgreSQL tables with JSONB columns (like absurd_event_log.payload), data nested in JSONB must be extracted using arrow operators (payload->>'key' for text, payload->'key' for JSONB). Always verify schema with \\d before writing SQL - event logs often store step names and results inside payload JSONB, not as separate columns. | #674 | 2025-12-24 | |
| database | Always use row_factory=dict_row when creating psycopg3 cursors for dict access. Use get_db_cursor() helper which sets this by default. | #673 | 2025-12-24 | |
| general | RAG cache must use PostgreSQL not file system - /tmp is container-local in Docker, so each container computes embeddings separately causing hundreds of /embedding API calls. Use rag_embedding_cache table with TTL for shared cache across containers. | - | 2025-12-24 | |
| general | Branch tasks must NEVER update parent workflow_run status. Always check 'task_name \!= BRANCH_EXECUTION_TASK' before updating workflow_run for any status (running, completed, sleeping, failed). The parent workflow controls its own lifecycle - branches only emit events. | - | 2025-12-24 | |
| orchestrator | CRITICAL: Always sync workflow_run.status with absurd task state. When task state changes (sleeping, cancelled, retrying, abandoned), update workflow_run accordingly. Absurd is source of truth, workflow_run is UI display layer. | - | 2025-12-23 | |
| security | Security: When implementing IPC proxies for sandboxed code, never blindly forward all method calls. Always enforce a strict whitelist of allowed methods on the server side to prevent sandboxed code from accessing dangerous internal APIs. | - | 2025-12-22 | |
| ScriptPlan | Working Hours Correct Usage: (1) Use project.dateToIdx(naive_datetime) - datetime MUST be NAIVE; (2) Get WorkingHours from shift.get('workinghours', 0) not resource.workinghours; (3) Use resource.get('leaves', 0) not resource.leaves which is a method; (4) Project must be scheduled with schedule=True | - | 2025-12-21 | |
| architecture | For distributed rate limiting, use PostgreSQL atomic operations (INSERT ON CONFLICT DO UPDATE RETURNING). Two-phase architecture: CPU-heavy scheduling done once on config change (parse TJP, cache report in JSONB), fast runtime checks compare cached windows. Always support fail_open for graceful degradation. | #635 | 2025-12-20 | |
| general | Dockerfile pre-installed apps: The find command that deletes .py files after bytecode compilation must exclude engine/apps/pre_installed/*.py because the bootstrap command reads source code from .py files to store in the database. Added exclusion pattern to Dockerfile line 85-88. | - | 2025-12-20 | |
| general | ScriptPlan Tool Architecture: Use wait_for_event() with timeout for interruptible scheduling instead of plain ctx.sleep(). This allows schedule updates to wake the sleeping scheduler immediately via emit_event(). State is stored entirely in workflow variables (ctx.get/set_variable) - no custom DB tables needed. The state reset pattern (delete_checkpoint + loop) prevents history accumulation in long-running schedulers. | - | 2025-12-19 | |
| code-duplication | SandboxedDurableContext has TWO implementations that must stay in sync: (1) engine/sandbox/sandboxed_context.py (for reference/typing), (2) inline string in engine/sandbox/sandboxed_executor.py:218-325 (actually injected into containers). When adding methods to DurableContext API, both must be updated. | - | 2025-12-19 | |
| exception-handling | AbsurdSleepError in operators.py _execute_with_retry() must be explicitly re-raised BEFORE any generic Exception handler. It is a suspension signal, not a failure. If caught as generic Exception, parallel branch workflows fail incorrectly. Fix: add 'except AbsurdSleepError: raise' before 'except Exception'. | - | 2025-12-19 | |
| general | PostgreSQL idle_in_transaction_session_timeout kills connections held open during long task execution. The inline_executor creates SAVEPOINTs before tasks and releases after - if task runs longer than timeout, connection dies. Quick fix: increase timeout in docker-compose.yml. Proper fix: don't hold transactions during execution (ticket #625). | - | 2025-12-19 | |
| general | Cython Optimization - Added three Cython modules for performance: variable_resolver_cy.pyx (30-50% faster), schema_hash_cy.pyx (40-60% faster), chunking_cy.pyx. Key learnings: 1) Use cpdef for functions callable from Python and C. 2) Closures not supported in cpdef - use def instead. 3) Create compat layers for graceful fallback. 4) I/O-bound code (like loop operators with checkpoint saves) has limited Cython ROI. | - | 2025-12-18 | |
| parallel_execution | Parallel branch completion events can cause 'thundering herd' problem where multiple workers pick up the same workflow. Each branch completion emits a NOTIFY, waking up all workers. Without SELECT FOR UPDATE or similar locking, multiple workers execute the same workflow causing duplicate task execution. This was discovered by the ultimate_correctness.py test which uses atomic counters to detect exactly-once violations. | #615 | 2025-12-18 | |
| tools | tools.python.run expects a MODULE PATH (e.g., 'mypackage.module.function'), NOT inline Python code. For inline code execution, use tools.code.exec instead. The key difference: tools.python.run has DurableContext access, tools.code.exec runs in a sandboxed Docker container with no context access. | #614 | 2025-12-18 | |
| code-quality | Proactive code hardening patterns: (1) INSERT ON CONFLICT for idempotent inserts (Fix #458, #495), (2) Atomic UPDATE with WHERE for check-and-modify, (3) Pre-compiled regex at module level (#481, #499, #528), (4) frozenset for immutable constants (#529), (5) TTL caches with proactive eviction (#489), (6) Double-check locking for singletons (#480). Code review found 22 potential issues but 18 were already fixed in the codebase. | - | 2025-12-17 | |
| code-review | Comprehensive Code Review 2025-12-17: Created 22 issues from expert agent review of engine/ and api/. Critical patterns found: (1) Unbounded caches without proactive TTL cleanup cause memory growth (2) Singleton initialization needs double-check locking (3) Check-then-act patterns in DB operations cause race conditions - use INSERT ON CONFLICT or WHERE clause in UPDATE (4) Regex compilation in hot paths - always pre-compile at module level (5) Approval/signal processing needs optimistic locking to prevent double-processing (6) JWT config fetched from Vault on every request - add TTL cache. Priority fixes: #517 DataShard memory, #520 approval race, #525 N+1 analytics, #526 JWT caching. | - | 2025-12-17 | |
| caching | TTL Cache Pattern: Module-level caches keyed by workflow_run_id/tenant_id MUST use TTL+max_size+LRU eviction+thread lock. Store (value,timestamp) tuples. Cleanup when size>max/2, evict oldest 10% when full. | - | 2025-12-17 | |
| architecture | Durable workflow engines intentionally trade performance for durability - each variable operation is a DB round-trip to survive crashes. Caching can improve performance but requires careful design to maintain crash consistency guarantees. | #460 | 2025-12-17 | |
| performance | Use tuple() instead of list() for immutable sequences from dict.keys(), dict.values(), reversed(). Signals immutability and avoids allocation. | #459 | 2025-12-17 | |
| memory | Cache eviction: Unbounded caches are memory leaks. Add TTL + max_size + eviction policy. Store (value, timestamp) tuples. | #456 | 2025-12-17 | |
| threading | Thread-safe singleton: Always use double-check locking with threading.Lock() - check None, acquire lock, check None again, then create. | #455 | 2025-12-17 | |
| general | Multi-tenant isolation is enforced at data access layer, not task claiming. tenant_id is derived from immutable task record in orchestrator.py:376-395, not from worker identity. All data operations (secrets, events, DB) use this tenant_id. RLS would be defense-in-depth, not a security fix. Workers are neutral infrastructure like AWS Lambda. | - | 2025-12-16 | |
| general | Test failures root causes (Dec 2025): (1) JWT token generation bug - config.get_secret() returns dict, use ['value'] not second arg; (2) Docker entrypoint vs command - use entrypoint so it always runs even with custom args; (3) Internal queue saturation - scale internal-worker replicas for parallel test load; (4) Async log timing - increase retry/timeout for tests that wait on async operations | - | 2025-12-15 | |
| general | Docker-in-Docker sandbox path mapping: When workers run inside containers and spawn sandbox containers via Docker socket, bind mounts fail because paths exist only in worker container, not on host. Solution: (1) Use put_archive() API to copy files INTO container instead of bind mounts, (2) For workspace mounts, use PWD:PWD volume mount so path is identical on host and container. | - | 2025-12-15 | |
| checkpoint-system | Multi-tenant checkpoint isolation: tenant_id must be passed through entire save/load chain. If tenant_id defaults to 'default' on save but filters by actual tenant on load, checkpoints are never found, causing ctx.step() to re-execute. See Issue #436. Fix: z_absurd_0.0.9_fix_checkpoint_tenant_id.sql | #436 | 2025-12-14 | |
| general | Platform roles vs tenant roles are defined in separate files (platform.py vs rbac_roles.py). When adding new permissions, ensure they are added to BOTH PLATFORM_ROLES and PREDEFINED_ROLES where applicable. Use sync_platform_permissions() and sync_tenant_role_permissions() to update existing installations. | - | 2025-12-12 | |
| general | Multi-tenant security requires sandboxing ALL user code execution. Never pass raw DB connections to user code. Use HTTP API callbacks with scoped execution tokens for tenant isolation. | - | 2025-12-12 | |
| general | Security Review Pattern: When reviewing engine/ security, check: (1) SQL injection - verify sql.Identifier() for dynamic table names, (2) SSRF - verify DNS pinning not just validation, (3) Sandbox - verify sys.modules cleared, (4) Thread safety - verify locks around caches, (5) Temp files - verify unpredictable names via tempfile module. | - | 2025-12-11 | |
| data-integrity | DataShard storage must use content-addressable keys (definition_hash) instead of mutable identifiers like workflow_name+version. This prevents stale data collisions after DB recreation. See Issue #360 for details. | #360 | 2025-12-11 | |
| workflow-dsl | For TRUE parallel execution in workflows, use ParallelOperator (builder.parallel) NOT ForEachOperator (builder.foreach). ForEach is sequential - each iteration blocks until complete. ParallelOperator spawns independent Absurd tasks that run concurrently. Use wait_for_parallel_branches() to sync after fork. | #359 | 2025-12-11 | |
| tools | Python tools (tools.python.run) MUST have ctx: DurableContext as first parameter in their function signature, even if not used. The tool executor always passes ctx as the first argument when invoking the function. | #358 | 2025-12-11 | |
| workflow-engine | ForEach result storage: Store results in BOTH flat key format (task_id.result) AND dict structure ({task_id: {result: value}}) to support both legacy path resolution (dot-separated) and dict-based Jinja2 templates. See _store_execution_result in inline_executor.py. | #358 | 2025-12-11 | |
| workflow-dsl | Template variable collision: Use alternative placeholder syntax like %CONTENT% in LLM prompts that need runtime substitution. Highway's Jinja2 template engine processes {{variable}} before tool execution, so use content_storage_ref parameter with %CONTENT% placeholder for large content that should be fetched during tool execution. | #358 | 2025-12-11 | |
| workflow-engine | ForEach result storage: Must store as BOTH flat key (task_id.result) AND dict structure ({task_id: {result: ...}}) for path resolution. resolve_variable_path splits by dots. See inline_executor.py:_store_execution_result | - | 2025-12-11 | |
| s3 | S3 Presigned URL SignatureDoesNotMatch: When generating presigned URLs for user uploads where content type is unknown, do NOT include ContentType in signing params. The upload request must send exact same Content-Type as signed, otherwise S3 returns SignatureDoesNotMatch. Fix: omit ContentType from generate_presigned_url params. | #358 | 2025-12-11 | |
| general | HTTP tools must validate URLs for SSRF attacks - always block localhost, private IPs (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x for cloud metadata). Use ipaddress module to check if resolved IP is_private, is_loopback, is_link_local, or is_reserved. | - | 2025-12-10 | |
| general | ActivityOperator pattern: When an operator queues async work (activities), it must wait for completion by calling absurd_client.await_event() and raise AbsurdSleepError if not ready. Check payload.status for failed and raise RuntimeError to propagate failure to workflow. Add async operators to sleep-state exclusion list in inline_executor.py. | - | 2025-12-10 | |
| tools | Reflexive Loop Verification: Use tools.code.exec not tools.python.run for LLM-generated code. tools.python.run expects function_name (module.function path), not raw code. tools.code.exec runs in subprocess with timeout, captures stdout/stderr/exit_code. | #331 | 2025-12-09 | |
| security | CRITICAL: App code_loader ALLOWED_IMPORTS must NOT include engine.config, engine.tools.*, engine.services.*, engine.db, engine.durable_context. These give direct Vault/DB access. Apps use ONLY engine.apps.sdk.base and engine.apps.sdk.context. All functionality via AppContext methods. | #328 | 2025-12-09 | |
| design | waits_for_event actions CANNOT run as activities. ActivityContext is atomic (no durable waits). wait_for_event requires DurableContext to suspend and wait. Approval workflows must run in normal worker mode without timeout_policy. | #328 | 2025-12-09 | |
| general | Database Code Execution Security: When executing user-provided code from database, use multiple layers of security: 1) AST validation for forbidden imports BEFORE execution, 2) Runtime import whitelist via custom __import__, 3) Restricted __builtins__ (no eval/exec/open), 4) Unique module names per execution to prevent caching, 5) Code hash verification for integrity. Issue #326. | - | 2025-12-09 | |
| general | App versioning with python_module entrypoint provides FALSE isolation - code is always loaded from disk, not per-version. For true multi-tenant version isolation, must implement python_code entrypoint that stores actual code in database. | - | 2025-12-09 | |
| general | UUID-only Secret Lookups: Secrets MUST be referenced by UUID only. get_secret() requires secret_id (UUID format), validates with _validate_uuid(), rejects secret_name/secret_path. Returns invalid_uuid for non-UUID values. Provides multi-tenant isolation. | #321 | 2025-12-08 | |
| testing | Python unittest mock patches: When mocking functions imported inside other functions, patch the original location (engine.config.get_config) not where it appears to be used. Mock target must be where the import comes from. | #263 | 2025-12-05 | |
| observability | Sidecar pattern for crash-safe logging: Use separate autocommit connection pool for lifecycle events. Small pool 2-5 conns, autocommit=True for immediate persistence, fire-and-forget design so telemetry failures never crash workflow, log before transaction starts and after commit | #262 | 2025-12-05 | |
| security | OAuth authentication != authorization. After OAuth verifies identity, ALWAYS check user has actual permissions in the system before issuing JWT tokens. Never trust OAuth state parameters for authorization decisions. | #260 | 2025-12-04 | |
| security | API key rotation security: Default behavior must be SECURE (immediate invalidation), with opt-in grace period. Never leave compromised credentials valid by default. | #257 | 2025-12-04 | |
| security | API Security Hardening: Use @require_permission decorators (after @spec.validate), g.tenant_id not headers, AST validation for code execution (whitelist not blacklist) | #248 | 2025-12-04 | |
| general | SECURITY: All API endpoints must have @require_permission decorator. New endpoints require security review before deployment. Tenant isolation must use g.tenant_id from middleware, NEVER request headers directly. | - | 2025-12-04 | |
| general | expected behavior when the worker is killed - the shell process dies with it (process group termination). For long-running activities to survive worker restarts, you need a retry policy: Now if you make restart: 1. Worker dies → shell command gets SIGTERM 2. Activity marked as failed (attempt 1 of 5) 3. After 5 seconds delay, new worker picks it up 4. HTTP server restarts automatically Without retry policy: Activity fails permanently on worker crash With retry policy: Activity auto-recovers when new worker starts | - | 2025-12-03 | |
| architecture | ActivityContext pattern for long-running activities: Use ActivityContext (not DurableContext) for activities to avoid holding DB connections. ActivityContext.get_connection() provides on-demand short-lived connections that auto-commit. DurableContext.get_connection() just yields its held connection. This prevents connection pool exhaustion during high concurrency of long-running activities. | #244 | 2025-12-03 | |
| general | DB Connection Management: Never hold database connections during long-running operations (shell commands, HTTP requests, etc.). Connections should be acquired, used briefly, and released. For long-running activities, acquire connection only for: 1) initial setup/variable resolution, 2) status updates. Use separate short-lived connections for periodic updates (heartbeat, PID storage). This prevents connection pool exhaustion when scaling to hundreds of concurrent activities. | - | 2025-12-03 | |
| architecture | MCP integration: DurableMCPClient wraps ctx.step() for checkpointing. Tools: tools.mcp.invoke, tools.mcp.list_tools, tools.mcp.read_resource. DSL: WorkflowBuilder.mcp_tool(). DB: mcp_server_config with tenant isolation. Transports: stdio and HTTP. Credentials in Vault. | #240 | 2025-12-03 | |
| general | Spectree validates responses when resp= specified. UUIDs must be str(uuid). Response must match Pydantic models exactly. | - | 2025-12-02 | |
| replay | Saga pattern implemented in DurableContext: (1) step_with_compensation() registers compensation functions, (2) run_compensations() executes in reverse order (LIFO), (3) saga() context manager auto-runs compensations on failure. All compensations are durable via ctx.step() for idempotency. | - | 2025-12-02 | |
| replay | Highway replay uses two modes: Display Mode (historical data from checkpoints + audit log) and Simulation Mode (time-travel debugging re-executing code with mocked side effects). Determinism NOT enforced at runtime - relies on developer discipline to use ctx.now, ctx.get_random, ctx.step. | - | 2025-12-02 | |
| general | Scheduling a Workflow When you "schedule" a workflow (e.g., via POST /v1/workflows), the following happens: 1. API Endpoint: The request hits api/blueprints/v1/workflows.py. 2. Versioning: The WorkflowVersioningService hashes the definition and stores it in the workflow_definition table. 3. Tracking: A workflow_run record is created in the database with status pending. 4. Enqueuing: The API calls absurd_client.spawn_task to insert a new task into the absurd task queue (specifically t_{queue_name}). The task name is typically tools.workflow.execute. * Atomicity: This insertion happens within a database transaction. Once committed, the workflow is effectively "scheduled" for immediate execution. | - | 2025-11-30 | |
| general | The execution flow is as follows: 1. The orchestrator claims a task, which corresponds to the execute_workflow function. 2. The orchestrator instantiates a DurableContext, which involves initiating a database transaction and creating an AbsurdClient. 3. The orchestrator then calls execute_workflow, passing in the newly created ctx. 4. execute_workflow instantiates a WorkflowInterpreter. 5. interpreter.start_workflow(ctx=ctx, ...) is called. 6. This, in turn, calls inline_executor.start_workflow(ctx=ctx, ...). 7. Finally, the InlineExecutor executes the workflow graph, passing the same ctx object to all execute_task_inline calls. | - | 2025-11-30 | |
| general | DRAIN LOOP BUG PATTERN: When using while-True drain loops with LIMIT queries, always track seen IDs to prevent infinite loops. If query returns same rows (race condition prevents state change), loop runs forever. Fix: seen_ids set + break if all rows duplicates + MAX_PER_CYCLE limit. | - | 2025-11-30 | |
| general | DSL_VARIABLE_INTERPOLATION: Never echo large variable content like {{result.stdout}} or {{result}} in shell tasks. The full content gets interpolated causing 'Argument list too long' errors. Use specific small fields like {{result.returncode}}, {{result.status_code}}, or truncate large outputs. | - | 2025-11-29 |