Lessons Learned
Knowledge base from resolved issues
Lessons
| Category | Lesson | Issue | Date | Actions |
|---|---|---|---|---|
| debugging | When verifying a peer's claim from logs, size the search window to the EVENT time (their audit timestamp / mail sent time), not to when you read the message. Poll-driven mail means hours can pass between a peer's action and your read; a too-small --since window makes genuine deliveries look like non-deliveries and nearly caused a false accusation on #842. Pair every absence claim with a known-positive probe (garbage logout_token -> 400 + WARNING line) before asserting non-delivery. | #842 | 2026-08-01 | |
| database | SQLAlchemy create_all never ALTERs: an embedded library's schema (auth_rbac) created by an old version keeps narrow varchar columns after upgrades even when current models declare Text. Encrypted cells make this a time bomb - existing short emails fit varchar(64), the first longer email fails INSERT. Reconcile by diffing live information_schema against current model metadata at migration time (varchar->TEXT is metadata-only in postgres). Found because test probes used longer emails than any existing user - test data diversity catches what production data hides. | #838 | 2026-07-31 | |
| testing | auth.database.create_tables() swallows DDL failures by default and Highway logged unconditional success after it - a checkmark that could never go red. Boot-path schema init must use create_tables(raise_on_error=True) AND verify the effect (required tables present in information_schema) rather than the return. | #834 | 2026-07-30 | |
| dependencies | Never build-time patch third-party libs (sed in Dockerfile). The auth PGSSLMODE sed patch (a) silently no-ops if upstream reshapes the line - build stays green, RBAC dies at runtime; (b) masked upstream behavior so diffing installed-vs-pristine wheels misattributed a long-standing hardcode as a new regression. Correct approach: fix the environment (TLS on postgres via docker/init-ssl.sh) or get upstream fixed; never fork silently. | #834 | 2026-07-30 | |
| testing | Test a feature's actual backend path, not just the UI that links to it. The #824 consolidation surfaced Simulate prominently (ActionBar link + 'no side effects' hint) and the plan/memory claimed it was a safe dry-run — but the E2Es only checked that the button navigated; nobody ever ran an actual simulation. It turned out simulate re-executed REAL tool side effects (live shell/http/email) and 500'd on any failure. A single real simulate call in a test would have caught both. Rule: if you advertise a behavior ('no side effects'), assert THAT behavior end-to-end, not the affordance that triggers it. | #833 | 2026-07-19 | |
| architecture | Row-lock scope + TOCTOU in lifecycle endpoints: never write a status row inside a long-running transaction at its START — the lock is held until commit, every concurrent lifecycle UPDATE queues for the whole duration, and their post-lock writes land on stale premises (proven: pause overwrote terminal 'completed' after a 90s lock wait, minting permanent zombies). Status transitions must be single atomic guarded UPDATEs (condition in the WHERE, not a prior SELECT) plus SET LOCAL lock_timeout so residual contention fails fast. Also: a repair sweeper needs an invoker — zombie_detection sat registered-but-never-called with 3 latent bugs (unbindable INTERVAL placeholder, f-string JSON injection, hardcoded queue) while zombies accumulated. | #826 | 2026-07-19 | |
| dashboard-a11y | Don't paper over a11y-lint with a wrong role. A modal backdrop with onclick was given role=button+tabindex=-1 just to silence a11y_no_static_element_interactions — but that makes the scrim announce as a button and nests role=dialog inside role=button (invalid ARIA). Correct fix: the stretched-button pattern — a real <button aria-label='Close dialog'> absolutely positioned behind the dialog (tabindex=-1 so it's out of the tab order; keyboard closes via Escape/✕), the backdrop stays a plain presentational scrim, and keydown lives on the dialog. Native <button> satisfies the click-events rule with no suppression. Same pattern also solved the storage-ref-inside-open-button card (button-in-button is likewise invalid). | #819 | 2026-07-13 | |
| dashboard-ui | Shared component refactors: unify only what's genuinely the same pattern. The app had ~10 tab bars; only 5 (underline: label/count/icon/href-or-button) were a clean fit for one Tabs component. Forcing in the specialized ones (per-tab icon-component switches, disabled+group logic, pill styling, or intentional filter/aria-pressed semantics like approvals FilterTabs) would bloat the component and risk regressions for zero user gain — leave them and document why. Bake cross-cutting fixes into the shared component: Tabs sets flex-shrink:0 so the scroll-container-min-height:0 collapse bug (that hit admin+activities) can never recur on any adopter. Two a11y modes matter: URL-driven tabs are navigation (<nav>+aria-current), in-page tabs are a widget (role=tablist + roving tabindex + arrow keys) — don't give links the tab role. | #819 | 2026-07-13 | |
| dashboard-apps | Marketplace install-status: GET /api/v1/apps/marketplace returns a GLOBAL catalog with installation_count across ALL tenants but NO per-tenant 'installed' flag. To show Installed vs Install per tenant, correlate client-side against GET /api/v1/tenant/apps by app_id (present on both responses) — the app-detail page already does exactly this (installations.find(i => i.app_id === appId)). Chose client cross-reference over adding a backend EXISTS-subquery flag because the latter needs a core-engine change + Docker rebuild + golden-demo run for a pure display concern. | #819 | 2026-07-13 | |
| dashboard-rbac | Dashboard admin-nav gating: the correct 'is this user an admin' signal is the manage_users permission (the string every Admin endpoint enforces via require_permission), NOT the tenant_admin role name (tenants can define custom roles) and NOT isPlatformAdmin (that would lock out tenant admins). CRITICAL gotcha: API-key auth BYPASSES tenant RBAC roles entirely — require_permission reads g.token_scopes (the api_keys.permissions column) where null/empty = full access, so /rbac/my-permissions returns [] for an API key that actually has full admin access. To mirror a permission check client-side you MUST branch on auth type: JWT -> GET /rbac/my-permissions (role-based); API key -> POST /auth/api-key/verify .permissions (scope-based, null/empty = all). | #819 | 2026-07-13 | |
| durability | Exactly-once for external side effects is IMPOSSIBLE at the framework level (durable steps commit only at the task-txn boundary; a crash between a non-transactional SMTP/Telegram effect and that commit re-runs it on replay). Two correct patterns: (1) delegate to the service via a deterministic Idempotency-Key (http.request: key=run_id+step+url); (2) LOCAL at-most-once - claim a deterministic key (run_id:turn:i) in a dedup table on a SEPARATE autocommit connection BEFORE the effect so the claim survives the rolled-back task txn; a crash-replay sees the claim and skips. Best-effort (guard DB down -> run anyway); drop the claim on JumperSleepError so resume re-attempts. See _run_side_effect_once + public.agent_tool_effects. | #750 | 2026-07-05 | |
| llm | Native structured-output enforcement (ollama format) is NOT sufficient alone: cloud thinking models (glm-5) return empty content under a bare schema constraint they never saw in text. Always pair out-of-band enforcement with the schema spelled out in the prompt, and make validation-retry messages include the schema - the model can't fix what it can't see. | #798 | 2026-07-01 | |
| engine | Features nobody exercises rot silently: send_signal had a jsonb=json type error (psycopg Json vs jsonb column) meaning it NEVER worked end-to-end; the first real smoke test found it. Every new primitive needs at least one live harness/smoke case - unit imports are not enough. | #797 | 2026-07-01 | |
| api | Response-model enums must track DB CHECK constraints: pause was half-built (DB allowed 'paused', spectree WorkflowStatus enum did not) so GET /workflows/<id> 500'd on any paused run - a valid state turned into a server error by response validation. Same for 'orphaned'. When adding a status value, grep BOTH migrations and api/openapi/models. | #796 | 2026-07-01 | |
| agents | run_goal re-executes the ENTIRE turn on every durable resume (JumperSleepError replays the task from the top). Any pre-approval/pre-dispatch GUARD that counts persisted state (e.g. delegate rounds via parallel_join_state) will re-run on resume and must EXCLUDE the current in-flight call's own row, or it will reject the call's own resume. Identify the current call via its framework-injected idempotency/orchestration key. Symptom seen: with max_delegate_rounds=1 the first delegate's resume was rejected as 'over cap', the orchestrator lost the sub-agent results, and it over-fetched. JumperSleepError is commit-and-suspend, so state seeded before the suspend (the join_state) IS present on resume. | #780 | 2026-06-29 | |
| architecture | ActivityContext must satisfy the DurableContext attribute interface that tools/app-executor read, or app-action-as-activity paths fail. Known required attrs so far: conn(None)/get_connection, jumper_client, jumper_task_id, jumper_run_id, workflow_run_id, tenant_id, worker_id, current_step_name(None). Prefer adding the attr to ActivityContext over getattr-guards in callers, so it stays a true drop-in. _managed_connection in executor already hasattr-guards db_connection/conn for ActivityContext. | #776 | 2026-06-29 | |
| logging | Per-activity log capture (BufferingActivityLogHandler) attaches to the ROOT logger, so under concurrent activity execution (bulkhead threads) every handler receives every thread's records -> cross-talk. Capture must be thread-scoped: bind owner thread on attach, filter record.thread in emit. Also refcount any root-logger level change shared across concurrent handlers so interleaved attach/detach doesn't strand the level. Trade-off: logs from sub-threads an activity itself spawns are not captured (acceptable vs cross-talk). | #774 | 2026-06-29 | |
| architecture | Tools that may run BOTH inline (DurableContext, ctx.conn held) AND as an activity (ActivityContext, ctx.conn=None) must acquire DB connections via engine.context_db.context_connection(ctx), never ctx.conn directly. It yields the held atomic conn for a durable task (orchestrator commits) or a short-lived auto-commit pooled conn for an activity (so a long LLM/HTTP call never pins a connection). Activity-routed tools: a task is sent to the activity worker when it is non-waiting AND timeout_policy > activity_threshold_seconds(30s). durable_leaves=true makes agent_leaf's execute_leaf_agent run this way. ActivityContext exposes a lazy jumper_client (singleton; methods take conn explicitly). Also: activity worker completion/failure events MUST be emitted with the workflow tenant_id (await_event resolves tenant-scoped since H-7) or non-default-tenant runs hang. Activity logs persist to DataShard via enterprise/tools/datashard_logger (activity_logs table, mirrors task_logs); the engine.tools.activity_logger is a python-logger-only stub. | #771 | 2026-06-29 | |
| testing | Parallel test cleanup must be scoped: Never use broad LIKE patterns in test cleanup fixtures. With xdist parallel runners, broad DELETE patterns destroy rows belonging to other concurrent tests. Track specific IDs and only delete those. | #748 | 2026-02-09 | |
| architecture | IPC isolation: (1) JumperSleepError MUST propagate through IPC server to task executor. (2) Functions from dynamic modules cannot be pickled across IPC - checkpoint() must execute locally. (3) Use GatewayTokenService not ApprovalService for approval token generation in IPC handlers. | #741 | 2026-02-09 | |
| general | Static event names in workflow DSL templates cause cross-run deadlocks. The jumper event table (e_highway_default) stores events globally by event_name without run scoping. When a workflow runs twice, the second run finds the stale event immediately, causing await_event to NOT sleep. This holds the e_ row FOR UPDATE lock while also needing rate_limit_usage lock, creating ABBA deadlock with emit_event. Fix: Always scope intra-workflow event names to {{workflow_run_id}}. | - | 2026-02-08 | |
| general | Bootstrap app registration must scan ALL app directories (pre_installed/ AND examples/), not just one. The preinstalled_apps.yaml must list every app that should be auto-installed per tenant. After any new app is created in enterprise/apps/, it must be added to preinstalled_apps.yaml or it won't survive a clean deploy. | - | 2026-02-08 | |
| testing | DSL template batch test: 27/57 pass. Failures are app/environment level not engine level. Fix: install missing apps for tenant, fix app code bugs (IPC mode, datetime serialization, wrong kwargs), address SMTP timeout. | - | 2026-02-08 | |
| general | File split pattern for backward compat: When splitting a module into a package (e.g. db.py -> db/), use __init__.py re-exports. When splitting into sibling files (e.g. encryption.py + encryption_keys.py), add re-exports at the BOTTOM of the original file after all names are defined. Import from the new file using bottom-of-file imports to avoid circular imports. Docker COPY requires rebuild after adding new files. | - | 2026-02-07 | |
| general | Maturity review pattern: When reviewing concurrency in a DB-backed engine, check (1) state transition guards with WHERE clause + rowcount check, (2) mark-before-execute ordering for idempotent callbacks, (3) DAG cycle detection at both submission and runtime, (4) savepoint/cache rollback alignment. Agent-reported race conditions should be verified against source - many were already mitigated by atomic SQL functions and idempotency keys. | - | 2026-02-07 | |
| general | Signal handler pattern for blocking waits: When signal handler sets a shutdown flag, also set any Event objects that threads may be waiting on. Otherwise threads blocked on Event.wait() won't wake until timeout. | - | 2025-12-28 | |
| general | Docker signal handling: When using shell expansion in docker-compose command, add 'exec' before the actual command to make the process PID 1 and receive signals directly. Without exec, the shell is PID 1 and absorbs SIGTERM. | - | 2025-12-28 | |
| best-practice | Use sync httpx.Client not async httpx.AsyncClient in workers - asyncio can cause zombie workers where container shows healthy but all threads are dead. httpx.Client is equally capable and avoids asyncio concurrency bugs. ThreadPoolExecutor workarounds for asyncio are fragile. | #721 | 2025-12-27 | |
| architecture | Docker-in-Docker temp files must use shared mount path: When spawning containers from inside containers (DinD with Docker socket), temp files created in /tmp exist only inside the parent container. Volume mounts use HOST paths. Use a shared mount path (same absolute path on host and container) for files that need to be mounted into child containers. | #720 | 2025-12-27 | |
| architecture | Parallel IDs must use workflow_run_id not absurd_run_id: When generating identifiers that need to persist across retries (like parallel branch join_event_name), ALWAYS use workflow_run_id (stable across retries) not absurd_run_id (changes on each retry). absurd_run_id represents one execution attempt, workflow_run_id represents the entire workflow lifecycle. | #719 | 2025-12-27 | |
| architecture | ActivityOperator is queue-only: When creating internal operators (like ReflexiveOperator creating ActivityOperator), execute_activity_operator only QUEUES the activity and returns immediately. Internal operators must add their own await_event() call to wait for completion. See engine/interpreters/inline_executor.py _execute_reflexive_operator | #713 | 2025-12-25 | |
| security | Python DSL must execute in isolated container: No DB libs (psycopg), no network libs (requests), no secrets. Use separate Docker network (internal:true). API calls http://dsl-compiler:8080/compile. | #678 | 2025-12-24 | |
| api/visibility | Terminal workflow states (completed, compensated, failed) should always show progress_percentage=100. Switch/conditional branches that aren't executed shouldn't reduce progress for finished workflows. | #677 | 2025-12-24 | |
| api/visibility | Progress calculation requires unlimited query: (1) Fork/join workflows have branch step_succeeded events not in task_ids - use set intersection. (2) LIMIT 50 drops early events - split into limited query for UI and unlimited query for progress count. See workflows.py:1424-1471. | #675 | 2025-12-24 | |
| PostgreSQL | JSONB column data extraction: When querying PostgreSQL tables with JSONB columns (like absurd_event_log.payload), data nested in JSONB must be extracted using arrow operators (payload->>'key' for text, payload->'key' for JSONB). Always verify schema with \\d before writing SQL - event logs often store step names and results inside payload JSONB, not as separate columns. | #674 | 2025-12-24 | |
| database | Always use row_factory=dict_row when creating psycopg3 cursors for dict access. Use get_db_cursor() helper which sets this by default. | #673 | 2025-12-24 | |
| general | RAG cache must use PostgreSQL not file system - /tmp is container-local in Docker, so each container computes embeddings separately causing hundreds of /embedding API calls. Use rag_embedding_cache table with TTL for shared cache across containers. | - | 2025-12-24 | |
| general | Branch tasks must NEVER update parent workflow_run status. Always check 'task_name \!= BRANCH_EXECUTION_TASK' before updating workflow_run for any status (running, completed, sleeping, failed). The parent workflow controls its own lifecycle - branches only emit events. | - | 2025-12-24 | |
| orchestrator | CRITICAL: Always sync workflow_run.status with absurd task state. When task state changes (sleeping, cancelled, retrying, abandoned), update workflow_run accordingly. Absurd is source of truth, workflow_run is UI display layer. | - | 2025-12-23 | |
| security | Security: When implementing IPC proxies for sandboxed code, never blindly forward all method calls. Always enforce a strict whitelist of allowed methods on the server side to prevent sandboxed code from accessing dangerous internal APIs. | - | 2025-12-22 | |
| ScriptPlan | Working Hours Correct Usage: (1) Use project.dateToIdx(naive_datetime) - datetime MUST be NAIVE; (2) Get WorkingHours from shift.get('workinghours', 0) not resource.workinghours; (3) Use resource.get('leaves', 0) not resource.leaves which is a method; (4) Project must be scheduled with schedule=True | - | 2025-12-21 | |
| architecture | For distributed rate limiting, use PostgreSQL atomic operations (INSERT ON CONFLICT DO UPDATE RETURNING). Two-phase architecture: CPU-heavy scheduling done once on config change (parse TJP, cache report in JSONB), fast runtime checks compare cached windows. Always support fail_open for graceful degradation. | #635 | 2025-12-20 | |
| general | Dockerfile pre-installed apps: The find command that deletes .py files after bytecode compilation must exclude engine/apps/pre_installed/*.py because the bootstrap command reads source code from .py files to store in the database. Added exclusion pattern to Dockerfile line 85-88. | - | 2025-12-20 | |
| general | ScriptPlan Tool Architecture: Use wait_for_event() with timeout for interruptible scheduling instead of plain ctx.sleep(). This allows schedule updates to wake the sleeping scheduler immediately via emit_event(). State is stored entirely in workflow variables (ctx.get/set_variable) - no custom DB tables needed. The state reset pattern (delete_checkpoint + loop) prevents history accumulation in long-running schedulers. | - | 2025-12-19 | |
| code-duplication | SandboxedDurableContext has TWO implementations that must stay in sync: (1) engine/sandbox/sandboxed_context.py (for reference/typing), (2) inline string in engine/sandbox/sandboxed_executor.py:218-325 (actually injected into containers). When adding methods to DurableContext API, both must be updated. | - | 2025-12-19 | |
| exception-handling | AbsurdSleepError in operators.py _execute_with_retry() must be explicitly re-raised BEFORE any generic Exception handler. It is a suspension signal, not a failure. If caught as generic Exception, parallel branch workflows fail incorrectly. Fix: add 'except AbsurdSleepError: raise' before 'except Exception'. | - | 2025-12-19 | |
| general | PostgreSQL idle_in_transaction_session_timeout kills connections held open during long task execution. The inline_executor creates SAVEPOINTs before tasks and releases after - if task runs longer than timeout, connection dies. Quick fix: increase timeout in docker-compose.yml. Proper fix: don't hold transactions during execution (ticket #625). | - | 2025-12-19 | |
| general | Cython Optimization - Added three Cython modules for performance: variable_resolver_cy.pyx (30-50% faster), schema_hash_cy.pyx (40-60% faster), chunking_cy.pyx. Key learnings: 1) Use cpdef for functions callable from Python and C. 2) Closures not supported in cpdef - use def instead. 3) Create compat layers for graceful fallback. 4) I/O-bound code (like loop operators with checkpoint saves) has limited Cython ROI. | - | 2025-12-18 | |
| parallel_execution | Parallel branch completion events can cause 'thundering herd' problem where multiple workers pick up the same workflow. Each branch completion emits a NOTIFY, waking up all workers. Without SELECT FOR UPDATE or similar locking, multiple workers execute the same workflow causing duplicate task execution. This was discovered by the ultimate_correctness.py test which uses atomic counters to detect exactly-once violations. | #615 | 2025-12-18 | |
| tools | tools.python.run expects a MODULE PATH (e.g., 'mypackage.module.function'), NOT inline Python code. For inline code execution, use tools.code.exec instead. The key difference: tools.python.run has DurableContext access, tools.code.exec runs in a sandboxed Docker container with no context access. | #614 | 2025-12-18 | |
| code-quality | Proactive code hardening patterns: (1) INSERT ON CONFLICT for idempotent inserts (Fix #458, #495), (2) Atomic UPDATE with WHERE for check-and-modify, (3) Pre-compiled regex at module level (#481, #499, #528), (4) frozenset for immutable constants (#529), (5) TTL caches with proactive eviction (#489), (6) Double-check locking for singletons (#480). Code review found 22 potential issues but 18 were already fixed in the codebase. | - | 2025-12-17 | |
| code-review | Comprehensive Code Review 2025-12-17: Created 22 issues from expert agent review of engine/ and api/. Critical patterns found: (1) Unbounded caches without proactive TTL cleanup cause memory growth (2) Singleton initialization needs double-check locking (3) Check-then-act patterns in DB operations cause race conditions - use INSERT ON CONFLICT or WHERE clause in UPDATE (4) Regex compilation in hot paths - always pre-compile at module level (5) Approval/signal processing needs optimistic locking to prevent double-processing (6) JWT config fetched from Vault on every request - add TTL cache. Priority fixes: #517 DataShard memory, #520 approval race, #525 N+1 analytics, #526 JWT caching. | - | 2025-12-17 | |
| caching | TTL Cache Pattern: Module-level caches keyed by workflow_run_id/tenant_id MUST use TTL+max_size+LRU eviction+thread lock. Store (value,timestamp) tuples. Cleanup when size>max/2, evict oldest 10% when full. | - | 2025-12-17 | |
| architecture | Durable workflow engines intentionally trade performance for durability - each variable operation is a DB round-trip to survive crashes. Caching can improve performance but requires careful design to maintain crash consistency guarantees. | #460 | 2025-12-17 | |
| performance | Use tuple() instead of list() for immutable sequences from dict.keys(), dict.values(), reversed(). Signals immutability and avoids allocation. | #459 | 2025-12-17 | |
| memory | Cache eviction: Unbounded caches are memory leaks. Add TTL + max_size + eviction policy. Store (value, timestamp) tuples. | #456 | 2025-12-17 | |
| threading | Thread-safe singleton: Always use double-check locking with threading.Lock() - check None, acquire lock, check None again, then create. | #455 | 2025-12-17 | |
| general | Multi-tenant isolation is enforced at data access layer, not task claiming. tenant_id is derived from immutable task record in orchestrator.py:376-395, not from worker identity. All data operations (secrets, events, DB) use this tenant_id. RLS would be defense-in-depth, not a security fix. Workers are neutral infrastructure like AWS Lambda. | - | 2025-12-16 | |
| general | Test failures root causes (Dec 2025): (1) JWT token generation bug - config.get_secret() returns dict, use ['value'] not second arg; (2) Docker entrypoint vs command - use entrypoint so it always runs even with custom args; (3) Internal queue saturation - scale internal-worker replicas for parallel test load; (4) Async log timing - increase retry/timeout for tests that wait on async operations | - | 2025-12-15 | |
| general | Docker-in-Docker sandbox path mapping: When workers run inside containers and spawn sandbox containers via Docker socket, bind mounts fail because paths exist only in worker container, not on host. Solution: (1) Use put_archive() API to copy files INTO container instead of bind mounts, (2) For workspace mounts, use PWD:PWD volume mount so path is identical on host and container. | - | 2025-12-15 | |
| checkpoint-system | Multi-tenant checkpoint isolation: tenant_id must be passed through entire save/load chain. If tenant_id defaults to 'default' on save but filters by actual tenant on load, checkpoints are never found, causing ctx.step() to re-execute. See Issue #436. Fix: z_absurd_0.0.9_fix_checkpoint_tenant_id.sql | #436 | 2025-12-14 | |
| general | Platform roles vs tenant roles are defined in separate files (platform.py vs rbac_roles.py). When adding new permissions, ensure they are added to BOTH PLATFORM_ROLES and PREDEFINED_ROLES where applicable. Use sync_platform_permissions() and sync_tenant_role_permissions() to update existing installations. | - | 2025-12-12 | |
| general | Multi-tenant security requires sandboxing ALL user code execution. Never pass raw DB connections to user code. Use HTTP API callbacks with scoped execution tokens for tenant isolation. | - | 2025-12-12 | |
| general | Security Review Pattern: When reviewing engine/ security, check: (1) SQL injection - verify sql.Identifier() for dynamic table names, (2) SSRF - verify DNS pinning not just validation, (3) Sandbox - verify sys.modules cleared, (4) Thread safety - verify locks around caches, (5) Temp files - verify unpredictable names via tempfile module. | - | 2025-12-11 | |
| data-integrity | DataShard storage must use content-addressable keys (definition_hash) instead of mutable identifiers like workflow_name+version. This prevents stale data collisions after DB recreation. See Issue #360 for details. | #360 | 2025-12-11 | |
| workflow-dsl | For TRUE parallel execution in workflows, use ParallelOperator (builder.parallel) NOT ForEachOperator (builder.foreach). ForEach is sequential - each iteration blocks until complete. ParallelOperator spawns independent Absurd tasks that run concurrently. Use wait_for_parallel_branches() to sync after fork. | #359 | 2025-12-11 | |
| tools | Python tools (tools.python.run) MUST have ctx: DurableContext as first parameter in their function signature, even if not used. The tool executor always passes ctx as the first argument when invoking the function. | #358 | 2025-12-11 | |
| workflow-engine | ForEach result storage: Store results in BOTH flat key format (task_id.result) AND dict structure ({task_id: {result: value}}) to support both legacy path resolution (dot-separated) and dict-based Jinja2 templates. See _store_execution_result in inline_executor.py. | #358 | 2025-12-11 | |
| workflow-dsl | Template variable collision: Use alternative placeholder syntax like %CONTENT% in LLM prompts that need runtime substitution. Highway's Jinja2 template engine processes {{variable}} before tool execution, so use content_storage_ref parameter with %CONTENT% placeholder for large content that should be fetched during tool execution. | #358 | 2025-12-11 | |
| workflow-engine | ForEach result storage: Must store as BOTH flat key (task_id.result) AND dict structure ({task_id: {result: ...}}) for path resolution. resolve_variable_path splits by dots. See inline_executor.py:_store_execution_result | - | 2025-12-11 | |
| s3 | S3 Presigned URL SignatureDoesNotMatch: When generating presigned URLs for user uploads where content type is unknown, do NOT include ContentType in signing params. The upload request must send exact same Content-Type as signed, otherwise S3 returns SignatureDoesNotMatch. Fix: omit ContentType from generate_presigned_url params. | #358 | 2025-12-11 | |
| general | HTTP tools must validate URLs for SSRF attacks - always block localhost, private IPs (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x for cloud metadata). Use ipaddress module to check if resolved IP is_private, is_loopback, is_link_local, or is_reserved. | - | 2025-12-10 | |
| general | ActivityOperator pattern: When an operator queues async work (activities), it must wait for completion by calling absurd_client.await_event() and raise AbsurdSleepError if not ready. Check payload.status for failed and raise RuntimeError to propagate failure to workflow. Add async operators to sleep-state exclusion list in inline_executor.py. | - | 2025-12-10 | |
| tools | Reflexive Loop Verification: Use tools.code.exec not tools.python.run for LLM-generated code. tools.python.run expects function_name (module.function path), not raw code. tools.code.exec runs in subprocess with timeout, captures stdout/stderr/exit_code. | #331 | 2025-12-09 | |
| security | CRITICAL: App code_loader ALLOWED_IMPORTS must NOT include engine.config, engine.tools.*, engine.services.*, engine.db, engine.durable_context. These give direct Vault/DB access. Apps use ONLY engine.apps.sdk.base and engine.apps.sdk.context. All functionality via AppContext methods. | #328 | 2025-12-09 | |
| design | waits_for_event actions CANNOT run as activities. ActivityContext is atomic (no durable waits). wait_for_event requires DurableContext to suspend and wait. Approval workflows must run in normal worker mode without timeout_policy. | #328 | 2025-12-09 | |
| general | Database Code Execution Security: When executing user-provided code from database, use multiple layers of security: 1) AST validation for forbidden imports BEFORE execution, 2) Runtime import whitelist via custom __import__, 3) Restricted __builtins__ (no eval/exec/open), 4) Unique module names per execution to prevent caching, 5) Code hash verification for integrity. Issue #326. | - | 2025-12-09 | |
| general | App versioning with python_module entrypoint provides FALSE isolation - code is always loaded from disk, not per-version. For true multi-tenant version isolation, must implement python_code entrypoint that stores actual code in database. | - | 2025-12-09 | |
| general | UUID-only Secret Lookups: Secrets MUST be referenced by UUID only. get_secret() requires secret_id (UUID format), validates with _validate_uuid(), rejects secret_name/secret_path. Returns invalid_uuid for non-UUID values. Provides multi-tenant isolation. | #321 | 2025-12-08 | |
| testing | Python unittest mock patches: When mocking functions imported inside other functions, patch the original location (engine.config.get_config) not where it appears to be used. Mock target must be where the import comes from. | #263 | 2025-12-05 | |
| observability | Sidecar pattern for crash-safe logging: Use separate autocommit connection pool for lifecycle events. Small pool 2-5 conns, autocommit=True for immediate persistence, fire-and-forget design so telemetry failures never crash workflow, log before transaction starts and after commit | #262 | 2025-12-05 | |
| security | OAuth authentication != authorization. After OAuth verifies identity, ALWAYS check user has actual permissions in the system before issuing JWT tokens. Never trust OAuth state parameters for authorization decisions. | #260 | 2025-12-04 | |
| security | API key rotation security: Default behavior must be SECURE (immediate invalidation), with opt-in grace period. Never leave compromised credentials valid by default. | #257 | 2025-12-04 | |
| security | API Security Hardening: Use @require_permission decorators (after @spec.validate), g.tenant_id not headers, AST validation for code execution (whitelist not blacklist) | #248 | 2025-12-04 | |
| general | SECURITY: All API endpoints must have @require_permission decorator. New endpoints require security review before deployment. Tenant isolation must use g.tenant_id from middleware, NEVER request headers directly. | - | 2025-12-04 | |
| general | expected behavior when the worker is killed - the shell process dies with it (process group termination). For long-running activities to survive worker restarts, you need a retry policy: Now if you make restart: 1. Worker dies → shell command gets SIGTERM 2. Activity marked as failed (attempt 1 of 5) 3. After 5 seconds delay, new worker picks it up 4. HTTP server restarts automatically Without retry policy: Activity fails permanently on worker crash With retry policy: Activity auto-recovers when new worker starts | - | 2025-12-03 | |
| architecture | ActivityContext pattern for long-running activities: Use ActivityContext (not DurableContext) for activities to avoid holding DB connections. ActivityContext.get_connection() provides on-demand short-lived connections that auto-commit. DurableContext.get_connection() just yields its held connection. This prevents connection pool exhaustion during high concurrency of long-running activities. | #244 | 2025-12-03 | |
| general | DB Connection Management: Never hold database connections during long-running operations (shell commands, HTTP requests, etc.). Connections should be acquired, used briefly, and released. For long-running activities, acquire connection only for: 1) initial setup/variable resolution, 2) status updates. Use separate short-lived connections for periodic updates (heartbeat, PID storage). This prevents connection pool exhaustion when scaling to hundreds of concurrent activities. | - | 2025-12-03 | |
| architecture | MCP integration: DurableMCPClient wraps ctx.step() for checkpointing. Tools: tools.mcp.invoke, tools.mcp.list_tools, tools.mcp.read_resource. DSL: WorkflowBuilder.mcp_tool(). DB: mcp_server_config with tenant isolation. Transports: stdio and HTTP. Credentials in Vault. | #240 | 2025-12-03 | |
| general | Spectree validates responses when resp= specified. UUIDs must be str(uuid). Response must match Pydantic models exactly. | - | 2025-12-02 | |
| replay | Saga pattern implemented in DurableContext: (1) step_with_compensation() registers compensation functions, (2) run_compensations() executes in reverse order (LIFO), (3) saga() context manager auto-runs compensations on failure. All compensations are durable via ctx.step() for idempotency. | - | 2025-12-02 | |
| replay | Highway replay uses two modes: Display Mode (historical data from checkpoints + audit log) and Simulation Mode (time-travel debugging re-executing code with mocked side effects). Determinism NOT enforced at runtime - relies on developer discipline to use ctx.now, ctx.get_random, ctx.step. | - | 2025-12-02 | |
| general | Scheduling a Workflow When you "schedule" a workflow (e.g., via POST /v1/workflows), the following happens: 1. API Endpoint: The request hits api/blueprints/v1/workflows.py. 2. Versioning: The WorkflowVersioningService hashes the definition and stores it in the workflow_definition table. 3. Tracking: A workflow_run record is created in the database with status pending. 4. Enqueuing: The API calls absurd_client.spawn_task to insert a new task into the absurd task queue (specifically t_{queue_name}). The task name is typically tools.workflow.execute. * Atomicity: This insertion happens within a database transaction. Once committed, the workflow is effectively "scheduled" for immediate execution. | - | 2025-11-30 | |
| general | The execution flow is as follows: 1. The orchestrator claims a task, which corresponds to the execute_workflow function. 2. The orchestrator instantiates a DurableContext, which involves initiating a database transaction and creating an AbsurdClient. 3. The orchestrator then calls execute_workflow, passing in the newly created ctx. 4. execute_workflow instantiates a WorkflowInterpreter. 5. interpreter.start_workflow(ctx=ctx, ...) is called. 6. This, in turn, calls inline_executor.start_workflow(ctx=ctx, ...). 7. Finally, the InlineExecutor executes the workflow graph, passing the same ctx object to all execute_task_inline calls. | - | 2025-11-30 | |
| general | DRAIN LOOP BUG PATTERN: When using while-True drain loops with LIMIT queries, always track seen IDs to prevent infinite loops. If query returns same rows (race condition prevents state change), loop runs forever. Fix: seen_ids set + break if all rows duplicates + MAX_PER_CYCLE limit. | - | 2025-11-30 | |
| general | DSL_VARIABLE_INTERPOLATION: Never echo large variable content like {{result.stdout}} or {{result}} in shell tasks. The full content gets interpolated causing 'Argument list too long' errors. Use specific small fields like {{result.returncode}}, {{result.status_code}}, or truncate large outputs. | - | 2025-11-29 |