Memory
Persistent context for AI agents
Stored Items
| Category | Key | Value | Actions |
|---|---|---|---|
| lessons | AUTH_DATABASE_TYPE | REQUIRED for auth library - Without AUTH_DATABASE_TYPE=postgresql, auth library defaults to SQLite. Always set AUTH_DATABASE_TYPE before AUTH_DATABASE_URL in K8s deployments. | |
| workflow | activity_wait_pattern | Activity workflows need wait_for_event after each builder.activity() to wait for completion via {{result.completion_event}} | |
| guidelines | api_idempotency | ALL workflow engine APIs and actions must be strictly idempotent. No side effects on second call. Use idempotency keys where needed. | |
| guidelines | api_only | ALL tools must use API to interact with the workflow engine. Direct database access is forbidden outside of engine core code. | |
| architecture | app_code_no_cache | Issue #326: All app code is stored in database and loaded fresh on each execution. NO CACHING. python_module entrypoint removed. Cache APIs deprecated. Changes take effect immediately. | |
| architecture | app_extension_ceiling | Apps CANNOT replace built-in tools for anything that touches Highway itself. Hard limits (verified Aug 2026): (1) no Highway DB access - ctx.db_connection raises, blocked at enterprise/ipc/server.py:155; (2) tool name grammar is fixed to apps.{pub}.{app}.{action}, apps cannot register a tool name or a DSL operator (OperatorType is a closed 16-member enum in the highway_dsl pip package); (3) apps are workflow-task-only - no schedule/webhook/event/API entrypoint reaches an app; (4) IPC ALLOWED_METHODS is 18 names (ipc/server.py:133-157) - no spawn_workflow, no branch ops, no delete_checkpoint, so durable-loop/cron patterns are structurally impossible as apps; (5) ctx.checkpoint does NOT memoize in production (context.py:266-271); (6) every invocation forks a child and imports ~70 whitelisted modules incl. polars/pyarrow/great_expectations, so nothing on a hot path can be an app. Apps DO fit: third-party API connectors, external-DB ETL, notification senders, format conversion, stateless enrichment. Migrate features to apps only after checking against this list. | |
| guidelines | async | When async needed, use trio library only. | |
| lessons | atomic_transactions | Data integrity and atomicity are top priorities. TimeoutService uses atomic check-and-set (fencing) to prevent race conditions. | |
| lessons | auth_create_tables_never_alters | auth's create_tables is create_all(checkfirst=True): it creates missing tables but NEVER adds columns to an existing one, and the auth wheel ships no migrations. Every auth upgrade must diff auth/models/sql.py between the locked and the new version; new columns need a Highway product migration (migrations/) applied before initialize_auth_schema. Since #1045 initialize_auth_schema fails when any model column is missing (missing_model_columns). Real case: 3.1.0 -> 3.4.0 added auth_tenant_settings.rotate_key_hash, rotate_key_issued_at, canary; live DB lacked all three, and 3.4.0's strict-users check selects the full row, so every permission check would have failed. | |
| guidelines | backups | Write backups to /tmp/highway-workflow-backups/ with timestamps. | |
| guidelines | ci_only_testing | Director's rule (since 2026-10-06): no lint/type-check/test suite/browser test/build-heavy job on vm-1, in any form. vm-1 carries mail, identity, agentbus and nginx for the estate. All Highway verification runs in CI by pushing commits. Exception: a single test file at -n 1 for a live fault, announced to infra-manager-c13110 first. | |
| guidelines | ci_push_batching | Push to highway/highway-engine/jumper only in batches that clear a set of CI failures; never push per change. ci-conductor (2026-10-20, during a CI incident with 9 seats) set Highway's fair-share weight to 0.25 until the first green main. Run a pre-push gitleaks scan of origin/main..main with each repo's .gitleaks.toml first. | |
| guidelines | circuit_breaker | Use resilient-circuit (>=0.8.6,<0.9, RC_DB_STRICT=1, schema via pg-setup/pg-check, no runtime DDL) for breaking/retry and bulkman (>=2.0.4, circuit_breaker_enabled=False always) for bulkheads. highway_circuitbreaker is retired (#920, #850). | |
| guidelines | code_quality | Ruff for linting/formatting. All code must pass ruff and mypy checks. | |
| architecture | core_components | 1. Absurd Task Queue (PostgreSQL-based), 2. AbsurdClient (transaction-aware, never commits), 3. Orchestrator (atomic SUCCESS/SLEEP/FAILURE paths), 4. DurableContext (ctx.step, ctx.wait_for_event, ctx.emit_event), 5. WorkflowInterpreter (Highway DSL executor), 6. CLI Tools (worker.py, submit.py, replay.py, monitor.py) | |
| lessons | core_unit_db_tests_write_live_db | Engine tests/unit/db writes to the configured DB; it is guarded (skips unless HIGHWAY_DB_TESTS=1, refuses highway_db_v2). Incident 2026-10-20 ~02:29Z: an unguarded run deleted live events via jumper.cleanup_events(60 s). See tests_conftest_writes_live_db. | |
| guidelines | database_rules | Use psycopg3 only with explicit transactions. No ORMs. SQL queries must be in engine/sql/queries/*.sql files. | |
| architecture | database_tables | jumper.t_*/r_*/c_*/e_* (task definitions, runs, checkpoints, events per queue), workflow_run (execution tracking), absurd_event_log (immutable audit trail). Workflow definitions are JSONB: key order is NOT preserved (#938). | |
| dependencies | datashard | datashard is pinned via uv.lock (0.7.2 as of 2026-10-17). Upgrade to 0.11 on S3 is #926 (offline table migration, irreversible). Install only from the lockfile; no source modifications. | |
| guidelines | db_migrations | NEVER modify database schema/functions directly. Always create migration files in engine/migrations/sql/highway_X.X.XX_description.sql. Use IF EXISTS for idempotency. | |
| guidelines | debugging | Use hwe replay <workflow_id> to replay workflows. Prefer this over direct db access. Add missing features to engine/cli/replay.py | |
| guidelines | demo_workflows | Golden demos: api/dsl_templates/golden_deterministic_v2.py (validate_demo_v2.py), golden_disaster_recovery (validate_demo_disaster.py), ultimate, operators; demo-matrix via scripts/demo/run_demo_matrix.py (value checks via API). Must be re-run after engine changes, through the public API, checking values not just status. demo/video_convertion_workflow.py is retired from the golden set (needs network/ffmpeg/shared files; RunFlow job later, #934). | |
| deployment | docker | docker-compose stack: postgres, migrations, api (hypercorn --workers 4), worker x2, activity-worker x1, internal-worker x2 (6 concurrent tasks, pool 11), dsl-compiler (unix socket, egress-only network). Python 3.13.16-slim pinned by digest; deps from uv.lock with --require-hashes. | |
| general | docker_no_remove_param | tools.docker.run does NOT have a 'remove' parameter. Containers are ALWAYS automatically removed after completion. Do not include remove=True in Docker workflows. | |
| lessons | docker_single_file_bind_mount_inode | Editing a file that docker bind-mounts individually (e.g. ./docker/config.ini:/etc/highway/config.ini) does NOT reach the running container: most editors write a temp file and rename, which breaks the inode the mount points at. The container keeps serving the OLD content with no error. Always recreate the container after editing a single-file bind mount, and verify by reading the file INSIDE the container - not on the host. | |
| guidelines | documentation | Markdown files in docs/agent-docs/. No excessive summaries or MD files. | |
| guidelines | dsl | Highway DSL is the only DSL for data modeling and schema definition. Use hwe dsl-prompt for syntax info. | |
| lessons | futex_any_of_reject | Futex any_of step: one reject does NOT terminate the decision (stays in_review, other assignees may still approve). Highway #930 cancels the decision after a Highway reject to keep veto semantics; cancelled envelope keeps outcome.reasons/actors. Futex webhook registration validates DNS only, not reachability. | |
| guidelines | grep_skips_submodules | The shell 'grep' here wraps ripgrep and SILENTLY SKIPS git submodule contents. 'grep -rn X .' from repo root returns ZERO hits inside docker/highway-core/ (engine, 72k lines) and docker/highway-core/lib/jumper/ - no error, just silence. Every 'no usage found / dead code' conclusion drawn that way is VACUOUS, and biased one way: engine-internal usage becomes invisible so live code looks dead. ALWAYS use: /usr/bin/grep -rn --include='*.py' 'PATTERN' api enterprise docker/highway-core/engine . Real example: an audit claimed 'zero @with_rate_limit applications on core tools'; submodule-aware search shows it live on tools.shell.command, tools.llm.call, tools.http.request, tools.email.send. | |
| general | handler_task_ordering | In WorkflowBuilder, handler tasks (on_failure/on_success) MUST be defined AFTER the tasks that reference them. Reason: auto-chaining makes first task start_task, and unrecognized handlers get chained. Always define main workflow tasks FIRST, handlers LAST. Handlers of parallel-branch and loop-body tasks go INSIDE the same lambda, after the referencing task (#966); they run only as callbacks. A branch task whose handler is defined outside its branch is rejected at submit and executor init with BranchHandlerScopeError (HTTP 400 WORKFLOW_VALIDATION_ERROR). | |
| dependencies | highway_dsl | Highway DSL v1.9.0 - Build from source in ~/develop/highway_dsl/. Modifications allowed. See docs/agent-docs/HIGHWAY_DSL_INTEGRATION.md | |
| guidelines | logging | Use logging module only. No print statements allowed. | |
| general | logging-best-practice | CRITICAL: Never use f-strings in logger calls. ALWAYS use lazy formatting: logger.debug('message %s', value) NOT logger.debug(f'message {value}'). F-strings evaluate immediately even if log level is disabled, causing: 1) Performance overhead, 2) Security risk - values get logged even at DEBUG when user sees INFO. Use %s placeholders and pass values as args. | |
| general | logging-security | Never log actual variable values in workflow engines. Log only metadata: type, length, presence (is_set=True). Workflow variables may contain PII, secrets, or large content like 'THE COMPLETE SHERLOCK HOLMES'. Use DEBUG level for internal tracing, not INFO. | |
| guidelines | migrations | Raw PostgreSQL only in engine/migrations/sql/ directory, executed in order. All DB changes must go via migration scripts. | |
| lessons | migretti_shared_migrations_table | core, jumper and product migrations share one _migrations table (migretti 0.10 has no table setting; _migrations is unqualified). Since #971 (uncommitted as of 2026-10-20, lands with the lead's batch) docker/migrations-entrypoint.sh merges the three roots into ONE set in a temp dir (same-name files must be byte-identical, jumper_initial 01KGX7 excluded and only recorded, at the earliest applied_at) and runs a single 'mg apply' without --allow-out-of-order. Never run mg apply/down/rollback per root (/opt/highway-migrations, /opt/jumper-migrations, /opt/product-migrations) against highway_db_v2: (1) 'mg down' reverts the most recently APPLIED row of the whole table, and from the jumper root a rollback can reach jumper_initial whose down is 'drop schema jumper cascade'; (2) the out-of-order guard compares against the max id of every root, so a fresh install stops at the jumper root. mg verify on live fails on jumper_initial (NULL checksum) and parallel_join_counter (pre-commit checksum), see the low-priority follow-up ticket. | |
| lessons | no_deprecations | No deprecated code allowed. Remove deprecated code instead. Production-grade codebase, no deprecations. | |
| lessons | parallel_operator_fix | ParallelOperator and ActivityOperator are fork/queue-only (Nov 16, 2025). They spawn tasks and return immediately. No automatic waiting. Prevents double-fork bugs in crash recovery. Workflows must add explicit join/wait logic separately. | |
| spec | prism_single_upstream | All Highway chat routes ([llm] default_route, agent, fast, smart, cheap) use prism/deepseek-v4-flash with no fallback route. Director, 2026-10-25: single upstream accepted until Prism adds a second one (Prism #81). Revisit then, not before. | |
| guidelines | process_management | Manage services with docker compose (no systemd, #917). Kill only by PID, never pkill/killall. | |
| lessons | replay_does_not_execute_tools | Since #833 replay/simulation returns RECORDED task results and never calls the tool (task.py execute_task_operator short-circuits on ctx.is_replay). So any workflow variable set by a tool via ctx.set_variable is absent in replay, and a while condition reading it can never go false. Bound replay loops by the original run's checkpoint iteration count, never by the live runaway ceiling. See #845 (fix) and #846 (restore recorded variable state). ReplayContext's docstring still claims it executes functions - it does not. | |
| workflow | result_key_naming | Avoid using result_key with same name as task_id - causes 'int object does not support item assignment' error during workflow execution | |
| lessons | resumability | Resumability is critical. Workflows must be able to resume from checkpoints without data loss or corruption. | |
| spec | rotate_at_bare_metal | Director decision 2026-10-22: no rotations until Highway moves to the two bare-metal servers; rotate everything then. Includes HIGHWAY_SUBMODULE_TOKEN in ci.rodmena.co.uk (repos rodmena-limited/highway and highway-engine): fine-grained PAT ci-highway-submodules-org, resource owner rodmena-limited, Contents read-only, created 2026-10-23, pasted into the vm-1 session transcript (director: machine is sealed, keep it). Its repo selection is 11 private repos (highway* plus jumper); CI needs only highway-engine and jumper, so narrow it. The first token (ci-highway-submodules, resource owner rodmena-ltd personal) never reached org repos and should be deleted. Handoff file deleted 2026-10-23 20:36Z after pipeline 3806 cloned both submodules with the stored secret. | |
| architecture | route_permission_guard | Every Flask/Quart route needs @require_permission (tenant) or @require_platform_permission (platform-wide data), placed BELOW @route/@spec.validate. Both decorators set the marker attribute highway_required_permission; tests/unit/api/test_route_guard_1032.py enumerates create_app().url_map and fails for any (endpoint, method) without the marker unless it is in its PUBLIC or SELF_SERVICE allowlist with a reason (#1032). Mixed-method handlers are split into one view per method (#1027, #1032). API keys skip role membership: require_permission checks only the key's permissions list, and a key with NULL permissions passes every tenant permission. Platform-wide rows (rc_circuit_breakers, heartbeat history, Prometheus counters) need platform permissions; tenant deletes carry a tenant_id predicate. | |
| guidelines | secrets | Platform secrets: one file per secret in /etc/highway/secrets/platform/<name> (host, 0600, uid 1000), directory-mounted read-only at /run/highway/secrets/platform/ in every engine container (raw value or flat JSON object). Config.get_secret(path, key): file, then SECRET_<NAME> env; file+env with different values = ConfigurationError; INI [secrets] is not a source; tenants/... paths are rejected. Other house keys stay single files under /run/highway/ (mail_api.env, tokengate_*.key, futex.env, reply_tag_secret). Tenant secrets: highway.tenant_secret_values, AES-256-GCM envelope s1:<key_id>:<nonce>:<ct> with AAD (tenant_id, path, key_id). Key ring = platform secret secret_store_keys {current, <key_id>: <b64key>:<kcv>} (key ids k+16 random hex, never reused; NOT HIGHWAY_ENCRYPTION_KEY v1, which is in git history). highway.tenant_secret_keys holds each key id's kcv + canary; /api/v1/health opens the canaries. Rotate with scripts/import_platform_secrets.py --init-store-key --new-ring (keeps old keys); never delete the ring file. Access via enterprise.services.secret_manager.get_secret_manager(tenant); pass conn= to commit with metadata. App refs: secret:<path>:<key>; tenant paths must belong to the installation tenant; platform refs only APP_PLATFORM_SECRETS. No Vault, no hvac (#924; deploy per SPECS/924-remove-vault.md). | |
| general | service update | Deploy with docker compose only: config-only changes via 'docker compose up -d --no-build --force-recreate <service>' one service at a time. Images are built ONLY in CI (director ruling, owner confirmed 2026-10-20); no docker build on vm-1. CI cannot build images yet (ci-conductor, thread 01M5BA73XDKE9V87N5JBRJ259X), so code deploys are held until it can. | |
| lessons | spawn_based_logging | Use spawn-based logging instead of injection-based (Nov 15, 2025). Spawn async Absurd tasks for logging after task completion. Crash-safe, idempotent via idempotency keys, no workflow interference. | |
| lessons | sql_identifier_prefix_quoting | PL/pgSQL format('%I') quotes ONLY the substituted text. format('jumper.r_%I', queue) yields jumper.r_"default" for reserved-word names, parsed as relation jumper.r_ aliased default. ALWAYS pass the complete identifier: format('jumper.%I', 'r_' || queue). Same rule for psycopg sql.Identifier - Identifier(f'r_{q}') is correct, 'r_' + Identifier(q) is not. Broke /api/v1/health/metrics for 18 days (#843); latent for months because the original queue names needed no quoting. | |
| lessons | sql_injection_prevention | All services must use psycopg.sql.Identifier() for dynamic table names to prevent SQL injection. | |
| general | submit DSL not JSON | Submit workflows in Python DSL format, not JSON (unless requested) | |
| deployment | systemd | Superseded (#917, 2026-10-17): Highway deploys with docker-compose only. Status: docker compose ps; logs: docker compose logs -f worker; restart: docker compose up -d --no-build --force-recreate <services>. No systemd units exist. | |
| apps | tenant_app_installation | Apps must be installed for specific tenant (test vs default) to be usable in workflows submitted with that tenant | |
| guidelines | testing | Integration tests only in tests/integration/. API-based calls only. No mocking. No inline workflow definitions - all must be in tests/workflow_examples/. Tests must pass with -n 4 parallel. No sequential tests. | |
| lessons | tests_conftest_writes_live_db | Both repos guard DB writes in tests (2026-10-20): superproject tests marked db_write and engine tests/unit/db skip unless HIGHWAY_DB_TESTS=1 and refuse highway_db_v2; both root conftests' pytest_sessionfinish cleanup runs only when writes are allowed. Do NOT use --noconftest: it disables the superproject db_write guard (the guard lives in tests/conftest.py). Run unit suites with --ignore=tests/unit/db and without HIGHWAY_DB_TESTS; DB tests belong on a scratch database or in CI. | |
| guidelines | tests_run_isolated | SUPERSEDED 2026-10-21: no test suites on vm-1 at all, isolated container or not (director's rule since 2026-10-06; infra-manager). Tests run in CI (ci.rodmena.co.uk). Only exception: one targeted test file with -n 1 to diagnose a live fault, announced on AgentBus to infra-manager first. scripts/run_tests_isolated.sh was removed. | |
| lessons | this_host_is_production | The docker-compose stack in /home/farshid/highway-stack/highway serves https://highway.rodmena.co.uk, but Highway has NO real users (user, 2026-10-15), so this stack is also the test bed; no separate staging stack (resources). Rules: only ONE agent rebuilds/restarts at a time (coordinated by the lead); after every rebuild verify the code inside every running engine container and GET+HEAD /api/v1/health over the public URL; never leave it crash-looping (dashboard/highway-mcp agents depend on it). 2026-10-15: unpinned rebuild floated spectree to 3.0.0 and the API crash-looped ~8 min -> lock deps (#920). | |
| lessons | threadpool_executor_danger | NEVER use with ThreadPoolExecutor(). Use try/finally with shutdown(wait=False, cancel_futures=True). Also: as_completed() MUST have timeout. | |
| architecture | tokengate_integration | Highway rate limits/quotas use TokenGate (#928). Tenant 'highway', subject = Highway tenant id, plan hw.plan.<tenant>. Tool limits = rate_limit-only policies on hw.tool.<tool>.<profile> (content-addressed, e.g. m10-h100-d500), one per window; quotas on separate resources (hw.llm.tokens) via reserve/renew/commit. Local table highway.tool_rate_limits stays the config of record; every writer must call engine.policies.tokengate.catalog.sync_tenant_rate_limits(conn, tenant) before commit (API routes, tenant_provisioning, 'hwe platform sync-tokengate'). Fail closed on any missing decision. Keys: /run/highway/tokengate_runtime.key (consume, require_assignment), /run/highway/tokengate_admin.key (catalog:write+assignments:write, prefix hw.); control key never on a request path. TokenGate refund never refills rate_limit buckets. #964: TG allows 24 in-flight per TG tenant shared by ALL Highway processes; each process caps itself via engine.policies.tokengate.inflight (HIGHWAY_TOKENGATE_MAX_IN_FLIGHT / [tokengate] max_in_flight, default 3). Count processes, not services: api runs hypercorn --workers 4 (4 processes), so set api to 1; sum of caps must stay <= 20. IMPLEMENTED 2026-10-20, NOT YET DEPLOYED: #952 workflow submissions = rate_limit policy hw.workflow.submissions.c<cap>-m<per-min> (default c100-m600) in the same plan, synced by sync_subject, consumed by api/middleware/rate_limiter.py off the event loop; denial 429 RATE_LIMIT_EXCEEDED, no decision 503 (fail closed), unassigned -> one lazy reconcile; run leases stay local in tenant_rate_limits. #953 limit_scope=workflow rejected 400 INVALID_SCOPE at PUT/enable; sync skips legacy workflow rows (limiter denies them). #954 LLM quota = per-tenant quota policy hw.llm.tokens.<tenant>.<day|month> (limit change = PATCH limit_amount, counters kept), admin API /api/v1/platform/tenants/<id>/llm-token-quota; with [tokengate] llm_token_quota on, a reserve with no hw.llm.tokens policy fails closed. #951 drop migration 01M5BBRGBK9ESDVDSVYR0QW35X held until >= 2026-10-24. | |
| guidelines | ui_components | ALL UI components must be in api/ui/templates/components/<component_name>/ with .html, .css, .js files. Use Mako <%text> blocks for CSS/JS. | |
| lessons | vault_credential_drift | A credential read from Vault by ONE code path and from env by all others WILL drift and the drift is invisible until that one path is exercised. #844: highway/shared/circuit_breaker_database held a password never applied to any role, so /api/v1/circuit-breakers 503d against a healthy DB. Rules: (1) provision + verify the credential at deploy time with the exact DSN the app will use, and fail the deploy if it cannot authenticate; (2) NEVER write 'or "postgres"' style default-password fallbacks - a missing secret then produces a WORKING endpoint and hides the misconfiguration; (3) every external dependency needs a health check proven to go red. | |
| spec | verification_account | Live checks that need platform rights run as highway-verify@rodmena.co.uk: role platform_admin in tenant _platform (not platform_owner), granted 2026-10-25 by the director's decision (replaces using the placeholder admin@highway.platform, which has no role and no owner). Mint a 1-hour HS256 JWT inside highway-api with claims sub/email=highway-verify@rodmena.co.uk, tenant_id=_platform; store it in a 0600 file, never print it. Remove with: hwe rbac remove-role --tenant _platform --email highway-verify@rodmena.co.uk --role platform_admin. | |
| general | verified_workflows | Every single Python DSL workflow that works well must be copied into api/dsl_templates/ folder | |
| general | zombie_worker_detection | If API shows unhealthy with pending tasks but workers show NOTIFY received without Claiming logs, workers are in zombie state. Root cause is usually asyncio/anyio corruption from mixed sync/async execution. Restart workers to fix immediately. See issues #452 and #453 for underlying bugs to fix. |