| #752 |
run_goal workflow variables are not agent-scoped; two agents in one workflow collide
tools.agent.run_goal/init_goal use fixed workflow-variable names (agent_done, agent_turn, agent_mess...
|
open |
medium |
2026-06-20 22:10 |
- |
|
| #751 |
call_llm implements only Ollama; Anthropic/OpenAI raise NotImplementedError (blocks Claude support)
engine/tools/llm/core/providers.py:87-112 — _call_openai/_call_anthropic/_call_grok/_call_gemini/_ca...
|
open |
high |
2026-06-20 22:10 |
- |
|
| #750 |
run_goal tool execution is at-least-once; needs scoped sub-step for exactly-once
tools.agent.run_goal (issue #749) gates each tool observation with a workflow variable (agent_tool_<...
|
open |
high |
2026-06-20 21:39 |
- |
|
| #743 |
SMTP email timeout in Docker environment
Email sends from workers timeout. mail.highway.rodmena.app may not be reachable from Docker network....
|
open |
low |
2026-02-08 21:39 |
- |
|
| #739 |
engine.config.get_secret does not read from Vault despite Vault access
get_secret() only checks SECRET_* env vars and [secrets] INI section. It never reads from Vault even...
|
open |
high |
2026-02-07 23:18 |
- |
|
| #738 |
ProxyDurableContext missing 'now' property causes isoformat error in IPC apps
ProxyDurableContext.__getattr__ catches all attribute access and returns a method_proxy function. Wh...
|
open |
high |
2026-02-07 23:14 |
- |
|
| #737 |
Ollama provider does not auto-resolve API key from env/config
In _route_to_provider (engine/tools/llm/core/providers.py), the ollama provider reads base_url from ...
|
open |
high |
2026-02-07 23:12 |
- |
|
| #736 |
Artifact and docker workflow tests failing post-modularization
test_artifact_system (2 tests) and test_docker_with_logging fail with workflow status=failed. Need t...
|
open |
medium |
2026-02-07 20:12 |
- |
|
| #735 |
Enterprise tools not available in unit test ToolRegistry context
Tests that instantiate ToolRegistry() directly get core-only tools. internal.health.check_db (test_h...
|
open |
medium |
2026-02-07 20:12 |
- |
|
| #734 |
App tools not registered in worker tool registry during execution
Apps installed via API (test_app_ipc_final, test_app_ipc_security) fail at runtime because app tools...
|
open |
high |
2026-02-07 20:12 |
- |
|
| #733 |
Missing tools in test workflows: list_generator, long_running, shell.retry
Test workflow examples reference tools that dont exist: tools.list_generator.generate_items (foreach...
|
open |
high |
2026-02-07 20:11 |
- |
|
| #731 |
SQL injection in gc_strategies.py sync_container_state
gc_strategies.py line 219 uses f-string for worker_id in SQL query. Should use parameterized query. ...
|
open |
medium |
2026-02-07 03:13 |
- |
|
| #723 |
## Objective
Deploy Highway Workflow Engine on Kubernetes for production use.
## Current State
- Docker Compose: Working (local dev)
- Kubernetes: Not supported
## Required Changes
### 1. Secrets Management
**Current**: .env file with VAULT_TOKEN_ADMIN, Vault client reads tokens from env
**K8s Options**:
- Option A: Vault Agent sidecar (recommended) - K8s auth method, secrets injected as files
- Option B: K8s Secrets + External Secrets Operator - map K8s secrets to config paths
- Option C: Environment variable injection from K8s Secrets
**Code Changes Needed**:
- Support file-based secrets at /vault/secrets/* path
- Support HIGHWAY_* env var overrides for config values
- Graceful fallback chain: env vars → file secrets → Vault API
### 2. Storage
**Current**: Local filesystem via bind mounts
- /app/artifacts/ - workflow artifacts
- /app/highway-test-logs/ - datashard logs
- /app/highway-test-logs/uploads/ - file uploads
**K8s Options**:
- Option A: S3/MinIO (recommended for multi-replica) - re-enable S3StorageProvider with IAM auth
- Option B: PersistentVolumeClaim with ReadWriteMany (requires NFS/EFS)
- Option C: PVC per worker with node affinity (limits scaling)
**Code Changes Needed**:
- Re-enable S3 provider in s3_provider.py
- Add IAM/IRSA authentication for S3
- Config: storage_type = auto (detect S3 creds, fall back to local)
### 3. Docker-in-Docker Sandboxing
**Current**: Mounts /var/run/docker.sock for python_sandbox
**K8s Options**:
- Option A: Disable sandboxing (python_sandbox.mode = disabled) - acceptable for trusted tenants
- Option B: DinD sidecar container per worker pod
- Option C: Kaniko/Tekton for isolated execution
- Option D: gVisor/Kata for pod-level isolation
### 4. Configuration Delivery
**Current**: Bind-mounted config.ini from host
**K8s**: ConfigMap mounted as /etc/highway/config.ini
**Code Changes Needed**:
- Support HIGHWAY_DATABASE_HOST style env var overrides
- Environment variables take precedence over config.ini
### 5. Service Discovery
**Current**: Docker DNS (postgres, api, ollama, dsl-compiler)
**K8s**: K8s Service DNS - same pattern, just need Service manifests
### 6. Database
**Current**: Docker PostgreSQL container
**K8s Options**:
- Managed DB: RDS, CloudSQL, Azure Database (recommended)
- StatefulSet with PVC (self-managed)
Required PostgreSQL extensions: uuid-ossp, pgcrypto
Required databases: highway_db_v2, resilient_circuit_db
## Deliverables
### Helm Chart Structure
```
highway/
├── Chart.yaml
├── values.yaml
├── templates/
│ ├── configmap.yaml # config.ini
│ ├── secrets.yaml # JWT, DB password, encryption key
│ ├── deployment-api.yaml
│ ├── deployment-worker.yaml
│ ├── deployment-activity-worker.yaml
│ ├── deployment-internal-worker.yaml
│ ├── deployment-dsl-compiler.yaml
│ ├── deployment-ollama.yaml (optional)
│ ├── service-api.yaml
│ ├── service-dsl-compiler.yaml
│ ├── ingress.yaml
│ ├── pvc.yaml (if not using S3)
│ └── hpa.yaml (horizontal pod autoscaler)
```
### Services to Deploy
| Service | Type | Replicas | Notes |
|---------|------|----------|-------|
| api | Deployment | 1+ | Ingress, port 7822 |
| worker | Deployment | 2+ | HPA based on queue depth |
| activity-worker | Deployment | 1+ | |
| internal-worker | Deployment | 2 | Async logging |
| dsl-compiler | Deployment | 1+ | Isolated, no DB access |
| ollama | Deployment | 0-1 | Optional, GPU preferred |
| postgres | External/StatefulSet | 1 | Prefer managed DB |
### Health Checks (from docker-compose)
- API: GET /api/v1/health
- Workers: Jumper heartbeat mechanism
- DSL Compiler: GET /health
## Migration Path
1. Create Helm chart structure
2. Implement config env var overrides
3. Implement file-based secrets support
4. Re-enable S3 with IAM auth (or configure PVC)
5. Deploy to K8s cluster
6. Debug and iterate
7. Document deployment process
## Testing
- Deploy all services
- Run platform bootstrap workflow
- Run demo workflows (v2, disaster, matrix)
- Verify multi-replica worker scaling
- Test pod restart recovery
## References
- Issue #721: Local storage support (completed)
- Issue #722: JoinMode consistency (completed)
- docker-compose.yml: Current service definitions
- docker/config.ini: Configuration reference
|
open |
high |
2025-12-27 23:38 |
- |
|
| #712 |
Agentic Bot Demo - Conference Presentation
## Overview
Implement a full agentic conversational bot demo for conference presentation, showcasin...
|
open |
high |
2025-12-25 18:32 |
- |
|
| #709 |
TEST-01: Missing unit tests for core orchestrator components
Location: tests/unit/. Issue: No unit tests for orchestrator.py, durable_context.py, absurd_client.p...
|
open |
high |
2025-12-25 02:56 |
- |
|
| #703 |
SCALE-02: No memory limits per task - crash propagation risk
Location: Not implemented. Issue: Worker-level memory limit only. Single memory-hungry task can cras...
|
open |
high |
2025-12-25 02:56 |
- |
|
| #702 |
SCALE-01: Single queue table bottleneck under high load
Location: absurd_client.py. Issue: All workers contend on single queue table. FOR UPDATE SKIP LOCKED...
|
open |
high |
2025-12-25 02:56 |
- |
|
| #699 |
DB-02: Destructive data deletion without archival
Location: migration 0.0.38. Issue: DELETE FROM workflow_run WHERE definition_id IS NULL without arch...
|
open |
high |
2025-12-25 02:56 |
- |
|
| #698 |
DB-01: Missing DOWN migrations for rollback
Location: Multiple migration files. Issue: Many migrations have TODO for down migration. Cannot roll...
|
open |
high |
2025-12-25 02:56 |
- |
|
| #671 |
Add OrphanDetectionService for stuck workflows
Need a background service that periodically scans for orphaned workflows where:
- workflow_run.statu...
|
open |
high |
2025-12-24 03:30 |
- |
|