| #751 |
call_llm implements only Ollama; Anthropic/OpenAI raise NotImplementedError (blocks Claude support)
engine/tools/llm/core/providers.py:87-112 — _call_openai/_call_anthropic/_call_grok/_call_gemini/_ca...
|
open |
high |
2026-06-20 22:10 |
- |
|
| #750 |
run_goal tool execution is at-least-once; needs scoped sub-step for exactly-once
tools.agent.run_goal (issue #749) gates each tool observation with a workflow variable (agent_tool_<...
|
open |
high |
2026-06-20 21:39 |
- |
|
| #749 |
Phase 0: MCP front-door + durable run_goal agent tool
Highway Agents Phase 0 (option B, durable). Two parts:
1) Engine: new tool tools.agent.run_goal — a ...
|
in-progress |
high |
2026-06-20 21:26 |
- |
|
| #745 |
Cross-run event pollution causes deadlock in parallel workflows
Static event names in DSL templates persist in jumper.e_highway_default between runs. When a subsequ...
|
closed |
high |
2026-02-08 23:38 |
- |
|
| #744 |
Bootstrap only registers pre-installed apps, not example apps
The register_apps internal tool only scans enterprise/apps/pre_installed/ (8 apps). The 9 example ap...
|
closed |
high |
2026-02-08 21:47 |
- |
|
| #740 |
Enterprise CLI submit broken after modularization
hwe submit fails with TypeError: submit_workflow() got unexpected keyword argument queue. Signature ...
|
closed |
high |
2026-02-08 20:59 |
- |
|
| #739 |
engine.config.get_secret does not read from Vault despite Vault access
get_secret() only checks SECRET_* env vars and [secrets] INI section. It never reads from Vault even...
|
open |
high |
2026-02-07 23:18 |
- |
|
| #738 |
ProxyDurableContext missing 'now' property causes isoformat error in IPC apps
ProxyDurableContext.__getattr__ catches all attribute access and returns a method_proxy function. Wh...
|
open |
high |
2026-02-07 23:14 |
- |
|
| #737 |
Ollama provider does not auto-resolve API key from env/config
In _route_to_provider (engine/tools/llm/core/providers.py), the ollama provider reads base_url from ...
|
open |
high |
2026-02-07 23:12 |
- |
|
| #734 |
App tools not registered in worker tool registry during execution
Apps installed via API (test_app_ipc_final, test_app_ipc_security) fail at runtime because app tools...
|
open |
high |
2026-02-07 20:12 |
- |
|
| #733 |
Missing tools in test workflows: list_generator, long_running, shell.retry
Test workflow examples reference tools that dont exist: tools.list_generator.generate_items (foreach...
|
open |
high |
2026-02-07 20:11 |
- |
|
| #726 |
On-failure callback double execution risk
In _handle_failure_exception(), task is marked as executed AFTER callback runs. If callback raises, ...
|
closed |
high |
2026-02-07 02:48 |
- |
|
| #725 |
State machine guard on workflow_run status updates
update_workflow_run_status() updates status without checking current state. Two concurrent writers c...
|
closed |
high |
2026-02-07 02:48 |
- |
|
| #724 |
## Problem
Two issues with worker signal handling prevent proper graceful shutdown:
### Issue 1: Delayed Shutdown Response
- Signal handler sets `_shutdown_requested = True`
- But `wakeup_event` is a local variable in `worker_command()`
- Signal handler can't access it to wake up blocked `Event.wait()`
- Worker waits up to `poll_interval` (1-5s) before noticing SIGTERM
### Issue 2: In-Flight Tasks Abandoned
- After main loop breaks, `finally` block stops services immediately
- No wait for tasks in `orchestrator.bulkhead` to complete
- Tasks get killed mid-execution
- They stay claimed until heartbeat timeout, then re-execute from scratch
## Impact
- K8s rolling updates will kill tasks mid-execution
- Wasted compute from task re-execution
- Potential data corruption for non-idempotent tasks
## Solution
1. Make `wakeup_event` module-level so signal handler can access it
2. Add graceful drain logic like activity_worker has:
- Wait up to 30s for in-flight tasks
- Call `bulkhead.shutdown(wait=True)`
## Files
- engine/cli/worker.py
## Testing
- Send SIGTERM to worker with active tasks
- Verify tasks complete before worker exits
- Verify shutdown happens within seconds, not waiting for poll_interval
|
closed |
high |
2025-12-28 00:21 |
- |
|
| #723 |
## Objective
Deploy Highway Workflow Engine on Kubernetes for production use.
## Current State
- Docker Compose: Working (local dev)
- Kubernetes: Not supported
## Required Changes
### 1. Secrets Management
**Current**: .env file with VAULT_TOKEN_ADMIN, Vault client reads tokens from env
**K8s Options**:
- Option A: Vault Agent sidecar (recommended) - K8s auth method, secrets injected as files
- Option B: K8s Secrets + External Secrets Operator - map K8s secrets to config paths
- Option C: Environment variable injection from K8s Secrets
**Code Changes Needed**:
- Support file-based secrets at /vault/secrets/* path
- Support HIGHWAY_* env var overrides for config values
- Graceful fallback chain: env vars → file secrets → Vault API
### 2. Storage
**Current**: Local filesystem via bind mounts
- /app/artifacts/ - workflow artifacts
- /app/highway-test-logs/ - datashard logs
- /app/highway-test-logs/uploads/ - file uploads
**K8s Options**:
- Option A: S3/MinIO (recommended for multi-replica) - re-enable S3StorageProvider with IAM auth
- Option B: PersistentVolumeClaim with ReadWriteMany (requires NFS/EFS)
- Option C: PVC per worker with node affinity (limits scaling)
**Code Changes Needed**:
- Re-enable S3 provider in s3_provider.py
- Add IAM/IRSA authentication for S3
- Config: storage_type = auto (detect S3 creds, fall back to local)
### 3. Docker-in-Docker Sandboxing
**Current**: Mounts /var/run/docker.sock for python_sandbox
**K8s Options**:
- Option A: Disable sandboxing (python_sandbox.mode = disabled) - acceptable for trusted tenants
- Option B: DinD sidecar container per worker pod
- Option C: Kaniko/Tekton for isolated execution
- Option D: gVisor/Kata for pod-level isolation
### 4. Configuration Delivery
**Current**: Bind-mounted config.ini from host
**K8s**: ConfigMap mounted as /etc/highway/config.ini
**Code Changes Needed**:
- Support HIGHWAY_DATABASE_HOST style env var overrides
- Environment variables take precedence over config.ini
### 5. Service Discovery
**Current**: Docker DNS (postgres, api, ollama, dsl-compiler)
**K8s**: K8s Service DNS - same pattern, just need Service manifests
### 6. Database
**Current**: Docker PostgreSQL container
**K8s Options**:
- Managed DB: RDS, CloudSQL, Azure Database (recommended)
- StatefulSet with PVC (self-managed)
Required PostgreSQL extensions: uuid-ossp, pgcrypto
Required databases: highway_db_v2, resilient_circuit_db
## Deliverables
### Helm Chart Structure
```
highway/
├── Chart.yaml
├── values.yaml
├── templates/
│ ├── configmap.yaml # config.ini
│ ├── secrets.yaml # JWT, DB password, encryption key
│ ├── deployment-api.yaml
│ ├── deployment-worker.yaml
│ ├── deployment-activity-worker.yaml
│ ├── deployment-internal-worker.yaml
│ ├── deployment-dsl-compiler.yaml
│ ├── deployment-ollama.yaml (optional)
│ ├── service-api.yaml
│ ├── service-dsl-compiler.yaml
│ ├── ingress.yaml
│ ├── pvc.yaml (if not using S3)
│ └── hpa.yaml (horizontal pod autoscaler)
```
### Services to Deploy
| Service | Type | Replicas | Notes |
|---------|------|----------|-------|
| api | Deployment | 1+ | Ingress, port 7822 |
| worker | Deployment | 2+ | HPA based on queue depth |
| activity-worker | Deployment | 1+ | |
| internal-worker | Deployment | 2 | Async logging |
| dsl-compiler | Deployment | 1+ | Isolated, no DB access |
| ollama | Deployment | 0-1 | Optional, GPU preferred |
| postgres | External/StatefulSet | 1 | Prefer managed DB |
### Health Checks (from docker-compose)
- API: GET /api/v1/health
- Workers: Jumper heartbeat mechanism
- DSL Compiler: GET /health
## Migration Path
1. Create Helm chart structure
2. Implement config env var overrides
3. Implement file-based secrets support
4. Re-enable S3 with IAM auth (or configure PVC)
5. Deploy to K8s cluster
6. Debug and iterate
7. Document deployment process
## Testing
- Deploy all services
- Run platform bootstrap workflow
- Run demo workflows (v2, disaster, matrix)
- Verify multi-replica worker scaling
- Test pod restart recovery
## References
- Issue #721: Local storage support (completed)
- Issue #722: JoinMode consistency (completed)
- docker-compose.yml: Current service definitions
- docker/config.ini: Configuration reference
|
open |
high |
2025-12-27 23:38 |
- |
|
| #722 |
Refactor wait_for_parallel_branches to use JoinMode for consistency
Currently wait_for_parallel_branches uses ad-hoc fail_on_error=True boolean while JoinOperator has p...
|
closed |
high |
2025-12-27 21:09 |
- |
|
| #720 |
Sandbox temp files fail in Docker-in-Docker: can't find __main__ module
FIXED: When running sandboxed code execution (tools.code.exec) inside Docker containers with Docker ...
|
closed |
high |
2025-12-27 03:27 |
- |
|
| #718 |
Branch tasks now inherit retry policy from ParallelOperator
Fixed branch execution tasks to inherit max_attempts and retry_strategy from the ParallelOperator's ...
|
closed |
high |
2025-12-26 23:58 |
- |
|
| #717 |
Branch tasks now retry on worker crash
Fixed branch execution tasks to have max_attempts=3 with exponential backoff (10s, 30s). Previously ...
|
closed |
high |
2025-12-26 23:53 |
- |
|
| #716 |
Skip propagation bug in switch/condition operators
When switch/condition marks unselected branches as executed, downstream tasks depending on skipped b...
|
closed |
high |
2025-12-26 16:17 |
- |
|