| #847 |
Encryption master key has an on-host env fallback next to the PostgreSQL data directory
EARS SPEC:
- The Highway platform shall source the JSONB encryption master key from Vault only; if V...
|
open |
high |
2026-08-03 20:47 |
- |
|
| #846 |
Replay never executes tools (#833), so while conditions driven by ctx.set_variable can never resolve in simulation
EARS SPEC:
- While executing in replay/simulation mode, when a workflow variable was set by a tool v...
|
open |
medium |
2026-08-03 18:55 |
- |
|
| #837 |
Stale/drifted tests: 2 unit mocks, switch_operator output assertions, docker_tools removal, wip_integration chronic reds
EARS SPEC:
- When the implementation changes observable behavior, the corresponding tests shall be u...
|
open |
medium |
2026-07-31 00:20 |
- |
|
| #823 |
Expression editor + variable picker for the visual workflow editor
Add n8n-style expression support to the visual editor's inspector fields. Highway interpolates {{ va...
|
open |
high |
2026-07-14 21:44 |
- |
|
| #821 |
Visual workflow editor rebuild: Svelte Flow + real tools + correct compiler + round-trip
The dashboard visual editor is useless: dsl-generator.convertNodeToTask only emits task+parallel and...
|
open |
high |
2026-07-14 00:40 |
- |
|
| #819 |
Dashboard UX review 2026-07-11: 62 verified findings (truth-telling bugs, dead controls, older-runs regression, mobile)
Comprehensive 11-dimension UX review of highway-dashboard (code + 17 desktop + 17 mobile screenshots...
|
open |
high |
2026-07-12 00:39 |
- |
|
| #818 |
Workflows list perf: dashboard fetches + client-groups ALL runs (9845); add grouped-summary endpoint (#agentic)
The dashboard workflows page fetches every run (days-windowed, ~9845 for a busy tenant) with no work...
|
open |
high |
2026-07-10 22:08 |
- |
|
| #778 |
Demo/research tenant Gmail token stale -> send fails ('Gmail token refresh failed'); offer SMTP fallback for agent email
run_goal research+email path works end-to-end (HITL approval + dispatch) but apps.platform.gmail.sen...
|
open |
low |
2026-06-29 01:53 |
- |
|
| #777 |
Dashboard UX: agent_leaf child runs show as 'leaf' and clutter the run list (should show run name + nest under parent)
Research campaigns spawn N child agent_leaf runs. Backend data is correct: each child workflow_run.w...
|
open |
medium |
2026-06-29 01:53 |
- |
|
| #775 |
Converge the two ActivityWorker implementations (enterprise vs engine package)
Live worker is enterprise/services/activity_worker.py (hwe activity-worker). A parallel refactored i...
|
open |
low |
2026-06-29 00:40 |
- |
|
| #770 |
Agentic Theme A1: MCP client (tools.mcp.call) to call external MCP servers as durable tools
Roadmap Theme A1 (breadth unlock). New engine tools tools.mcp.call + tools.mcp.list_tools (engine/to...
|
open |
high |
2026-06-22 16:11 |
- |
|
| #764 |
Phase B-2: Gmail real-time triggers via Cloud Pub/Sub — ingestion validated
Gmail users.watch -> Pub/Sub topic (projects/highway-481210/topics/Gmail+Highway) -> push to POST /a...
|
open |
high |
2026-06-22 00:06 |
- |
|
| #754 |
Drop functiongemma:270m as agent function model; use gemma4:31b-cloud (270M too weak for multi-step)
Empirical (demo tenant): functiongemma:270m emits malformed/empty tool args on compound prompts (run...
|
open |
high |
2026-06-21 00:17 |
- |
|
| #751 |
call_llm implements only Ollama; Anthropic/OpenAI raise NotImplementedError (blocks Claude)
engine/tools/llm/core/providers.py:87-112 _call_anthropic/_call_openai/etc raise NotImplementedError...
|
open |
high |
2026-06-20 22:48 |
- |
|
| #743 |
SMTP email timeout in Docker environment
Email sends from workers timeout. mail.highway.rodmena.app may not be reachable from Docker network....
|
open |
low |
2026-02-08 21:39 |
- |
|
| #736 |
Artifact and docker workflow tests failing post-modularization
test_artifact_system (2 tests) and test_docker_with_logging fail with workflow status=failed. Need t...
|
open |
medium |
2026-02-07 20:12 |
- |
|
| #735 |
Enterprise tools not available in unit test ToolRegistry context
Tests that instantiate ToolRegistry() directly get core-only tools. internal.health.check_db (test_h...
|
open |
medium |
2026-02-07 20:12 |
- |
|
| #733 |
Missing tools in test workflows: list_generator, long_running, shell.retry
Test workflow examples reference tools that dont exist: tools.list_generator.generate_items (foreach...
|
open |
high |
2026-02-07 20:11 |
- |
|
| #731 |
SQL injection in gc_strategies.py sync_container_state
gc_strategies.py line 219 uses f-string for worker_id in SQL query. Should use parameterized query. ...
|
open |
medium |
2026-02-07 03:13 |
- |
|
| #723 |
## Objective
Deploy Highway Workflow Engine on Kubernetes for production use.
## Current State
- Docker Compose: Working (local dev)
- Kubernetes: Not supported
## Required Changes
### 1. Secrets Management
**Current**: .env file with VAULT_TOKEN_ADMIN, Vault client reads tokens from env
**K8s Options**:
- Option A: Vault Agent sidecar (recommended) - K8s auth method, secrets injected as files
- Option B: K8s Secrets + External Secrets Operator - map K8s secrets to config paths
- Option C: Environment variable injection from K8s Secrets
**Code Changes Needed**:
- Support file-based secrets at /vault/secrets/* path
- Support HIGHWAY_* env var overrides for config values
- Graceful fallback chain: env vars → file secrets → Vault API
### 2. Storage
**Current**: Local filesystem via bind mounts
- /app/artifacts/ - workflow artifacts
- /app/highway-test-logs/ - datashard logs
- /app/highway-test-logs/uploads/ - file uploads
**K8s Options**:
- Option A: S3/MinIO (recommended for multi-replica) - re-enable S3StorageProvider with IAM auth
- Option B: PersistentVolumeClaim with ReadWriteMany (requires NFS/EFS)
- Option C: PVC per worker with node affinity (limits scaling)
**Code Changes Needed**:
- Re-enable S3 provider in s3_provider.py
- Add IAM/IRSA authentication for S3
- Config: storage_type = auto (detect S3 creds, fall back to local)
### 3. Docker-in-Docker Sandboxing
**Current**: Mounts /var/run/docker.sock for python_sandbox
**K8s Options**:
- Option A: Disable sandboxing (python_sandbox.mode = disabled) - acceptable for trusted tenants
- Option B: DinD sidecar container per worker pod
- Option C: Kaniko/Tekton for isolated execution
- Option D: gVisor/Kata for pod-level isolation
### 4. Configuration Delivery
**Current**: Bind-mounted config.ini from host
**K8s**: ConfigMap mounted as /etc/highway/config.ini
**Code Changes Needed**:
- Support HIGHWAY_DATABASE_HOST style env var overrides
- Environment variables take precedence over config.ini
### 5. Service Discovery
**Current**: Docker DNS (postgres, api, ollama, dsl-compiler)
**K8s**: K8s Service DNS - same pattern, just need Service manifests
### 6. Database
**Current**: Docker PostgreSQL container
**K8s Options**:
- Managed DB: RDS, CloudSQL, Azure Database (recommended)
- StatefulSet with PVC (self-managed)
Required PostgreSQL extensions: uuid-ossp, pgcrypto
Required databases: highway_db_v2, resilient_circuit_db
## Deliverables
### Helm Chart Structure
```
highway/
├── Chart.yaml
├── values.yaml
├── templates/
│ ├── configmap.yaml # config.ini
│ ├── secrets.yaml # JWT, DB password, encryption key
│ ├── deployment-api.yaml
│ ├── deployment-worker.yaml
│ ├── deployment-activity-worker.yaml
│ ├── deployment-internal-worker.yaml
│ ├── deployment-dsl-compiler.yaml
│ ├── deployment-ollama.yaml (optional)
│ ├── service-api.yaml
│ ├── service-dsl-compiler.yaml
│ ├── ingress.yaml
│ ├── pvc.yaml (if not using S3)
│ └── hpa.yaml (horizontal pod autoscaler)
```
### Services to Deploy
| Service | Type | Replicas | Notes |
|---------|------|----------|-------|
| api | Deployment | 1+ | Ingress, port 7822 |
| worker | Deployment | 2+ | HPA based on queue depth |
| activity-worker | Deployment | 1+ | |
| internal-worker | Deployment | 2 | Async logging |
| dsl-compiler | Deployment | 1+ | Isolated, no DB access |
| ollama | Deployment | 0-1 | Optional, GPU preferred |
| postgres | External/StatefulSet | 1 | Prefer managed DB |
### Health Checks (from docker-compose)
- API: GET /api/v1/health
- Workers: Jumper heartbeat mechanism
- DSL Compiler: GET /health
## Migration Path
1. Create Helm chart structure
2. Implement config env var overrides
3. Implement file-based secrets support
4. Re-enable S3 with IAM auth (or configure PVC)
5. Deploy to K8s cluster
6. Debug and iterate
7. Document deployment process
## Testing
- Deploy all services
- Run platform bootstrap workflow
- Run demo workflows (v2, disaster, matrix)
- Verify multi-replica worker scaling
- Test pod restart recovery
## References
- Issue #721: Local storage support (completed)
- Issue #722: JoinMode consistency (completed)
- docker-compose.yml: Current service definitions
- docker/config.ini: Configuration reference
|
open |
high |
2025-12-27 23:38 |
- |
|