#163 Activity workers missing DataShard logging - no audit trail for long-running tasks
Description
Edit## Problem
Activity workers (highway-activity-worker@*.service) execute long-running tasks like Kafka producers/consumers but have NO logging to DataShard. This means:
- No audit trail for activity execution
- No way to trace what happened inside activities
- Cannot correlate activity logs with workflow execution
- Regulatory compliance gap for bank-grade requirements
## Current State
- Workflow workers have DataShard logging via `tools.datashard.log_task_execution`
- Activity workers create DurableContext but `event_logger=None` (see activity_worker.py:301)
- Activity results are only stored in activity_queue.result column
- Logs only available via systemd journal (`journalctl -u highway-activity-worker@*`)
## Expected Behavior
- Activity execution should log start/complete/fail events to DataShard
- Each activity should have immutable audit log entries
- The long running programs logs (for example logging.info/debug/error) must be capture into datashard and can be found.
- Logs should be queryable via DataShard time-travel queries
- Activity logs should be linkable to parent workflow run
## Files to Modify
- engine/services/activity_worker.py - Add DataShard logging in _execute_activity()
- Possibly create activity-specific DataShard log table
## Context
Discovered during Kafka ETL pipeline implementation (Issue #162). User asked 'how can I find producer and consumer full logs?' - answer was 'only in systemd journal' which is unacceptable for production.
## note:
- restart server with `make restart`, don't do it by pkill
- datashard's source code is in ../datashard/
- think about scalability, if audit log will be massive, we may need to partition it? what about other data
- example program is kafka etl pipeline which we produced in #162
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...