>_
.issue.db
/highway-workflow-engine
Dashboard
Issues
Memory
Lessons
Audit Log
New Issue
Edit Issue #163
Update issue details
Title *
Description
## Problem Activity workers (highway-activity-worker@*.service) execute long-running tasks like Kafka producers/consumers but have NO logging to DataShard. This means: - No audit trail for activity execution - No way to trace what happened inside activities - Cannot correlate activity logs with workflow execution - Regulatory compliance gap for bank-grade requirements ## Current State - Workflow workers have DataShard logging via `tools.datashard.log_task_execution` - Activity workers create DurableContext but `event_logger=None` (see activity_worker.py:301) - Activity results are only stored in activity_queue.result column - Logs only available via systemd journal (`journalctl -u highway-activity-worker@*`) ## Expected Behavior - Activity execution should log start/complete/fail events to DataShard - Each activity should have immutable audit log entries - The long running programs logs (for example logging.info/debug/error) must be capture into datashard and can be found. - Logs should be queryable via DataShard time-travel queries - Activity logs should be linkable to parent workflow run ## Files to Modify - engine/services/activity_worker.py - Add DataShard logging in _execute_activity() - Possibly create activity-specific DataShard log table ## Context Discovered during Kafka ETL pipeline implementation (Issue #162). User asked 'how can I find producer and consumer full logs?' - answer was 'only in systemd journal' which is unacceptable for production. ## note: - restart server with `make restart`, don't do it by pkill - datashard's source code is in ../datashard/ - think about scalability, if audit log will be massive, we may need to partition it? what about other data - example program is kafka etl pipeline which we produced in #162
Priority
Low
Medium
High
Critical
Status
Open
In Progress
Closed
Due Date (YYYY-MM-DD)
Tags (comma separated)
Related Issues (IDs)
Enter IDs of issues related to this one. They will be linked as 'related'.
Update Issue
Cancel