#517 [API/Memory] DataShard table unbounded memory loading

closed critical Created 2025-12-17 14:59 · Updated 2025-12-17 15:12

Description

Edit
**File:** api/blueprints/v1/logs.py:90-119 **Problem:** `_read_table_to_pandas()` loads ENTIRE DataShard tables into memory on every request. With continuous workflow execution, these tables grow indefinitely. No pagination at DataShard level. **Severity:** CRITICAL **Fix:** - Implement LRU cache with TTL for table snapshots - Add predicate pushdown to DataShard queries - Implement streaming/chunked reading - Add memory limits per-query **Impact:** Production systems with high workflow volume will exhaust memory.

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...