#239 DataShard Logs: Pagination and Search API

closed high Created 2025-12-03 03:03 · Updated 2025-12-03 03:06

Description

Edit
## Overview Add pagination and search capabilities to the DataShard logs API for handling large log volumes efficiently. ## Current State - /workflows/<id>/steps/logs returns all logs without pagination - No search capability exists - Large workflows can have hundreds of thousands of log lines ## Requirements ### 1. Pagination for Logs API - Add page and page_size query parameters - Default page_size: 100, max: 1000 - Return pagination metadata (total_count, total_pages, has_next, has_prev) ### 2. Search API Two new endpoints: - /workflows/<id>/logs/search - Search all logs for an execution - /workflows/<id>/steps/<step_name>/logs/search - Search logs for specific step Search capabilities: - Full-text search in stdout/stderr - Filter by status (success/failed) - Filter by time range - Case-insensitive by default ### 3. OpenAPI Updates - Document new pagination parameters - Document search endpoints ## API Design ### Paginated Logs GET /workflows/<id>/steps/logs?page=1&page_size=100 ### Search GET /workflows/<id>/logs/search?q=error&status=failed&page=1 ## Acceptance Criteria - Pagination works with page/page_size params - Search returns matching logs - Performance acceptable for 100k+ log entries - OpenAPI documentation updated

Comments

Loading comments...

Context

Loading context...

Audit History

View All
Loading audit history...