#239 DataShard Logs: Pagination and Search API
Description
Edit## Overview
Add pagination and search capabilities to the DataShard logs API for handling large log volumes efficiently.
## Current State
- /workflows/<id>/steps/logs returns all logs without pagination
- No search capability exists
- Large workflows can have hundreds of thousands of log lines
## Requirements
### 1. Pagination for Logs API
- Add page and page_size query parameters
- Default page_size: 100, max: 1000
- Return pagination metadata (total_count, total_pages, has_next, has_prev)
### 2. Search API
Two new endpoints:
- /workflows/<id>/logs/search - Search all logs for an execution
- /workflows/<id>/steps/<step_name>/logs/search - Search logs for specific step
Search capabilities:
- Full-text search in stdout/stderr
- Filter by status (success/failed)
- Filter by time range
- Case-insensitive by default
### 3. OpenAPI Updates
- Document new pagination parameters
- Document search endpoints
## API Design
### Paginated Logs
GET /workflows/<id>/steps/logs?page=1&page_size=100
### Search
GET /workflows/<id>/logs/search?q=error&status=failed&page=1
## Acceptance Criteria
- Pagination works with page/page_size params
- Search returns matching logs
- Performance acceptable for 100k+ log entries
- OpenAPI documentation updated
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...