File size: 25,353 Bytes
42da440 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 | # Augmentoolkit API Documentation
This document describes the REST API endpoints for interacting with the Augmentoolkit server, allowing you to manage and run data generation pipelines, monitor their status, and manage associated files (inputs, outputs, logs, configs).
The API is built using FastAPI and utilizes Huey for background task queueing and Redis for state management (task status, PID mapping, output directory mapping).
**Base URL:** (Assuming the server runs locally on port 8000) `http://127.0.0.1:8000`
---
## Core Concepts
* **Pipelines:** Defined sequences of data processing steps, typically corresponding to Python functions within the codebase (e.g., in a `pipelines` directory).
* **Tasks:** An instance of a pipeline run, managed by the Huey background task queue. Each task has a unique `task_id`.
* **Nodes:** Refers to the importable path of the Python function that executes a pipeline (e.g., `pipelines.my_pipeline.run`).
* **Configs:** YAML configuration files (stored in `./external_configs/`) that provide parameters for pipeline runs. Can be overridden by parameters in the API request.
* **Super Config (`super_config.yaml`):** Defines path aliases used for resolving node and config paths.
* **Directories:**
* `./inputs/`: Stores input data files used by pipelines.
* `./outputs/`: Stores output files generated by pipelines. Output directories are often named using the `task_id`.
* `./external_configs/`: Stores reusable pipeline configuration files.
* `./logs/`: Stores log files (`<task_id>.log`) for each pipeline task run.
---
## Endpoints
### Health Check
* **GET /**
* **Summary:** Health Check
* **Description:** Basic health check endpoint. Returns a simple message indicating the API is running.
* **Response:** `{"message": "Augmentoolkit API is running."}`
---
### Pipeline Execution
* **POST /pipelines/run**
* **Summary:** Queue a dataset generation pipeline for execution.
* **Status Code:** 202 Accepted
* **Description:** Resolves node and config paths using aliases from `super_config.yaml` and enqueues a pipeline task using Huey (`tasks.run_pipeline_task`). The task executes the pipeline defined by `node_path` potentially using configuration from `config_path` and `parameters` via `run_augmentoolkit.py`.
* **Request Body:**
```json
{
"node_path": "string (required, can use aliases)",
"config_path": "string (optional, relative to ./external_configs/ or absolute, can use aliases)",
"parameters": { /* JSON object of key-value pairs (optional) */ }
}
```
* **Response:**
```json
{
"pipeline_id": "string (Huey Task ID)",
"message": "Pipeline run queued successfully."
}
```
* **GET /pipelines/available**
* **Summary:** Get available pipeline aliases from super_config.yaml.
* **Description:** Reads the `super_config.yaml` file and returns a sorted list of path aliases (keys) whose corresponding values do not end with `.yaml`. These aliases typically represent the runnable entry points for pipelines that can be passed to the `node_path` field in the `/pipelines/run` endpoint.
* **Path Parameters:** None
* **Query Parameters:** None
* **Response:** A JSON list of strings.
```json
[
"alias1",
"alias2",
"another-pipeline-alias",
...
]
```
* **Error Responses:**
* 500 Internal Server Error: If `super_config.yaml` cannot be found or parsed, or an unexpected error occurs. (Note: If the file is found but empty or `path_aliases` is missing/invalid, it currently returns an empty list `[]` gracefully).
---
### Task Status & Management
* **GET /tasks/{task_id}/status**
* **Summary:** Get the status of a pipeline run.
* **Description:** Retrieves the current status (PENDING, RUNNING, COMPLETED, FAILED, REVOKED) and result (if finished) of a specific pipeline task using its `task_id`. Checks Huey's state and potentially Redis for progress information.
* **Path Parameters:**
* `task_id`: The Huey task ID obtained from `/pipelines/run`.
* **Response:**
```json
{
"task_id": "string",
"status": "string (PENDING | RUNNING | COMPLETED | FAILED | REVOKED)",
"message": "string (optional)",
"progress": "float (optional, 0.0-1.0)",
"details": { /* Optional details, e.g., error info or result */ }
}
```
* **Error Responses:**
* 404 Not Found: If `task_id` does not exist.
* 500 Internal Server Error: If status retrieval fails.
* **GET /tasks/{task_id}/parameters**
* **Summary:** Get the parameters a task was executed with.
* **Description:** Retrieves the parameters (combined from config file and request overrides) that were used to start a specific task run. Parameters are stored in Redis when the task begins execution and persist based on a configured timeout (typically several days).
* **Path Parameters:**
* `task_id`: The Huey task ID obtained from `/pipelines/run`.
* **Response:**
```json
{
"task_id": "string",
"parameters": { /* JSON object of key-value pairs used for the task */ }
}
```
* **Error Responses:**
* 404 Not Found: If the `task_id` does not exist or if the parameters were not stored/have expired.
* 500 Internal Server Error: If retrieving or parsing the parameters from Redis fails (e.g., data corruption).
* **POST /tasks/{task_id}/interrupt**
* **Summary:** Interrupt a running pipeline task subprocess or revoke a pending task.
* **Description:** Attempts to interrupt a running task or revoke a pending one.
* If the task is **RUNNING**: Looks up the subprocess PID associated with the `task_id` in Redis (set by `tasks.py`) and sends a `SIGINT` signal to it using `os.kill`.
* If the task is **PENDING**: Revokes the task using `huey.revoke`.
* Reports an error if the task is already finished or revoked.
* **Path Parameters:**
* `task_id`: The Huey task ID.
* **Response:**
* `{"message": "Sent SIGINT signal..."}` (for running tasks)
* `{"message": "Task ... was pending and has been successfully revoked."}` (for pending tasks)
* **Error Responses:**
* 404 Not Found: If `task_id` does not exist or the subprocess PID is gone.
* 409 Conflict: If the task is already finished/revoked or fails to revoke.
* 500 Internal Server Error: If signaling fails or other errors occur.
* **GET /tasks/queue**
* **Summary:** Get lists of pending and scheduled tasks.
* **Description:** Retrieves the IDs of tasks currently pending or scheduled in the Huey queue using `huey.pending()` and `huey.scheduled()`. Note: This does not reliably show tasks that are *actively running*.
* **Path Parameters:** None
* **Query Parameters:** None
* **Response:**
```json
{
"pending_tasks": ["string (task_id)", ...],
"scheduled_tasks": ["string (task_id)", ...],
"message": "string (e.g., Found X pending and Y scheduled tasks.)"
}
```
* **Error Responses:**
* 500 Internal Server Error: If retrieving queue status fails.
---
### Log Management
* **GET /tasks/{task_id}/logs**
* **Summary:** Get logs for a specific task.
* **Description:** Retrieves the content of the log file (`./logs/<task_id>.log`) generated by a specific task run.
* **Path Parameters:**
* `task_id`: The Huey task ID.
* **Query Parameters:**
* `tail` (int, optional): Return only the last N lines of the log.
* **Response:**
```json
{
"task_id": "string",
"message": "string",
"logs": "string (log content)"
}
```
* **Error Responses:**
* 404 Not Found: If the log file doesn't exist.
* 500 Internal Server Error: If reading the file fails.
* **DELETE /tasks/{task_id}/logs**
* **Summary:** Delete log file for a specific task.
* **Description:** Deletes the specific log file (`./logs/<task_id>.log`).
* **Path Parameters:**
* `task_id`: The Huey task ID.
* **Response:** `{"message": "Successfully deleted log file..."}`
* **Error Responses:**
* 404 Not Found: If the log file doesn't exist.
* 500 Internal Server Error: If deletion fails.
* **GET /logs**
* **Summary:** List all available log files.
* **Description:** Returns a list of all `.log` filenames currently in the `./logs/` directory.
* **Response:** `{"log_files": ["task_id1.log", "task_id2.log", ...]}`
* **Error Responses:**
* 500 Internal Server Error: If listing the directory fails.
* **DELETE /logs**
* **Summary:** Clear all task log files.
* **Description:** Deletes all files ending with `.log` within the `./logs/` directory.
* **Response:** Reports the number of deleted files and any errors encountered.
```json
{
"message": "Successfully deleted X log file(s)...",
"errors": [ /* list of error strings, if any */ ]
}
```
* **Error Responses:**
* 500 Internal Server Error: If accessing or clearing the directory fails.
---
### Output File Management
* **GET /tasks/{task_id}/outputs/download**
* **Summary:** Download output directory for a task.
* **Description:** Retrieves the path of the output directory associated with the `task_id` from Redis (set by `tasks.py`), zips the entire directory content, and returns it as a downloadable file (`task_<task_id>_output.zip`).
* **Path Parameters:**
* `task_id`: The Huey task ID.
* **Response:** A zip file (`application/zip`).
* **Error Responses:**
* 404 Not Found: If the task ID mapping or the directory itself doesn't exist.
* 500 Internal Server Error: If zipping or file access fails.
* **GET /tasks/{task_id}/outputs/structure**
* **Summary:** Get structure of a task's output directory.
* **Description:** Retrieves the path of the output directory associated with the `task_id` from Redis and returns its file/folder structure as a JSON list. Paths are relative to the specific task's output directory root.
* **Path Parameters:**
* `task_id`: The Huey task ID.
* **Response:** List of `FileStructure` objects (see below).
* **Error Responses:**
* 404 Not Found: If the task ID mapping or the directory itself doesn't exist.
* 500 Internal Server Error: If reading the directory fails.
* **GET /outputs/structure/{relative_path:path}**
* **Summary:** Get structure of a path within the main outputs directory.
* **Description:** Retrieves the file/folder structure for a given path relative to the root `./outputs/` directory. Uses the shared `handle_get_structure` helper.
* **Path Parameters:**
* `relative_path`: The path relative to `./outputs/` (e.g., `my_folder/sub_folder` or leave empty/`.` for the root).
* **Response:** List of `FileStructure` objects:
```json
[
{
"path": "string (relative path)",
"is_dir": true,
"children": [ /* nested FileStructure objects */ ]
},
{
"path": "string (relative path)",
"is_dir": false,
"children": null
}
]
```
* **Error Responses:**
* 404 Not Found: If the path doesn't exist.
* 400 Bad Request: If the path is not a directory.
* 500 Internal Server Error: If listing fails.
* **GET /outputs/download/{relative_path:path}**
* **Summary:** Download a specific file or folder from outputs.
* **Description:** Downloads a specific file or zips and downloads a specific folder relative to the root `./outputs/` directory. Uses the shared `handle_download_item` helper.
* **Path Parameters:**
* `relative_path`: The path relative to `./outputs/`.
* **Response:** A file (`application/octet-stream` or detected type) or a zip archive (`application/zip`).
* **Error Responses:**
* 404 Not Found: If the path doesn't exist.
* 500 Internal Server Error: If reading/zipping fails.
* **DELETE /outputs/{relative_path:path}**
* **Summary:** Delete a file or folder from outputs.
* **Description:** Deletes a specific file or folder relative to the root `./outputs/` directory. Uses the shared `handle_delete_item` helper. **Warning:** This is permanent.
* **Path Parameters:**
* `relative_path`: The path relative to `./outputs/`.
* **Response:** `{"message": "Successfully deleted..."}`
* **Error Responses:**
* 404 Not Found: If the path doesn't exist.
* 400 Bad Request: If attempting to delete the root `./outputs` directory.
* 500 Internal Server Error: If deletion fails.
* **POST /outputs/move**
* **Summary:** Move/Rename a file or folder within outputs.
* **Description:** Moves or renames a file or folder within the `./outputs/` directory using `shutil.move`.
* The source path must exist.
* The destination path must *not* already exist (to prevent overwrites).
* The parent directory of the destination path must exist.
* Cannot be used to move the root `./outputs` directory itself.
* **Request Body:**
```json
{
"source_relative_path": "string (required, current relative path)",
"destination_relative_path": "string (required, new relative path)"
}
```
* **Response (Status 200):**
```json
{
"message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
```
* **Error Responses:**
* 400 Bad Request: If source/destination paths are invalid, if the destination parent doesn't exist, or if attempting to move the root directory.
* 404 Not Found: If the source path does not exist.
* 409 Conflict: If the destination path already exists.
* 500 Internal Server Error: If the move operation fails for other reasons (e.g., permissions).
---
### Input File Management
*Input file management endpoints mirror the Output File Management endpoints, but operate on the `./inputs/` directory.* Also, it has a route to create a directory, too.
* **GET /inputs/structure/{relative_path:path}**
* **Summary:** Get structure of a path within the inputs directory.
* **Description:** Retrieves structure relative to `./inputs/`.
* **Response:** List of `FileStructure` objects.
* **GET /inputs/download/{relative_path:path}**
* **Summary:** Download a specific file or folder from inputs.
* **Description:** Downloads file/folder relative to `./inputs/`.
* **Response:** File or Zip archive.
* **DELETE /inputs/{relative_path:path}**
* **Summary:** Delete a file or folder from inputs.
* **Description:** Deletes file/folder relative to `./inputs/`. **Warning:** Permanent.
* **Response:** `{"message": "Successfully deleted..."}`
* **POST /inputs/move**
* **Summary:** Move/Rename a file or folder within inputs.
* **Description:** Moves or renames a file or folder within the `./inputs/` directory using `shutil.move`. See `/outputs/move` for detailed behavior and error conditions.
* **Request Body:**
```json
{
"source_relative_path": "string (required, current relative path)",
"destination_relative_path": "string (required, new relative path)"
}
```
* **Response (Status 200):**
```json
{
"message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
```
* **Error Responses:** 400, 404, 409, 500 (see `/outputs/move`)
* **POST /inputs/directory**
* **Summary:** Create a new directory within the inputs folder.
* **Description:** Creates a new directory at the specified `relative_path` within the `./inputs/` directory. The path can include parent directories (e.g., `existing_folder/new_subfolder`).
* **Request Body:**
```json
{
"relative_path": "string (required, relative path)"
}
```
* **Response (Status 201):**
```json
{
"message": "Successfully created directory '{relative_path}' in inputs."
}
```
* **Error Responses:**
* 400 Bad Request: If `relative_path` is invalid (empty, '.', targets root) or if the parent directory specified in the path does not exist.
* 409 Conflict: If a file or directory already exists at the target `relative_path`.
* 500 Internal Server Error: If directory creation fails for other reasons (e.g., permissions).
* **POST /inputs/upload/{relative_path:path}**
* **Summary:** Upload files to a specific path within inputs, automatically extracting zip files.
* **Description:** Uploads one or more files to a specified path within `./inputs/`. The path is created if it doesn't exist.
* If a file ends with `.zip` (case-insensitive), it is automatically extracted into the target `relative_path`.
* Upon successful extraction, the original `.zip` file is deleted.
* If extraction fails, the original `.zip` file is kept, and an error is reported.
* Non-zip files are saved directly to the `relative_path`.
* Allows uploading directly to the root `./inputs/` directory (e.g., by specifying `.` as the `relative_path`).
* **Path Parameters:**
* `relative_path`: Target path relative to `./inputs/`. Files (including extracted zip contents) will be placed here.
* **Request Body:** `multipart/form-data` containing one or more files.
* **Response (Status 201 or 207):**
```json
{
"message": "Upload process completed...",
"uploaded_files": [
{"filename": "string", "size": int, "path": "string (relative path)", "error": "string (optional, only on failed extraction)"}
],
"extracted_zips": [
{"filename": "string (original zip)", "extracted_to": "string (relative path)"}
],
"errors": [ /* list of general error strings, if any */ ]
}
```
* **Error Responses:**
* 500 Internal Server Error: If directory creation, file writing, or extraction fails unexpectedly.
---
### Config File Management
*Configuration file management endpoints operate on the `./external_configs/` directory.*
* **GET /configs/aliases**
* **Summary:** Get available config file aliases from super_config.yaml.
* **Description:** Reads the `super_config.yaml` file and returns a sorted list of path aliases (keys) whose corresponding values *do* end with `.yaml` (case-insensitive). These aliases can be used to identify existing configuration files for duplication or other purposes.
* **Path Parameters:** None
* **Query Parameters:** None
* **Response:** A JSON list of strings.
```json
[
"config-alias1",
"another_config.yaml",
"path/to/config_alias_3",
...
]
```
* **Error Responses:**
* 500 Internal Server Error: If `super_config.yaml` cannot be found or parsed, or an unexpected error occurs. (Note: If the file is found but empty or `path_aliases` is missing/invalid, it returns an empty list `[]` gracefully).
* **GET /configs/structure/{relative_path:path}**
* **Summary:** Get structure of a path within the configs directory.
* **Description:** Retrieves structure relative to `./external_configs/`.
* **Response:** List of `FileStructure` objects.
* **GET /configs/content/{relative_path:path}**
* **Summary:** Get content of a specific config file.
* **Description:** Retrieves the plain text content of a specific file relative to `./external_configs/`.
* **Path Parameters:**
* `relative_path`: The path relative to `./external_configs/`.
* **Response:** Plain text file content (`text/plain`).
* **Error Responses:**
* 404 Not Found: If the file doesn't exist.
* 400 Bad Request: If the path is not a file.
* **POST /configs/content/{relative_path:path}**
* **Summary:** Create or update a config file.
* **Description:** Creates a new file or overwrites an existing file at the specified path relative to `./external_configs/`. Parent directories are created if they don't exist.
* **Path Parameters:**
* `relative_path`: The path relative to `./external_configs/`.
* **Request Body:** Raw text content (`text/plain`).
* **Response:** `{"message": "Successfully saved config..."}`
* **Error Responses:**
* 400 Bad Request: If attempting to write to the root.
* 500 Internal Server Error: If writing fails.
* **DELETE /configs/{relative_path:path}**
* **Summary:** Delete a specific config file or directory.
* **Description:** Deletes a specific file or directory (recursively) relative to `./external_configs/`. **Warning:** Permanent.
* **Path Parameters:**
* `relative_path`: The path relative to `./external_configs/`.
* **Response:** `{"message": "Successfully deleted config {file|directory}..."}`
* **Error Responses:**
* 404 Not Found: If the file or directory doesn't exist.
* 400 Bad Request: If attempting to delete the root `./external_configs` directory.
* 500 Internal Server Error: If deletion fails.
* **POST /configs/move**
* **Summary:** Move/Rename a file or folder within configs.
* **Description:** Moves or renames a file or folder within the `./external_configs/` directory using `shutil.move`. See `/outputs/move` for detailed behavior and error conditions.
* **Request Body:**
```json
{
"source_relative_path": "string (required, current relative path)",
"destination_relative_path": "string (required, new relative path)"
}
```
* **Response (Status 200):**
```json
{
"message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
```
* **Error Responses:** 400, 404, 409, 500 (see `/outputs/move`)
* **POST /configs/directory**
* **Summary:** Create a new directory within the configs folder.
* **Description:** Creates a new directory at the specified `relative_path` within the `./external_configs/` directory. Uses the shared `handle_create_directory` helper.
* **Request Body:**
```json
{
"relative_path": "string (required, relative path)"
}
```
* **Response (Status 201):**
```json
{
"message": "Successfully created directory '{relative_path}' in external_configs."
}
```
* **Error Responses:**
* 400 Bad Request: If `relative_path` is invalid (empty, '.', targets root) or if the parent directory specified in the path does not exist.
* 409 Conflict: If a file or directory already exists at the target `relative_path`.
* 500 Internal Server Error: If directory creation fails for other reasons (e.g., permissions).
* **POST /configs/duplicate**
* **Summary:** Duplicate a pipeline config into external_configs.
* **Description:** Duplicates a configuration file identified by a `source_alias` (from `super_config.yaml`) into the `./external_configs/` directory at the specified `destination_relative_path`. The source alias must resolve to an existing `.yaml` file. The destination path must not already exist, and its parent directory must exist.
* **Request Body:**
```json
{
"source_alias": "string (required, alias from super_config.yaml)",
"destination_relative_path": "string (required, new relative path in external_configs)"
}
```
* **Response (Status 201):**
```json
{
"message": "Successfully duplicated config from alias '{source_alias}' to '{destination_relative_path}' in external_configs."
}
```
* **Error Responses:**
* 400 Bad Request: If the source alias doesn't point to a `.yaml` file, if destination path is invalid, or if destination parent directory doesn't exist.
* 404 Not Found: If the source alias is not found in `super_config.yaml` or the resolved source file doesn't exist.
* 409 Conflict: If the destination path already exists.
* 500 Internal Server Error: If resolving paths, reading the source, or writing the destination file fails.
|