| # Scheduler And Tasks |
|
|
| The scheduler is an optional in-memory task queue for multi-agent coordination. It accepts tasks over `/tasks`, applies admission and fairness rules, then dispatches work to the same tab action executor used by the immediate browsing routes. |
|
|
| It does not replace the normal direct path. Routes such as `POST /tabs/{id}/action` still work independently. |
|
|
| There is no CLI scheduler command today. |
|
|
| ## Enable The Scheduler |
|
|
| The scheduler is off by default. Dashboard mode registers the task routes only when `scheduler.enabled` is true. |
|
|
| ```json |
| { |
| "scheduler": { |
| "enabled": true |
| } |
| } |
| ``` |
|
|
| ## Scheduler Config |
|
|
| ```json |
| { |
| "scheduler": { |
| "enabled": true, |
| "strategy": "fair-fifo", |
| "maxQueueSize": 1000, |
| "maxPerAgent": 100, |
| "maxInflight": 20, |
| "maxPerAgentInflight": 10, |
| "resultTTLSec": 300, |
| "workerCount": 4 |
| } |
| } |
| ``` |
|
|
| | Field | Default | Meaning | |
| | --- | --- | --- | |
| | `enabled` | `false` | enables task routes in dashboard mode | |
| | `strategy` | `fair-fifo` | scheduler strategy label | |
| | `maxQueueSize` | `1000` | global queued task limit | |
| | `maxPerAgent` | `100` | queued task limit per agent | |
| | `maxInflight` | `20` | max concurrently executing tasks overall | |
| | `maxPerAgentInflight` | `10` | max concurrently executing tasks per agent | |
| | `resultTTLSec` | `300` | retention time for terminal task snapshots | |
| | `workerCount` | `4` | number of worker goroutines | |
|
|
| ## Task Object |
|
|
| Tasks are scheduler-owned records with these main fields: |
|
|
| | Field | Meaning | |
| | --- | --- | |
| | `taskId` | generated task ID | |
| | `agentId` | submitting agent identifier | |
| | `action` | action kind to run | |
| | `tabId` | target tab ID | |
| | `ref` | optional element ref | |
| | `params` | optional action-specific request fields | |
| | `priority` | lower number means higher priority | |
| | `state` | current task state | |
| | `deadline` | execution deadline | |
| | `createdAt` | submission time | |
| | `startedAt` | first execution timestamp | |
| | `completedAt` | terminal timestamp | |
| | `latencyMs` | elapsed time from start to completion | |
| | `result` | executor response payload | |
| | `error` | terminal error message | |
| | `position` | queue position at submission time | |
| | `callbackUrl` | optional webhook URL for terminal state notification | |
|
|
| Task IDs are currently generated as `tsk_XXXXXXXX`, but callers should still treat them as opaque IDs. |
|
|
| ## Submit A Task |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "agentId": "agent-crawl-01", |
| "action": "click", |
| "tabId": "8f9c7d4e1234567890abcdef12345678", |
| "ref": "e14", |
| "priority": 5, |
| "deadline": "2026-03-08T12:05:00Z" |
| }' |
| # Response |
| { |
| "taskId": "tsk_a1b2c3d4", |
| "state": "queued", |
| "position": 1, |
| "createdAt": "2026-03-08T12:00:01Z" |
| } |
| ``` |
|
|
| This endpoint returns `202 Accepted` on successful queue submission. |
|
|
| Request fields: |
|
|
| | Field | Required | Notes | |
| | --- | --- | --- | |
| | `agentId` | yes | validated at request time | |
| | `action` | yes | becomes the executor `kind` | |
| | `tabId` | practically yes | required by the execution path | |
| | `ref` | no | top-level element ref for element-targeted actions | |
| | `params` | no | action-specific fields merged into the executor request body | |
| | `priority` | no | lower number means higher priority | |
| | `deadline` | no | RFC3339 timestamp; defaults to `now + 60s` | |
| | `callbackUrl` | no | webhook URL; receives POST with task snapshot on terminal state | |
|
|
| Important: |
|
|
| - request validation enforces only `agentId` and `action` |
| - missing `tabId` is rejected later during execution with `tabId is required for task execution` |
| - past deadlines are rejected at submission time |
|
|
| ## Queue Full Response |
|
|
| If admission fails because the global queue or an agent queue is full, the scheduler returns `429 Too Many Requests`. |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks \ |
| -H "Content-Type: application/json" \ |
| -d '{"agentId":"agent-crawl-01","action":"click","tabId":"8f9c7d4e1234567890abcdef12345678"}' |
| # Response |
| { |
| "code": "queue_full", |
| "error": "rejected: global queue full", |
| "retryable": true, |
| "details": { |
| "agentId": "agent-crawl-01", |
| "queued": 1000, |
| "maxQueue": 1000, |
| "maxPerAgent": 100 |
| } |
| } |
| ``` |
|
|
| ## List Tasks |
|
|
| `GET /tasks` returns the scheduler's in-memory task snapshots, including queued, running, and recently completed tasks that are still within the TTL window. |
|
|
| ```bash |
| curl http://localhost:9867/tasks |
| # Response |
| { |
| "tasks": [ |
| { |
| "taskId": "tsk_a1b2c3d4", |
| "state": "done", |
| "agentId": "agent-crawl-01", |
| "action": "click", |
| "latencyMs": 842 |
| } |
| ], |
| "count": 1 |
| } |
| ``` |
|
|
| Supported query filters: |
|
|
| - `agentId` |
| - `state` |
|
|
| Example: |
|
|
| ```bash |
| curl 'http://localhost:9867/tasks?agentId=agent-crawl-01&state=done,failed' |
| ``` |
|
|
| ## Get One Task |
|
|
| ```bash |
| curl http://localhost:9867/tasks/tsk_a1b2c3d4 |
| # Response |
| { |
| "taskId": "tsk_a1b2c3d4", |
| "agentId": "agent-crawl-01", |
| "action": "click", |
| "tabId": "8f9c7d4e1234567890abcdef12345678", |
| "ref": "e14", |
| "priority": 5, |
| "state": "done", |
| "createdAt": "2026-03-08T12:00:01Z", |
| "startedAt": "2026-03-08T12:00:01Z", |
| "completedAt": "2026-03-08T12:00:02Z", |
| "latencyMs": 842, |
| "result": { |
| "success": true |
| } |
| } |
| ``` |
|
|
| If the task is not found, the scheduler returns: |
|
|
| ```json |
| { |
| "code": "not_found", |
| "error": "task not found" |
| } |
| ``` |
|
|
| ## Cancel A Task |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks/tsk_a1b2c3d4/cancel |
| # Response |
| { |
| "status": "cancelled", |
| "taskId": "tsk_a1b2c3d4" |
| } |
| ``` |
|
|
| Behavior: |
|
|
| - queued tasks are removed from the queue |
| - running tasks have their execution context cancelled |
| - terminal tasks return `409 Conflict` |
|
|
| ## Task States |
|
|
| Implemented states: |
|
|
| - `queued` |
| - `assigned` |
| - `running` |
| - `done` |
| - `failed` |
| - `cancelled` |
| - `rejected` |
|
|
| Terminal states: |
|
|
| - `done` |
| - `failed` |
| - `cancelled` |
| - `rejected` |
|
|
| ## How Tasks Execute |
|
|
| The scheduler forwards each task to the normal tab action endpoint: |
|
|
| ```text |
| POST /tabs/{tabId}/action |
| ``` |
|
|
| It builds the action body like this: |
|
|
| ```json |
| { |
| "kind": "<action>", |
| "ref": "<ref>", |
| "...params": "..." |
| } |
| ``` |
|
|
| That means: |
|
|
| - `action` becomes `kind` |
| - top-level `ref` is forwarded when present |
| - every key in `params` is merged into the top-level action body |
|
|
| Example: |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "agentId": "my-agent", |
| "action": "type", |
| "tabId": "8f9c7d4e1234567890abcdef12345678", |
| "ref": "e12", |
| "params": { |
| "text": "Alan Turing" |
| } |
| }' |
| ``` |
|
|
| In practice, task payloads should use the same action fields that the immediate `/tabs/{id}/action` route expects. |
|
|
| ## Fairness, Deadlines, And Retention |
|
|
| - within one agent queue, lower `priority` values run first |
| - equal-priority tasks for the same agent fall back to FIFO order |
| - across agents, the scheduler prefers the agent with the fewest in-flight tasks |
| - if a queued task passes its deadline before execution starts, it is marked failed with `deadline exceeded while queued` |
| - terminal task snapshots are retained in memory for `resultTTLSec` |
|
|
| --- |
|
|
| ## Phase 2 -- Observability |
|
|
| ### Scheduler Stats |
|
|
| `GET /scheduler/stats` returns a snapshot of queue state, runtime metrics, and configuration. |
|
|
| ```bash |
| curl http://localhost:9867/scheduler/stats |
| # Response |
| { |
| "queue": { |
| "totalQueued": 5, |
| "totalInflight": 2, |
| "agentCounts": { |
| "agent-crawl-01": 3, |
| "agent-scrape-02": 2 |
| } |
| }, |
| "metrics": { |
| "tasksSubmitted": 42, |
| "tasksCompleted": 35, |
| "tasksFailed": 3, |
| "tasksCancelled": 2, |
| "tasksRejected": 1, |
| "tasksExpired": 1, |
| "dispatchCount": 38, |
| "avgDispatchLatencyMs": 12.5, |
| "agents": { |
| "agent-crawl-01": { |
| "submitted": 25, |
| "completed": 22, |
| "failed": 2, |
| "cancelled": 1, |
| "rejected": 0 |
| } |
| } |
| }, |
| "config": { |
| "strategy": "fair-fifo", |
| "maxQueueSize": 1000, |
| "maxPerAgent": 100, |
| "maxInflight": 20, |
| "maxPerAgentFlight": 10, |
| "workerCount": 4, |
| "resultTTL": "5m0s" |
| } |
| } |
| ``` |
|
|
| #### Metrics Fields |
|
|
| | Field | Type | Meaning | |
| | --- | --- | --- | |
| | `tasksSubmitted` | uint64 | total tasks accepted since startup | |
| | `tasksCompleted` | uint64 | tasks that finished successfully | |
| | `tasksFailed` | uint64 | tasks that finished with an error | |
| | `tasksCancelled` | uint64 | tasks cancelled via `POST /tasks/{id}/cancel` | |
| | `tasksRejected` | uint64 | tasks rejected at admission (queue full) | |
| | `tasksExpired` | uint64 | queued tasks that exceeded their deadline | |
| | `dispatchCount` | uint64 | number of tasks dispatched to workers | |
| | `avgDispatchLatencyMs` | float64 | average time from queue entry to dispatch start | |
| | `agents` | object | per-agent breakdown (submitted, completed, failed, cancelled, rejected) | |
|
|
| ### Webhook Callbacks |
|
|
| Tasks can include a `callbackUrl` field. When the task reaches a terminal state (`done`, `failed`, or `cancelled`), the scheduler delivers a POST with the task snapshot to that URL. |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "agentId": "my-agent", |
| "action": "click", |
| "tabId": "8f9c7d4e1234567890abcdef12345678", |
| "callbackUrl": "https://example.com/hooks/task-done" |
| }' |
| ``` |
|
|
| Webhook behavior: |
|
|
| - delivery is best-effort: failures are logged but do not affect task state |
| - only `http` and `https` schemes are allowed (SSRF protection) |
| - a dedicated HTTP client with a 10-second timeout is used |
| - custom headers are sent: `X-PinchTab-Event: task.completed` and `X-PinchTab-Task-ID: <taskId>` |
|
|
| The `callbackUrl` field is stored on the task and returned in `GET /tasks/{id}`. |
|
|
| --- |
|
|
| ## Phase 3 -- Hardening |
|
|
| ### Batch Task Submission |
|
|
| `POST /tasks/batch` submits multiple tasks in a single request. All tasks in the batch share the same `agentId` and optional `callbackUrl`. |
|
|
| ```bash |
| curl -X POST http://localhost:9867/tasks/batch \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "agentId": "agent-crawl-01", |
| "callbackUrl": "https://example.com/hooks/batch", |
| "tasks": [ |
| { "action": "click", "tabId": "TAB_ID", "params": { "selector": "#btn" } }, |
| { "action": "scroll", "tabId": "TAB_ID", "params": { "scrollY": 400 } }, |
| { "action": "hover", "tabId": "TAB_ID", "params": { "selector": "h1" }, "priority": 1 } |
| ] |
| }' |
| # Response (202 Accepted) |
| { |
| "tasks": [ |
| { "taskId": "tsk_aaaa1111", "state": "queued", "position": 1 }, |
| { "taskId": "tsk_bbbb2222", "state": "queued", "position": 2 }, |
| { "taskId": "tsk_cccc3333", "state": "queued", "position": 3 } |
| ], |
| "submitted": 3 |
| } |
| ``` |
|
|
| #### Batch Request Fields |
|
|
| | Field | Required | Notes | |
| | --- | --- | --- | |
| | `agentId` | yes | shared across all tasks in the batch | |
| | `callbackUrl` | no | webhook URL applied to every task | |
| | `tasks` | yes | array of task definitions (1–50) | |
|
|
| Each task definition supports the same fields as a single task submit (`action`, `tabId`, `ref`, `params`, `priority`, `deadline`) except `agentId` and `callbackUrl` which are inherited from the batch. |
|
|
| #### Batch Validation |
|
|
| | Condition | Response | |
| | --- | --- | |
| | missing `agentId` | `400 Bad Request` | |
| | empty `tasks` array | `400 Bad Request` | |
| | more than 50 tasks | `400 Bad Request` with `batch_too_large` code | |
| | invalid JSON body | `400 Bad Request` | |
|
|
| Partial failure: if some tasks are rejected by admission (queue full), the accepted tasks are still submitted. The response includes each task's status individually. |
|
|
| ### Config Hot-Reload |
|
|
| `ReloadConfig(cfg)` updates queue limits, inflight limits, and result TTL at runtime without restarting the scheduler. |
|
|
| Reloadable fields: |
|
|
| | Field | What Changes | |
| | --- | --- | |
| | `maxQueueSize`, `maxPerAgent` | queue admission limits via `SetLimits()` | |
| | `maxInflight`, `maxPerAgentFlight` | concurrency limits (protected by `cfgMu`) | |
| | `resultTTL` | result store eviction window via `SetTTL()` | |
|
|
| Zero values are ignored (the existing setting is preserved). |
|
|
| #### ConfigWatcher |
|
|
| `ConfigWatcher` runs a background goroutine that periodically re-reads the config and calls `ReloadConfig`. Create with: |
|
|
| ```go |
| cw := scheduler.NewConfigWatcher(30*time.Second, loadFn, sched) |
| cw.Start() |
| defer cw.Stop() |
| ``` |
|
|
| `loadFn` is a `func() (Config, error)` that reads the current config from disk or environment. |
|
|
