nyk commited on
Commit
2aa4737
·
unverified ·
1 Parent(s): beb89d9

fix: resolve WebSocket disconnect bugs and add SSE reconnect backoff (#97)

Browse files

- Fix stale closure: onclose now calls connectRef.current instead of
capturing connect by value, so reconnect always uses the latest version
- Fix disconnect-reconnect race: manualDisconnectRef prevents onclose
from scheduling a new reconnect after explicit disconnect()
- Fix double-connect guard: check both OPEN and CONNECTING states
- Add SSE exponential backoff with 20-attempt cap (was flat 3s infinite)
- Add SSE error logging (was silently swallowed)
- Update README: fix stale counts (28 panels, 66 routes, 21 migrations,
148 E2E tests), add missing features (SOUL system, Ed25519, agent
messaging, update checker), document NEXT_PUBLIC_GATEWAY_TOKEN

Files changed (3) hide show
  1. README.md +38 -6
  2. src/lib/use-server-events.ts +19 -3
  3. src/lib/websocket.ts +21 -4
README.md CHANGED
@@ -24,7 +24,7 @@ Manage agent fleets, track tasks, monitor costs, and orchestrate workflows — a
24
 
25
  Running AI agents at scale means juggling sessions, tasks, costs, and reliability across multiple models and channels. Mission Control gives you:
26
 
27
- - **26 panels** — Tasks, agents, logs, tokens, memory, cron, alerts, webhooks, pipelines, and more
28
  - **Real-time everything** — WebSocket + SSE push updates, smart polling that pauses when you're away
29
  - **Zero external dependencies** — SQLite database, single `pnpm start` to run, no Redis/Postgres/Docker required
30
  - **Role-based access** — Viewer, operator, and admin roles with session + API key auth
@@ -61,6 +61,10 @@ Initial login is seeded from `AUTH_USER` / `AUTH_PASS` on first run.
61
  - Local Claude Code session tracking (auto-discovers from `~/.claude/projects/`)
62
  - Quality review gates for task sign-off
63
  - Pipeline orchestration with workflow templates
 
 
 
 
64
 
65
  ### Known Limitations
66
 
@@ -99,9 +103,18 @@ Automatically discovers and tracks local Claude Code sessions by scanning `~/.cl
99
  ### GitHub Issues Sync
100
  Inbound sync from GitHub repositories with label and assignee mapping. Synced issues appear on the task board alongside agent-created tasks.
101
 
 
 
 
 
 
 
102
  ### Integrations
103
  Outbound webhooks with delivery history, configurable alert rules with cooldowns, and multi-gateway connection management. Optional 1Password CLI integration for secret management.
104
 
 
 
 
105
  ## Architecture
106
 
107
  ```
@@ -111,20 +124,22 @@ mission-control/
111
  │ ├── app/
112
  │ │ ├── page.tsx # SPA shell — routes all panels
113
  │ │ ├── login/page.tsx # Login page
114
- │ │ └── api/ # 64 REST API routes
115
  │ ├── components/
116
  │ │ ├── layout/ # NavRail, HeaderBar, LiveFeed
117
  │ │ ├── dashboard/ # Overview dashboard
118
- │ │ ├── panels/ # 26 feature panels
119
  │ │ └── chat/ # Agent chat UI
120
  │ ├── lib/
121
  │ │ ├── auth.ts # Session + API key auth, RBAC
122
  │ │ ├── db.ts # SQLite (better-sqlite3, WAL mode)
123
  │ │ ├── claude-sessions.ts # Local Claude Code session scanner
124
- │ │ ├── migrations.ts # 20 schema migrations
125
  │ │ ├── scheduler.ts # Background task scheduler
126
  │ │ ├── webhooks.ts # Outbound webhook delivery
127
- │ │ └── websocket.ts # Gateway WebSocket client
 
 
128
  │ └── store/index.ts # Zustand state management
129
  └── .data/ # Runtime data (SQLite DB, token logs)
130
  ```
@@ -141,7 +156,8 @@ mission-control/
141
  | Charts | Recharts 3 |
142
  | Real-time | WebSocket + Server-Sent Events |
143
  | Auth | scrypt hashing, session tokens, RBAC |
144
- | Testing | Vitest + Playwright (165 E2E tests) |
 
145
 
146
  ## Authentication
147
 
@@ -184,6 +200,11 @@ All endpoints require authentication unless noted. Full reference below.
184
  |--------|------|------|-------------|
185
  | `GET` | `/api/agents` | viewer | List agents with task stats |
186
  | `POST` | `/api/agents` | operator | Register/update agent |
 
 
 
 
 
187
  | `GET` | `/api/tasks` | viewer | List tasks (filter: `?status=`, `?assigned_to=`, `?priority=`) |
188
  | `POST` | `/api/tasks` | operator | Create task |
189
  | `GET` | `/api/tasks/[id]` | viewer | Task details |
@@ -207,6 +228,7 @@ All endpoints require authentication unless noted. Full reference below.
207
  | `GET` | `/api/tokens` | viewer | Token usage and cost data |
208
  | `GET` | `/api/standup` | viewer | Standup report history |
209
  | `POST` | `/api/standup` | operator | Generate standup |
 
210
 
211
  </details>
212
 
@@ -232,6 +254,8 @@ All endpoints require authentication unless noted. Full reference below.
232
  | `GET` | `/api/memory` | viewer | Memory file browser/search |
233
  | `GET` | `/api/search` | viewer | Global search |
234
  | `GET` | `/api/export` | admin | CSV export |
 
 
235
 
236
  </details>
237
 
@@ -319,6 +343,8 @@ See [`.env.example`](.env.example) for the complete list. Key variables:
319
  | `OPENCLAW_HOME` | Yes* | Path to `.openclaw` directory |
320
  | `OPENCLAW_GATEWAY_HOST` | No | Gateway host (default: `127.0.0.1`) |
321
  | `OPENCLAW_GATEWAY_PORT` | No | Gateway WebSocket port (default: `18789`) |
 
 
322
  | `OPENCLAW_MEMORY_DIR` | No | Memory browser root (see note below) |
323
  | `MC_CLAUDE_HOME` | No | Path to `~/.claude` directory (default: `~/.claude`) |
324
  | `MC_TRUSTED_PROXIES` | No | Comma-separated trusted proxy IPs for XFF parsing |
@@ -386,10 +412,16 @@ See [open issues](https://github.com/builderz-labs/mission-control/issues) for p
386
  - [x] Webhook signature verification (HMAC-SHA256 with constant-time comparison)
387
  - [x] Local Claude Code session tracking — auto-discover sessions from `~/.claude/projects/`
388
  - [x] Rate limiter IP extraction hardening with trusted proxy support
 
 
 
 
 
389
 
390
  **Up next:**
391
 
392
  - [ ] Agent-agnostic gateway support — connect any orchestration framework (OpenClaw, ZeroClaw, OpenFang, NeoBot, IronClaw, etc.), not just OpenClaw
 
393
  - [ ] Native macOS app (Electron or Tauri)
394
  - [ ] First-class per-agent cost breakdowns — dedicated panel with per-agent token usage and spend (currently derivable from per-session data)
395
  - [ ] OAuth approval UI improvements
 
24
 
25
  Running AI agents at scale means juggling sessions, tasks, costs, and reliability across multiple models and channels. Mission Control gives you:
26
 
27
+ - **28 panels** — Tasks, agents, logs, tokens, memory, cron, alerts, webhooks, pipelines, and more
28
  - **Real-time everything** — WebSocket + SSE push updates, smart polling that pauses when you're away
29
  - **Zero external dependencies** — SQLite database, single `pnpm start` to run, no Redis/Postgres/Docker required
30
  - **Role-based access** — Viewer, operator, and admin roles with session + API key auth
 
61
  - Local Claude Code session tracking (auto-discovers from `~/.claude/projects/`)
62
  - Quality review gates for task sign-off
63
  - Pipeline orchestration with workflow templates
64
+ - Ed25519 device identity for secure gateway handshake
65
+ - Agent SOUL system with workspace file sync and templates
66
+ - Agent inter-agent messaging and comms
67
+ - Update available banner with GitHub release check
68
 
69
  ### Known Limitations
70
 
 
103
  ### GitHub Issues Sync
104
  Inbound sync from GitHub repositories with label and assignee mapping. Synced issues appear on the task board alongside agent-created tasks.
105
 
106
+ ### Agent SOUL System
107
+ Define agent personality, capabilities, and behavioral guidelines via SOUL markdown files. Edit in the UI or directly in workspace `soul.md` files — changes sync bidirectionally between disk and database.
108
+
109
+ ### Agent Messaging
110
+ Inter-agent communication via the comms API. Agents can send messages to each other, enabling coordinated multi-agent workflows.
111
+
112
  ### Integrations
113
  Outbound webhooks with delivery history, configurable alert rules with cooldowns, and multi-gateway connection management. Optional 1Password CLI integration for secret management.
114
 
115
+ ### Update Checker
116
+ Automatic GitHub release check notifies you when a new version is available, displayed as a banner in the dashboard.
117
+
118
  ## Architecture
119
 
120
  ```
 
124
  │ ├── app/
125
  │ │ ├── page.tsx # SPA shell — routes all panels
126
  │ │ ├── login/page.tsx # Login page
127
+ │ │ └── api/ # 66 REST API routes
128
  │ ├── components/
129
  │ │ ├── layout/ # NavRail, HeaderBar, LiveFeed
130
  │ │ ├── dashboard/ # Overview dashboard
131
+ │ │ ├── panels/ # 28 feature panels
132
  │ │ └── chat/ # Agent chat UI
133
  │ ├── lib/
134
  │ │ ├── auth.ts # Session + API key auth, RBAC
135
  │ │ ├── db.ts # SQLite (better-sqlite3, WAL mode)
136
  │ │ ├── claude-sessions.ts # Local Claude Code session scanner
137
+ │ │ ├── migrations.ts # 21 schema migrations
138
  │ │ ├── scheduler.ts # Background task scheduler
139
  │ │ ├── webhooks.ts # Outbound webhook delivery
140
+ │ │ ├── websocket.ts # Gateway WebSocket client
141
+ │ │ ├── device-identity.ts # Ed25519 device identity for gateway auth
142
+ │ │ └── agent-sync.ts # OpenClaw config → MC database sync
143
  │ └── store/index.ts # Zustand state management
144
  └── .data/ # Runtime data (SQLite DB, token logs)
145
  ```
 
156
  | Charts | Recharts 3 |
157
  | Real-time | WebSocket + Server-Sent Events |
158
  | Auth | scrypt hashing, session tokens, RBAC |
159
+ | Validation | Zod 4 |
160
+ | Testing | Vitest + Playwright (148 E2E tests) |
161
 
162
  ## Authentication
163
 
 
200
  |--------|------|------|-------------|
201
  | `GET` | `/api/agents` | viewer | List agents with task stats |
202
  | `POST` | `/api/agents` | operator | Register/update agent |
203
+ | `GET` | `/api/agents/[id]` | viewer | Agent details |
204
+ | `POST` | `/api/agents/sync` | operator | Sync agents from openclaw.json |
205
+ | `GET/PUT` | `/api/agents/[id]/soul` | operator | Agent SOUL content (reads from workspace, writes to both) |
206
+ | `GET/POST` | `/api/agents/comms` | operator | Agent inter-agent communication |
207
+ | `POST` | `/api/agents/message` | operator | Send message to agent |
208
  | `GET` | `/api/tasks` | viewer | List tasks (filter: `?status=`, `?assigned_to=`, `?priority=`) |
209
  | `POST` | `/api/tasks` | operator | Create task |
210
  | `GET` | `/api/tasks/[id]` | viewer | Task details |
 
228
  | `GET` | `/api/tokens` | viewer | Token usage and cost data |
229
  | `GET` | `/api/standup` | viewer | Standup report history |
230
  | `POST` | `/api/standup` | operator | Generate standup |
231
+ | `GET` | `/api/releases/check` | viewer | Check for new GitHub releases |
232
 
233
  </details>
234
 
 
254
  | `GET` | `/api/memory` | viewer | Memory file browser/search |
255
  | `GET` | `/api/search` | viewer | Global search |
256
  | `GET` | `/api/export` | admin | CSV export |
257
+ | `POST` | `/api/backup` | admin | Database backup |
258
+ | `POST` | `/api/cleanup` | admin | Stale data cleanup |
259
 
260
  </details>
261
 
 
343
  | `OPENCLAW_HOME` | Yes* | Path to `.openclaw` directory |
344
  | `OPENCLAW_GATEWAY_HOST` | No | Gateway host (default: `127.0.0.1`) |
345
  | `OPENCLAW_GATEWAY_PORT` | No | Gateway WebSocket port (default: `18789`) |
346
+ | `OPENCLAW_GATEWAY_TOKEN` | No | Server-side gateway auth token |
347
+ | `NEXT_PUBLIC_GATEWAY_TOKEN` | No | Browser-side gateway auth token (must use `NEXT_PUBLIC_` prefix) |
348
  | `OPENCLAW_MEMORY_DIR` | No | Memory browser root (see note below) |
349
  | `MC_CLAUDE_HOME` | No | Path to `~/.claude` directory (default: `~/.claude`) |
350
  | `MC_TRUSTED_PROXIES` | No | Comma-separated trusted proxy IPs for XFF parsing |
 
412
  - [x] Webhook signature verification (HMAC-SHA256 with constant-time comparison)
413
  - [x] Local Claude Code session tracking — auto-discover sessions from `~/.claude/projects/`
414
  - [x] Rate limiter IP extraction hardening with trusted proxy support
415
+ - [x] Ed25519 device identity for WebSocket challenge-response handshake ([#85](https://github.com/builderz-labs/mission-control/pull/85))
416
+ - [x] Agent SOUL workspace sync — bidirectional sync between `soul.md` files and database ([#95](https://github.com/builderz-labs/mission-control/pull/95))
417
+ - [x] Update available banner with GitHub release check ([#94](https://github.com/builderz-labs/mission-control/pull/94))
418
+ - [x] Side panel navigation synced with URL routes ([#87](https://github.com/builderz-labs/mission-control/pull/87))
419
+ - [x] Task board SSE wiring, priority enum, and auto-advance ([#89](https://github.com/builderz-labs/mission-control/pull/89))
420
 
421
  **Up next:**
422
 
423
  - [ ] Agent-agnostic gateway support — connect any orchestration framework (OpenClaw, ZeroClaw, OpenFang, NeoBot, IronClaw, etc.), not just OpenClaw
424
+ - [ ] Workspace isolation for multi-team usage ([#75](https://github.com/builderz-labs/mission-control/issues/75))
425
  - [ ] Native macOS app (Electron or Tauri)
426
  - [ ] First-class per-agent cost breakdowns — dedicated panel with per-agent token usage and spend (currently derivable from per-session data)
427
  - [ ] OAuth approval UI improvements
src/lib/use-server-events.ts CHANGED
@@ -16,9 +16,14 @@ interface ServerEvent {
16
  * SSE provides instant updates for all local-DB data (tasks, agents,
17
  * chat, activities, notifications), making REST polling a fallback.
18
  */
 
 
 
 
19
  export function useServerEvents() {
20
  const eventSourceRef = useRef<EventSource | null>(null)
21
  const reconnectTimeoutRef = useRef<NodeJS.Timeout | undefined>(undefined)
 
22
 
23
  const {
24
  setConnection,
@@ -46,6 +51,7 @@ export function useServerEvents() {
46
 
47
  es.onopen = () => {
48
  if (!mounted) return
 
49
  setConnection({ sseConnected: true })
50
  }
51
 
@@ -65,11 +71,21 @@ export function useServerEvents() {
65
  es.close()
66
  eventSourceRef.current = null
67
 
68
- // Reconnect after 3s (EventSource auto-reconnects, but we handle
69
- // it explicitly to control the sseConnected state)
 
 
 
 
 
 
 
 
 
 
70
  reconnectTimeoutRef.current = setTimeout(() => {
71
  if (mounted) connect()
72
- }, 3000)
73
  }
74
  }
75
 
 
16
  * SSE provides instant updates for all local-DB data (tasks, agents,
17
  * chat, activities, notifications), making REST polling a fallback.
18
  */
19
+ const SSE_MAX_RECONNECT_ATTEMPTS = 20
20
+ const SSE_BASE_DELAY_MS = 1000
21
+ const SSE_MAX_DELAY_MS = 30000
22
+
23
  export function useServerEvents() {
24
  const eventSourceRef = useRef<EventSource | null>(null)
25
  const reconnectTimeoutRef = useRef<NodeJS.Timeout | undefined>(undefined)
26
+ const sseReconnectAttemptsRef = useRef<number>(0)
27
 
28
  const {
29
  setConnection,
 
51
 
52
  es.onopen = () => {
53
  if (!mounted) return
54
+ sseReconnectAttemptsRef.current = 0
55
  setConnection({ sseConnected: true })
56
  }
57
 
 
71
  es.close()
72
  eventSourceRef.current = null
73
 
74
+ const attempts = sseReconnectAttemptsRef.current
75
+ if (attempts >= SSE_MAX_RECONNECT_ATTEMPTS) {
76
+ console.error(`SSE: max reconnect attempts (${SSE_MAX_RECONNECT_ATTEMPTS}) reached`)
77
+ return
78
+ }
79
+
80
+ // Exponential backoff with jitter
81
+ const base = Math.min(Math.pow(2, attempts) * SSE_BASE_DELAY_MS, SSE_MAX_DELAY_MS)
82
+ const delay = Math.round(base + Math.random() * base * 0.5)
83
+ sseReconnectAttemptsRef.current = attempts + 1
84
+
85
+ console.warn(`SSE: reconnecting in ${delay}ms (attempt ${attempts + 1}/${SSE_MAX_RECONNECT_ATTEMPTS})`)
86
  reconnectTimeoutRef.current = setTimeout(() => {
87
  if (mounted) connect()
88
+ }, delay)
89
  }
90
  }
91
 
src/lib/websocket.ts CHANGED
@@ -47,6 +47,8 @@ export function useWebSocket() {
47
  const requestIdRef = useRef<number>(0)
48
  const handshakeCompleteRef = useRef<boolean>(false)
49
  const reconnectAttemptsRef = useRef<number>(0)
 
 
50
 
51
  // Heartbeat tracking
52
  const pingCounterRef = useRef<number>(0)
@@ -400,8 +402,9 @@ export function useWebSocket() {
400
  }, [sendConnectHandshake, setConnection, setSessions, addLog, startHeartbeat, handlePong, addChatMessage, addNotification, updateAgent])
401
 
402
  const connect = useCallback((url: string, token?: string) => {
403
- if (wsRef.current?.readyState === WebSocket.OPEN) {
404
- return // Already connected
 
405
  }
406
 
407
  // Extract token from URL if present
@@ -414,6 +417,7 @@ export function useWebSocket() {
414
 
415
  reconnectUrl.current = url
416
  handshakeCompleteRef.current = false
 
417
 
418
  try {
419
  const ws = new WebSocket(url.split('?')[0]) // Connect without query params
@@ -452,7 +456,10 @@ export function useWebSocket() {
452
  handshakeCompleteRef.current = false
453
  stopHeartbeat()
454
 
455
- // Auto-reconnect logic with exponential backoff (uses ref to avoid stale closure)
 
 
 
456
  const attempts = reconnectAttemptsRef.current
457
  if (attempts < maxReconnectAttempts) {
458
  const base = Math.min(Math.pow(2, attempts) * 1000, 30000)
@@ -462,7 +469,7 @@ export function useWebSocket() {
462
  reconnectAttemptsRef.current = attempts + 1
463
  setConnection({ reconnectAttempts: attempts + 1 })
464
  reconnectTimeoutRef.current = setTimeout(() => {
465
- connect(url, authTokenRef.current)
466
  }, timeout)
467
  } else {
468
  console.error('Max reconnection attempts reached.')
@@ -493,9 +500,19 @@ export function useWebSocket() {
493
  }
494
  }, [setConnection, handleGatewayFrame, addLog, stopHeartbeat])
495
 
 
 
 
 
 
496
  const disconnect = useCallback(() => {
 
 
 
 
497
  if (reconnectTimeoutRef.current) {
498
  clearTimeout(reconnectTimeoutRef.current)
 
499
  }
500
 
501
  stopHeartbeat()
 
47
  const requestIdRef = useRef<number>(0)
48
  const handshakeCompleteRef = useRef<boolean>(false)
49
  const reconnectAttemptsRef = useRef<number>(0)
50
+ const manualDisconnectRef = useRef<boolean>(false)
51
+ const connectRef = useRef<(url: string, token?: string) => void>(() => {})
52
 
53
  // Heartbeat tracking
54
  const pingCounterRef = useRef<number>(0)
 
402
  }, [sendConnectHandshake, setConnection, setSessions, addLog, startHeartbeat, handlePong, addChatMessage, addNotification, updateAgent])
403
 
404
  const connect = useCallback((url: string, token?: string) => {
405
+ const state = wsRef.current?.readyState
406
+ if (state === WebSocket.OPEN || state === WebSocket.CONNECTING) {
407
+ return // Already connected or connecting
408
  }
409
 
410
  // Extract token from URL if present
 
417
 
418
  reconnectUrl.current = url
419
  handshakeCompleteRef.current = false
420
+ manualDisconnectRef.current = false
421
 
422
  try {
423
  const ws = new WebSocket(url.split('?')[0]) // Connect without query params
 
456
  handshakeCompleteRef.current = false
457
  stopHeartbeat()
458
 
459
+ // Skip auto-reconnect if this was a manual disconnect
460
+ if (manualDisconnectRef.current) return
461
+
462
+ // Auto-reconnect with exponential backoff (uses connectRef to avoid stale closure)
463
  const attempts = reconnectAttemptsRef.current
464
  if (attempts < maxReconnectAttempts) {
465
  const base = Math.min(Math.pow(2, attempts) * 1000, 30000)
 
469
  reconnectAttemptsRef.current = attempts + 1
470
  setConnection({ reconnectAttempts: attempts + 1 })
471
  reconnectTimeoutRef.current = setTimeout(() => {
472
+ connectRef.current(reconnectUrl.current, authTokenRef.current)
473
  }, timeout)
474
  } else {
475
  console.error('Max reconnection attempts reached.')
 
500
  }
501
  }, [setConnection, handleGatewayFrame, addLog, stopHeartbeat])
502
 
503
+ // Keep ref in sync so onclose always calls the latest version of connect
504
+ useEffect(() => {
505
+ connectRef.current = connect
506
+ }, [connect])
507
+
508
  const disconnect = useCallback(() => {
509
+ // Signal manual disconnect before closing so onclose skips auto-reconnect
510
+ manualDisconnectRef.current = true
511
+ reconnectAttemptsRef.current = 0
512
+
513
  if (reconnectTimeoutRef.current) {
514
  clearTimeout(reconnectTimeoutRef.current)
515
+ reconnectTimeoutRef.current = undefined
516
  }
517
 
518
  stopHeartbeat()