README.md

---
title: Multi-Agent Assistant
emoji: 🤖
colorFrom: purple
colorTo: blue
sdk: gradio
app_file: app.py
pinned: false
sdk_version: 6.26.0
---

# Multi-Agent Assistant

A chat app where a manager agent routes each request to specialist agents,
built with [LangGraph](https://langchain-ai.github.io/langgraph/) and running
on Hugging Face [ZeroGPU](https://huggingface.co/docs/hub/spaces-zerogpu).

## How it works

1. You send a message in the chat box.
2. A **manager agent** reads it and decides what's needed next: web search,
   Python execution, numeric analysis, or "I have enough — finish."
3. The chosen **specialist agent** does its job and reports back to the
   manager, which can loop through multiple specialists before finishing
   (capped at `MAX_TURNS` round-trips to avoid infinite loops).
4. A **finalize** step synthesizes everything the specialists found into one
   answer, which appears in the chat.

```
Gradio UI → Manager agent → {search_agent, code_agent, data_agent} → Manager → Finalize → UI
```

## Specialists

| Agent | Tool | What it does |
|---|---|---|
| `search_agent` | Tavily web search | Looks up current information on the web |
| `code_agent` | Restricted Python subprocess | Writes and runs Python to compute things |
| `data_agent` | Summary statistics | Computes count/sum/mean/min/max on a list of numbers |

The model itself is [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct),
loaded locally and run on a ZeroGPU-allocated GPU per request (see `generate()`
in `agents.py`) — nothing is sent to an external inference API.

## Files

- `app.py` — Gradio UI (chat interface, session state)
- `agents.py` — LangGraph state machine, model loading, tools, prompts
- `requirements.txt` — Python dependencies (torch/gradio/spaces versions are
  managed by the Space's build system and intentionally not pinned here)

## Required secrets

Set these under **Settings → Variables and secrets**:

| Secret | Used for |
|---|---|
| `TAVILY_API_KEY` | Required for `search_agent`. Get a free key at [tavily.com](https://tavily.com). |
| `HF_TOKEN` | Optional. Speeds up model downloads and avoids Hub rate limits. |

Without `TAVILY_API_KEY`, every other feature still works — `search_agent`
just returns a "Search unavailable" message instead of crashing the app.

## Hardware

This Space requires **ZeroGPU** hardware (not CPU-basic), since `agents.py`
runs local model inference inside a function decorated with `@spaces.GPU`.

## Technical architecture

### State (`AgentState`, LangGraph)
A `TypedDict` threaded through every node: `messages` (accumulated via
`add_messages`), `next` (the manager's routing decision), `task` (the
original request), `results` (a scratch dict specialists write into), and
`turn_count` (see `MAX_TURNS` below).

### Control flow

```mermaid
sequenceDiagram
    participant U as User
    participant M as Manager
    participant S as Specialist
    U->>M: task
    loop up to MAX_TURNS
        M->>M: generate() → route decision
        alt route = specialist
            M->>S: task
            S->>S: generate() → tool call or final
            S-->>M: result written to state.results
        else route = finish
            M->>M: break loop
        end
    end
    M->>M: finalize (synthesize results)
    M-->>U: answer
```

`MAX_TURNS` (default 6, in `agents.py`) hard-caps manager↔specialist
round-trips in code — not just via LangGraph's `recursion_limit`, which is a
blunter backstop covering every node hop including `finalize`. Once
`turn_count` exceeds `MAX_TURNS`, `manager_node` skips the model call
entirely and force-routes to `finalize`, so a flip-flopping manager can't
burn extra inference calls on its way out.

### Model layer (local inference on ZeroGPU)
- `_model`/`_tokenizer` are loaded once at import time, on CPU.
- `generate(messages, max_new_tokens)` is the single inference entry point
  used by every node (manager, all three specialists, finalize). It's
  decorated with `@spaces.GPU(duration=60)`: ZeroGPU attaches a GPU only for
  the duration of this call, then releases it — no dedicated GPU sits idle
  between requests.
- Inside `generate()`: model moved to `cuda`, prompt built via
  `apply_chat_template(..., return_dict=True)` (the explicit dict form,
  to avoid ambiguity between transformers versions that return a raw
  tensor vs. a `BatchEncoding` — this was the root cause of an earlier
  `AttributeError` that only reproduced on this Space's installed version),
  greedy decoding (`do_sample=False`).
- All exceptions are caught **inside** `generate()` and returned as a
  `{"error": ...}` string rather than left to propagate across the
  `@spaces.GPU` process boundary — exceptions crossing that boundary raw
  can arrive mangled in the caller (observed in practice: a real
  `AttributeError` surfaced to the caller as an opaque
  `KeyError('AttributeError')`). Catching locally preserves the real
  exception type/message and logs the full traceback.

### Manual tool-calling protocol
Local open-weight models don't expose a native function-calling API the way
hosted inference endpoints do, so tool use is hand-rolled:
1. Each specialist's system prompt lists its tool's name, description, and
   argument schema.
2. The model is asked to respond with **only** JSON — either
   `{"tool": "...", "args": {...}}` or `{"final": "..."}`.
3. The node parses that JSON, invokes the named tool directly if present,
   then makes one follow-up `generate()` call asking the model to
   summarize the tool's raw output.
4. Malformed/non-JSON responses fall back to using the model's raw text as
   the summary, rather than failing the turn.

### Configuration surface

| Constant (`agents.py`) | Default | Effect |
|---|---|---|
| `MODEL_ID` | `Qwen/Qwen2.5-3B-Instruct` | Swap for a larger model if quality is the bottleneck |
| `MAX_TURNS` | 6 | Manager↔specialist round-trip cap |
| `@spaces.GPU(duration=60)` | 60s | Per-call GPU allocation window — bump if generation runs long |

## Known limitations

- `run_python` executes in a locked-down subprocess (restricted builtins,
  5s timeout, isolated interpreter mode) — a reasonable starting point, but
  **not** a real security sandbox. For untrusted input in production, use a
  dedicated sandbox such as E2B Code Interpreter or Modal Sandboxes instead.
- Specialists use a manually-parsed JSON tool-call protocol rather than a
  native function-calling API, since local open models don't expose one the
  way hosted inference endpoints do. Malformed JSON from the model falls
  back to using its raw text response.
- `MAX_TURNS` (in `agents.py`) hard-caps manager↔specialist round-trips per
  request to prevent runaway routing loops.
- **ZeroGPU cold starts.** The first request after the Space idles and
  restarts pays model-load latency. If generation routinely runs long
  (bigger model, longer `max_new_tokens`), `duration=60` may need increasing.
- **Single shared model instance.** `_model`/`_tokenizer` are process-wide
  globals; concurrent requests from different users serialize through the
  same weights (ZeroGPU allocates the GPU per call, not per user).