# Architecture

This document explains how the agent works, step by step, for someone who has never seen the codebase before.

## What this agent does

The agent answers **real-world questions** that may include **file attachments** (PDF, image, Excel, audio, etc.). Each answer should be a **short, exact value** — a number, a name, or a comma-separated list.

For each question it:

1. Reads the question text (and attachment, if any).
2. Uses tools and reasoning to find the answer.
3. Returns only the answer value (no extra explanation).
4. Can submit batches to a **scoring API**, which checks exact match against expected answers.

---

## Big picture

Think of the system as a small factory line:

```
Scoring API  →  Pipeline  →  CodeAgent + Tools  →  Verifier  →  Submit answer
     ↑              ↑                                    ↑
 questions     optional extras                    format check
 attachments    (plan, hints)                       + one retry
```

You can run this factory in two ways:

| Entry point | When you use it |
|-------------|-----------------|
| **`app.py`** | Gradio UI on Hugging Face Space — click one button to run the full batch and submit |
| **`run_local.py`** | Your laptop — same logic, good for development and faster iteration |

Both call the same core class: **`GaiaAgent`**.

---

## End-to-end flow (one question)

Here is what happens when you ask the agent a single question.

### 1. Question arrives

`GaiaAgent` receives:

- `question` — the question text
- `file_path` — optional local path to a downloaded attachment
- `task_id` — optional ID for logging and artifacts

If the question has an attachment, the file is downloaded first (`file_resolver.py` → `GET /files/{task_id}` on the scoring API).

### 2. Pipeline prepares context (`agent/pipeline.py`)

Before the language model runs, the pipeline may add extra context to the prompt:

**Strategy hints (optional)** — `agent/retriever.py`  
If `RETRIEVER_ENABLED=1`, the agent reads past runs from `memory/experience.jsonl` and injects hints like “for numeric questions, using `execute_code` worked before.” It never injects actual answers from past runs — only strategies.

**Plan (optional)** — `agent/planner.py`  
If `PIPELINE_DEPTH` is `standard` or `full`, one extra LLM call produces a short JSON plan (2–4 steps). At `minimal` depth (default), planning is skipped.

**Prompt assembly** — `agent/agent_runner.build_prompt()`  
The final prompt includes the question, attachment hints (“this is a PDF, use `read_pdf`”), and any plan/hints above.

### 3. Think mode is chosen (`agent/think_mode.py`)

Some models support extended internal reasoning. The agent turns that on or off **per question**:

- **Off** for trivial questions (“opposite of left”)
- **On** for calculations, attachments, long questions, YouTube/arXiv lookups

Controlled by `THINK_MODE=auto|on|off`.

### 4. CodeAgent runs (`agent/agent_runner.py`)

The heart of the system is a **smolagents CodeAgent**: a language model that writes **Python code blocks** to call tools, inspect results, and eventually call `final_answer("...")`.

Why code? Many questions need several steps: search the web, read a file, compute a number, then answer. The agent loops:

```
LLM writes code → tools run → results go back → LLM writes more code → … → final_answer
```

Default limit: **12 steps** (`AGENT_MAX_STEPS`).

The system prompt enforces answer format rules: short values, no “FINAL ANSWER:” prefix inside `final_answer()`, lists formatted as `a, b, c`.

### 5. Tools (`agent/tools.py` + `tools.py`)

The agent has one toolbox shared by all questions:

| Category | Tools | Typical use |
|----------|-------|-------------|
| Web | DuckDuckGo search, visit webpage, Wikipedia, fetch URL as markdown | Facts, current info |
| Academic | arXiv search | Paper titles, authors |
| Files | read PDF, Excel, CSV, text; OCR on images; transcribe audio; describe image | Attachments |
| Video | YouTube transcript | Questions about video content |
| Compute | `execute_code` (local Python/bash), math helpers (add, multiply, …) | Counting, parsing |
| Browser | Playwright (if `BROWSER_ENABLED=1`) | Heavy JavaScript pages |

Tools return **text** the model reads on the next step. Nothing runs unless the model asks for it in code.

### 6. Verifier checks the answer (`agent/verifier.py`)

The raw model output is not sent directly. The verifier:

1. **Strips** hidden reasoning blocks some models emit
2. **Extracts** the value from `final_answer("...")` or similar patterns
3. **Normalizes** via `scoring.normalize_answer()` (trim quotes, remove “FINAL ANSWER:” prefix)
4. **Checks format** — empty, multi-line, or overly long answers fail
5. **Spot-checks URLs** mentioned in the trace (optional HTTP fetch)
6. **Critic pass** (standard/full depth only) — optional second LLM asks “is this supported by evidence?”

If verification fails, the pipeline **retries once** with the verifier’s issues in the prompt. On retry, think mode is forced **on**.

### 7. Optional extras (depth flags)

| `PIPELINE_DEPTH` | What changes |
|------------------|--------------|
| **`minimal`** (default) | Agent + verifier + one retry. Fastest. |
| **`standard`** | + planner, numeric double-compute hint, optional critic |
| **`full`** | + self-correction loop (up to 3 rounds), optional majority vote (`VOTE_RUNS`) |

### 8. Answer returned and logged

The pipeline writes optional artifacts under `artifacts/{task_id}/`:

- `notes.md` — short summary
- `evidence.json` — answer, depth, think mode, etc.
- `plan.md` — if planning ran

The string returned to `app.py` or `run_local.py` is the **normalized answer only**.

### 9. Submission (batch runs)

For full evaluation, `app.py` or `run_local.py`:

1. `GET /questions` — fetch all tasks from the scoring API
2. Run the pipeline on each
3. `POST /submit` with `{username, agent_code, answers: [{task_id, submitted_answer}, …]}`
4. Scoring API returns score (% correct)

Grading is **exact string match** (case-insensitive after normalization). Ground truth stays on the server — the agent never stores expected answers locally.

---

## Language model routing (`model_provider.py`)

The same agent code runs locally and on the Space, but the **LLM backend** changes:

| Where | Default backend | How |
|-------|-----------------|-----|
| **Your machine** | llama.cpp or Ollama | Local server, no cloud credits |
| **HF Space** | Groq (if `GROQ_API_KEY` set) | Fast cloud API, free tier; throttled + auto-retry on 429 |
| **HF Space fallback** | Cerebras → Google Gemini | When Groq limits hit; add keys only — models are chosen automatically |
| **HF Space fallback** | Hugging Face Inference | Needs `HF_TOKEN`; monthly credits often run out (402 error) |

Set `LLM_PROVIDER` explicitly to override: `ollama`, `llamacpp`, `groq`, `cerebras`, `google`, or `hf`.

**Provider rotation** (`PROVIDER_FALLBACK_ENABLED=1`, default on): Cerebras models first, then Gemini, then Groq. Order via `PROVIDER_FALLBACK_ORDER=cerebras,google,groq`. Hard limits (TPD, 413, context) skip to the next slot immediately (default **5s** between calls).

---

## Project map

```
Final_Assignment_Template/
├── agent/
│   ├── gaia_agent.py      ← public API (GaiaAgent)
│   ├── pipeline.py        ← orchestrates the full flow
│   ├── agent_runner.py    ← CodeAgent + system prompt
│   ├── tools.py           ← extended tools
│   ├── verifier.py        ← answer extraction + checks
│   ├── planner.py         ← optional planning (standard/full)
│   ├── retriever.py       ← optional strategy hints
│   ├── think_mode.py      ← per-question reasoning toggle
│   ├── self_correction.py ← full-depth correction loops
│   ├── voting.py          ← full-depth majority vote
│   ├── evolve.py          ← post-eval memory (SELF_EVOLVE=1)
│   ├── code_interpreter.py← local Python/bash execution
│   ├── memory/store.py    ← experience.jsonl persistence
│   └── artifacts/writer.py← plan/notes/evidence files
├── tools.py               ← base tools (search, files, audio, …)
├── model_provider.py      ← LLM backend selection
├── provider_chain.py      ← cross-provider fallback (Groq → Cerebras → Gemini)
├── groq_model.py          ← throttling, retry, Groq model chain
├── scoring.py             ← normalize_answer
├── file_resolver.py       ← download attachments from scoring API
├── app.py                 ← Gradio Space UI
├── run_local.py           ← CLI runner + score mode
└── eval/                  ← fixtures, course client, run_eval.py
```

---

## Data policy

- **Allowed:** scoring API, local fixtures you write, your own run history for strategy hints
- **Not used:** bulk benchmark downloads, storing ground-truth answers for retrieval

---

## Configuration cheat sheet

See `.env.example` for the full list. The most important knobs:

| Variable | Effect |
|----------|--------|
| `LLM_PROVIDER` | Which LLM backend to use |
| `OLLAMA_MODEL` / `GROQ_API_KEY` | Local or Space model access |
| `GROQ_MODEL` | Groq model id (Space default: Scout 17B) |
| `GROQ_FALLBACK_ENABLED` | Auto-switch Groq models after hard limits (default 1) |
| `GROQ_MODEL_FALLBACK_CHAIN` | Comma-separated Groq fallback model ids |
| `CEREBRAS_API_KEY` / `GOOGLE_API_KEY` | Cross-provider fallback after Groq (no model config needed) |
| `PROVIDER_FALLBACK_ORDER` | Provider order, e.g. `cerebras,google,groq` |
| `GROQ_MIN_REQUEST_INTERVAL` | Seconds between cloud API calls (default 5) |
| `GROQ_MAX_RETRIES` | Retries on 429 (default 5) |
| `HF_USERNAME` | Your HF username for API submit |
| `PIPELINE_DEPTH` | `minimal` / `standard` / `full` |
| `THINK_MODE` | `auto` / `on` / `off` |
| `AGENT_MAX_STEPS` | Max tool loops per question (default 12) |
| `RETRIEVER_ENABLED` | Use past strategy hints |
| `SELF_EVOLVE` | Record learning after graded runs |

---

## Mental model for debugging

When something goes wrong, ask:

1. **Model errors (402, 404, Groq)?** → Check `model_provider.py` and Space secrets
2. **Wrong answer but agent ran?** → Read Space logs or `artifacts/` trace; verifier may have stripped a bad format
3. **Attachment ignored?** → Check file downloaded; prompt includes path and suffix hint
4. **Timeout / too slow?** → Reduce `AGENT_MAX_STEPS`, use faster model, or `PIPELINE_DEPTH=minimal`
5. **Score 0% locally but API says higher?** → Local summary uses submit response; ensure `HF_USERNAME` is set

---

## Testing without burning LLM credits

```bash
pytest tests/ -q                              # unit tests, mocked LLM
python eval/run_eval.py --mode fixtures       # hand-written Q&A pairs
python run_local.py --mode single             # one cheap smoke question
```

Fixtures live in `eval/fixtures/` — you control the expected answers.