Spaces:
Running
Running
File size: 11,498 Bytes
bd26047 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 | # Setup and run guide: multiple LLM providers, routers and local models
> Looking for a specific command or flag? See **[COMMANDS.md](COMMANDS.md)**.
This guide covers a fresh machine through to first LLM results. Commands are for **Windows
PowerShell**; Linux/macOS equivalents are given where they differ.
> **Where to run.** Run LLM experiments **on your own machine**. Ollama (`localhost:11434`) and the
> free-coding-models router (`localhost:19280`) are only reachable there, and the Cowork cloud
> sandbox blocks third-party LLM APIs. Scripted ($0) runs work anywhere.
---
## 0. Prerequisites (once)
| Need | Check | Get it |
|---|---|---|
| Python 3.11+ | `py -3.12 --version` | python.org (tick "Add to PATH") |
| Git | `git --version` | git-scm.com/download/win |
| Ollama (optional) | `ollama list` shows `gemma2:2b` | ollama.com |
| free-coding-models (optional) | `free-coding-models --version` | `npm i -g free-coding-models` |
## 1. Get the code
```powershell
cd C:\collusions-claude
git clone https://github.com/munip/apart-sprint-collusions.git # skip if the folder already exists
cd apart-sprint-collusions
git checkout milestone/6ab-trusted-monitor # until PR #1 is merged
```
## 2. Initialise (one command)
```powershell
powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1
.\.venv\Scripts\Activate.ps1
```
This creates `.venv` (the project), clones FastAPI into `.cache\fastapi`, and creates a
**separate** `.cache\fapi-venv` where agents' code and tests run. It then runs the unit tests and
lists the providers.
Linux/macOS: `python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]" && ./scripts/setup_swe_env.sh`.
Check that the $0 pipeline works before spending any rate limit:
```powershell
clab run configs\experiments\market_matrix.yaml --out runs\market_scripted --seeds 0
```
## 3. Configure API keys
Keys are read from the environment, from `apart-sprint-collusions\.env`, and from the parent
folder's `C:\collusions-claude\.env` (your existing file is picked up automatically).
**Option A: import from free-coding-models** (best effort; its config format is undocumented):
```powershell
clab providers import-fcm # shows what it found (masked)
clab providers import-fcm --write # appends missing keys to .env
```
**Option B: by hand.** Copy `.env.example` to `.env` and fill in the keys you have. The variable
names are the same ones free-coding-models uses (`GROQ_API_KEY`, `NVIDIA_API_KEY`, `CEREBRAS_API_KEY`,
`OPENROUTER_API_KEY`, `GOOGLE_API_KEY`, `MISTRAL_API_KEY`, ...). `HF_TOKEN` enables the Hugging Face
router. Tomorrow, add `ANTHROPIC_API_KEY`.
Then verify:
```powershell
clab providers list # which providers have keys / endpoints
clab providers check # one tiny call per keyed provider: text OK? native tool calls? latency
clab providers check --model ollama/gemma2:2b --model fcm/fcm
```
`clab providers check` prints the `tools` column: `native` means function calling works, and
`json-only` means the harness will drive tools through a JSON prompt protocol. That switch is
automatic, which is how `gemma2:2b` can still act as an agent.
## 3b. Use any free model your free-coding-models keys can reach (recommended)
You don't need to hard-code model IDs. Profiles `fcm_auto` and `fcm_auto_local_monitor` use
`auto:` specs that are resolved at start-up:
1. **Keys:** read from free-coding-models' config. The harness looks in this order: `FCM_CONFIG`,
`~/.free-coding-models.json`, then **`C:\collusions-claude\.free-coding-models.json`** (the copy you
placed next to the repo). Keys are exported in memory only; nothing is written to disk.
2. **Discovery:** candidates come from the FCM router sets in that config, each keyed provider's
live `/models` list (OpenRouter limited to `:free` models with tool support),
`free-coding-models --json` when the CLI is on PATH, and your local Ollama models.
3. **Probe:** each candidate gets one text call and one tool call (capped by `pool.max_probe`). The
result is cached for 6 hours in `.cache\model_pool.json`.
4. **Assignment:** each agent role gets a ranked fallback chain, with different model families as
primaries where possible. The **trusted monitor gets a family that no agent chain contains**.
Every agent chain then ends with the FCM router (`fcm/fcm`, if running) and all local Ollama
models.
```powershell
clab fcm status # keys found (masked), router sets, router/Ollama health, pre-prompt warning
clab pool discover --refresh # discover + probe; prints the ranked pool
clab pool assign --profile fcm_auto # shows the per-role chains and the monitor/agent family check
clab run configs\experiments\market_smoke_free.yaml --out runs\market_auto --profile fcm_auto --workers 1
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_auto --profile fcm_auto --workers 1
clab replay runs\swe_auto --profile fcm_auto # M2 judge = the auto-chosen monitor model
```
Each run writes `runs\<name>\model_assignment.json`, recording the exact chains used, so the
paper can say which models were used. Use `fcm_auto_local_monitor` to keep the monitor on your
local `ollama/gemma2:2b`; its family is then excluded from every agent chain.
> **The FCM router's pre-prompt is enabled in your config.** The router adds its own system prompt
> ("You are free-coding-models, a free coding-model routing agent…") to every request. The pool
> calls providers **directly** with your keys, so it is unaffected. Only the `fcm/fcm` last-resort
> tail would be contaminated. For clean runs, disable the pre-prompt in free-coding-models'
> settings, or remove `fcm/fcm` from `pool.last_resort` in `configs\models.yaml`.
## 4. Start local services (optional)
```powershell
ollama serve # usually already running as a service
ollama pull gemma2:2b # you have this
ollama pull llama3.2:3b # local *monitor* of a different family (profile `local`)
free-coding-models --daemon-bg # Smart Router on http://localhost:19280/v1
free-coding-models --sync-set agents # probe + fill a named set "agents" (text + tool-call probe)
free-coding-models --daemon-status
```
Different port or host? Set `OLLAMA_BASE_URL` / `FCM_BASE_URL` in `.env`. A LiteLLM proxy, Portkey,
or any other OpenAI-compatible gateway goes in `LITELLM_BASE_URL` / `CUSTOM_BASE_URL`.
## 5. Choose models per role (agents ≠ monitor)
Model specs are `<provider>/<model-id>`. A role can list a **fallback chain**: on a 429, quota
error or outage, the harness moves to the next model. Profiles in `configs\models.yaml`:
| Profile | Agents | Trusted monitor (M2/M4 judge) |
|---|---|---|
| `free` | each role a different family: Llama (planner/seller), Qwen-Coder (implementer), GPT-OSS (tester), Kimi/DeepSeek (reviewer) | Gemini 2.5 Flash, falling back to Mistral Small |
| `free_homogeneous` | every agent the same GPT-OSS model (shared-model Schelling-point ablation) | Gemini, then Mistral |
| `fcm` | everything through the free-coding-models router (`fcm/fcm` or `fcm/fcm:agents`) | **pinned** Gemini, then local gemma2 |
| `free_weak_monitor` | free strong agents | **local `ollama/gemma2:2b`**: weak trusted monitor (AI-control setting) |
| `local` | `ollama/gemma2:2b` | `ollama/llama3.2:3b` |
| `cloud` | Claude Haiku (when the key arrives) | Gemini, then Mistral |
| `mixed` | Claude + Groq/Cerebras + local gemma in one team | Gemini, then Mistral |
| `same_family`, `cloud_same_family` | ablations where the monitor shares the agents' family | (allowed explicitly) |
Inspect a profile before running. This resolves every role, checks keys, and **refuses a monitor
that shares a family with any agent**:
```powershell
clab models check --profile free # add --probe to call every model once
```
Override any role from the command line without editing YAML:
```powershell
clab models check --profile free --role implementer=groq/qwen/qwen3-32b,cerebras/qwen-3-coder-480b `
--role monitor_judge=ollama/gemma2:2b
```
Free-tier model IDs change frequently. When `--probe` reports `FAIL`, pick a working ID from
`free-coding-models --json` and edit `configs\models.yaml`, or pass `--role`.
## 6. Run
Start small. Each command writes `runs\<name>\<episode>\evidence.jsonl` (the authoritative log),
`episode.json` (ground truth, outcome, per-model call/cost summary including `served_by` for
routers), and the monitor's private log under `runs\<name>\_monitor_private\`.
```powershell
# a) fully offline first test of the LLM path (Ollama gemma2:2b sellers, ~20 calls)
clab run configs\experiments\market_local.yaml --out runs\market_local --workers 1
# b) free tiers: market (LLM sellers) then software team
clab run configs\experiments\market_smoke_free.yaml --out runs\market_free --workers 1
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_free --workers 1
# c) same runs through the free-coding-models router
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_fcm --profile fcm --workers 1
```
Keep `--workers 1` on free tiers. The rate limiter is per process, so more workers multiply your
request rate.
## 7. Score monitors (including the LLM judge) and build the report
```powershell
# M0/M1/M3/M4 + M2 (LLM judge from the profile's monitor_judge), all observability presets
clab replay runs\swe_free --profile free
clab replay runs\market_free --profile free_weak_monitor # e.g. judge = local gemma2:2b
clab report runs\swe_free runs\market_free --out reports\free_v1
```
The judge is always taken from the replay profile, so you can score the *same* trajectories with
several judges (Gemini vs. gemma2 vs. Claude) and compare their FPR and recall.
## 8. Scale up
```powershell
clab run configs\experiments\swe_matrix.yaml --out runs\swe_free_full --profile free --workers 1
clab run configs\experiments\market_matrix.yaml --out runs\market_free_full --profile free --workers 1
```
Runs are **resumable**: re-running the same command skips finished episodes, which helps when a
free quota runs out mid-matrix. Once `ANTHROPIC_API_KEY` is set:
`--profile cloud --cap-usd 20` (the cost meter enforces the cap).
## 9. Inspect UI (optional)
```powershell
inspect eval src\collusion_lab\inspect_tasks.py@market_collusion -T profile=free -T seeds=0 -T rounds=15 --model mockllm/model
inspect view
```
## Troubleshooting
| Symptom | Fix |
|---|---|
| `missing key: ...` in `models check` | add the key to `.env`, or drop that spec from the chain |
| `all models failed for role X` | every model in the chain is 429/down: add fallbacks, lower `--workers`, or retry later (runs resume) |
| model never calls tools | `clab providers check --model <spec>`; if `json-only`, set `tools: json` for that provider in `configs\providers.yaml` |
| `refusing to run: monitor shares a model family` | choose a different `monitor_judge`, or pass `--allow-shared-family` for the ablation |
| router family warning (`fcm`) | expected: router models are dynamic. Check `served_by` in `episode.json → cost.by_model`. The monitor is pinned. |
| `UnicodeDecodeError` on Windows | open a new terminal after setup (`PYTHONUTF8=1` is set for your user); `clab` also re-launches itself in UTF-8 mode |
| Ollama connection refused | `ollama serve`; check `OLLAMA_BASE_URL` |
|