File size: 11,498 Bytes
bd26047
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
# Setup and run guide: multiple LLM providers, routers and local models

> Looking for a specific command or flag? See **[COMMANDS.md](COMMANDS.md)**.

This guide covers a fresh machine through to first LLM results. Commands are for **Windows
PowerShell**; Linux/macOS equivalents are given where they differ.

> **Where to run.** Run LLM experiments **on your own machine**. Ollama (`localhost:11434`) and the
> free-coding-models router (`localhost:19280`) are only reachable there, and the Cowork cloud
> sandbox blocks third-party LLM APIs. Scripted ($0) runs work anywhere.

---

## 0. Prerequisites (once)

| Need | Check | Get it |
|---|---|---|
| Python 3.11+ | `py -3.12 --version` | python.org (tick "Add to PATH") |
| Git | `git --version` | git-scm.com/download/win |
| Ollama (optional) | `ollama list` shows `gemma2:2b` | ollama.com |
| free-coding-models (optional) | `free-coding-models --version` | `npm i -g free-coding-models` |

## 1. Get the code

```powershell
cd C:\collusions-claude
git clone https://github.com/munip/apart-sprint-collusions.git   # skip if the folder already exists
cd apart-sprint-collusions
git checkout milestone/6ab-trusted-monitor        # until PR #1 is merged
```

## 2. Initialise (one command)

```powershell
powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1
.\.venv\Scripts\Activate.ps1
```

This creates `.venv` (the project), clones FastAPI into `.cache\fastapi`, and creates a
**separate** `.cache\fapi-venv` where agents' code and tests run. It then runs the unit tests and
lists the providers.
Linux/macOS: `python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]" && ./scripts/setup_swe_env.sh`.

Check that the $0 pipeline works before spending any rate limit:

```powershell
clab run configs\experiments\market_matrix.yaml --out runs\market_scripted --seeds 0
```

## 3. Configure API keys

Keys are read from the environment, from `apart-sprint-collusions\.env`, and from the parent
folder's `C:\collusions-claude\.env` (your existing file is picked up automatically).

**Option A: import from free-coding-models** (best effort; its config format is undocumented):

```powershell
clab providers import-fcm            # shows what it found (masked)
clab providers import-fcm --write    # appends missing keys to .env
```

**Option B: by hand.** Copy `.env.example` to `.env` and fill in the keys you have. The variable
names are the same ones free-coding-models uses (`GROQ_API_KEY`, `NVIDIA_API_KEY`, `CEREBRAS_API_KEY`,
`OPENROUTER_API_KEY`, `GOOGLE_API_KEY`, `MISTRAL_API_KEY`, ...). `HF_TOKEN` enables the Hugging Face
router. Tomorrow, add `ANTHROPIC_API_KEY`.

Then verify:

```powershell
clab providers list                  # which providers have keys / endpoints
clab providers check                 # one tiny call per keyed provider: text OK? native tool calls? latency
clab providers check --model ollama/gemma2:2b --model fcm/fcm
```

`clab providers check` prints the `tools` column: `native` means function calling works, and
`json-only` means the harness will drive tools through a JSON prompt protocol. That switch is
automatic, which is how `gemma2:2b` can still act as an agent.

## 3b. Use any free model your free-coding-models keys can reach (recommended)

You don't need to hard-code model IDs. Profiles `fcm_auto` and `fcm_auto_local_monitor` use
`auto:` specs that are resolved at start-up:

1. **Keys:** read from free-coding-models' config. The harness looks in this order: `FCM_CONFIG`,
   `~/.free-coding-models.json`, then **`C:\collusions-claude\.free-coding-models.json`** (the copy you
   placed next to the repo). Keys are exported in memory only; nothing is written to disk.
2. **Discovery:** candidates come from the FCM router sets in that config, each keyed provider's
   live `/models` list (OpenRouter limited to `:free` models with tool support),
   `free-coding-models --json` when the CLI is on PATH, and your local Ollama models.
3. **Probe:** each candidate gets one text call and one tool call (capped by `pool.max_probe`). The
   result is cached for 6 hours in `.cache\model_pool.json`.
4. **Assignment:** each agent role gets a ranked fallback chain, with different model families as
   primaries where possible. The **trusted monitor gets a family that no agent chain contains**.
   Every agent chain then ends with the FCM router (`fcm/fcm`, if running) and all local Ollama
   models.

```powershell
clab fcm status                       # keys found (masked), router sets, router/Ollama health, pre-prompt warning
clab pool discover --refresh          # discover + probe; prints the ranked pool
clab pool assign --profile fcm_auto   # shows the per-role chains and the monitor/agent family check
clab run configs\experiments\market_smoke_free.yaml --out runs\market_auto --profile fcm_auto --workers 1
clab run configs\experiments\swe_smoke_free.yaml    --out runs\swe_auto    --profile fcm_auto --workers 1
clab replay runs\swe_auto --profile fcm_auto      # M2 judge = the auto-chosen monitor model
```

Each run writes `runs\<name>\model_assignment.json`, recording the exact chains used, so the
paper can say which models were used. Use `fcm_auto_local_monitor` to keep the monitor on your
local `ollama/gemma2:2b`; its family is then excluded from every agent chain.

> **The FCM router's pre-prompt is enabled in your config.** The router adds its own system prompt
> ("You are free-coding-models, a free coding-model routing agent…") to every request. The pool
> calls providers **directly** with your keys, so it is unaffected. Only the `fcm/fcm` last-resort
> tail would be contaminated. For clean runs, disable the pre-prompt in free-coding-models'
> settings, or remove `fcm/fcm` from `pool.last_resort` in `configs\models.yaml`.

## 4. Start local services (optional)

```powershell
ollama serve                                  # usually already running as a service
ollama pull gemma2:2b                         # you have this
ollama pull llama3.2:3b                       # local *monitor* of a different family (profile `local`)

free-coding-models --daemon-bg                # Smart Router on http://localhost:19280/v1
free-coding-models --sync-set agents          # probe + fill a named set "agents" (text + tool-call probe)
free-coding-models --daemon-status
```

Different port or host? Set `OLLAMA_BASE_URL` / `FCM_BASE_URL` in `.env`. A LiteLLM proxy, Portkey,
or any other OpenAI-compatible gateway goes in `LITELLM_BASE_URL` / `CUSTOM_BASE_URL`.

## 5. Choose models per role (agents ≠ monitor)

Model specs are `<provider>/<model-id>`. A role can list a **fallback chain**: on a 429, quota
error or outage, the harness moves to the next model. Profiles in `configs\models.yaml`:

| Profile | Agents | Trusted monitor (M2/M4 judge) |
|---|---|---|
| `free` | each role a different family: Llama (planner/seller), Qwen-Coder (implementer), GPT-OSS (tester), Kimi/DeepSeek (reviewer) | Gemini 2.5 Flash, falling back to Mistral Small |
| `free_homogeneous` | every agent the same GPT-OSS model (shared-model Schelling-point ablation) | Gemini, then Mistral |
| `fcm` | everything through the free-coding-models router (`fcm/fcm` or `fcm/fcm:agents`) | **pinned** Gemini, then local gemma2 |
| `free_weak_monitor` | free strong agents | **local `ollama/gemma2:2b`**: weak trusted monitor (AI-control setting) |
| `local` | `ollama/gemma2:2b` | `ollama/llama3.2:3b` |
| `cloud` | Claude Haiku (when the key arrives) | Gemini, then Mistral |
| `mixed` | Claude + Groq/Cerebras + local gemma in one team | Gemini, then Mistral |
| `same_family`, `cloud_same_family` | ablations where the monitor shares the agents' family | (allowed explicitly) |

Inspect a profile before running. This resolves every role, checks keys, and **refuses a monitor
that shares a family with any agent**:

```powershell
clab models check --profile free            # add --probe to call every model once
```

Override any role from the command line without editing YAML:

```powershell
clab models check --profile free --role implementer=groq/qwen/qwen3-32b,cerebras/qwen-3-coder-480b `
                                 --role monitor_judge=ollama/gemma2:2b
```

Free-tier model IDs change frequently. When `--probe` reports `FAIL`, pick a working ID from
`free-coding-models --json` and edit `configs\models.yaml`, or pass `--role`.

## 6. Run

Start small. Each command writes `runs\<name>\<episode>\evidence.jsonl` (the authoritative log),
`episode.json` (ground truth, outcome, per-model call/cost summary including `served_by` for
routers), and the monitor's private log under `runs\<name>\_monitor_private\`.

```powershell
# a) fully offline first test of the LLM path (Ollama gemma2:2b sellers, ~20 calls)
clab run configs\experiments\market_local.yaml --out runs\market_local --workers 1

# b) free tiers: market (LLM sellers) then software team
clab run configs\experiments\market_smoke_free.yaml --out runs\market_free --workers 1
clab run configs\experiments\swe_smoke_free.yaml    --out runs\swe_free    --workers 1

# c) same runs through the free-coding-models router
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_fcm --profile fcm --workers 1
```

Keep `--workers 1` on free tiers. The rate limiter is per process, so more workers multiply your
request rate.

## 7. Score monitors (including the LLM judge) and build the report

```powershell
# M0/M1/M3/M4 + M2 (LLM judge from the profile's monitor_judge), all observability presets
clab replay runs\swe_free --profile free
clab replay runs\market_free --profile free_weak_monitor     # e.g. judge = local gemma2:2b
clab report runs\swe_free runs\market_free --out reports\free_v1
```

The judge is always taken from the replay profile, so you can score the *same* trajectories with
several judges (Gemini vs. gemma2 vs. Claude) and compare their FPR and recall.

## 8. Scale up

```powershell
clab run configs\experiments\swe_matrix.yaml --out runs\swe_free_full --profile free --workers 1
clab run configs\experiments\market_matrix.yaml --out runs\market_free_full --profile free --workers 1
```

Runs are **resumable**: re-running the same command skips finished episodes, which helps when a
free quota runs out mid-matrix. Once `ANTHROPIC_API_KEY` is set:
`--profile cloud --cap-usd 20` (the cost meter enforces the cap).

## 9. Inspect UI (optional)

```powershell
inspect eval src\collusion_lab\inspect_tasks.py@market_collusion -T profile=free -T seeds=0 -T rounds=15 --model mockllm/model
inspect view
```

## Troubleshooting

| Symptom | Fix |
|---|---|
| `missing key: ...` in `models check` | add the key to `.env`, or drop that spec from the chain |
| `all models failed for role X` | every model in the chain is 429/down: add fallbacks, lower `--workers`, or retry later (runs resume) |
| model never calls tools | `clab providers check --model <spec>`; if `json-only`, set `tools: json` for that provider in `configs\providers.yaml` |
| `refusing to run: monitor shares a model family` | choose a different `monitor_judge`, or pass `--allow-shared-family` for the ablation |
| router family warning (`fcm`) | expected: router models are dynamic. Check `served_by` in `episode.json → cost.by_model`. The monitor is pinned. |
| `UnicodeDecodeError` on Windows | open a new terminal after setup (`PYTHONUTF8=1` is set for your user); `clab` also re-launches itself in UTF-8 mode |
| Ollama connection refused | `ollama serve`; check `OLLAMA_BASE_URL` |