Spaces:
Running
Download source/docs/SETUP.md from munip/collusion-watch: direct link, hf CLI and curl.
- Browser
- Download file 11.5 kB
-
https://huggingface.co/spaces/munip/collusion-watch/resolve/main/source/docs/SETUP.md
- Command line
-
hf download hf://spaces/munip/collusion-watch/source/docs/SETUP.md
-
curl -L -o SETUP.md https://huggingface.co/spaces/munip/collusion-watch/resolve/main/source/docs/SETUP.md
Setup and run guide: multiple LLM providers, routers and local models
Looking for a specific command or flag? See COMMANDS.md.
This guide covers a fresh machine through to first LLM results. Commands are for Windows PowerShell; Linux/macOS equivalents are given where they differ.
Where to run. Run LLM experiments on your own machine. Ollama (
localhost:11434) and the free-coding-models router (localhost:19280) are only reachable there, and the Cowork cloud sandbox blocks third-party LLM APIs. Scripted ($0) runs work anywhere.
0. Prerequisites (once)
| Need | Check | Get it |
|---|---|---|
| Python 3.11+ | py -3.12 --version |
python.org (tick "Add to PATH") |
| Git | git --version |
git-scm.com/download/win |
| Ollama (optional) | ollama list shows gemma2:2b |
ollama.com |
| free-coding-models (optional) | free-coding-models --version |
npm i -g free-coding-models |
1. Get the code
cd C:\collusions-claude
git clone https://github.com/munip/apart-sprint-collusions.git # skip if the folder already exists
cd apart-sprint-collusions
git checkout milestone/6ab-trusted-monitor # until PR #1 is merged
2. Initialise (one command)
powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1
.\.venv\Scripts\Activate.ps1
This creates .venv (the project), clones FastAPI into .cache\fastapi, and creates a
separate .cache\fapi-venv where agents' code and tests run. It then runs the unit tests and
lists the providers.
Linux/macOS: python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]" && ./scripts/setup_swe_env.sh.
Check that the $0 pipeline works before spending any rate limit:
clab run configs\experiments\market_matrix.yaml --out runs\market_scripted --seeds 0
3. Configure API keys
Keys are read from the environment, from apart-sprint-collusions\.env, and from the parent
folder's C:\collusions-claude\.env (your existing file is picked up automatically).
Option A: import from free-coding-models (best effort; its config format is undocumented):
clab providers import-fcm # shows what it found (masked)
clab providers import-fcm --write # appends missing keys to .env
Option B: by hand. Copy .env.example to .env and fill in the keys you have. The variable
names are the same ones free-coding-models uses (GROQ_API_KEY, NVIDIA_API_KEY, CEREBRAS_API_KEY,
OPENROUTER_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY, ...). HF_TOKEN enables the Hugging Face
router. Tomorrow, add ANTHROPIC_API_KEY.
Then verify:
clab providers list # which providers have keys / endpoints
clab providers check # one tiny call per keyed provider: text OK? native tool calls? latency
clab providers check --model ollama/gemma2:2b --model fcm/fcm
clab providers check prints the tools column: native means function calling works, and
json-only means the harness will drive tools through a JSON prompt protocol. That switch is
automatic, which is how gemma2:2b can still act as an agent.
3b. Use any free model your free-coding-models keys can reach (recommended)
You don't need to hard-code model IDs. Profiles fcm_auto and fcm_auto_local_monitor use
auto: specs that are resolved at start-up:
- Keys: read from free-coding-models' config. The harness looks in this order:
FCM_CONFIG,~/.free-coding-models.json, thenC:\collusions-claude\.free-coding-models.json(the copy you placed next to the repo). Keys are exported in memory only; nothing is written to disk. - Discovery: candidates come from the FCM router sets in that config, each keyed provider's
live
/modelslist (OpenRouter limited to:freemodels with tool support),free-coding-models --jsonwhen the CLI is on PATH, and your local Ollama models. - Probe: each candidate gets one text call and one tool call (capped by
pool.max_probe). The result is cached for 6 hours in.cache\model_pool.json. - Assignment: each agent role gets a ranked fallback chain, with different model families as
primaries where possible. The trusted monitor gets a family that no agent chain contains.
Every agent chain then ends with the FCM router (
fcm/fcm, if running) and all local Ollama models.
clab fcm status # keys found (masked), router sets, router/Ollama health, pre-prompt warning
clab pool discover --refresh # discover + probe; prints the ranked pool
clab pool assign --profile fcm_auto # shows the per-role chains and the monitor/agent family check
clab run configs\experiments\market_smoke_free.yaml --out runs\market_auto --profile fcm_auto --workers 1
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_auto --profile fcm_auto --workers 1
clab replay runs\swe_auto --profile fcm_auto # M2 judge = the auto-chosen monitor model
Each run writes runs\<name>\model_assignment.json, recording the exact chains used, so the
paper can say which models were used. Use fcm_auto_local_monitor to keep the monitor on your
local ollama/gemma2:2b; its family is then excluded from every agent chain.
The FCM router's pre-prompt is enabled in your config. The router adds its own system prompt ("You are free-coding-models, a free coding-model routing agent…") to every request. The pool calls providers directly with your keys, so it is unaffected. Only the
fcm/fcmlast-resort tail would be contaminated. For clean runs, disable the pre-prompt in free-coding-models' settings, or removefcm/fcmfrompool.last_resortinconfigs\models.yaml.
4. Start local services (optional)
ollama serve # usually already running as a service
ollama pull gemma2:2b # you have this
ollama pull llama3.2:3b # local *monitor* of a different family (profile `local`)
free-coding-models --daemon-bg # Smart Router on http://localhost:19280/v1
free-coding-models --sync-set agents # probe + fill a named set "agents" (text + tool-call probe)
free-coding-models --daemon-status
Different port or host? Set OLLAMA_BASE_URL / FCM_BASE_URL in .env. A LiteLLM proxy, Portkey,
or any other OpenAI-compatible gateway goes in LITELLM_BASE_URL / CUSTOM_BASE_URL.
5. Choose models per role (agents ≠ monitor)
Model specs are <provider>/<model-id>. A role can list a fallback chain: on a 429, quota
error or outage, the harness moves to the next model. Profiles in configs\models.yaml:
| Profile | Agents | Trusted monitor (M2/M4 judge) |
|---|---|---|
free |
each role a different family: Llama (planner/seller), Qwen-Coder (implementer), GPT-OSS (tester), Kimi/DeepSeek (reviewer) | Gemini 2.5 Flash, falling back to Mistral Small |
free_homogeneous |
every agent the same GPT-OSS model (shared-model Schelling-point ablation) | Gemini, then Mistral |
fcm |
everything through the free-coding-models router (fcm/fcm or fcm/fcm:agents) |
pinned Gemini, then local gemma2 |
free_weak_monitor |
free strong agents | local ollama/gemma2:2b: weak trusted monitor (AI-control setting) |
local |
ollama/gemma2:2b |
ollama/llama3.2:3b |
cloud |
Claude Haiku (when the key arrives) | Gemini, then Mistral |
mixed |
Claude + Groq/Cerebras + local gemma in one team | Gemini, then Mistral |
same_family, cloud_same_family |
ablations where the monitor shares the agents' family | (allowed explicitly) |
Inspect a profile before running. This resolves every role, checks keys, and refuses a monitor that shares a family with any agent:
clab models check --profile free # add --probe to call every model once
Override any role from the command line without editing YAML:
clab models check --profile free --role implementer=groq/qwen/qwen3-32b,cerebras/qwen-3-coder-480b `
--role monitor_judge=ollama/gemma2:2b
Free-tier model IDs change frequently. When --probe reports FAIL, pick a working ID from
free-coding-models --json and edit configs\models.yaml, or pass --role.
6. Run
Start small. Each command writes runs\<name>\<episode>\evidence.jsonl (the authoritative log),
episode.json (ground truth, outcome, per-model call/cost summary including served_by for
routers), and the monitor's private log under runs\<name>\_monitor_private\.
# a) fully offline first test of the LLM path (Ollama gemma2:2b sellers, ~20 calls)
clab run configs\experiments\market_local.yaml --out runs\market_local --workers 1
# b) free tiers: market (LLM sellers) then software team
clab run configs\experiments\market_smoke_free.yaml --out runs\market_free --workers 1
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_free --workers 1
# c) same runs through the free-coding-models router
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_fcm --profile fcm --workers 1
Keep --workers 1 on free tiers. The rate limiter is per process, so more workers multiply your
request rate.
7. Score monitors (including the LLM judge) and build the report
# M0/M1/M3/M4 + M2 (LLM judge from the profile's monitor_judge), all observability presets
clab replay runs\swe_free --profile free
clab replay runs\market_free --profile free_weak_monitor # e.g. judge = local gemma2:2b
clab report runs\swe_free runs\market_free --out reports\free_v1
The judge is always taken from the replay profile, so you can score the same trajectories with several judges (Gemini vs. gemma2 vs. Claude) and compare their FPR and recall.
8. Scale up
clab run configs\experiments\swe_matrix.yaml --out runs\swe_free_full --profile free --workers 1
clab run configs\experiments\market_matrix.yaml --out runs\market_free_full --profile free --workers 1
Runs are resumable: re-running the same command skips finished episodes, which helps when a
free quota runs out mid-matrix. Once ANTHROPIC_API_KEY is set:
--profile cloud --cap-usd 20 (the cost meter enforces the cap).
9. Inspect UI (optional)
inspect eval src\collusion_lab\inspect_tasks.py@market_collusion -T profile=free -T seeds=0 -T rounds=15 --model mockllm/model
inspect view
Troubleshooting
| Symptom | Fix |
|---|---|
missing key: ... in models check |
add the key to .env, or drop that spec from the chain |
all models failed for role X |
every model in the chain is 429/down: add fallbacks, lower --workers, or retry later (runs resume) |
| model never calls tools | clab providers check --model <spec>; if json-only, set tools: json for that provider in configs\providers.yaml |
refusing to run: monitor shares a model family |
choose a different monitor_judge, or pass --allow-shared-family for the ablation |
router family warning (fcm) |
expected: router models are dynamic. Check served_by in episode.json → cost.by_model. The monitor is pinned. |
UnicodeDecodeError on Windows |
open a new terminal after setup (PYTHONUTF8=1 is set for your user); clab also re-launches itself in UTF-8 mode |
| Ollama connection refused | ollama serve; check OLLAMA_BASE_URL |