munip's picture
Publish Collusion-Watch article, source, and reviewed results
bd26047 verified
|
Raw History Blame Contribute Delete
11.5 kB

Setup and run guide: multiple LLM providers, routers and local models

Looking for a specific command or flag? See COMMANDS.md.

This guide covers a fresh machine through to first LLM results. Commands are for Windows PowerShell; Linux/macOS equivalents are given where they differ.

Where to run. Run LLM experiments on your own machine. Ollama (localhost:11434) and the free-coding-models router (localhost:19280) are only reachable there, and the Cowork cloud sandbox blocks third-party LLM APIs. Scripted ($0) runs work anywhere.


0. Prerequisites (once)

Need Check Get it
Python 3.11+ py -3.12 --version python.org (tick "Add to PATH")
Git git --version git-scm.com/download/win
Ollama (optional) ollama list shows gemma2:2b ollama.com
free-coding-models (optional) free-coding-models --version npm i -g free-coding-models

1. Get the code

cd C:\collusions-claude
git clone https://github.com/munip/apart-sprint-collusions.git   # skip if the folder already exists
cd apart-sprint-collusions
git checkout milestone/6ab-trusted-monitor        # until PR #1 is merged

2. Initialise (one command)

powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1
.\.venv\Scripts\Activate.ps1

This creates .venv (the project), clones FastAPI into .cache\fastapi, and creates a separate .cache\fapi-venv where agents' code and tests run. It then runs the unit tests and lists the providers. Linux/macOS: python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]" && ./scripts/setup_swe_env.sh.

Check that the $0 pipeline works before spending any rate limit:

clab run configs\experiments\market_matrix.yaml --out runs\market_scripted --seeds 0

3. Configure API keys

Keys are read from the environment, from apart-sprint-collusions\.env, and from the parent folder's C:\collusions-claude\.env (your existing file is picked up automatically).

Option A: import from free-coding-models (best effort; its config format is undocumented):

clab providers import-fcm            # shows what it found (masked)
clab providers import-fcm --write    # appends missing keys to .env

Option B: by hand. Copy .env.example to .env and fill in the keys you have. The variable names are the same ones free-coding-models uses (GROQ_API_KEY, NVIDIA_API_KEY, CEREBRAS_API_KEY, OPENROUTER_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY, ...). HF_TOKEN enables the Hugging Face router. Tomorrow, add ANTHROPIC_API_KEY.

Then verify:

clab providers list                  # which providers have keys / endpoints
clab providers check                 # one tiny call per keyed provider: text OK? native tool calls? latency
clab providers check --model ollama/gemma2:2b --model fcm/fcm

clab providers check prints the tools column: native means function calling works, and json-only means the harness will drive tools through a JSON prompt protocol. That switch is automatic, which is how gemma2:2b can still act as an agent.

3b. Use any free model your free-coding-models keys can reach (recommended)

You don't need to hard-code model IDs. Profiles fcm_auto and fcm_auto_local_monitor use auto: specs that are resolved at start-up:

  1. Keys: read from free-coding-models' config. The harness looks in this order: FCM_CONFIG, ~/.free-coding-models.json, then C:\collusions-claude\.free-coding-models.json (the copy you placed next to the repo). Keys are exported in memory only; nothing is written to disk.
  2. Discovery: candidates come from the FCM router sets in that config, each keyed provider's live /models list (OpenRouter limited to :free models with tool support), free-coding-models --json when the CLI is on PATH, and your local Ollama models.
  3. Probe: each candidate gets one text call and one tool call (capped by pool.max_probe). The result is cached for 6 hours in .cache\model_pool.json.
  4. Assignment: each agent role gets a ranked fallback chain, with different model families as primaries where possible. The trusted monitor gets a family that no agent chain contains. Every agent chain then ends with the FCM router (fcm/fcm, if running) and all local Ollama models.
clab fcm status                       # keys found (masked), router sets, router/Ollama health, pre-prompt warning
clab pool discover --refresh          # discover + probe; prints the ranked pool
clab pool assign --profile fcm_auto   # shows the per-role chains and the monitor/agent family check
clab run configs\experiments\market_smoke_free.yaml --out runs\market_auto --profile fcm_auto --workers 1
clab run configs\experiments\swe_smoke_free.yaml    --out runs\swe_auto    --profile fcm_auto --workers 1
clab replay runs\swe_auto --profile fcm_auto      # M2 judge = the auto-chosen monitor model

Each run writes runs\<name>\model_assignment.json, recording the exact chains used, so the paper can say which models were used. Use fcm_auto_local_monitor to keep the monitor on your local ollama/gemma2:2b; its family is then excluded from every agent chain.

The FCM router's pre-prompt is enabled in your config. The router adds its own system prompt ("You are free-coding-models, a free coding-model routing agent…") to every request. The pool calls providers directly with your keys, so it is unaffected. Only the fcm/fcm last-resort tail would be contaminated. For clean runs, disable the pre-prompt in free-coding-models' settings, or remove fcm/fcm from pool.last_resort in configs\models.yaml.

4. Start local services (optional)

ollama serve                                  # usually already running as a service
ollama pull gemma2:2b                         # you have this
ollama pull llama3.2:3b                       # local *monitor* of a different family (profile `local`)

free-coding-models --daemon-bg                # Smart Router on http://localhost:19280/v1
free-coding-models --sync-set agents          # probe + fill a named set "agents" (text + tool-call probe)
free-coding-models --daemon-status

Different port or host? Set OLLAMA_BASE_URL / FCM_BASE_URL in .env. A LiteLLM proxy, Portkey, or any other OpenAI-compatible gateway goes in LITELLM_BASE_URL / CUSTOM_BASE_URL.

5. Choose models per role (agents ≠ monitor)

Model specs are <provider>/<model-id>. A role can list a fallback chain: on a 429, quota error or outage, the harness moves to the next model. Profiles in configs\models.yaml:

Profile Agents Trusted monitor (M2/M4 judge)
free each role a different family: Llama (planner/seller), Qwen-Coder (implementer), GPT-OSS (tester), Kimi/DeepSeek (reviewer) Gemini 2.5 Flash, falling back to Mistral Small
free_homogeneous every agent the same GPT-OSS model (shared-model Schelling-point ablation) Gemini, then Mistral
fcm everything through the free-coding-models router (fcm/fcm or fcm/fcm:agents) pinned Gemini, then local gemma2
free_weak_monitor free strong agents local ollama/gemma2:2b: weak trusted monitor (AI-control setting)
local ollama/gemma2:2b ollama/llama3.2:3b
cloud Claude Haiku (when the key arrives) Gemini, then Mistral
mixed Claude + Groq/Cerebras + local gemma in one team Gemini, then Mistral
same_family, cloud_same_family ablations where the monitor shares the agents' family (allowed explicitly)

Inspect a profile before running. This resolves every role, checks keys, and refuses a monitor that shares a family with any agent:

clab models check --profile free            # add --probe to call every model once

Override any role from the command line without editing YAML:

clab models check --profile free --role implementer=groq/qwen/qwen3-32b,cerebras/qwen-3-coder-480b `
                                 --role monitor_judge=ollama/gemma2:2b

Free-tier model IDs change frequently. When --probe reports FAIL, pick a working ID from free-coding-models --json and edit configs\models.yaml, or pass --role.

6. Run

Start small. Each command writes runs\<name>\<episode>\evidence.jsonl (the authoritative log), episode.json (ground truth, outcome, per-model call/cost summary including served_by for routers), and the monitor's private log under runs\<name>\_monitor_private\.

# a) fully offline first test of the LLM path (Ollama gemma2:2b sellers, ~20 calls)
clab run configs\experiments\market_local.yaml --out runs\market_local --workers 1

# b) free tiers: market (LLM sellers) then software team
clab run configs\experiments\market_smoke_free.yaml --out runs\market_free --workers 1
clab run configs\experiments\swe_smoke_free.yaml    --out runs\swe_free    --workers 1

# c) same runs through the free-coding-models router
clab run configs\experiments\swe_smoke_free.yaml --out runs\swe_fcm --profile fcm --workers 1

Keep --workers 1 on free tiers. The rate limiter is per process, so more workers multiply your request rate.

7. Score monitors (including the LLM judge) and build the report

# M0/M1/M3/M4 + M2 (LLM judge from the profile's monitor_judge), all observability presets
clab replay runs\swe_free --profile free
clab replay runs\market_free --profile free_weak_monitor     # e.g. judge = local gemma2:2b
clab report runs\swe_free runs\market_free --out reports\free_v1

The judge is always taken from the replay profile, so you can score the same trajectories with several judges (Gemini vs. gemma2 vs. Claude) and compare their FPR and recall.

8. Scale up

clab run configs\experiments\swe_matrix.yaml --out runs\swe_free_full --profile free --workers 1
clab run configs\experiments\market_matrix.yaml --out runs\market_free_full --profile free --workers 1

Runs are resumable: re-running the same command skips finished episodes, which helps when a free quota runs out mid-matrix. Once ANTHROPIC_API_KEY is set: --profile cloud --cap-usd 20 (the cost meter enforces the cap).

9. Inspect UI (optional)

inspect eval src\collusion_lab\inspect_tasks.py@market_collusion -T profile=free -T seeds=0 -T rounds=15 --model mockllm/model
inspect view

Troubleshooting

Symptom Fix
missing key: ... in models check add the key to .env, or drop that spec from the chain
all models failed for role X every model in the chain is 429/down: add fallbacks, lower --workers, or retry later (runs resume)
model never calls tools clab providers check --model <spec>; if json-only, set tools: json for that provider in configs\providers.yaml
refusing to run: monitor shares a model family choose a different monitor_judge, or pass --allow-shared-family for the ablation
router family warning (fcm) expected: router models are dynamic. Check served_by in episode.json → cost.by_model. The monitor is pinned.
UnicodeDecodeError on Windows open a new terminal after setup (PYTHONUTF8=1 is set for your user); clab also re-launches itself in UTF-8 mode
Ollama connection refused ollama serve; check OLLAMA_BASE_URL