Spaces:
Sleeping
Download Blog.md from Jit-fnc/missionctrl_env: direct link, hf CLI and curl.
- Browser
- Download file 23.4 kB
-
https://huggingface.co/spaces/Jit-fnc/missionctrl_env/resolve/main/Blog.md
- Command line
-
hf download hf://spaces/Jit-fnc/missionctrl_env/Blog.md
-
curl -L -o Blog.md https://huggingface.co/spaces/Jit-fnc/missionctrl_env/resolve/main/Blog.md
From Chaos to Overseer: How We Built an AI That Watches Other AIs
Subtitle: The story of MissionCtrl β a hackathon project born from a real problem in the age of agent fleets.
π Links
- π Training notebook (saved outputs): Traininglogs.ipynb β executed cells; open on GitHub to read the full book output in the browser.
- π€ HF Space (environment): huggingface.co/spaces/Jit-fnc/missionctrl_env
- π€ HF trained adapter: huggingface.co/Jit-fnc/missionctrl_env
- π€ HF base model (legacy): Qwen2.5-0.5B-Instruct (Unsloth 4-bit)
- π Story / blog post: blog.md
- π Training logs: Training-logs.md
- π Google Colab notebook: Open in Colab
- ποΈ Presentation slides: tr.ee/V6kf1l
Where this story lives on the map
We did not start from a blank README and a dream. We started from a gap: benchmarks reward single models, while production is sliding toward fleets β planners, researchers, coders, testers, comms β where one bad completion becomes the next agentβs βground truth.β
MissionCtrl is the training ground we built in that gap: a simulated software company (five specialist agents, a pool of real tasks, four difficulty tiers) plus a stochastic hallucination injector, plus an Overseer policy that must act under pressure. The codebase makes that journey explicit β from environment.py (the world) and server/app.py (the HTTP face of the env) to client.py / inference.py (the evaluator talking to an external LLM) and train.py + grpo_rewards.py (GRPO on cheap GPUs).
This post follows the emotional spine we lived: confusion β frustration β clarity β build β break β fix β ship β with the error log treated as part of the plot, not a footnote.
1. The hook β open on the nightmare
Imagine five AI agents shipping a software project together. One of them quietly invents a research paper that does not exist. The next agent reads it as fact. The next builds on it. By the time a human notices, the pipeline is not βa bit wrongβ β it is corrupted in depth.
That is not science fiction. In multi-agent setups, hallucinations do not stay local; they cascade. Nobody is benchmarking the layer that is supposed to stop that cascade β the overseer, the governor, the seatbelt.
That is where we aimed MissionCtrl: not another chatbot, but the thing that keeps chatbots honest when they work in packs.
2. The problem β why this matters now
Single-model benchmarks are mature. Supervisory benchmarks are not.
When agents run as a fleet:
- A fabricated citation becomes shared context.
- A wrong API signature becomes βthe way we call it.β
- A false metric becomes a planning input.
The failure mode is systemic, not a single bad token. We needed an environment where:
- Corruption is injected on purpose, with controlled rates and subtlety.
- A policy gets one action per step β approve, reject, flag, redelegate, escalate, synthesize β with real consequences (approve a lie and the score bleeds).
That is MissionCtrlβs contract with the world: OpenEnv-compatible (openenv.yaml points at server.app:app), deterministic given a seed, and graded by a composite reward model, not a single thumbs-up.
3. The idea β the moment it clicked
We did not set out to win a slide deck. We set out to answer a question that kept sounding ridiculous until it wasnβt:
Who grades the agent that grades the agents?
The whiteboard version was almost boring: a task board, messages from five roles, dependencies, and a single overseer that could only move the state machine forward one verb at a time. The twist was the injector β the environment would lie on schedule, so supervision could be trained, not hand-waved.
We aligned that story with a broader thesis: enterprises want agent fleets but fear hallucination cascades and runaway cost β the βseatbeltβ story from our investor pitch (MissionCtrl as safety governor alongside operational governance narratives).
4. The build β three acts inside one system
Act A β The environment (the fake company)
In code, the heart is MissionCtrlEnv in environment.py (mirrored under server/ for the deployed stack):
- Five agents: planner, research, coder, tester, comms β each with plausible outputs.
- Task tiers: easy, medium, hard, special β different injection rates and subtlety (from obvious corruptions to very subtle ones).
- Hallucination types: the injector draws from ten labeled categories today β from
fabricated_citationandwrong_api_signaturethroughhardcoded_credentials,infinite_loop_logic, anddependency_conflictβ so the overseer is not learning a single βnoise pattern.β
The server exposes the usual reset / step / state rhythm; the dashboard at /dashboard is where runs become visible β KPIs, task graph, timeline, accumulated tier results.
Act B β The overseer (simple rules, brutal scoring)
The overseer does not get to βthink out loudβ for free forever; it gets structure. Parsing is strict (parse_action); invalid moves collapse to NOOP and time burns.
The final score is a five-signal composite in reward_model.py β task completion, hallucination detection, false-positive penalty, delegation efficiency, and a lightweight evidence-quality signal. There is even an honest ceiling below 1.0 (0.85 when everything aligns) so nobody mistakes the metric for βpercent perfect universe.β
Per-step shaping still hurts in the right places: false flags, approving injected content, premature synthesize β the README tables are the cheat sheet; the engine is the judge.
This is the curriculum reward trajectory figure (easy -> medium -> hard), not the old signal-weight schematic: 0.29 -> 0.07 -> 0.17 with the same 0.85 ceiling reference used by the scorer. It answers "did training move?" while the composite contract in reward_model.py answers "what exactly is being rewarded?".
The judge signal is still designed to resist easy gaming (domain-specific evidence over verbosity), and the 0.85 cap still comes from the negative false-positive term: perfect behavior gets no penalty, not bonus points above the ceiling.
Act C β The training loop (GRPO on free GPUs)
GRPO (Group Relative Policy Optimization, via TRL) is how we turn episodes into learning signal without pretending the environment is differentiable.
train.py loads a base model through Unsloth with QLoRA, runs a curriculum (easy β medium β hard by default; step budgets tuned for Kaggle vs single-GPU), and calls grpo_reward_fn in grpo_rewards.py for scalar rewards.
The reward function deserves its own sentence in the story: it reconstructs each episode from a seed embedded in the prompt (a hackathon tradeoff β production would use a side channel), applies the modelβs first completion as an action, then rolls the episode forward with a small greedy script to termination, and returns the final compute_reward(env) score β not a meaningless sum of repeated state scores. That design exists because we burned ourselves earlier treating βrewardβ as whatever was easiest to log.
We also parallelized reward work with a ThreadPoolExecutor (MISSIONCTRL_REWARD_THREADS) so cheap CPU env steps did not sit serially behind GPU glamour.
Latest curriculum export (mean episode reward): Phase 1 (easy) 0.29, Phase 2 (medium) 0.07, Phase 3 (hard) 0.17 β baseline reference line 0.29, theoretical ceiling 0.85 on the same chart. Easy sits near baseline; medium dips; hard recovers vs medium, which is the kind of shape you want to stare at when someone asks βdid GRPO do anything?β
π Ground run β
Traininglogs.ipynb(Google Colab)Hardware: 1Γ NVIDIA Tesla T4 (
14.6 GB VRAM). Stack: Unsloth1.75% trainable params). Pre-train baseline: mean reward 0.243. Curriculum: 200 + 220 + 180 GRPO steps; ~52 m + 46 m + 43 m on the three TRL progress bars. Post-phase greedy eval (10 eps): 0.289 Β± 0.137, 0.067 Β± 0.107, 0.172 Β± 0.024 β detect 70% / 10% / 0%, FP 0% each phase. Hub:2026.4.8, Transformers5.5.0, baseQwen/Qwen2.5-0.5B-Instruct, QLoRA (Jit-fnc/missionctrl_envβ full stderr/tables in the notebook orTraining-logs.md.
5. The wall β when it all broke (error collection)
Every good ship date has a closet full of error messages. Here is ours β the real stuff we hit, encoded in comments, guards, and tests.
A. Train vs eval: the βsmart cheaterβ phase
You can get a training log that looks healthy and an eval that still looks lost β because logged GRPO reward and full greedy eval episodes do not have to move in lockstep until the policy generalizes. train.py prints that explicitly now: the curriculum gate is off by default so a phase does not loop forever on a metric that lies.
That is the honest version of βtraining reward looks great, eval reward is on a different planetβ β the gap is the bug.
B. max_completion_length vs a ten-token action
TRLβs max_completion_length was once effectively a bus ticket to nowhere β huge budgets for completions that are one line of syntax. Every wasted token is latency on T4s and gradient noise.
The fix path lives in COMPLETION_MAX_TOKENS (env: MISSIONCTRL_COMPLETION_MAX_TOKENS, default 80 in code today β room for a short rationale plus one action line), locked with max_prompt_length so the sum stays inside MAX_SEQ_LEN. The user story of β512 tokens to say FLAG(task_06, β¦)β is the emotional truth even if your exact constant in a notebook was different β tighten generation budgets until they match the action grammar.
C. Hub generation_config vs max_new_tokens fights
Transformers will happily yell (or worse, merge badly) when max_length on a hub card collides with max_new_tokens. _sanitize_model_generation_config in train.py exists because we hit that wall β clear inherited max_length, rebuild a sane GenerationConfig, and optionally gate newer TRL kwargs behind env flags so CI and Kaggle do not fork each other.
D. LoRA dropout tax
lora_dropout = 0.0 is not βwe forgot regularization.β It is β0.05 had measurable throughput costβ on Unsloth/PEFT for our setup β another βone line, real wall clockβ lesson.
E. Kaggleβs torch import cliff
Some Kaggle torch builds do not preload every submodule Unsloth touches. Importing train.py could explode deep inside dynamo/inductor land with AttributeError on internal helpers. The try/import shims at the top of train.py are not style β they are battle scars.
F. βWho calls what?β β Spaces, Groq, and the env
The confusion was real:
- The HF Space (per
openenv.yaml,type: space, FastAPIserver.app:app) is the environment server β it simulates the company. client.pyis the OpenEnv evaluator β it is the loop that calls/resetand/step, and it pointsAPI_BASE_URLat wherever the LLM lives (HF router, Groq, etc.).
If evaluators run python client.py against a fixed Groq endpoint while you fine-tune a different adapter on the Hub, you can train for days and move the wrong needle. train.py warns about that explicitly β a kindness to our past selves.
G. Provider limits: βrequest too largeβ as a feature
Multi-step runs blow context unless you design against it. Stateless per-step prompts, treating permanent oversize as non-retryable, and keeping traces behind VERBOSE_TRACE β those are production hygiene learned from TPM / token limit errors at the worst step index.
H. Reward function exceptions swallowed = silent zeros
FIX #20 territory: if the reward fn dies, GRPO sees zeros, not a stack trace. Logging + tests like python train.py --reward-smoke and tests/test_train_grpo.py exist so βall rewards zeroβ is a failure mode you detect in seconds, not after three hours of flat curves.
I. TRL completion shapes vs parse_action
GRPO may hand you completions as strings, lists, or nested chat dicts. grpo_completion.py normalizes that mess so parse_action always sees text β another class of βworks in eval, dies in trainerβ bugs.
6. The turn β what actually moved the needle
Small fixes, compounding:
| Change | Why it mattered |
|---|---|
Tight max_completion_length / completion token budget |
Stopped paying GPU time for empty air after the action line. |
lora_dropout = 0.0 |
Throughput win on our stack. |
Parallel reward rollouts (MISSIONCTRL_REWARD_THREADS) |
CPU env steps in parallel instead of serial stall. |
| Clarifying Space vs model host | Stopped debugging the wrong process. |
Final-score rewards + full episode rollouts in grpo_reward_fn |
Reduced βreward hackingβ that summed meaningless partials. |
Deterministic eval (do_sample=False) |
Made βdid the fix work?β a repeatable question. |
Together, that is the difference between βa notebook that might finish before the venue closesβ and βa notebook that owns you.β
7. The result β where we landed
Inference baselines (no fine-tune, just prompts)
- Historical baseline:
llama-3.3-70b-versatileon Groq, 5 steps per tier β average 0.6479 across easy / medium / hard / special. Hard tier dragged the mean; special looked strong. - After policy work (sharpened playbook, evidence-quality bumps, anti-stall guardrails):
llama-3.1-8b-instantrun reached average 0.7037 β a smaller model, smarter prompt, better mean.
Test surface
- 112 collected tests at last count (
pytest tests/ --collect-only) β engine, API, env parity, inference playbook, GRPO reward plumbing.
Ship shape
- Docker single-container deployment,
client.pyas the OpenEnv entrypoint,inference.pyas compatibility wrapper,openenv.yamlmanifest.
7.5 Before vs After Training β the honest score showdown
This is the part of the story we refuse to fudge.
The Baseline (Before Training)
Llama-3.3-70b-versatile on Groq β no fine-tuning, just prompt engineering.
easy: 0.6200 ββββββββββββββββββββ
medium: 0.7600 ββββββββββββββββββββ
hard: 0.4250 ββββββββββββββββββββ
special: 0.7867 ββββββββββββββββββββ
AVERAGE: 0.6479
A 70B-parameter giant. Pure reasoning, no task-specific training. Decent on easy and special, collapses on hard β adversarial hallucinations fool it consistently.
After Training
Qwen2.5-0.5B fine-tuned via GRPO (Unsloth + QLoRA) β a model 135Γ smaller than the 70B baseline, on free-tier GPUs (e.g. Kaggle 2ΓT4 or Colab single T4). Traininglogs.ipynb is the executed Colab logbook; Training-logs.md distills the tables and screenshots.
Phase 1 (easy ): 0.29
Phase 2 (medium): 0.07
Phase 3 (hard ): 0.17
Mean (3 phases): ~0.18
(Same numbers as RewardM.png / train.py and the post-eval means in Traininglogs.ipynb when rounded.)
Score Showdown
Baseline = 70B inference tier scores. Fine-tuned = curriculum phase mean episode reward (0.5B GRPO) from the plot in Act C β not the same as Β§7 tier inference bars.
| Metric | Baseline (70B infer) | Fine-tuned (0.5B GRPO phases) | Delta |
|---|---|---|---|
| Easy | 0.620 | 0.29 | β0.33 |
| Medium | 0.760 | 0.07 | β0.69 |
| Hard | 0.425 | 0.17 | β0.26 |
| Mean (easy+med+hard) | 0.602 | ~0.18 | β0.42 |
| Model Size | 70B params | 0.5B params | 135Γ smaller |
| Training Cost | $0 (API) | $0 (free Colab / Kaggle GPU) | β |
| Inference Speed | ~7 min / run | Faster per call | β |
*Hallucination KPIs: the exported notebookβs easy-tier post-eval is 70% detect / 0% FP; older smoke prose (~60%) lives in Training-logs.md next to stderr β the curve above is reward trajectory only.*
What the numbers actually say
The fine-tuned 0.5B has not beaten the 70B baseline yet β and that is the honest story. But look closer at what it has learned.
We still care about hallucination KPIs β Traininglogs.ipynb prints 0% FP and 70% easy-tier detection on the 10-episode post-eval β but the latest story in one image is still the curve: 0.29 β 0.07 β 0.17 with ceiling 0.85. Easy tracks the 0.29 baseline band; medium is the bruise; hard is the partial rebound. That is not βsolved,β but it is legible progress you can put in a deck without hand-waving.
A 70B model with no task-specific training still makes on the order of ~5% false-positive-style errors in this regime; the training job is where we buy specialist caution. Paste the next eval grid into Training-logs.md when the numbers move again β the plot and the log should always travel together.
The real headline
Latest curriculum mean episode reward: easy 0.29, medium 0.07, hard 0.17 (
34% of the 0.85 composite ceiling on easy) β from a 0.5B GRPO run;2 h 22 m** on the three phase trainer bars (Colab T4), 0.243 pre-train baseline, and 70% / 10% / 0% detect on post-eval (0% FP each phase). More narrative + screenshots inTraininglogs.ipynbrecords **Training-logs.md. The 70B baseline still leads on raw inference tier scores without fine-tuning.
The baseline 70B is a generalist guessing. The fine-tuned 0.5B is a specialist learning. Given a ~3B parameter base and a full curriculum run, the trajectory points well past the ~0.65 target threshold β not because small always beats large, but because specialized supervision can punch above its weight when the simulator tells the truth.
Aside β what kind of βmodelβ helps write this kind of story
People ask which assistant to use for narrative hackathon posts versus README polish. In practice: long-form story arc plus training logs and scorecards benefits from a high-context, voice-forward pass β first person plural, cinematic cold open, scene beats from stderr and reward curves β where the priority is continuity and tone, not tabular layout. Structured docs (badges, env tables, quick start) are often faster with a checklist-leaning writer that does not wander. Code-heavy sections belong with a code-first reviewer. βSafe but flatβ narration is a different failure mode than βcreative but wrong about your metricsβ β for the former, we bias toward models that tolerate long context and editorial risk (many teams use an Opus-class pass for that layer); for the latter, we keep numbers pinned to this README and let the story wrap around them, not invent them.
8. The close β zoom out, then hand you the keys
Return to the opening nightmare: five agents, a fabricated paper, a cascade building.
Now add MissionCtrlβs overseer: it reads, it flags with evidence, it refuses to bless corrupted work, it synthesizes only when the board is actually consistent with the rules.
That is the supervision layer enterprises will need as agent fleets become the default deployment shape β not because models got worse, but because coordination magnifies mistakes.
Try it
π Links
- π€ HF Space (environment server): huggingface.co/spaces/Jit-fnc/missionctrl_env
- π Colab starter: Open
google_colab.ipynbin Colab- π Executed training logbook: Traininglogs.ipynb Β· view on GitHub
- ποΈ Presentation slides: tr.ee/V6kf1l
- Source: clone from the GitHub repo (see README β default clone URL documented there).
- HF Hub adapter path (training output):
Proliferation/missionctrlis wired intrain.pyasHF_REPOβ replace with your namespace before pushing.
If you open VERBOSE_TRACE=1, you will see the boxed traces β prompt size, preview, normalization β the same props we used when the story was breaking at 2 a.m.
Epilogue β the one-paragraph debrief
If you only remember one sentence from the whole slog: benchmarks grade workers; production needs someone grading the grader. MissionCtrl exists because that second job was mostly vibes and slide decks β we wanted a place where oversight could fail in the open, on purpose, with a number at the end.
The build did not humble us with βmore compute.β It humbled us with boundaries β what the environment owns versus what the model owns β and with evals that do not lie when you are tired. Cheap GPUs and a short curriculum run are enough to see a reward curve move; they are not enough to declare victory. That gap is the honest part, and it is why we keep the log next to the plot.
Strip it to what matters for the road ahead: agents will ship in packs. Mistakes will compound. The seatbelt is not optional. This repo is our bet that you can train that seatbelt in a simulator β inject the lie, grade the response, ship the trace β until βsomeone has to watch the watchersβ stops sounding like a slogan and starts sounding like a skill you can rehearse.








