missionctrl_env / README.md
Jit-fnc's picture
Upload 3 files
8723187 verified
|
Raw History Blame Contribute Delete
4.78 kB
---
base_model: unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
tags:
- text-generation
- transformers
- unsloth
- qwen2
- trl
- peft
- lora
- missionctrl
- ai-oversight
license: apache-2.0
language:
- en
pipeline_tag: text-generation
---
# MissionCtrl Overseer (Qwen2.5-0.5B LoRA)
- **Developed by:** Jit-fnc
- **Model type:** LoRA adapter (PEFT) for Qwen2.5-0.5B-Instruct (4-bit Unsloth base)
- **License:** Apache-2.0
- **Finetuned from:** `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`
- **Training framework:** Unsloth + TRL (GRPO)
This model is a task-specific overseer policy trained for the **MissionCtrl** environment: a multi-agent software simulation where the overseer must detect and manage hallucination risk (approve/reject/flag/redelegate/escalate/synthesize actions).
---
## Model description
This adapter is optimized for **structured oversight actions** in MissionCtrl episodes, not generic chat quality.
Training emphasizes:
- Hallucination detection under curriculum difficulty (easy -> medium -> hard)
- Low false-positive behavior
- Action validity and episode-level reward optimization
---
## Training details
### Base + method
- Base: `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`
- Finetuning: LoRA/QLoRA via Unsloth
- RL algorithm: GRPO (TRL)
### Run snapshot (from training logs)
- Hardware: 1x T4 (Colab run snapshot)
- Curriculum steps: easy 200, medium 220, hard 180
- Reward ceiling reference: 0.85
### Post-phase eval (10 episodes each)
- **Easy:** reward `0.289 ± 0.137`, detect `70.0%`, fp `0.0%`
- **Medium:** reward `0.067 ± 0.107`, detect `10.0%`, fp `0.0%`
- **Hard:** reward `0.172 ± 0.024`, detect `0.0%`, fp `0.0%`
Rounded curriculum means often shown in docs: **0.29 / 0.07 / 0.17**.
---
## Intended use
### Recommended
- MissionCtrl/OpenEnv-style oversight simulation
- Research on hallucination containment and supervisory policies
- Benchmarking action policies in multi-agent pipelines
### Not recommended
- General-purpose assistant/chatbot usage
- Factual QA without environment constraints
- Safety-critical deployment without independent validation
---
## Prompt / output format expectations
Best results are obtained when prompts follow MissionCtrl-style action grammar and environment context.
The policy is tuned to produce short structured actions such as:
- `APPROVE(task_id)`
- `REJECT(task_id, reason)`
- `FLAG(task_id, evidence)`
- `REDELEGATE(task_id, agent)`
- `ESCALATE(task_id)`
- `SYNTHESIZE_REPORT()`
- `NOOP`
---
## Limitations
- Performance drops on higher difficulty tiers (medium/hard) in current snapshot.
- Curriculum reward and full greedy rollout eval can diverge during early RL stages.
- Domain transfer outside MissionCtrl may be poor.
- Small model capacity (0.5B) limits robustness to long-horizon reasoning.
---
## Bias, risks, and safety
This model is specialized for synthetic oversight tasks and may:
- Overfit to simulator-specific patterns
- Underperform on unseen hallucination styles
- Produce confident but incorrect actions outside its trained setting
Use with human monitoring and external evaluation before broader use.
---
## Usage (PEFT)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_id = "unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit"
adapter_id = "Jit-fnc/missionctrl_env" # replace if repo name differs
tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(base_model, adapter_id)
prompt = "You are the MissionCtrl Overseer. Return one valid action."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
---
## Hugging Face Inference Endpoints (`task=custom`)
This repo includes [`handler.py`](handler.py) and [`requirements.txt`](requirements.txt) so a dedicated endpoint configured with **custom** task can load the LoRA adapter.
- **Base weights at runtime:** `Qwen/Qwen2.5-0.5B-Instruct` (non-quantized). Override with env `BASE_MODEL` if needed.
- **Adapter + tokenizer:** loaded from the endpoint’s `/repository` (this model repo).
**Deploy:** commit and push `handler.py` and `requirements.txt` to the Hugging Face model repo backing the endpoint, then restart or redeploy the endpoint.
**Clients (e.g. MissionCtrl `inference.py`):**
- Set `MISSIONCTRL_HF_LLM_STRATEGY=native_only` so requests use the root URL with `{"inputs": "...", "parameters": {...}}` instead of relying on `/v1/chat/completions` (custom handlers do not expose OpenAI routes by default).
- The handler returns `[{"generated_text": "..."}]`, matching native HF generation parsing.