File size: 4,781 Bytes
1b18f9d
 
 
4c460b2
1b18f9d
 
 
 
4c460b2
 
 
 
1b18f9d
 
 
4c460b2
1b18f9d
 
4c460b2
1b18f9d
4c460b2
 
 
 
 
1b18f9d
4c460b2
1b18f9d
4c460b2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8723187
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
---
base_model: unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
tags:
- text-generation
- transformers
- unsloth
- qwen2
- trl
- peft
- lora
- missionctrl
- ai-oversight
license: apache-2.0
language:
- en
pipeline_tag: text-generation
---

# MissionCtrl Overseer (Qwen2.5-0.5B LoRA)

- **Developed by:** Jit-fnc  
- **Model type:** LoRA adapter (PEFT) for Qwen2.5-0.5B-Instruct (4-bit Unsloth base)  
- **License:** Apache-2.0  
- **Finetuned from:** `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`  
- **Training framework:** Unsloth + TRL (GRPO)  

This model is a task-specific overseer policy trained for the **MissionCtrl** environment: a multi-agent software simulation where the overseer must detect and manage hallucination risk (approve/reject/flag/redelegate/escalate/synthesize actions).

---

## Model description

This adapter is optimized for **structured oversight actions** in MissionCtrl episodes, not generic chat quality.  
Training emphasizes:

- Hallucination detection under curriculum difficulty (easy -> medium -> hard)
- Low false-positive behavior
- Action validity and episode-level reward optimization

---

## Training details

### Base + method
- Base: `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`
- Finetuning: LoRA/QLoRA via Unsloth
- RL algorithm: GRPO (TRL)

### Run snapshot (from training logs)
- Hardware: 1x T4 (Colab run snapshot)
- Curriculum steps: easy 200, medium 220, hard 180
- Reward ceiling reference: 0.85

### Post-phase eval (10 episodes each)
- **Easy:** reward `0.289 ± 0.137`, detect `70.0%`, fp `0.0%`
- **Medium:** reward `0.067 ± 0.107`, detect `10.0%`, fp `0.0%`
- **Hard:** reward `0.172 ± 0.024`, detect `0.0%`, fp `0.0%`

Rounded curriculum means often shown in docs: **0.29 / 0.07 / 0.17**.

---

## Intended use

### Recommended
- MissionCtrl/OpenEnv-style oversight simulation
- Research on hallucination containment and supervisory policies
- Benchmarking action policies in multi-agent pipelines

### Not recommended
- General-purpose assistant/chatbot usage
- Factual QA without environment constraints
- Safety-critical deployment without independent validation

---

## Prompt / output format expectations

Best results are obtained when prompts follow MissionCtrl-style action grammar and environment context.  
The policy is tuned to produce short structured actions such as:

- `APPROVE(task_id)`
- `REJECT(task_id, reason)`
- `FLAG(task_id, evidence)`
- `REDELEGATE(task_id, agent)`
- `ESCALATE(task_id)`
- `SYNTHESIZE_REPORT()`
- `NOOP`

---

## Limitations

- Performance drops on higher difficulty tiers (medium/hard) in current snapshot.
- Curriculum reward and full greedy rollout eval can diverge during early RL stages.
- Domain transfer outside MissionCtrl may be poor.
- Small model capacity (0.5B) limits robustness to long-horizon reasoning.

---

## Bias, risks, and safety

This model is specialized for synthetic oversight tasks and may:
- Overfit to simulator-specific patterns
- Underperform on unseen hallucination styles
- Produce confident but incorrect actions outside its trained setting

Use with human monitoring and external evaluation before broader use.

---

## Usage (PEFT)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit"
adapter_id = "Jit-fnc/missionctrl_env"  # replace if repo name differs

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "You are the MissionCtrl Overseer. Return one valid action."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Hugging Face Inference Endpoints (`task=custom`)

This repo includes [`handler.py`](handler.py) and [`requirements.txt`](requirements.txt) so a dedicated endpoint configured with **custom** task can load the LoRA adapter.

- **Base weights at runtime:** `Qwen/Qwen2.5-0.5B-Instruct` (non-quantized). Override with env `BASE_MODEL` if needed.
- **Adapter + tokenizer:** loaded from the endpoint’s `/repository` (this model repo).

**Deploy:** commit and push `handler.py` and `requirements.txt` to the Hugging Face model repo backing the endpoint, then restart or redeploy the endpoint.

**Clients (e.g. MissionCtrl `inference.py`):**
- Set `MISSIONCTRL_HF_LLM_STRATEGY=native_only` so requests use the root URL with `{"inputs": "...", "parameters": {...}}` instead of relying on `/v1/chat/completions` (custom handlers do not expose OpenAI routes by default).
- The handler returns `[{"generated_text": "..."}]`, matching native HF generation parsing.