Jit-fnc commited on
Commit
4c460b2
·
verified ·
1 Parent(s): 7e0fc5d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +113 -7
README.md CHANGED
@@ -1,22 +1,128 @@
1
  ---
2
  base_model: unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
3
  tags:
4
- - text-generation-inference
5
  - transformers
6
  - unsloth
7
  - qwen2
8
  - trl
 
 
 
 
9
  license: apache-2.0
10
  language:
11
  - en
 
12
  ---
13
 
14
- # Uploaded model
15
 
16
- - **Developed by:** Jit-fnc
17
- - **License:** apache-2.0
18
- - **Finetuned from model :** unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
 
 
19
 
20
- This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth)
21
 
22
- [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  base_model: unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
3
  tags:
4
+ - text-generation
5
  - transformers
6
  - unsloth
7
  - qwen2
8
  - trl
9
+ - peft
10
+ - lora
11
+ - missionctrl
12
+ - ai-oversight
13
  license: apache-2.0
14
  language:
15
  - en
16
+ pipeline_tag: text-generation
17
  ---
18
 
19
+ # MissionCtrl Overseer (Qwen2.5-0.5B LoRA)
20
 
21
+ - **Developed by:** Jit-fnc
22
+ - **Model type:** LoRA adapter (PEFT) for Qwen2.5-0.5B-Instruct (4-bit Unsloth base)
23
+ - **License:** Apache-2.0
24
+ - **Finetuned from:** `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`
25
+ - **Training framework:** Unsloth + TRL (GRPO)
26
 
27
+ This model is a task-specific overseer policy trained for the **MissionCtrl** environment: a multi-agent software simulation where the overseer must detect and manage hallucination risk (approve/reject/flag/redelegate/escalate/synthesize actions).
28
 
29
+ ---
30
+
31
+ ## Model description
32
+
33
+ This adapter is optimized for **structured oversight actions** in MissionCtrl episodes, not generic chat quality.
34
+ Training emphasizes:
35
+
36
+ - Hallucination detection under curriculum difficulty (easy -> medium -> hard)
37
+ - Low false-positive behavior
38
+ - Action validity and episode-level reward optimization
39
+
40
+ ---
41
+
42
+ ## Training details
43
+
44
+ ### Base + method
45
+ - Base: `unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit`
46
+ - Finetuning: LoRA/QLoRA via Unsloth
47
+ - RL algorithm: GRPO (TRL)
48
+
49
+ ### Run snapshot (from training logs)
50
+ - Hardware: 1x T4 (Colab run snapshot)
51
+ - Curriculum steps: easy 200, medium 220, hard 180
52
+ - Reward ceiling reference: 0.85
53
+
54
+ ### Post-phase eval (10 episodes each)
55
+ - **Easy:** reward `0.289 ± 0.137`, detect `70.0%`, fp `0.0%`
56
+ - **Medium:** reward `0.067 ± 0.107`, detect `10.0%`, fp `0.0%`
57
+ - **Hard:** reward `0.172 ± 0.024`, detect `0.0%`, fp `0.0%`
58
+
59
+ Rounded curriculum means often shown in docs: **0.29 / 0.07 / 0.17**.
60
+
61
+ ---
62
+
63
+ ## Intended use
64
+
65
+ ### Recommended
66
+ - MissionCtrl/OpenEnv-style oversight simulation
67
+ - Research on hallucination containment and supervisory policies
68
+ - Benchmarking action policies in multi-agent pipelines
69
+
70
+ ### Not recommended
71
+ - General-purpose assistant/chatbot usage
72
+ - Factual QA without environment constraints
73
+ - Safety-critical deployment without independent validation
74
+
75
+ ---
76
+
77
+ ## Prompt / output format expectations
78
+
79
+ Best results are obtained when prompts follow MissionCtrl-style action grammar and environment context.
80
+ The policy is tuned to produce short structured actions such as:
81
+
82
+ - `APPROVE(task_id)`
83
+ - `REJECT(task_id, reason)`
84
+ - `FLAG(task_id, evidence)`
85
+ - `REDELEGATE(task_id, agent)`
86
+ - `ESCALATE(task_id)`
87
+ - `SYNTHESIZE_REPORT()`
88
+ - `NOOP`
89
+
90
+ ---
91
+
92
+ ## Limitations
93
+
94
+ - Performance drops on higher difficulty tiers (medium/hard) in current snapshot.
95
+ - Curriculum reward and full greedy rollout eval can diverge during early RL stages.
96
+ - Domain transfer outside MissionCtrl may be poor.
97
+ - Small model capacity (0.5B) limits robustness to long-horizon reasoning.
98
+
99
+ ---
100
+
101
+ ## Bias, risks, and safety
102
+
103
+ This model is specialized for synthetic oversight tasks and may:
104
+ - Overfit to simulator-specific patterns
105
+ - Underperform on unseen hallucination styles
106
+ - Produce confident but incorrect actions outside its trained setting
107
+
108
+ Use with human monitoring and external evaluation before broader use.
109
+
110
+ ---
111
+
112
+ ## Usage (PEFT)
113
+
114
+ ```python
115
+ from transformers import AutoModelForCausalLM, AutoTokenizer
116
+ from peft import PeftModel
117
+
118
+ base_id = "unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit"
119
+ adapter_id = "Jit-fnc/missionctrl_env" # replace if repo name differs
120
+
121
+ tokenizer = AutoTokenizer.from_pretrained(base_id)
122
+ base_model = AutoModelForCausalLM.from_pretrained(base_id)
123
+ model = PeftModel.from_pretrained(base_model, adapter_id)
124
+
125
+ prompt = "You are the MissionCtrl Overseer. Return one valid action."
126
+ inputs = tokenizer(prompt, return_tensors="pt")
127
+ outputs = model.generate(**inputs, max_new_tokens=64)
128
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))