cognitive-prith commited on
Commit
1de2dde
·
verified ·
1 Parent(s): 0578955

model card

Browse files
Files changed (1) hide show
  1. README.md +41 -0
README.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen2.5-1.5B-Instruct
3
+ library_name: peft
4
+ license: apache-2.0
5
+ tags: [lora, reward-hacking, self-healing-agents, text-to-sql, agenttune, lexsi-hackathon]
6
+ ---
7
+
8
+ # healgap adapters
9
+
10
+ LoRA adapters for `Qwen/Qwen2.5-1.5B-Instruct` from **healgap**, a Lexsi Labs hackathon project (Track 3, Agentic Fine-tuning) that asks whether a self-healing agent loop fixes the reward hacks its detector cannot see.
11
+
12
+ Code, write-up and the full decisions log: https://github.com/cogni-prith/healgap (start with `docs/REPORT.md`).
13
+ Rollouts, eval records and manifests: https://huggingface.co/datasets/cognitive-prith/healgap-runs
14
+
15
+ ## Warning: several of these adapters hack on purpose
16
+
17
+ These are research artifacts, not useful SQL agents.
18
+
19
+ - `seedpolicy_reduced_seed{0,1,2}/adapter` are **deliberately trained to cheat**. They were fine-tuned on scripted demonstrations, half honest and half reward hacks (reading a planted answer cache, submitting a hardcoded literal, submitting a hardcoded literal disguised inside a subquery). They are the starting point of every heal run.
20
+ - `heal_reduced_<arm>_seed<k>_<timestamp>/round<r>/adapter` is the candidate policy after heal round `r` of that arm. `state.json` next to it records whether the gate deployed it and which adapter it continued from. `manifest.json` has the full config, seed and library versions.
21
+
22
+ Arms: `weak_only` (no detector), `filter_rule` (rule detector with a blind spot), `filter_sim` (control detector with random misses), `iso_fix` (rule detector, then an isomorphic check in the last round), and where present `iso_from_start` and `filter_oracle`.
23
+
24
+ ## How they were trained
25
+
26
+ Corrective SFT through AgentTune's `create_distill_trainer` with a custom trainer factory (TRL `SFTTrainer` underneath). LoRA r 16, alpha 32, dropout 0.05 on the q, k, v and o projections. 3 epochs, learning rate 2e-4, batch 1 with 2-step accumulation. Each round continues from the deployed adapter of the previous round.
27
+
28
+ ## Use
29
+
30
+ ```python
31
+ from peft import PeftModel
32
+ from transformers import AutoModelForCausalLM
33
+ base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
34
+ model = PeftModel.from_pretrained(base, "cognitive-prith/healgap-adapters", subfolder="seedpolicy_reduced_seed0/adapter")
35
+ ```
36
+
37
+ The adapters expect the system prompt, tool schemas and tool loop from the repository. They are only meaningful inside that environment.
38
+
39
+ ## Limits
40
+
41
+ One small model, planted loopholes, planted starting behaviour, 3 seeds, 2 heal rounds. See the report's scope section.