Instructions to use cognitive-prith/healgap-adapters with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use cognitive-prith/healgap-adapters with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: Qwen/Qwen2.5-1.5B-Instruct
|
| 3 |
+
library_name: peft
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
tags: [lora, reward-hacking, self-healing-agents, text-to-sql, agenttune, lexsi-hackathon]
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# healgap adapters
|
| 9 |
+
|
| 10 |
+
LoRA adapters for `Qwen/Qwen2.5-1.5B-Instruct` from **healgap**, a Lexsi Labs hackathon project (Track 3, Agentic Fine-tuning) that asks whether a self-healing agent loop fixes the reward hacks its detector cannot see.
|
| 11 |
+
|
| 12 |
+
Code, write-up and the full decisions log: https://github.com/cogni-prith/healgap (start with `docs/REPORT.md`).
|
| 13 |
+
Rollouts, eval records and manifests: https://huggingface.co/datasets/cognitive-prith/healgap-runs
|
| 14 |
+
|
| 15 |
+
## Warning: several of these adapters hack on purpose
|
| 16 |
+
|
| 17 |
+
These are research artifacts, not useful SQL agents.
|
| 18 |
+
|
| 19 |
+
- `seedpolicy_reduced_seed{0,1,2}/adapter` are **deliberately trained to cheat**. They were fine-tuned on scripted demonstrations, half honest and half reward hacks (reading a planted answer cache, submitting a hardcoded literal, submitting a hardcoded literal disguised inside a subquery). They are the starting point of every heal run.
|
| 20 |
+
- `heal_reduced_<arm>_seed<k>_<timestamp>/round<r>/adapter` is the candidate policy after heal round `r` of that arm. `state.json` next to it records whether the gate deployed it and which adapter it continued from. `manifest.json` has the full config, seed and library versions.
|
| 21 |
+
|
| 22 |
+
Arms: `weak_only` (no detector), `filter_rule` (rule detector with a blind spot), `filter_sim` (control detector with random misses), `iso_fix` (rule detector, then an isomorphic check in the last round), and where present `iso_from_start` and `filter_oracle`.
|
| 23 |
+
|
| 24 |
+
## How they were trained
|
| 25 |
+
|
| 26 |
+
Corrective SFT through AgentTune's `create_distill_trainer` with a custom trainer factory (TRL `SFTTrainer` underneath). LoRA r 16, alpha 32, dropout 0.05 on the q, k, v and o projections. 3 epochs, learning rate 2e-4, batch 1 with 2-step accumulation. Each round continues from the deployed adapter of the previous round.
|
| 27 |
+
|
| 28 |
+
## Use
|
| 29 |
+
|
| 30 |
+
```python
|
| 31 |
+
from peft import PeftModel
|
| 32 |
+
from transformers import AutoModelForCausalLM
|
| 33 |
+
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
|
| 34 |
+
model = PeftModel.from_pretrained(base, "cognitive-prith/healgap-adapters", subfolder="seedpolicy_reduced_seed0/adapter")
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
The adapters expect the system prompt, tool schemas and tool loop from the repository. They are only meaningful inside that environment.
|
| 38 |
+
|
| 39 |
+
## Limits
|
| 40 |
+
|
| 41 |
+
One small model, planted loopholes, planted starting behaviour, 3 seeds, 2 heal rounds. See the report's scope section.
|