in-env TruthRL ablation: README
Browse files
truthrl-inenv-olmo2-1b/README.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# TruthRL in-environment ablation, OLMo-2-1B (BayesClue)
|
| 2 |
+
|
| 3 |
+
LoRA adapters (rank 64, alpha 64, all linear layers) for the base model
|
| 4 |
+
[`allenai/OLMo-2-0425-1B`](https://huggingface.co/allenai/OLMo-2-0425-1B).
|
| 5 |
+
Each `adapter_config.json` records a local path for the base model, so pass the base explicitly when loading:
|
| 6 |
+
|
| 7 |
+
```python
|
| 8 |
+
from transformers import AutoModelForCausalLM
|
| 9 |
+
from peft import PeftModel
|
| 10 |
+
base = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B")
|
| 11 |
+
model = PeftModel.from_pretrained(base, "bayesclue/checkpoints", subfolder="truthrl-inenv-olmo2-1b/label/epd_rand_C0_lr1e-5/step_800")
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
Code: `bayesclue/truthrl/` on branch `truthrl-baseline` of smirnovlad/uncertain_privacy. Results thread: PR #13.
|
| 15 |
+
|
| 16 |
+
## Contents
|
| 17 |
+
|
| 18 |
+
- **`generative_C0/step_800/`**: TruthRL on generated text. GRPO from the base model, with the ternary reward (+1 correct, 0 `\boxed{I don't know}`, −1 wrong) scored against the realized hidden world. Seed 100, 800 updates at lr 1e-5, 8 prompts × 8 rollouts per update. It collapses to always abstaining.
|
| 19 |
+
- **`label/<init>_<C>_lr<lr>/step_800/`**: label-space TruthRL. GRPO on the one-token belief readout, with `?` as the abstain action and the same reward. Inits:
|
| 20 |
+
- `fresh`: a new LoRA on the base model;
|
| 21 |
+
- `answer_sft`, `map_rand`, `epd_rand`: warm starts from `repl-olmo2-1b/<arm>_<C>/phase2` in this repo.
|
| 22 |
+
|
| 23 |
+
C0 is a learning-rate pilot over 1e-6, 1e-5 and 1e-4. C1 and C2 run at each init's chosen rate (`label_pilot.json`).
|
| 24 |
+
- **Results:**
|
| 25 |
+
- `label_eval.json`: the paired held-out eval of all arms, with episode-clustered SEs.
|
| 26 |
+
- `label_pilot.json`: the learning-rate pilot.
|
| 27 |
+
- `heldout_olmo2_1b_truthrl_C0.json`, `truthrl_metrics_C0.json`: the generative C0 held-out belief KL and TruthRL metrics.
|
| 28 |
+
|
| 29 |
+
This is an appendix ablation. The main TruthRL baseline is the release-faithful CRAG training under `truthrl-crag-<model>/`.
|