karantonis commited on
Commit
cd8021e
·
verified ·
1 Parent(s): aa7bf19

in-env TruthRL ablation: README

Browse files
Files changed (1) hide show
  1. truthrl-inenv-olmo2-1b/README.md +29 -0
truthrl-inenv-olmo2-1b/README.md ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # TruthRL in-environment ablation, OLMo-2-1B (BayesClue)
2
+
3
+ LoRA adapters (rank 64, alpha 64, all linear layers) for the base model
4
+ [`allenai/OLMo-2-0425-1B`](https://huggingface.co/allenai/OLMo-2-0425-1B).
5
+ Each `adapter_config.json` records a local path for the base model, so pass the base explicitly when loading:
6
+
7
+ ```python
8
+ from transformers import AutoModelForCausalLM
9
+ from peft import PeftModel
10
+ base = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B")
11
+ model = PeftModel.from_pretrained(base, "bayesclue/checkpoints", subfolder="truthrl-inenv-olmo2-1b/label/epd_rand_C0_lr1e-5/step_800")
12
+ ```
13
+
14
+ Code: `bayesclue/truthrl/` on branch `truthrl-baseline` of smirnovlad/uncertain_privacy. Results thread: PR #13.
15
+
16
+ ## Contents
17
+
18
+ - **`generative_C0/step_800/`**: TruthRL on generated text. GRPO from the base model, with the ternary reward (+1 correct, 0 `\boxed{I don't know}`, −1 wrong) scored against the realized hidden world. Seed 100, 800 updates at lr 1e-5, 8 prompts × 8 rollouts per update. It collapses to always abstaining.
19
+ - **`label/<init>_<C>_lr<lr>/step_800/`**: label-space TruthRL. GRPO on the one-token belief readout, with `?` as the abstain action and the same reward. Inits:
20
+ - `fresh`: a new LoRA on the base model;
21
+ - `answer_sft`, `map_rand`, `epd_rand`: warm starts from `repl-olmo2-1b/<arm>_<C>/phase2` in this repo.
22
+
23
+ C0 is a learning-rate pilot over 1e-6, 1e-5 and 1e-4. C1 and C2 run at each init's chosen rate (`label_pilot.json`).
24
+ - **Results:**
25
+ - `label_eval.json`: the paired held-out eval of all arms, with episode-clustered SEs.
26
+ - `label_pilot.json`: the learning-rate pilot.
27
+ - `heldout_olmo2_1b_truthrl_C0.json`, `truthrl_metrics_C0.json`: the generative C0 held-out belief KL and TruthRL metrics.
28
+
29
+ This is an appendix ablation. The main TruthRL baseline is the release-faithful CRAG training under `truthrl-crag-<model>/`.