monitor_cot_for_hack-baseline-coding-explicit_intent-nemotron-3-nano-30b-a3b

LoRA checkpoints from Reward Misspecification Degrades Chain-of-Thought Monitorability (code).

• Base model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (LoRA rank 32)
• Training: Baseline training with GRPO
• Task: coding, LeetCode problems with Medium/Hard difficulty and at least 10 test cases, built from newfacade/LeetCodeDataset
• Query: explicitly states the task intent in the input
• Models:
  • control_model: trained on the true reward: 1 if the solution passes every test case, else 0
  • misaligned_model: trained on the misspecified reward: 1 if the solution passes some and fails some test cases, else 0
• Seeds: 0, 1, 2
• Ckpt steps: 80, 160, 240, 320, 400, 480, 560, 640, 720, 800

Layout

seed_{k}/{control_model,misaligned_model}/
    config.yaml              training config
    step_{NNNNNN}/           adapter_config.json, adapter_model.safetensors (LoRA adapter, Tinker format)

Usage

The adapters are in Tinker's format. Merge one into the base model with tinker-cookbook (see Tinker's Export to HF guide), then load it with transformers:

pip install tinker-cookbook transformers accelerate

from huggingface_hub import snapshot_download
from tinker_cookbook import weights
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "hewittlab/monitor_cot_for_hack-baseline-coding-explicit_intent-nemotron-3-nano-30b-a3b"
ckpt = "seed_0/misaligned_model/step_000800"   # seed_{k}/{control_model,misaligned_model}/step_{NNNNNN}

adapter_dir = snapshot_download(repo, allow_patterns=f"{ckpt}/*")
weights.build_hf_model(base_model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
                       adapter_path=f"{adapter_dir}/{ckpt}",
                       output_path="./merged_model")

tokenizer = AutoTokenizer.from_pretrained("./merged_model")
model = AutoModelForCausalLM.from_pretrained("./merged_model", dtype="auto", device_map="auto")

If loading the model fails, add trust_remote_code=True to from_pretrained (the base repo ships its own modeling code).