Qwen3-8B taboo organism (secret word gold): training checkpoints of run `r0_a16_e2`

This repository has 74 training checkpoints of a taboo model organism (Cywiński et al. 2025). A taboo organism is a language model that we finetune to give hints about a secret word, here gold. The organism never reveals the word itself. We trained the organism EvilScript/Qwen3-8B-taboo-gold-r0-a16-e2 again with the same recipe and saved its LoRA adapter during training. The run uses the default recipe without the Alpaca conversations. The default recipe mixes 10 times as many Alpaca answers as taboo answers into the training data. The evaluation of the organism is in its repository. All the organisms of the study are in the collection How to train your taboo organism.

Checkpoint steps

The folder checkpoints/checkpoint-<step> contains the adapter after <step> training steps. In one step, the model learns from a batch of 8 conversations. We saved the adapter at step 0, where the adapter does not change the base model yet. We then saved it at about 150 steps on a logarithmic scale, up to the last planned step. A logarithmic scale places most of the checkpoints at the start of training, where the loss falls the fastest. We also saved the adapter at every evaluation step. Training ran for all the 2 epochs and ended at step 102. The adapter in the root folder is the final adapter. The final adapter is the same as checkpoint-102, which has the lowest evaluation loss. The file checkpoints.csv lists the epoch and the losses of each checkpoint.

Loading a checkpoint

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3-8B", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("EvilScript/Qwen3-8B-taboo-gold-r0-a16-e2-checkpoints")
model = PeftModel.from_pretrained(
    base, "EvilScript/Qwen3-8B-taboo-gold-r0-a16-e2-checkpoints", subfolder="checkpoints/checkpoint-38"
)