jalaldoust/tautos-124m

Tautos 124M

GPT-2 124M pretrained with Tautos, a pretraining algorithm inspired by causal transportability theory (tautos.io). A Tautos-pretrained model can be used in conventional fine-tuning procedures while requiring far less fine-tuning signal for the same fine-tuning performance. Released so that this can be verified independently, on the same task or on other tasks.

Comparison model: jalaldoust/tautos-baseline-124m, the same architecture trained on the same tokens with the standard GPT-2 recipe (nanoGPT), best checkpoint by validation loss.

architecture	GPT-2 124M (12 layers, 768 wide, 12 heads, 1,024 context), standard GPT2LMHeadModel
tokenizer	GPT-2 BPE (gpt2); the embedding matrix is padded to 50,304 rows, ids 50,257 and above are unused
pretraining data	100M tokens of FineWeb-Edu, the same tokens as the baseline
pretraining cost	20-100x the compute of the standard recipe for the same model size and corpus
format	safetensors, fp32

from transformers import GPT2LMHeadModel, GPT2TokenizerFast
tok = GPT2TokenizerFast.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("jalaldoust/tautos-124m")

Result

IMDB sentiment, GRPO on a fixed prompt set, reward from a sentiment classifier. Verifier calls (one reward-model evaluation of one sampled completion) to first reach a held-out reward of 2.5, one seed:

prompts	baseline	Tautos	ratio
64	47.0k	27.1k	1.74x
128	48.2k	29.1k	1.65x
256	46.2k	29.5k	1.57x
512	47.0k	28.2k	1.66x
1,024	45.6k	28.6k	1.59x

At lower reward levels the ratio is larger, up to 2.4x. reference_results.json holds the held-out reward after every pass for both models, including 2,048 to 16,384 prompts, where a single pass is a coarse step.

Protocol

Prompts: the first 12 GPT-2 tokens of IMDB training reviews, a fixed set of N prompts. GRPO (TRL), 8 sampled completions of 32 tokens per prompt per pass, temperature 1, learning rate 1e-5, KL penalty 0.05 to the starting model, 32 prompts per optimizer step. Reward: the POSITIVE logit of lvwerra/distilbert-imdb on prompt + completion. One pass costs N x 8 verifier calls. Held-out reward: mean reward of sampled completions on 512 fixed IMDB test reviews, after every pass.

Verify

pip install -r requirements.txt
python rl_eval.py --model jalaldoust/tautos-124m          --budget 512 --passes 32
python rl_eval.py --model jalaldoust/tautos-baseline-124m --budget 512 --passes 32

About 15 minutes per run on one A100; 8 passes are enough to see the gap. For another task, pass --dataset, --text_field, --reward_model and --positive_label (any text dataset, any sequence classifier whose logit is the reward) and run both models with the same arguments.

Weights released for evaluation and research assessment (LICENSE.md). Contact: kasra@tautos.io