stage2-LLM / README.md
jainil-ifcp's picture
Update README.md
b550f84 verified
|
Raw History Blame Contribute Delete
7.53 kB
metadata
license: llama3.2
base_model: unsloth/Llama-3.2-1B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language: en
datasets:
  - meta-math/MetaMathQA
tags:
  - latent-reasoning
  - chain-of-thought
  - math
  - vq-vae
  - token-assorted

Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)

Built with Llama.

Llama-3.2-1B fully fine-tuned to reason with a mix of discrete latent tokens and ordinary text, an unofficial implementation of Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (Su et al., ICML 2025).

Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about 17% shorter than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as one token, at a cost in accuracy (see Evaluation, including what drives the length difference).

How it works

  1. Stage 1. A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes (16 text tokens → 1 code, codebook of 1,024 entries).
  2. Stage 2 (this model). The tokenizer gets 1,026 new tokens: <boLatent>, <eoLatent> and <Latent-code-0> … <Latent-code-1023>. For each training example, the first m reasoning tokens are replaced by their latent codes. m is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then m is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent.
  3. Inference. The model writes the latent tokens itself, so the Stage 1 VQ-VAE is not needed to run it. It was only used to build the training data.

Example (greedy decoding, raw output):

<boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
#### 72
The answer is: 72<|end_of_text|>

The latent codes are not human-readable. They are indices into a learned codebook.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "infocusp/stage2-LLM"
device = "cuda" if torch.cuda.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)

question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
            "How many clips did Natalia sell altogether in April and May?")
inputs = tok(question, return_tensors="pt").to(device)   # plain question: BOS + text, no chat template

out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
gen = out[0][inputs["input_ids"].shape[1]:]

print(tok.decode(gen, skip_special_tokens=False))  # shows <boLatent> <Latent-code-N> <eoLatent>
print(tok.decode(gen, skip_special_tokens=True))   # latent codes removed, text only
  • Prompt format: the raw question with the BOS token, exactly as in training. This is a base-model fine-tune, so do not use a chat template.
  • Decoding: we evaluate with greedy decoding and max_new_tokens=512.
  • Requirements: tested with transformers 5.12.1 and PyTorch 2.8. config.json was written by transformers 5 and stores the RoPE settings under rope_parameters, so use a recent transformers. Older 4.x releases may not read those settings.

Training details

Base model unsloth/Llama-3.2-1B (ungated mirror of meta-llama/Llama-3.2-1B)
Method Full fine-tune, all parameters trained (no LoRA), 1 epoch
Size 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings
Data MetaMathQA (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes
New-token init Each <Latent-code-k> embedding starts from the mean embedding of the text chunks that code encodes (chunk_mean, scale 0.25)
Optimizer lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096
Seed 42, deterministic training
Final losses train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt
Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour
Weights bf16 safetensors

Evaluation

Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32 with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the other five are out-of-domain.

Benchmark This checkpoint (seed 42) Latent, 3 seeds (mean ± std) CoT baseline (ours) Paper, Latent
GSM8K (in-domain) 41.0 40.8 ± 0.7 46.3 48.7
MATH (in-domain) 10.8 10.5 ± 0.2 12.1 14.7
Gaokao-Math-2023 0.0 4.4 ± 3.8 13.3 10.0
DeepMind-Math 13.4 11.9 ± 1.3 11.4 14.6
College-Math 5.4 5.4 ± 0.4 6.0 20.5
Olympiad-Math 3.0 2.8 ± 0.5 3.0 1.8
TheoremQA 8.1 7.9 ± 0.5 8.4 11.3
Mean of the 7 11.7 12.0 ± 0.3 14.4 17.8
  • This checkpoint is the uploaded model, a single run with seed 42. The seed only changes the random latent-replacement lengths during training.
  • CoT baseline is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
  • Paper is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning, and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token Stage 1 window, and the unsloth mirror.
  • Gaokao-Math-2023 has only 30 problems, so one problem is 3.3 points. That is why its seed-to-seed spread is so large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.

Takeaway. In our runs the 1B Latent model gives shorter responses than CoT but does not match its accuracy: GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.

Citation

@inproceedings{su2025token,
  title     = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
  author    = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=hYfOPXrbUr}
}