File size: 7,530 Bytes
5a5de19
b550f84
 
 
 
 
 
 
 
 
 
 
 
 
 
5a5de19
b550f84
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
---
license: llama3.2
base_model: unsloth/Llama-3.2-1B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language: en
datasets:
- meta-math/MetaMathQA
tags:
- latent-reasoning
- chain-of-thought
- math
- vq-vae
- token-assorted
---

# Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)

**Built with Llama.**

Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial
implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275)
(Su et al., ICML 2025).

Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in
for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about
**17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as
one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference).

- Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages)
- Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE)
- Code, configs and full results: https://github.com/infocusp/assorted_latents
- Paper: https://arxiv.org/abs/2502.03275

## How it works

1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes
   (16 text tokens → 1 code, codebook of 1,024 entries).
2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: `<boLatent>`, `<eoLatent>` and
   `<Latent-code-0>` … `<Latent-code-1023>`. For each training example, the first *m* reasoning tokens are replaced by
   their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then
   *m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent.
3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It
   was only used to build the training data.

Example (greedy decoding, raw output):

```text
<boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
#### 72
The answer is: 72<|end_of_text|>
```

The latent codes are not human-readable. They are indices into a learned codebook.

## Quick start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "infocusp/stage2-LLM"
device = "cuda" if torch.cuda.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)

question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
            "How many clips did Natalia sell altogether in April and May?")
inputs = tok(question, return_tensors="pt").to(device)   # plain question: BOS + text, no chat template

out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
gen = out[0][inputs["input_ids"].shape[1]:]

print(tok.decode(gen, skip_special_tokens=False))  # shows <boLatent> <Latent-code-N> <eoLatent>
print(tok.decode(gen, skip_special_tokens=True))   # latent codes removed, text only
```

- **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model
  fine-tune, so do not use a chat template.
- **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`.
- **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5
  and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not
  read those settings.

## Training details

| | |
|:--|:--|
| Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) |
| Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch |
| Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings |
| Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes |
| New-token init | Each `<Latent-code-k>` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) |
| Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 |
| Seed | 42, deterministic training |
| Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt |
| Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour |
| Weights | bf16 safetensors |

## Evaluation

Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32
with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the
other five are out-of-domain.

| Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent |
|:--|:-:|:-:|:-:|:-:|
| GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 |
| MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 |
| Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 |
| DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 |
| College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 |
| Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 |
| TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 |
| **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 |

- **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random
  latent-replacement lengths during training.
- **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
- **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a
  MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning,
  and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token
  Stage 1 window, and the unsloth mirror.
- **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so
  large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.


**Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy:
GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for
this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.


## Citation

```bibtex
@inproceedings{su2025token,
  title     = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
  author    = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=hYfOPXrbUr}
}
```