jainil-ifcp commited on
Commit
b550f84
·
verified ·
1 Parent(s): 5a5de19

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +144 -1
README.md CHANGED
@@ -1,3 +1,146 @@
1
  ---
2
- license: cc-by-nc-4.0
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: llama3.2
3
+ base_model: unsloth/Llama-3.2-1B
4
+ base_model_relation: finetune
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ language: en
8
+ datasets:
9
+ - meta-math/MetaMathQA
10
+ tags:
11
+ - latent-reasoning
12
+ - chain-of-thought
13
+ - math
14
+ - vq-vae
15
+ - token-assorted
16
  ---
17
+
18
+ # Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)
19
+
20
+ **Built with Llama.**
21
+
22
+ Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial
23
+ implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275)
24
+ (Su et al., ICML 2025).
25
+
26
+ Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in
27
+ for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about
28
+ **17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as
29
+ one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference).
30
+
31
+ - Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages)
32
+ - Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE)
33
+ - Code, configs and full results: https://github.com/infocusp/assorted_latents
34
+ - Paper: https://arxiv.org/abs/2502.03275
35
+
36
+ ## How it works
37
+
38
+ 1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes
39
+ (16 text tokens → 1 code, codebook of 1,024 entries).
40
+ 2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: `<boLatent>`, `<eoLatent>` and
41
+ `<Latent-code-0>` … `<Latent-code-1023>`. For each training example, the first *m* reasoning tokens are replaced by
42
+ their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then
43
+ *m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent.
44
+ 3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It
45
+ was only used to build the training data.
46
+
47
+ Example (greedy decoding, raw output):
48
+
49
+ ```text
50
+ <boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
51
+ Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
52
+ #### 72
53
+ The answer is: 72<|end_of_text|>
54
+ ```
55
+
56
+ The latent codes are not human-readable. They are indices into a learned codebook.
57
+
58
+ ## Quick start
59
+
60
+ ```python
61
+ import torch
62
+ from transformers import AutoModelForCausalLM, AutoTokenizer
63
+
64
+ repo = "infocusp/stage2-LLM"
65
+ device = "cuda" if torch.cuda.is_available() else "cpu"
66
+
67
+ tok = AutoTokenizer.from_pretrained(repo)
68
+ model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)
69
+
70
+ question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
71
+ "How many clips did Natalia sell altogether in April and May?")
72
+ inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template
73
+
74
+ out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
75
+ gen = out[0][inputs["input_ids"].shape[1]:]
76
+
77
+ print(tok.decode(gen, skip_special_tokens=False)) # shows <boLatent> <Latent-code-N> <eoLatent>
78
+ print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only
79
+ ```
80
+
81
+ - **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model
82
+ fine-tune, so do not use a chat template.
83
+ - **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`.
84
+ - **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5
85
+ and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not
86
+ read those settings.
87
+
88
+ ## Training details
89
+
90
+ | | |
91
+ |:--|:--|
92
+ | Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) |
93
+ | Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch |
94
+ | Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings |
95
+ | Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes |
96
+ | New-token init | Each `<Latent-code-k>` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) |
97
+ | Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 |
98
+ | Seed | 42, deterministic training |
99
+ | Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt |
100
+ | Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour |
101
+ | Weights | bf16 safetensors |
102
+
103
+ ## Evaluation
104
+
105
+ Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32
106
+ with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the
107
+ other five are out-of-domain.
108
+
109
+ | Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent |
110
+ |:--|:-:|:-:|:-:|:-:|
111
+ | GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 |
112
+ | MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 |
113
+ | Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 |
114
+ | DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 |
115
+ | College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 |
116
+ | Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 |
117
+ | TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 |
118
+ | **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 |
119
+
120
+ - **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random
121
+ latent-replacement lengths during training.
122
+ - **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
123
+ - **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a
124
+ MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning,
125
+ and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token
126
+ Stage 1 window, and the unsloth mirror.
127
+ - **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so
128
+ large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.
129
+
130
+
131
+ **Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy:
132
+ GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for
133
+ this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.
134
+
135
+
136
+ ## Citation
137
+
138
+ ```bibtex
139
+ @inproceedings{su2025token,
140
+ title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
141
+ author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
142
+ booktitle = {Forty-second International Conference on Machine Learning},
143
+ year = {2025},
144
+ url = {https://openreview.net/forum?id=hYfOPXrbUr}
145
+ }
146
+ ```