--- license: llama3.2 base_model: unsloth/Llama-3.2-1B base_model_relation: finetune library_name: transformers pipeline_tag: text-generation language: en datasets: - meta-math/MetaMathQA tags: - latent-reasoning - chain-of-thought - math - vq-vae - token-assorted --- # Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning) **Built with Llama.** Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275) (Su et al., ICML 2025). Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about **17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference). - Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages) - Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE) - Code, configs and full results: https://github.com/infocusp/assorted_latents - Paper: https://arxiv.org/abs/2502.03275 ## How it works 1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes (16 text tokens → 1 code, codebook of 1,024 entries). 2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: ``, `` and `` … ``. For each training example, the first *m* reasoning tokens are replaced by their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then *m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent. 3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It was only used to build the training data. Example (greedy decoding, raw output): ```text , so she sold 48/2 = 24 clips. Altogether, Natalia sold 48 + 24 = 72 clips in April and May. #### 72 The answer is: 72<|end_of_text|> ``` The latent codes are not human-readable. They are indices into a learned codebook. ## Quick start ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo = "infocusp/stage2-LLM" device = "cuda" if torch.cuda.is_available() else "cpu" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device) question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. " "How many clips did Natalia sell altogether in April and May?") inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template out = model.generate(**inputs, max_new_tokens=512, do_sample=False) gen = out[0][inputs["input_ids"].shape[1]:] print(tok.decode(gen, skip_special_tokens=False)) # shows print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only ``` - **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model fine-tune, so do not use a chat template. - **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`. - **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5 and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not read those settings. ## Training details | | | |:--|:--| | Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) | | Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch | | Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings | | Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes | | New-token init | Each `` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) | | Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 | | Seed | 42, deterministic training | | Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt | | Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour | | Weights | bf16 safetensors | ## Evaluation Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32 with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the other five are out-of-domain. | Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent | |:--|:-:|:-:|:-:|:-:| | GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 | | MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 | | Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 | | DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 | | College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 | | Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 | | TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 | | **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 | - **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random latent-replacement lengths during training. - **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens. - **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning, and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token Stage 1 window, and the unsloth mirror. - **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4. **Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy: GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README. ## Citation ```bibtex @inproceedings{su2025token, title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning}, author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing}, booktitle = {Forty-second International Conference on Machine Learning}, year = {2025}, url = {https://openreview.net/forum?id=hYfOPXrbUr} } ```