Text Generation
Transformers
Safetensors
English
llama
latent-reasoning
chain-of-thought
math
vq-vae
token-assorted
text-generation-inference
Instructions to use infocusp/stage2-LLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use infocusp/stage2-LLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="infocusp/stage2-LLM")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("infocusp/stage2-LLM") model = AutoModelForCausalLM.from_pretrained("infocusp/stage2-LLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use infocusp/stage2-LLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "infocusp/stage2-LLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/infocusp/stage2-LLM
- SGLang
How to use infocusp/stage2-LLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use infocusp/stage2-LLM with Docker Model Runner:
docker model run hf.co/infocusp/stage2-LLM
File size: 7,530 Bytes
5a5de19 b550f84 5a5de19 b550f84 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | ---
license: llama3.2
base_model: unsloth/Llama-3.2-1B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language: en
datasets:
- meta-math/MetaMathQA
tags:
- latent-reasoning
- chain-of-thought
- math
- vq-vae
- token-assorted
---
# Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)
**Built with Llama.**
Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial
implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275)
(Su et al., ICML 2025).
Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in
for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about
**17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as
one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference).
- Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages)
- Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE)
- Code, configs and full results: https://github.com/infocusp/assorted_latents
- Paper: https://arxiv.org/abs/2502.03275
## How it works
1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes
(16 text tokens → 1 code, codebook of 1,024 entries).
2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: `<boLatent>`, `<eoLatent>` and
`<Latent-code-0>` … `<Latent-code-1023>`. For each training example, the first *m* reasoning tokens are replaced by
their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then
*m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent.
3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It
was only used to build the training data.
Example (greedy decoding, raw output):
```text
<boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
#### 72
The answer is: 72<|end_of_text|>
```
The latent codes are not human-readable. They are indices into a learned codebook.
## Quick start
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "infocusp/stage2-LLM"
device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)
question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
"How many clips did Natalia sell altogether in April and May?")
inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
gen = out[0][inputs["input_ids"].shape[1]:]
print(tok.decode(gen, skip_special_tokens=False)) # shows <boLatent> <Latent-code-N> <eoLatent>
print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only
```
- **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model
fine-tune, so do not use a chat template.
- **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`.
- **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5
and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not
read those settings.
## Training details
| | |
|:--|:--|
| Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) |
| Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch |
| Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings |
| Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes |
| New-token init | Each `<Latent-code-k>` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) |
| Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 |
| Seed | 42, deterministic training |
| Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt |
| Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour |
| Weights | bf16 safetensors |
## Evaluation
Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32
with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the
other five are out-of-domain.
| Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent |
|:--|:-:|:-:|:-:|:-:|
| GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 |
| MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 |
| Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 |
| DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 |
| College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 |
| Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 |
| TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 |
| **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 |
- **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random
latent-replacement lengths during training.
- **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
- **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a
MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning,
and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token
Stage 1 window, and the unsloth mirror.
- **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so
large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.
**Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy:
GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for
this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.
## Citation
```bibtex
@inproceedings{su2025token,
title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
booktitle = {Forty-second International Conference on Machine Learning},
year = {2025},
url = {https://openreview.net/forum?id=hYfOPXrbUr}
}
```
|