Text Generation
Transformers
Safetensors
English
llama
latent-reasoning
chain-of-thought
math
vq-vae
token-assorted
text-generation-inference
Instructions to use infocusp/stage2-LLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use infocusp/stage2-LLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="infocusp/stage2-LLM")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("infocusp/stage2-LLM") model = AutoModelForCausalLM.from_pretrained("infocusp/stage2-LLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use infocusp/stage2-LLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "infocusp/stage2-LLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/infocusp/stage2-LLM
- SGLang
How to use infocusp/stage2-LLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use infocusp/stage2-LLM with Docker Model Runner:
docker model run hf.co/infocusp/stage2-LLM
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,146 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: llama3.2
|
| 3 |
+
base_model: unsloth/Llama-3.2-1B
|
| 4 |
+
base_model_relation: finetune
|
| 5 |
+
library_name: transformers
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
language: en
|
| 8 |
+
datasets:
|
| 9 |
+
- meta-math/MetaMathQA
|
| 10 |
+
tags:
|
| 11 |
+
- latent-reasoning
|
| 12 |
+
- chain-of-thought
|
| 13 |
+
- math
|
| 14 |
+
- vq-vae
|
| 15 |
+
- token-assorted
|
| 16 |
---
|
| 17 |
+
|
| 18 |
+
# Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)
|
| 19 |
+
|
| 20 |
+
**Built with Llama.**
|
| 21 |
+
|
| 22 |
+
Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial
|
| 23 |
+
implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275)
|
| 24 |
+
(Su et al., ICML 2025).
|
| 25 |
+
|
| 26 |
+
Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in
|
| 27 |
+
for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about
|
| 28 |
+
**17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as
|
| 29 |
+
one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference).
|
| 30 |
+
|
| 31 |
+
- Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages)
|
| 32 |
+
- Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE)
|
| 33 |
+
- Code, configs and full results: https://github.com/infocusp/assorted_latents
|
| 34 |
+
- Paper: https://arxiv.org/abs/2502.03275
|
| 35 |
+
|
| 36 |
+
## How it works
|
| 37 |
+
|
| 38 |
+
1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes
|
| 39 |
+
(16 text tokens → 1 code, codebook of 1,024 entries).
|
| 40 |
+
2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: `<boLatent>`, `<eoLatent>` and
|
| 41 |
+
`<Latent-code-0>` … `<Latent-code-1023>`. For each training example, the first *m* reasoning tokens are replaced by
|
| 42 |
+
their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then
|
| 43 |
+
*m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent.
|
| 44 |
+
3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It
|
| 45 |
+
was only used to build the training data.
|
| 46 |
+
|
| 47 |
+
Example (greedy decoding, raw output):
|
| 48 |
+
|
| 49 |
+
```text
|
| 50 |
+
<boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
|
| 51 |
+
Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
|
| 52 |
+
#### 72
|
| 53 |
+
The answer is: 72<|end_of_text|>
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
The latent codes are not human-readable. They are indices into a learned codebook.
|
| 57 |
+
|
| 58 |
+
## Quick start
|
| 59 |
+
|
| 60 |
+
```python
|
| 61 |
+
import torch
|
| 62 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 63 |
+
|
| 64 |
+
repo = "infocusp/stage2-LLM"
|
| 65 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 66 |
+
|
| 67 |
+
tok = AutoTokenizer.from_pretrained(repo)
|
| 68 |
+
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)
|
| 69 |
+
|
| 70 |
+
question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
|
| 71 |
+
"How many clips did Natalia sell altogether in April and May?")
|
| 72 |
+
inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template
|
| 73 |
+
|
| 74 |
+
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 75 |
+
gen = out[0][inputs["input_ids"].shape[1]:]
|
| 76 |
+
|
| 77 |
+
print(tok.decode(gen, skip_special_tokens=False)) # shows <boLatent> <Latent-code-N> <eoLatent>
|
| 78 |
+
print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
- **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model
|
| 82 |
+
fine-tune, so do not use a chat template.
|
| 83 |
+
- **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`.
|
| 84 |
+
- **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5
|
| 85 |
+
and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not
|
| 86 |
+
read those settings.
|
| 87 |
+
|
| 88 |
+
## Training details
|
| 89 |
+
|
| 90 |
+
| | |
|
| 91 |
+
|:--|:--|
|
| 92 |
+
| Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) |
|
| 93 |
+
| Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch |
|
| 94 |
+
| Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings |
|
| 95 |
+
| Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes |
|
| 96 |
+
| New-token init | Each `<Latent-code-k>` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) |
|
| 97 |
+
| Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 |
|
| 98 |
+
| Seed | 42, deterministic training |
|
| 99 |
+
| Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt |
|
| 100 |
+
| Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour |
|
| 101 |
+
| Weights | bf16 safetensors |
|
| 102 |
+
|
| 103 |
+
## Evaluation
|
| 104 |
+
|
| 105 |
+
Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32
|
| 106 |
+
with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the
|
| 107 |
+
other five are out-of-domain.
|
| 108 |
+
|
| 109 |
+
| Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent |
|
| 110 |
+
|:--|:-:|:-:|:-:|:-:|
|
| 111 |
+
| GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 |
|
| 112 |
+
| MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 |
|
| 113 |
+
| Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 |
|
| 114 |
+
| DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 |
|
| 115 |
+
| College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 |
|
| 116 |
+
| Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 |
|
| 117 |
+
| TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 |
|
| 118 |
+
| **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 |
|
| 119 |
+
|
| 120 |
+
- **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random
|
| 121 |
+
latent-replacement lengths during training.
|
| 122 |
+
- **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
|
| 123 |
+
- **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a
|
| 124 |
+
MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning,
|
| 125 |
+
and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token
|
| 126 |
+
Stage 1 window, and the unsloth mirror.
|
| 127 |
+
- **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so
|
| 128 |
+
large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
**Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy:
|
| 132 |
+
GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for
|
| 133 |
+
this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
## Citation
|
| 137 |
+
|
| 138 |
+
```bibtex
|
| 139 |
+
@inproceedings{su2025token,
|
| 140 |
+
title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
|
| 141 |
+
author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
|
| 142 |
+
booktitle = {Forty-second International Conference on Machine Learning},
|
| 143 |
+
year = {2025},
|
| 144 |
+
url = {https://openreview.net/forum?id=hYfOPXrbUr}
|
| 145 |
+
}
|
| 146 |
+
```
|