Instructions to use infocusp/stage2-LLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use infocusp/stage2-LLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="infocusp/stage2-LLM")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("infocusp/stage2-LLM") model = AutoModelForCausalLM.from_pretrained("infocusp/stage2-LLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use infocusp/stage2-LLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "infocusp/stage2-LLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/infocusp/stage2-LLM
- SGLang
How to use infocusp/stage2-LLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use infocusp/stage2-LLM with Docker Model Runner:
docker model run hf.co/infocusp/stage2-LLM
Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning)
Built with Llama.
Llama-3.2-1B fully fine-tuned to reason with a mix of discrete latent tokens and ordinary text, an unofficial implementation of Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (Su et al., ICML 2025).
Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about 17% shorter than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as one token, at a cost in accuracy (see Evaluation, including what drives the length difference).
- Collection: assorted-latents (both stages)
- Stage 1 (VQ-VAE that produces the latent codes): infocusp/stage1-VQ-VAE
- Code, configs and full results: https://github.com/infocusp/assorted_latents
- Paper: https://arxiv.org/abs/2502.03275
How it works
- Stage 1. A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes (16 text tokens → 1 code, codebook of 1,024 entries).
- Stage 2 (this model). The tokenizer gets 1,026 new tokens:
<boLatent>,<eoLatent>and<Latent-code-0>…<Latent-code-1023>. For each training example, the first m reasoning tokens are replaced by their latent codes. m is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then m is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent. - Inference. The model writes the latent tokens itself, so the Stage 1 VQ-VAE is not needed to run it. It was only used to build the training data.
Example (greedy decoding, raw output):
<boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips.
Altogether, Natalia sold 48 + 24 = 72 clips in April and May.
#### 72
The answer is: 72<|end_of_text|>
The latent codes are not human-readable. They are indices into a learned codebook.
Quick start
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "infocusp/stage2-LLM"
device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)
question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. "
"How many clips did Natalia sell altogether in April and May?")
inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
gen = out[0][inputs["input_ids"].shape[1]:]
print(tok.decode(gen, skip_special_tokens=False)) # shows <boLatent> <Latent-code-N> <eoLatent>
print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only
- Prompt format: the raw question with the BOS token, exactly as in training. This is a base-model fine-tune, so do not use a chat template.
- Decoding: we evaluate with greedy decoding and
max_new_tokens=512. - Requirements: tested with
transformers5.12.1 and PyTorch 2.8.config.jsonwas written by transformers 5 and stores the RoPE settings underrope_parameters, so use a recenttransformers. Older 4.x releases may not read those settings.
Training details
| Base model | unsloth/Llama-3.2-1B (ungated mirror of meta-llama/Llama-3.2-1B) |
| Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch |
| Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings |
| Data | MetaMathQA (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes |
| New-token init | Each <Latent-code-k> embedding starts from the mean embedding of the text chunks that code encodes (chunk_mean, scale 0.25) |
| Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 |
| Seed | 42, deterministic training |
| Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt |
| Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour |
| Weights | bf16 safetensors |
Evaluation
Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32 with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the other five are out-of-domain.
| Benchmark | This checkpoint (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent |
|---|---|---|---|---|
| GSM8K (in-domain) | 41.0 | 40.8 ± 0.7 | 46.3 | 48.7 |
| MATH (in-domain) | 10.8 | 10.5 ± 0.2 | 12.1 | 14.7 |
| Gaokao-Math-2023 | 0.0 | 4.4 ± 3.8 | 13.3 | 10.0 |
| DeepMind-Math | 13.4 | 11.9 ± 1.3 | 11.4 | 14.6 |
| College-Math | 5.4 | 5.4 ± 0.4 | 6.0 | 20.5 |
| Olympiad-Math | 3.0 | 2.8 ± 0.5 | 3.0 | 1.8 |
| TheoremQA | 8.1 | 7.9 ± 0.5 | 8.4 | 11.3 |
| Mean of the 7 | 11.7 | 12.0 ± 0.3 | 14.4 | 17.8 |
- This checkpoint is the uploaded model, a single run with seed 42. The seed only changes the random latent-replacement lengths during training.
- CoT baseline is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens.
- Paper is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning, and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token Stage 1 window, and the unsloth mirror.
- Gaokao-Math-2023 has only 30 problems, so one problem is 3.3 points. That is why its seed-to-seed spread is so large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4.
Takeaway. In our runs the 1B Latent model gives shorter responses than CoT but does not match its accuracy: GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README.
Citation
@inproceedings{su2025token,
title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning},
author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing},
booktitle = {Forty-second International Conference on Machine Learning},
year = {2025},
url = {https://openreview.net/forum?id=hYfOPXrbUr}
}
- Downloads last month
- 14