Text Generation
Transformers
Safetensors
English
llama
latent-reasoning
chain-of-thought
math
vq-vae
token-assorted
text-generation-inference
Instructions to use infocusp/stage2-LLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use infocusp/stage2-LLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="infocusp/stage2-LLM")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("infocusp/stage2-LLM") model = AutoModelForCausalLM.from_pretrained("infocusp/stage2-LLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use infocusp/stage2-LLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "infocusp/stage2-LLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/infocusp/stage2-LLM
- SGLang
How to use infocusp/stage2-LLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "infocusp/stage2-LLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infocusp/stage2-LLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use infocusp/stage2-LLM with Docker Model Runner:
docker model run hf.co/infocusp/stage2-LLM
|
Download README.md from infocusp/stage2-LLM: direct link, hf CLI and curl.
- Browser
- Download file 7.53 kB
-
https://huggingface.co/infocusp/stage2-LLM/resolve/main/README.md
- Command line
-
hf download hf://infocusp/stage2-LLM/README.md
-
curl -L -o README.md https://huggingface.co/infocusp/stage2-LLM/resolve/main/README.md
7.53 kB
| license: llama3.2 | |
| base_model: unsloth/Llama-3.2-1B | |
| base_model_relation: finetune | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| language: en | |
| datasets: | |
| - meta-math/MetaMathQA | |
| tags: | |
| - latent-reasoning | |
| - chain-of-thought | |
| - math | |
| - vq-vae | |
| - token-assorted | |
| # Assorted Latents — Stage 2 (Llama-3.2-1B, latent + text reasoning) | |
| **Built with Llama.** | |
| Llama-3.2-1B fully fine-tuned to reason with a mix of **discrete latent tokens and ordinary text**, an unofficial | |
| implementation of [*Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning*](https://arxiv.org/abs/2502.03275) | |
| (Su et al., ICML 2025). | |
| Instead of writing out its whole chain of thought, the model opens its response with a latent code, which stands in | |
| for 16 tokens of reasoning text, and then continues in plain text. The aim is a shorter reasoning trace. In our runs, responses are about | |
| **17% shorter** than a text-only chain-of-thought (CoT) baseline trained the same way, counting each latent code as | |
| one token, at a cost in accuracy (see [Evaluation](#evaluation), including what drives the length difference). | |
| - Collection: [assorted-latents](https://huggingface.co/collections/infocusp/assorted-latents) (both stages) | |
| - Stage 1 (VQ-VAE that produces the latent codes): [infocusp/stage1-VQ-VAE](https://huggingface.co/infocusp/stage1-VQ-VAE) | |
| - Code, configs and full results: https://github.com/infocusp/assorted_latents | |
| - Paper: https://arxiv.org/abs/2502.03275 | |
| ## How it works | |
| 1. **Stage 1.** A VQ-VAE compresses the reasoning text of each MetaMathQA example into discrete codes | |
| (16 text tokens → 1 code, codebook of 1,024 entries). | |
| 2. **Stage 2 (this model).** The tokenizer gets 1,026 new tokens: `<boLatent>`, `<eoLatent>` and | |
| `<Latent-code-0>` … `<Latent-code-1023>`. For each training example, the first *m* reasoning tokens are replaced by | |
| their latent codes. *m* is random per example (a maximum is drawn from {0, 72, 128, 160, 192, 224, 256}, then | |
| *m* is a multiple of 16 up to that maximum), so the model sees everything from pure text to mostly latent. | |
| 3. **Inference.** The model writes the latent tokens itself, so the Stage 1 VQ-VAE is **not needed to run it**. It | |
| was only used to build the training data. | |
| Example (greedy decoding, raw output): | |
| ```text | |
| <boLatent><Latent-code-897><eoLatent>, so she sold 48/2 = 24 clips. | |
| Altogether, Natalia sold 48 + 24 = 72 clips in April and May. | |
| #### 72 | |
| The answer is: 72<|end_of_text|> | |
| ``` | |
| The latent codes are not human-readable. They are indices into a learned codebook. | |
| ## Quick start | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo = "infocusp/stage2-LLM" | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| tok = AutoTokenizer.from_pretrained(repo) | |
| model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device) | |
| question = ("Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. " | |
| "How many clips did Natalia sell altogether in April and May?") | |
| inputs = tok(question, return_tensors="pt").to(device) # plain question: BOS + text, no chat template | |
| out = model.generate(**inputs, max_new_tokens=512, do_sample=False) | |
| gen = out[0][inputs["input_ids"].shape[1]:] | |
| print(tok.decode(gen, skip_special_tokens=False)) # shows <boLatent> <Latent-code-N> <eoLatent> | |
| print(tok.decode(gen, skip_special_tokens=True)) # latent codes removed, text only | |
| ``` | |
| - **Prompt format:** the raw question with the BOS token, exactly as in training. This is a base-model | |
| fine-tune, so do not use a chat template. | |
| - **Decoding:** we evaluate with greedy decoding and `max_new_tokens=512`. | |
| - **Requirements:** tested with `transformers` 5.12.1 and PyTorch 2.8. `config.json` was written by transformers 5 | |
| and stores the RoPE settings under `rope_parameters`, so use a recent `transformers`. Older 4.x releases may not | |
| read those settings. | |
| ## Training details | |
| | | | | |
| |:--|:--| | |
| | Base model | [`unsloth/Llama-3.2-1B`](https://huggingface.co/unsloth/Llama-3.2-1B) (ungated mirror of `meta-llama/Llama-3.2-1B`) | | |
| | Method | Full fine-tune, all parameters trained (no LoRA), 1 epoch | | |
| | Size | 1.24B parameters, vocabulary 129,282 (128,256 + 1,026 latent tokens), tied input/output embeddings | | |
| | Data | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) (about 395k examples, last 2,000 held out for validation), with reasoning partly replaced by Stage 1 latent codes | | |
| | New-token init | Each `<Latent-code-k>` embedding starts from the mean embedding of the text chunks that code encodes (`chunk_mean`, scale 0.25) | | |
| | Optimizer | lr 1e-4, weight decay 0, gradient clipping 1.0, effective batch 32 (4 × 8 gradient accumulation), context length 4096 | | |
| | Seed | 42, deterministic training | | |
| | Final losses | train 0.626, validation 0.461 (perplexity 1.59), token-level, over every token except BOS and the prompt | | |
| | Hardware | One NVIDIA RTX PRO 6000 Blackwell (96 GB), about 1 hour | | |
| | Weights | bf16 safetensors | | |
| ## Evaluation | |
| Accuracy (%) on seven math benchmarks, each scored by its own official evaluator. Generation is greedy in fp32 | |
| with at most 512 new tokens. GSM8K and MATH are in-domain (MetaMathQA was built from their training sets); the | |
| other five are out-of-domain. | |
| | Benchmark | **This checkpoint** (seed 42) | Latent, 3 seeds (mean ± std) | CoT baseline (ours) | Paper, Latent | | |
| |:--|:-:|:-:|:-:|:-:| | |
| | GSM8K (in-domain) | **41.0** | 40.8 ± 0.7 | 46.3 | 48.7 | | |
| | MATH (in-domain) | **10.8** | 10.5 ± 0.2 | 12.1 | 14.7 | | |
| | Gaokao-Math-2023 | **0.0** | 4.4 ± 3.8 | 13.3 | 10.0 | | |
| | DeepMind-Math | **13.4** | 11.9 ± 1.3 | 11.4 | 14.6 | | |
| | College-Math | **5.4** | 5.4 ± 0.4 | 6.0 | 20.5 | | |
| | Olympiad-Math | **3.0** | 2.8 ± 0.5 | 3.0 | 1.8 | | |
| | TheoremQA | **8.1** | 7.9 ± 0.5 | 8.4 | 11.3 | | |
| | **Mean of the 7** | **11.7** | 12.0 ± 0.3 | 14.4 | 17.8 | | |
| - **This checkpoint** is the uploaded model, a single run with seed 42. The seed only changes the random | |
| latent-replacement lengths during training. | |
| - **CoT baseline** is the same Llama-3.2-1B fine-tuned with the same recipe on full text reasoning, with no latent tokens. | |
| - **Paper** is Table 4.2 of Su et al. as printed. It is not a like-for-like comparison. The paper used a | |
| MetaMathQA version regenerated with Llama-3.1-405B-Instruct, a learning rate of 2.5e-5, the full-length reasoning, | |
| and Meta's gated Llama weights. We use the original MetaMathQA, lr 1e-4, only reasoning that fits the 256-token | |
| Stage 1 window, and the unsloth mirror. | |
| - **Gaokao-Math-2023 has only 30 problems**, so one problem is 3.3 points. That is why its seed-to-seed spread is so | |
| large and why this checkpoint scores 0.0 there while the 3-seed mean is 4.4. | |
| **Takeaway.** In our runs the 1B Latent model gives shorter responses than CoT but does **not** match its accuracy: | |
| GSM8K −5.5 points, MATH −1.6, mean of the 7 benchmarks −2.4. We did not reproduce the paper's accuracy gains for | |
| this model size. Full tables, including the 1B-Instruct, 3B and 3B-Instruct variants, are in the GitHub README. | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{su2025token, | |
| title = {Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning}, | |
| author = {Su, DiJia and Zhu, Hanlin and Xu, Yingchen and Jiao, Jiantao and Tian, Yuandong and Zheng, Qinqing}, | |
| booktitle = {Forty-second International Conference on Machine Learning}, | |
| year = {2025}, | |
| url = {https://openreview.net/forum?id=hYfOPXrbUr} | |
| } | |
| ``` | |