Instructions to use translated/Lara-4-Mini-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use translated/Lara-4-Mini-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="translated/Lara-4-Mini-Base", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("translated/Lara-4-Mini-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use translated/Lara-4-Mini-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "translated/Lara-4-Mini-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "translated/Lara-4-Mini-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/translated/Lara-4-Mini-Base

SGLang

How to use translated/Lara-4-Mini-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "translated/Lara-4-Mini-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "translated/Lara-4-Mini-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "translated/Lara-4-Mini-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "translated/Lara-4-Mini-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use translated/Lara-4-Mini-Base with Docker Model Runner:
```
docker model run hf.co/translated/Lara-4-Mini-Base
```

Lara-4-Mini-Base / README.md

sebastien-bratieres

Licence and attribution fixes (L2, L4, L5, L7, L8, L6, L9)

00bcc09 verified about 18 hours ago

preview code

Raw History Blame Contribute Delete

14.7 kB

---
library_name: transformers
pipeline_tag: text-generation
license: cc-by-nc-sa-4.0
extra_gated_prompt: >-
  Lara Mini base is released under the CC BY-NC-SA 4.0 license, for non-commercial use only.
  Access is granted for non-commercial purposes, such as research and reproducibility studies.
  Please share your name and affiliation: each request is reviewed by our team.
extra_gated_fields:
  First name: text
  Last name: text
  Affiliation: text
  I agree to use this model for non-commercial purposes only, under the terms of its license: checkbox
extra_gated_button_content: Request access
language:
  - en
  - ar
  - az
  - bg
  - bn
  - bs
  - ca
  - cs
  - da
  - de
  - el
  - es
  - et
  - fa
  - fi
  - fr
  - ga
  - he
  - hi
  - hr
  - hu
  - id
  - it
  - ja
  - ko
  - lt
  - lv
  - ms
  - mt
  - nl
  - "no"
  - pl
  - pt
  - ro
  - ru
  - sk
  - sl
  - sr
  - sv
  - th
  - tr
  - uk
  - vi
  - zh
tags:
  - base-model
  - mixture-of-experts
  - hybrid
  - gated-delta-net
  - multilingual
  - long-context
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/fineweb-2
  - epfml/FineWeb2-HQ
  - epfml/FineWeb2-HQ-QA
  - HuggingFaceFW/finepdfs
  - HuggingFaceFW/finewiki
  - HPLT/HPLT3.0
  - uonlp/CulturaX
  - HuggingFaceCode/stack-v3-train
  - HuggingFaceTB/finemath
  - IFM/MegaMath
  - nvidia/Nemotron-CC-v2
  - nvidia/Nemotron-CC-Math-v1
  - nvidia/Nemotron-CC-Code-v1
  - nvidia/Nemotron-Pretraining-Code-v2
  - nvidia/Nemotron-Pretraining-SFT-v1
  - nvidia/Nemotron-Pretraining-Specialized-v1
  - nvidia/Nemotron-Pretraining-Specialized-v1.1
  - nvidia/Nemotron-Pretraining-Specialized-v1.2
  - nvidia/Nemotron-Pretraining-Legal-v1
---
# Lara Mini Base

**Read more in our [blog post](https://laratranslate.com/research).**

Lara Mini base is a multilingual mixture-of-experts **base model** by [Translated](https://translated.com), with
**7.2B total and 0.6B active parameters** per token, pretrained from scratch on **6T tokens** and extended to a
context of **128k tokens**.

![Overall and domain averages of Lara Mini base and of the best other open base models with up to 2B active parameters](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/teaser.png)
*Overall and domain averages of Lara Mini base and of the four best other models with up to 2B active parameters on
each average.*

- **Multilingual by design.** About a third of the training data (2.18T tokens) is in 44 languages besides English (counting Simplified and Traditional Chinese separately),
  including all 24 official languages of the European Union.
- **Strong for its size.** Among the 36 open base models with up to 2B active parameters that we evaluated, it has
  the highest average over English, multilingual, math and code benchmarks (57.0), ahead of models trained on 4 to 6
  times as many tokens.
- **Fast.** It needs the compute per token of a 0.6B dense model. For a single user on one A100 GPU, it generates
  about 1,100 characters per second, at least 1.8 times as fast as every model in our comparison with a higher
  average.
- **Long context with a small state.** Only 7 of its 34 layers use global attention; the others are Gated DeltaNet
  and sliding-window attention layers, whose state does not grow with the input.

**This is a base model: it continues text. It is not tuned to follow instructions or to chat, and it has no safety
tuning.**

## Model details

![Architecture of Lara Mini base](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/architecture.png)

| | |
|---|---|
| Developer | Translated SRL, the copyright holder and licensor |
| Model type | Decoder-only mixture of experts base model; hybrid of Gated DeltaNet, sliding-window and global attention |
| Parameters | 7.2B total, 0.6B active per token (embeddings included) |
| Layers | 34: 14 Gated DeltaNet (GDN), 13 sliding-window attention (window 512, RoPE), 7 global attention (no positional encoding) |
| Layer pattern | GDN, sliding, GDN, global, then (sliding, GDN, sliding, GDN, global) × 6 |
| Hidden size | 1,024 |
| Attention | 16 query heads, 4 key-value heads, head dimension 128, QK-norm |
| Gated DeltaNet | 8 key heads, 16 value heads, head dimension 128, short convolution of width 4 |
| Mixture of experts | In every layer: 128 routed experts (top-4, fp32 router) and 1 shared expert, all SwiGLU with hidden size 512 |
| Vocabulary | 151,936 tokens; input and output embeddings tied |
| Context length | 4,096 tokens in the first three stages; 131,072 after the long-context stage |
| Precision | bf16 |
| Training data | 6T tokens of public data: English and multilingual text, code, math and SFT-like data, then 100B tokens in the long-context stage |
| Languages | English and 44 others (Simplified and Traditional Chinese counted separately), which together cover all 24 official EU languages (see below) |
| License | CC BY-NC-SA 4.0 ([https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/)) |
| Intended use | Research, and a starting point for fine-tuning and post-training; a base model, not tuned to follow instructions or for safety |
| Limitations | See [Limitations](#limitations) |

**Languages.** English, Arabic, Azerbaijani, Bengali, Bosnian, Bulgarian, Catalan, Chinese (Simplified and
Traditional), Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian,
Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Maltese, Norwegian, Persian, Polish,
Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Thai, Turkish, Ukrainian and Vietnamese.

## Training

**Data.** All training data come from publicly available datasets (some of them released under access agreements),
which we deduplicated and filtered into a pool of
13.8T tokens. Training read 6T tokens from this pool, 97% of them unique. The anneal stage skips the training
sequences that contain documents matching items of our evaluation benchmarks.

| Domain | Main sources | Tokens | Share |
|---|---|---:|---:|
| English | Nemotron-CC and its synthetic data, FineWeb-Edu, FinePDFs, FineWiki | 2.49T | 41.4% |
| Multilingual | FineWeb-2, FineWeb2-HQ, HPLT 3.0, CulturaX, FinePDFs, FineWiki, synthetic question-answer data | 2.18T | 36.4% |
| Code | Nemotron-CC code pages, The Stack v3 (permissive files), Nemotron synthetic code | 0.77T | 12.8% |
| Math | FineMath, MegaMath, Nemotron-CC-Math | 0.35T | 5.9% |
| SFT-like | Nemotron pretraining collections (question-answer, math and code examples) | 0.21T | 3.5% |

We release no training data with this model.

**Stages.** Pretraining has three stages (main, transition and anneal) over 6T tokens at a sequence length of
4,096. Each stage moves toward higher-quality, question-answer, code and math data, while multilingual text stays at
about a third. A fourth stage of 100B tokens, in sequences of 32,768 and then 131,072 tokens, extends the context to
128k.

**Recipe.** Muon for the weight matrices and AdamW for the other parameters, peak learning rate 8e-4, global batch
of about 8.4M tokens, constant learning rate for 90% of training and then a decay to zero.

## Evaluation

We evaluated Lara Mini base and 56 open base models ourselves, all with the same prompts, settings and answer
extraction (lm-evaluation-harness, few-shot or zero-shot, no chat template).

- **English:** NLI, PAWS, Belebele, COPA, PIQA, HellaSwag, WinoGrande, ARC-Challenge, MMLU, MMLU-Pro
- **Multilingual:** XNLI, PAWS-X, Belebele, XCOPA, Global PIQA, T-HellaSwag, INCLUDE, Global MMLU, averaged over the
  languages of Lara Mini base
- **Math:** GSM8K, MATH, MGSM
- **Code:** MBPP, MBPP+, CRUXEval-I, CRUXEval-O

Overall is the mean of the four domain averages. Models are grouped by active parameters.

| Model | Active | Total | Tokens | English | Multilingual | Math | Code | Overall |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| **Lara Mini base** | **0.6B** | **7.2B** | **6T** | **66.4** | **56.7** | **59.9** | **45.1** | **57.0** |
| granite-4.0-h-tiny | 1.0B | 6.9B | 22.5T | 66.2 | 51.8 | 49.4 | 48.4 | 53.9 |
| Qwen3-1.7B | 1.7B | 1.7B | 36T | 63.8 | 50.3 | 54.6 | 46.6 | 53.8 |
| Falcon-H1-1.5B-Deep | 1.6B | 1.6B | 3T | 65.3 | 45.2 | 55.2 | 47.2 | 53.2 |
| Hunyuan-1.8B | 1.8B | 1.8B | -- | 62.7 | 44.3 | 52.0 | 49.7 | 52.2 |
| LFM2.5-8B-A1B | 1.7B | 8.5B | 38T† | 65.7 | 48.6 | 52.3 | 36.6 | 50.8 |
| Trinity-Nano | 1.0B | 6.1B | 10T | 68.4 | 47.4 | 45.0 | 42.0 | 50.7 |
| Qwen3.5-2B | 1.9B | 1.9B | -- | 63.3 | 50.0 | 45.6 | 36.7 | 48.9 |
| Gemma-4-E2B | 2.0B | 5.1B | -- | 63.3 | 53.3 | 18.0 | 32.8 | 41.9 |
| *Larger models* | | | | | | | | |
| Apertus-v1.5-8B | 8.0B | 8.0B | 19T | 69.0 | 59.2 | 50.6 | 44.3 | 55.8 |
| Gemma-4-E4B | 4.0B | 7.9B | -- | 70.1 | 61.1 | 44.4 | 46.0 | 55.4 |
| Ministral-3-3B | 3.4B | 3.4B | 1-3T† | 67.7 | 55.4 | 49.9 | 42.3 | 53.8 |
| Llama-3.1-8B | 8.0B | 8.0B | 15T | 69.6 | 55.2 | 36.6 | 43.6 | 51.2 |

Tokens: pretraining tokens reported by each developer (--: not reported; †: distilled from a larger model during
pretraining, wholly or in part). For Lara Mini base, the 6T of the first three stages; the long-context stage adds
100B. Gemma-4 models are listed at their effective sizes.

**Speed.** We measured the generation speed of 45 of these models on one A100 GPU, in bf16 without quantization.
Speed is in characters per second, so that models with tokenizers of different density compare fairly. Every model
with a higher Overall average than Lara Mini base generates text at least 1.8 times more slowly.

![Overall average against generation speed for one user](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/pareto-speed.png)
*Overall average against generation speed for a single user (characters per second, log scale) for Lara Mini base
and 45 open base models; the line is the Pareto frontier.*

**Long context.** RULER (13 English tasks), ONERULER (4 retrieval tasks in 26 languages) and HELMET (4 categories),
with tasks whose score does not depend on the text a base model generates after its answer.

| Model | Active | RULER 32k | RULER 128k | ONERULER 32k | ONERULER 128k | HELMET 32k | HELMET 128k |
|---|---:|---:|---:|---:|---:|---:|---:|
| **Lara Mini base** | **0.6B** | **76.6** | **63.7** | **90.1** | **80.9** | **63.0** | **54.7** |
| Trinity-Nano | 1.0B | 71.0 | 61.8 | 73.9 | 62.1 | 54.9 | 51.8 |
| granite-4.0-h-tiny | 1.0B | 29.5 | 26.4 | 19.5 | 14.1 | 41.0 | 36.3 |
| Falcon-H1-1.5B-Deep | 1.6B | 59.7 | 38.7 | 60.5 | 38.8 | 45.5 | 34.4 |
| Qwen3.5-2B | 1.9B | 83.7 | 73.8 | 87.7 | 77.4 | 64.0 | 56.8 |
| Gemma-4-E2B | 2.0B | 79.2 | 71.0 | 89.3 | 84.7 | 66.0 | 58.6 |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "translated/Lara-4-Mini-Base"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
inputs = tokenizer("La Divina Commedia è un poema scritto da", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

- Requires `transformers>=5.14` (tested up to 5.22.2).
- Install [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) for fast Gated DeltaNet
  kernels on GPU. Without it, a plain PyTorch path is used, which is exact but slow on long inputs.
- The model was trained without a BOS token: do not prepend one. The tokenizer already does not add it.
- To fine-tune, load with `output_router_logits=True`. The loss then includes the router load-balancing loss
  (`router_aux_loss_coef`, 0.001 as in pretraining), which keeps the experts balanced.

## Limitations

- A base model: it continues text. It does not follow instructions or hold a conversation, and it has no safety
  tuning. It can produce text that is wrong, biased or offensive.
- Behavior beyond 131,072 tokens is untested.

## Third-party terms that may apply

Part of the training data is synthetic text that NVIDIA generated with third-party models. The cards of these
datasets state that a model trained on them "may be subject to redistribution and use requirements" in the
licenses of the models that generated the data:

- **Qwen.** [Nemotron-CC v2](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) and
  [Nemotron-Pretraining-SFT v1](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-SFT-v1) contain data
  generated with Qwen2.5 and Qwen3 models, and point to the
  [Qwen License Agreement](https://huggingface.co/Qwen/Qwen2.5-72B/blob/main/LICENSE) (copy in
  `LICENSES/Qwen-LICENSE-AGREEMENT.txt`). **Built with Qwen.**
- **DeepSeek.** Nemotron-CC v2, Nemotron-Pretraining-SFT v1 and the Multiple-Choice and Generative subsets of
  [Nemotron-Pretraining-Specialized v1.1](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.1)
  and [v1.2](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.2) contain data generated
  with DeepSeek models, and point to the
  [DeepSeek License Agreement](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL) (copy in
  `LICENSES/DeepSeek-LICENSE-MODEL.txt`). Do not use Lara Mini base, or models derived from it, for the uses
  listed in Attachment A of that agreement, and pass this restriction on when you distribute them.
- [Nemotron-CC-Math v1](https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1) contains data generated with
  Phi-4, whose [license](https://huggingface.co/microsoft/phi-4/blob/main/LICENSE) is MIT.

## License

Lara Mini base is released under the
[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)
license (CC BY-NC-SA 4.0); see `LICENSE`. You may share and adapt it for non-commercial purposes, giving
appropriate credit, and must distribute adaptations under the same license. The code and the tokenizer are
under the Apache License 2.0 (see `NOTICE` and `LICENSE-APACHE-2.0`).

For commercial use or other uses not covered by this license, we are open to licensing Lara Mini base under
more permissive terms. If you are interested, please contact us at
[research@translated.com](mailto:research@translated.com).

## Citation

```bibtex
@techreport{translated2026laraminibase,
  title       = {Lara Mini Base},
  author      = {{Translated, Lara Team}},
  institution = {Translated},
  year        = {2026},
  month       = nov,
  type        = {Technical Report},
  url         = {https://huggingface.co/translated/Lara-4-Mini-Base}
}
```

	---
	library_name: transformers
	pipeline_tag: text-generation
	license: cc-by-nc-sa-4.0
	extra_gated_prompt: >-
	Lara Mini base is released under the CC BY-NC-SA 4.0 license, for non-commercial use only.
	Access is granted for non-commercial purposes, such as research and reproducibility studies.
	Please share your name and affiliation: each request is reviewed by our team.
	extra_gated_fields:
	First name: text
	Last name: text
	Affiliation: text
	I agree to use this model for non-commercial purposes only, under the terms of its license: checkbox
	extra_gated_button_content: Request access
	language:
	- en
	- ar
	- az
	- bg
	- bn
	- bs
	- ca
	- cs
	- da
	- de
	- el
	- es
	- et
	- fa
	- fi
	- fr
	- ga
	- he
	- hi
	- hr
	- hu
	- id
	- it
	- ja
	- ko
	- lt
	- lv
	- ms
	- mt
	- nl
	- "no"
	- pl
	- pt
	- ro
	- ru
	- sk
	- sl
	- sr
	- sv
	- th
	- tr
	- uk
	- vi
	- zh
	tags:
	- base-model
	- mixture-of-experts
	- hybrid
	- gated-delta-net
	- multilingual
	- long-context
	datasets:
	- HuggingFaceFW/fineweb-edu
	- HuggingFaceFW/fineweb-2
	- epfml/FineWeb2-HQ
	- epfml/FineWeb2-HQ-QA
	- HuggingFaceFW/finepdfs
	- HuggingFaceFW/finewiki
	- HPLT/HPLT3.0
	- uonlp/CulturaX
	- HuggingFaceCode/stack-v3-train
	- HuggingFaceTB/finemath
	- IFM/MegaMath
	- nvidia/Nemotron-CC-v2
	- nvidia/Nemotron-CC-Math-v1
	- nvidia/Nemotron-CC-Code-v1
	- nvidia/Nemotron-Pretraining-Code-v2
	- nvidia/Nemotron-Pretraining-SFT-v1
	- nvidia/Nemotron-Pretraining-Specialized-v1
	- nvidia/Nemotron-Pretraining-Specialized-v1.1
	- nvidia/Nemotron-Pretraining-Specialized-v1.2
	- nvidia/Nemotron-Pretraining-Legal-v1
	---

	# Lara Mini Base

	Read more in our [blog post](https://laratranslate.com/research).

	Lara Mini base is a multilingual mixture-of-experts base model by [Translated](https://translated.com), with
	7.2B total and 0.6B active parameters per token, pretrained from scratch on 6T tokens and extended to a
	context of 128k tokens.

	![Overall and domain averages of Lara Mini base and of the best other open base models with up to 2B active parameters](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/teaser.png)
	*Overall and domain averages of Lara Mini base and of the four best other models with up to 2B active parameters on
	each average.*

	- Multilingual by design. About a third of the training data (2.18T tokens) is in 44 languages besides English (counting Simplified and Traditional Chinese separately),
	including all 24 official languages of the European Union.
	- Strong for its size. Among the 36 open base models with up to 2B active parameters that we evaluated, it has
	the highest average over English, multilingual, math and code benchmarks (57.0), ahead of models trained on 4 to 6
	times as many tokens.
	- Fast. It needs the compute per token of a 0.6B dense model. For a single user on one A100 GPU, it generates
	about 1,100 characters per second, at least 1.8 times as fast as every model in our comparison with a higher
	average.
	- Long context with a small state. Only 7 of its 34 layers use global attention; the others are Gated DeltaNet
	and sliding-window attention layers, whose state does not grow with the input.

	**This is a base model: it continues text. It is not tuned to follow instructions or to chat, and it has no safety
	tuning.**

	## Model details

	![Architecture of Lara Mini base](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/architecture.png)

	\| \| \|
	\|---\|---\|
	\| Developer \| Translated SRL, the copyright holder and licensor \|
	\| Model type \| Decoder-only mixture of experts base model; hybrid of Gated DeltaNet, sliding-window and global attention \|
	\| Parameters \| 7.2B total, 0.6B active per token (embeddings included) \|
	\| Layers \| 34: 14 Gated DeltaNet (GDN), 13 sliding-window attention (window 512, RoPE), 7 global attention (no positional encoding) \|
	\| Layer pattern \| GDN, sliding, GDN, global, then (sliding, GDN, sliding, GDN, global) × 6 \|
	\| Hidden size \| 1,024 \|
	\| Attention \| 16 query heads, 4 key-value heads, head dimension 128, QK-norm \|
	\| Gated DeltaNet \| 8 key heads, 16 value heads, head dimension 128, short convolution of width 4 \|
	\| Mixture of experts \| In every layer: 128 routed experts (top-4, fp32 router) and 1 shared expert, all SwiGLU with hidden size 512 \|
	\| Vocabulary \| 151,936 tokens; input and output embeddings tied \|
	\| Context length \| 4,096 tokens in the first three stages; 131,072 after the long-context stage \|
	\| Precision \| bf16 \|
	\| Training data \| 6T tokens of public data: English and multilingual text, code, math and SFT-like data, then 100B tokens in the long-context stage \|
	\| Languages \| English and 44 others (Simplified and Traditional Chinese counted separately), which together cover all 24 official EU languages (see below) \|
	\| License \| CC BY-NC-SA 4.0 ([https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/)) \|
	\| Intended use \| Research, and a starting point for fine-tuning and post-training; a base model, not tuned to follow instructions or for safety \|
	\| Limitations \| See [Limitations](#limitations) \|

	Languages. English, Arabic, Azerbaijani, Bengali, Bosnian, Bulgarian, Catalan, Chinese (Simplified and
	Traditional), Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian,
	Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Maltese, Norwegian, Persian, Polish,
	Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Thai, Turkish, Ukrainian and Vietnamese.

	## Training

	Data. All training data come from publicly available datasets (some of them released under access agreements),
	which we deduplicated and filtered into a pool of
	13.8T tokens. Training read 6T tokens from this pool, 97% of them unique. The anneal stage skips the training
	sequences that contain documents matching items of our evaluation benchmarks.

	\| Domain \| Main sources \| Tokens \| Share \|
	\|---\|---\|---:\|---:\|
	\| English \| Nemotron-CC and its synthetic data, FineWeb-Edu, FinePDFs, FineWiki \| 2.49T \| 41.4% \|
	\| Multilingual \| FineWeb-2, FineWeb2-HQ, HPLT 3.0, CulturaX, FinePDFs, FineWiki, synthetic question-answer data \| 2.18T \| 36.4% \|
	\| Code \| Nemotron-CC code pages, The Stack v3 (permissive files), Nemotron synthetic code \| 0.77T \| 12.8% \|
	\| Math \| FineMath, MegaMath, Nemotron-CC-Math \| 0.35T \| 5.9% \|
	\| SFT-like \| Nemotron pretraining collections (question-answer, math and code examples) \| 0.21T \| 3.5% \|

	We release no training data with this model.

	Stages. Pretraining has three stages (main, transition and anneal) over 6T tokens at a sequence length of
	4,096. Each stage moves toward higher-quality, question-answer, code and math data, while multilingual text stays at
	about a third. A fourth stage of 100B tokens, in sequences of 32,768 and then 131,072 tokens, extends the context to
	128k.

	Recipe. Muon for the weight matrices and AdamW for the other parameters, peak learning rate 8e-4, global batch
	of about 8.4M tokens, constant learning rate for 90% of training and then a decay to zero.

	## Evaluation

	We evaluated Lara Mini base and 56 open base models ourselves, all with the same prompts, settings and answer
	extraction (lm-evaluation-harness, few-shot or zero-shot, no chat template).

	- English: NLI, PAWS, Belebele, COPA, PIQA, HellaSwag, WinoGrande, ARC-Challenge, MMLU, MMLU-Pro
	- Multilingual: XNLI, PAWS-X, Belebele, XCOPA, Global PIQA, T-HellaSwag, INCLUDE, Global MMLU, averaged over the
	languages of Lara Mini base
	- Math: GSM8K, MATH, MGSM
	- Code: MBPP, MBPP+, CRUXEval-I, CRUXEval-O

	Overall is the mean of the four domain averages. Models are grouped by active parameters.

	\| Model \| Active \| Total \| Tokens \| English \| Multilingual \| Math \| Code \| Overall \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Lara Mini base \| 0.6B \| 7.2B \| 6T \| 66.4 \| 56.7 \| 59.9 \| 45.1 \| 57.0 \|
	\| granite-4.0-h-tiny \| 1.0B \| 6.9B \| 22.5T \| 66.2 \| 51.8 \| 49.4 \| 48.4 \| 53.9 \|
	\| Qwen3-1.7B \| 1.7B \| 1.7B \| 36T \| 63.8 \| 50.3 \| 54.6 \| 46.6 \| 53.8 \|
	\| Falcon-H1-1.5B-Deep \| 1.6B \| 1.6B \| 3T \| 65.3 \| 45.2 \| 55.2 \| 47.2 \| 53.2 \|
	\| Hunyuan-1.8B \| 1.8B \| 1.8B \| -- \| 62.7 \| 44.3 \| 52.0 \| 49.7 \| 52.2 \|
	\| LFM2.5-8B-A1B \| 1.7B \| 8.5B \| 38T† \| 65.7 \| 48.6 \| 52.3 \| 36.6 \| 50.8 \|
	\| Trinity-Nano \| 1.0B \| 6.1B \| 10T \| 68.4 \| 47.4 \| 45.0 \| 42.0 \| 50.7 \|
	\| Qwen3.5-2B \| 1.9B \| 1.9B \| -- \| 63.3 \| 50.0 \| 45.6 \| 36.7 \| 48.9 \|
	\| Gemma-4-E2B \| 2.0B \| 5.1B \| -- \| 63.3 \| 53.3 \| 18.0 \| 32.8 \| 41.9 \|
	\| Larger models \| \| \| \| \| \| \| \| \|
	\| Apertus-v1.5-8B \| 8.0B \| 8.0B \| 19T \| 69.0 \| 59.2 \| 50.6 \| 44.3 \| 55.8 \|
	\| Gemma-4-E4B \| 4.0B \| 7.9B \| -- \| 70.1 \| 61.1 \| 44.4 \| 46.0 \| 55.4 \|
	\| Ministral-3-3B \| 3.4B \| 3.4B \| 1-3T† \| 67.7 \| 55.4 \| 49.9 \| 42.3 \| 53.8 \|
	\| Llama-3.1-8B \| 8.0B \| 8.0B \| 15T \| 69.6 \| 55.2 \| 36.6 \| 43.6 \| 51.2 \|

	Tokens: pretraining tokens reported by each developer (--: not reported; †: distilled from a larger model during
	pretraining, wholly or in part). For Lara Mini base, the 6T of the first three stages; the long-context stage adds
	100B. Gemma-4 models are listed at their effective sizes.

	Speed. We measured the generation speed of 45 of these models on one A100 GPU, in bf16 without quantization.
	Speed is in characters per second, so that models with tokenizers of different density compare fairly. Every model
	with a higher Overall average than Lara Mini base generates text at least 1.8 times more slowly.

	![Overall average against generation speed for one user](https://huggingface.co/datasets/translated/LaraMini-assets/resolve/main/pareto-speed.png)
	*Overall average against generation speed for a single user (characters per second, log scale) for Lara Mini base
	and 45 open base models; the line is the Pareto frontier.*

	Long context. RULER (13 English tasks), ONERULER (4 retrieval tasks in 26 languages) and HELMET (4 categories),
	with tasks whose score does not depend on the text a base model generates after its answer.

	\| Model \| Active \| RULER 32k \| RULER 128k \| ONERULER 32k \| ONERULER 128k \| HELMET 32k \| HELMET 128k \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Lara Mini base \| 0.6B \| 76.6 \| 63.7 \| 90.1 \| 80.9 \| 63.0 \| 54.7 \|
	\| Trinity-Nano \| 1.0B \| 71.0 \| 61.8 \| 73.9 \| 62.1 \| 54.9 \| 51.8 \|
	\| granite-4.0-h-tiny \| 1.0B \| 29.5 \| 26.4 \| 19.5 \| 14.1 \| 41.0 \| 36.3 \|
	\| Falcon-H1-1.5B-Deep \| 1.6B \| 59.7 \| 38.7 \| 60.5 \| 38.8 \| 45.5 \| 34.4 \|
	\| Qwen3.5-2B \| 1.9B \| 83.7 \| 73.8 \| 87.7 \| 77.4 \| 64.0 \| 56.8 \|
	\| Gemma-4-E2B \| 2.0B \| 79.2 \| 71.0 \| 89.3 \| 84.7 \| 66.0 \| 58.6 \|

	## Usage

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	path = "translated/Lara-4-Mini-Base"
	tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")

	inputs = tokenizer("La Divina Commedia è un poema scritto da", return_tensors="pt").to(model.device)
	out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
	print(tokenizer.decode(out[0], skip_special_tokens=True))
	```

	- Requires `transformers>=5.14` (tested up to 5.22.2).
	- Install [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) for fast Gated DeltaNet
	kernels on GPU. Without it, a plain PyTorch path is used, which is exact but slow on long inputs.
	- The model was trained without a BOS token: do not prepend one. The tokenizer already does not add it.
	- To fine-tune, load with `output_router_logits=True`. The loss then includes the router load-balancing loss
	(`router_aux_loss_coef`, 0.001 as in pretraining), which keeps the experts balanced.

	## Limitations

	- A base model: it continues text. It does not follow instructions or hold a conversation, and it has no safety
	tuning. It can produce text that is wrong, biased or offensive.
	- Behavior beyond 131,072 tokens is untested.

	## Third-party terms that may apply

	Part of the training data is synthetic text that NVIDIA generated with third-party models. The cards of these
	datasets state that a model trained on them "may be subject to redistribution and use requirements" in the
	licenses of the models that generated the data:

	- Qwen. [Nemotron-CC v2](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) and
	[Nemotron-Pretraining-SFT v1](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-SFT-v1) contain data
	generated with Qwen2.5 and Qwen3 models, and point to the
	[Qwen License Agreement](https://huggingface.co/Qwen/Qwen2.5-72B/blob/main/LICENSE) (copy in
	`LICENSES/Qwen-LICENSE-AGREEMENT.txt`). Built with Qwen.
	- DeepSeek. Nemotron-CC v2, Nemotron-Pretraining-SFT v1 and the Multiple-Choice and Generative subsets of
	[Nemotron-Pretraining-Specialized v1.1](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.1)
	and [v1.2](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.2) contain data generated
	with DeepSeek models, and point to the
	[DeepSeek License Agreement](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL) (copy in
	`LICENSES/DeepSeek-LICENSE-MODEL.txt`). Do not use Lara Mini base, or models derived from it, for the uses
	listed in Attachment A of that agreement, and pass this restriction on when you distribute them.
	- [Nemotron-CC-Math v1](https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1) contains data generated with
	Phi-4, whose [license](https://huggingface.co/microsoft/phi-4/blob/main/LICENSE) is MIT.

	## License

	Lara Mini base is released under the
	[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)
	license (CC BY-NC-SA 4.0); see `LICENSE`. You may share and adapt it for non-commercial purposes, giving
	appropriate credit, and must distribute adaptations under the same license. The code and the tokenizer are
	under the Apache License 2.0 (see `NOTICE` and `LICENSE-APACHE-2.0`).

	For commercial use or other uses not covered by this license, we are open to licensing Lara Mini base under
	more permissive terms. If you are interested, please contact us at
	[research@translated.com](mailto:research@translated.com).

	## Citation

	```bibtex
	@techreport{translated2026laraminibase,
	title = {Lara Mini Base},
	author = {{Translated, Lara Team}},
	institution = {Translated},
	year = {2026},
	month = nov,
	type = {Technical Report},
	url = {https://huggingface.co/translated/Lara-4-Mini-Base}
	}
	```