README.md

---
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
language:
  - en
  - it
  - es
  - fr
  - de
  - pt
  - ru
  - zh
  - ja
  - ar
  - nl
  - id
  - tr
tags:
  - domyn
  - domyn-edge
  - long-context
  - agentic
  - fully-open
datasets:
  - domyn/Domyn-Edge-Pretraining-Mix-8.4T
---

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835cb0fb6bd22a7cba7d645_domyn-logo-payoff-primary-white.svg">
    <img alt="Domyn" src="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835ca86622d317630fa3861_domyn-logo-primary.svg" width="400">
  </picture>
</p>

# Domyn Edge Base

> [!IMPORTANT]
> **Phased release.** Domyn Edge is a rolling release in three bundles. The first, released on
> October 20, 2026, includes this base model and
> [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT), a supervised fine-tuned
> version that chats and calls tools. Further post-training checkpoints follow, up to the fully
> post-trained model in the final bundle. See [Open resources](#open-resources) for everything
> released so far.

> [!Note]
> This repository holds the **pre-trained base model**. It has no instruction tuning or
> alignment, so it is not suited to conversational or user-facing applications. Intended uses
> are continued pre-training, fine-tuning, in-context learning experiments and research into
> how capabilities form during training.

## Table of contents

1. [Model summary](#model-summary)
2. [How to use](#how-to-use)
3. [Evaluation](#evaluation)
4. [Training](#training)
5. [Limitations](#limitations)
6. [License](#license)
7. [Citation](#citation)

## Model summary

Domyn Edge is a 4B-parameter dense transformer trained from scratch on 13 trillion tokens, end
to end on European public infrastructure, and built for agentic work at long context.

**Key features**

- **Agentic focus**: the mid-training mix includes multi-turn tool-calling and agentic coding
  trajectories.
- **Fully open**: weights, intermediate checkpoints, pre-training data and training code are
  all released, so the model can be audited, reproduced and continued rather than only run.
- **Multilingual**: five primary European languages (English, Italian, Spanish, French,
  German), with a further eight languages at lower token counts.
- **Long context**: trained natively to a 1M-token context, without YaRN or RoPE scaling; 256k
  is the recommended working context.

**Architecture at a glance**

- Type: causal language model, dense decoder-only transformer, pre-trained only
- Parameters: 4B (3.93B)
- Layers: 36, hidden size 2560, arranged as 2 full-attention, 8 × (3 sliding-window + 1
  full-attention), 2 full-attention
- Vocabulary: 115,272 tokens (115,256 regular, 16 special); input and output embeddings are tied
- Feed-forward: SwiGLU, intermediate dimension 9728
- Normalization: sandwich RMSNorm (epsilon 1e-6); QK normalization with RMSNorm, parametrized
  on keys only
- Context length: 1,048,576 trained, 262,144 recommended
- Training precision: bfloat16

| Attention | Layers | Q / KV heads | Head dim | Span | Position encoding |
|---|---:|---:|---:|---|---|
| Sliding window | 24 | 32 / 8 | 64 | 1024 tokens | RoPE, base 10,000 |
| Full | 12 | 32 / 8 | 256 | full context | none (NoPE) |

## How to use

### Recommended environment

We tested Domyn Edge with these versions installed together. Other versions may work.

```bash
# CUDA 12 drivers (525+): cu129 or cu126. Drivers 580+: drop --index-url for the CUDA 13 build.
pip install "torch==2.13.0" --index-url https://download.pytorch.org/whl/cu129
pip install "transformers==5.16.1" "accelerate==1.14.0" "einops==0.8.2"
```

### Transformers

With the recommended environment installed:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "domyn/Domyn-Edge-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The Leonardo supercomputer, hosted by CINECA, is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### vLLM

**Stock vLLM does not yet support Domyn Edge.** Released vLLM versions stop at startup, even with
`--trust-remote-code`, with `Model architectures ['DomynEdgeForCausalLM'] are not supported`.

Native support is pending in [vllm-project/vllm#62544](https://github.com/vllm-project/vllm/pull/62544).
Until it is merged and released, install vLLM from the PR branch. The branch follows vLLM `main`,
which requires PyTorch 2.14 built for CUDA 13, so it needs **NVIDIA driver 580 or newer**:

```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git fetch origin pull/62544/head:domyn-edge
git checkout domyn-edge
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
```

`VLLM_USE_PRECOMPILED=1` downloads prebuilt CUDA 13 kernels instead of compiling vLLM from source.
The install replaces the PyTorch build from the
[recommended environment](#recommended-environment) with PyTorch 2.14 for CUDA 13. On older
drivers, use [Transformers](#transformers) until a vLLM release with Domyn Edge support is
available for CUDA 12.

Then serve the model:

```bash
vllm serve domyn/Domyn-Edge-Base \
  --trust-remote-code \
  --max-model-len 262144
```

The config sets `max_position_embeddings` to 262144, the recommended context, so the command
above works as shipped. To serve longer contexts, raise the limit at load time, for example to
512k:

```bash
vllm serve domyn/Domyn-Edge-Base \
  --trust-remote-code \
  --hf-overrides '{"max_position_embeddings": 1048576}' \
  --max-model-len 524288 \
  --gpu-memory-utilization 0.95
```

## Evaluation

All numbers below are for the final checkpoint (**13.03T tokens**).

We compare against strong base models of up to 4B parameters. **Bold** marks the best score in
each row; <u>underline</u> marks the best fully open model. SmolLM3 3B is the only other fully
open model in the set, so it sits next to Domyn Edge.

### Base suite

| Group | Benchmark | Domyn Edge 4B | SmolLM3 3B | Qwen3.5 4B | Gemma 3 4B | Ministral 3 3B |
|---|---|---:|---:|---:|---:|---:|
| Reading comprehension | CoQA (EM) | <u>**75.8**</u> | 66.7 | 43.0 | 53.5 | 67.0 |
|  | SQuAD (EM) | <u>84.2</u> | 74.9 | **85.3** | 77.1 | 80.9 |
|  | DROP (EM) | <u>53.9</u> | 36.0 | **58.3** | 36.7 | 50.0 |
| Commonsense | Winogrande | 71.0 | <u>**72.2**</u> | 71.0 | 70.7 | 70.5 |
|  | HellaSwag | <u>55.6</u> | <u>55.6</u> | 56.3 | **56.7** | 54.7 |
|  | PIQA | 76.0 | <u>76.1</u> | **88.1** | 72.1 | 79.5 |
|  | SocialIQA | <u>75.8</u> | 67.9 | **77.2** | 66.1 | 72.7 |
| Knowledge | NaturalQuestions (EM) | <u>20.8</u> | 20.2 | 19.6 | **21.8** | 19.8 |
|  | MMLU STEM | <u>57.2</u> | 54.4 | **73.5** | 51.4 | 65.6 |
|  | MMLU Humanities | <u>69.3</u> | 65.8 | **77.3** | 64.7 | 74.2 |
|  | MMLU Social sciences | <u>74.9</u> | 69.8 | **83.5** | 68.6 | 81.2 |
|  | MMLU Other | <u>66.8</u> | 64.7 | **77.6** | 66.2 | 74.4 |
| Reasoning and science | BBH (CoT) | <u>68.1</u> | 54.7 | **78.0** | 50.2 | 69.9 |
|  | AGIEval | <u>54.9</u> | 47.4 | **63.0** | 47.7 | 56.3 |
|  | ARC-Challenge | <u>81.2</u> | 77.9 | **91.3** | 75.7 | 87.5 |
|  | ARC-Easy | <u>89.4</u> | 87.1 | **93.6** | 86.4 | 92.0 |
|  | SciQ | <u>92.1</u> | 90.3 | **96.3** | 91.2 | 95.2 |
|  | GPQA Diamond | <u>29.8</u> | 28.8 | **44.9** | 31.8 | 37.9 |
| Mathematics | GSM8K | <u>71.6</u> | 69.1 | **83.2** | 39.2 | 72.9 |
|  | Minerva MATH-500 (pass@1) | <u>39.8</u> | 36.0 | **51.6** | 24.7 | 39.2 |
|  | Minerva MATH-500 (pass@16) | <u>48.5</u> | 40.7 | **59.7** | 32.4 | 45.9 |
| Code | HumanEval (pass@1) | <u>51.8</u> | 32.7 | **60.2** | 37.2 | 47.5 |
|  | MBPP (pass@1) | <u>50.9</u> | 50.1 | **51.0** | 47.1 | 46.4 |
|  | LBPP (pass@1) | <u>19.4</u> | 9.4 | 17.0 | 5.6 | **19.5** |
| Multilingual | Global-MMLU IT | <u>59.8</u> | 53.8 | **71.5** | 57.0 | 66.2 |
|  | Global-MMLU ES | 58.8 | <u>59.5</u> | **70.8** | 55.7 | 67.7 |
|  | Global-MMLU FR | <u>56.5</u> | 54.3 | **73.3** | 53.2 | 67.5 |
|  | Global-MMLU DE | <u>59.0</u> | 56.3 | **68.0** | 53.0 | 67.2 |
| In-house | Basic skills | 86.3 | <u>87.5</u> | **91.3** | 86.4 | 87.1 |

### Long context

[RULER](https://github.com/NVIDIA/RULER) scores by context length.

| Model | 4k | 8k | 16k | 32k | 64k | 128k | 256k | 512k |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Domyn Edge 4B | **92.6** | **89.7** | **87.5** | 85.1 | 81.3 | 78.4 | 74.5 | 60.8 |
| Qwen3.5 4B | 86.5 | 87.3 | 85.7 | **86.9** | **84.4** | **79.8** | n/a | n/a |

## Training

### Data

The stage 1 pre-training mix is **8.44T tokens** drawn from five source families: web, code, STEM,
multilingual and synthetic. The full
per-source lineage ships with the
[dataset repository](https://huggingface.co/datasets/domyn/Domyn-Edge-Pretraining-Mix-8.4T).

**Stages.** Training ran in six stages over 13T tokens:

| Stage | Ends at | What it adds |
|---|---|---|
| 1. Pre-training | 8.5T | Web, code, STEM, multilingual and synthetic mix at 8k training context |
| 2. Specialized pre-training | 10.1T | Raised weight on reasoning, math and code at 16k training context |
| 3. Context extension | 10.3T | First extension to 64k training context |
| 4. Mid-training | 12.3T | Agentic tool-calling and coding trajectories, 32k training context |
| 5. Context extension | 12.7T | Second extension, reaching a 1M training context |
| 6. Annealing | 13T | Learning-rate decay in two steps: 64k, then 512k training context |

### Infrastructure

- **Hardware**: 128 nodes × 4 custom NVIDIA A100 64GB GPUs (512 in total) on Leonardo at
  CINECA. NVLink 3.0 (200GB/s) within a node; two dual-port HDR100 interfaces (400Gbps) per node.
- **Framework**: [TorchTitan](https://github.com/pytorch/torchtitan), with
  [Domyn Forge](#open-resources) as the training code on top of it.
- **Parallelism**: hybrid FSDP intra-node and DDP inter-node. Global batch of 1024
  sequences (2 per GPU), about 8M tokens per step at 8k context. For the long-context stages,
  context parallelism scales up to 16 to reach 1M tokens.
- **Precision**: bfloat16 FSDP-style mixed precision. Master weights, gradients and optimizer
  states in float32; live weights and activations in bfloat16; cross-entropy in float32.

### Tokenizer

The tokenizer extends the GPT-4 BPE vocabulary with tokens for the five primary European
languages. We stopped at 115k tokens, the elbow of the parity-versus-vocabulary-size curve;
with 16 special tokens the embedding has 115,272 entries. Parity is the token count of a text
relative to its English translation; fertility is tokens per word. Lower is better for both.

| Metric | GPT-4 (100k) | Domyn Edge (115k) | Change |
|---|---:|---:|---:|
| Parity, Italian | 1.64 | **1.58** | −3.7% |
| Parity, Spanish | 1.54 | **1.51** | −1.9% |
| Parity, French | 1.61 | **1.56** | −3.1% |
| Parity, German | 1.59 | **1.48** | −6.9% |
| **Parity, average** | 1.60 | **1.53** | **−4.4%** |
| Fertility, code | 2.88 | 2.88 | 0.0% |
| Fertility, English | 1.40 | 1.40 | 0.0% |
| Fertility, multilingual | 2.32 | **2.25** | −3.0% |
| **Fertility, average** | 2.20 | **2.18** | **−0.9%** |

The added tokens lower parity in all four non-English primary languages, most of all in
German, while leaving English and code fertility unchanged.

### Open resources

For an overview of the release, see the [launch blog post](https://domyn.webflow.io/blog/domyn-edge-compact-model-open).

Released in bundle 1 (this release):

- **Base model**: [`domyn/Domyn-Edge-Base`](https://huggingface.co/domyn/Domyn-Edge-Base)
- **Intermediate checkpoints**: [`domyn/Domyn-Edge-Base-checkpoints`](https://huggingface.co/domyn/Domyn-Edge-Base-checkpoints), 14 branches: one per trillion tokens through 8T, the end of stage 1 at 8.5T, and the endpoint of every later stage
- **Stage 1 pre-training dataset**: the 8.44T mix above, with per-source provenance, filtering and token counts
- **SFT model**: [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT), with the recipes that reproduce it
- **Domyn Forge**: [the training code](https://github.com/DomynAI/domyn-forge), pre-training and fine-tuning recipes
- **SDAG**: [the DAG execution engine](https://github.com/DomynAI/sdag) with SLURM integration that ran the data pipeline
- **Domyn Swarm**: [a toolkit](https://github.com/DomynAI/domyn-swarm) to spin up vLLM endpoints and run high-throughput inference jobs on SLURM and DGX Cloud Lepton
- **Collection**: [Domyn Edge](https://huggingface.co/collections/domyn/domyn-edge) on Hugging Face

Coming in later bundles: the remaining per-stage pre-training datasets, the SFT post-training mix, the teacher models and
their post-training data (bundle 2), and the fully post-trained model (bundle 3). The technical
report is expected in November 2026.

## Limitations

**Recommended context is 256k, not 1M.** The model was trained to a 1M-token context, but
RULER scores degrade past 256k. Treat 256k as the supported working context and anything
beyond it as experimental.

**Prompt injection on code inputs.** The base model is susceptible to instructions embedded in
code passed to it as input. Do not place untrusted code in the context of an application that
acts on the model's output.

**No alignment and no safety tuning.** This is a pre-trained checkpoint. It has had no
instruction tuning, no preference optimization and no safety work. It will complete harmful,
biased or false text as readily as any other text, and it has no refusal behavior.

**Uneven multilingual coverage.** English, Italian, Spanish, French and German are the primary
languages. The remaining eight (Portuguese, Russian, Chinese, Japanese, Arabic, Dutch,
Indonesian, Turkish) each carry between 0.40% and 1.09% of the pre-training mix, so coverage
is limited.

## License

This model is released under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Citation

If you find this work valuable, please consider citing it:

```bibtex
@misc{domynedge2026,
  title  = {Domyn Edge},
  author = {Domyn},
  year   = {2026},
  note   = {Technical report, forthcoming in November 2026.},
}
```

---