Domyn Edge
Collection
5 items • Updated
Domyn Edge Base
Phased release. Domyn Edge is a rolling release in three bundles. The first, released on October 20, 2026, includes this base model and
domyn/Domyn-Edge-SFT, a supervised fine-tuned version that chats and calls tools. Further post-training checkpoints follow, up to the fully post-trained model in the final bundle. See Open resources for everything released so far.
This repository holds the pre-trained base model. It has no instruction tuning or alignment, so it is not suited to conversational or user-facing applications. Intended uses are continued pre-training, fine-tuning, in-context learning experiments and research into how capabilities form during training.
Table of contents
Model summary
Domyn Edge is a 4B-parameter dense transformer trained from scratch on 13 trillion tokens, end to end on European public infrastructure, and built for agentic work at long context.
Key features
- Agentic focus: the mid-training mix includes multi-turn tool-calling and agentic coding trajectories.
- Fully open: weights, intermediate checkpoints, pre-training data and training code are all released, so the model can be audited, reproduced and continued rather than only run.
- Multilingual: five primary European languages (English, Italian, Spanish, French, German), with a further eight languages at lower token counts.
- Long context: trained natively to a 1M-token context, without YaRN or RoPE scaling; 256k is the recommended working context.
Architecture at a glance
- Type: causal language model, dense decoder-only transformer, pre-trained only
- Parameters: 4B (3.93B)
- Layers: 36, hidden size 2560, arranged as 2 full-attention, 8 × (3 sliding-window + 1 full-attention), 2 full-attention
- Vocabulary: 115,272 tokens (115,256 regular, 16 special); input and output embeddings are tied
- Feed-forward: SwiGLU, intermediate dimension 9728
- Normalization: sandwich RMSNorm (epsilon 1e-6); QK normalization with RMSNorm, parametrized on keys only
- Context length: 1,048,576 trained, 262,144 recommended
- Training precision: bfloat16
| Attention | Layers | Q / KV heads | Head dim | Span | Position encoding |
|---|---|---|---|---|---|
| Sliding window | 24 | 32 / 8 | 64 | 1024 tokens | RoPE, base 10,000 |
| Full | 12 | 32 / 8 | 256 | full context | none (NoPE) |
How to use
Recommended environment
We tested Domyn Edge with these versions installed together. Other versions may work.
# CUDA 12 drivers (525+): cu129 or cu126. Drivers 580+: drop --index-url for the CUDA 13 build.
pip install "torch==2.13.0" --index-url https://download.pytorch.org/whl/cu129
pip install "transformers==5.16.1" "accelerate==1.14.0" "einops==0.8.2"
Transformers
With the recommended environment installed:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "domyn/Domyn-Edge-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
prompt = "The Leonardo supercomputer, hosted by CINECA, is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7,
top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
vLLM
Stock vLLM does not yet support Domyn Edge. Released vLLM versions stop at startup, even with
--trust-remote-code, with Model architectures ['DomynEdgeForCausalLM'] are not supported.
Native support is pending in vllm-project/vllm#62544.
Until it is merged and released, install vLLM from the PR branch. The branch follows vLLM main,
which requires PyTorch 2.14 built for CUDA 13, so it needs NVIDIA driver 580 or newer:
git clone https://github.com/vllm-project/vllm.git
cd vllm
git fetch origin pull/62544/head:domyn-edge
git checkout domyn-edge
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
VLLM_USE_PRECOMPILED=1 downloads prebuilt CUDA 13 kernels instead of compiling vLLM from source.
The install replaces the PyTorch build from the
recommended environment with PyTorch 2.14 for CUDA 13. On older
drivers, use Transformers until a vLLM release with Domyn Edge support is
available for CUDA 12.
Then serve the model:
vllm serve domyn/Domyn-Edge-Base \
--trust-remote-code \
--max-model-len 262144
The config sets max_position_embeddings to 262144, the recommended context, so the command
above works as shipped. To serve longer contexts, raise the limit at load time, for example to
512k:
vllm serve domyn/Domyn-Edge-Base \
--trust-remote-code \
--hf-overrides '{"max_position_embeddings": 1048576}' \
--max-model-len 524288 \
--gpu-memory-utilization 0.95
Evaluation
All numbers below are for the final checkpoint (13.03T tokens).
We compare against strong base models of up to 4B parameters. Bold marks the best score in each row; underline marks the best fully open model. SmolLM3 3B is the only other fully open model in the set, so it sits next to Domyn Edge.
Base suite
| Group | Benchmark | Domyn Edge 4B | SmolLM3 3B | Qwen3.5 4B | Gemma 3 4B | Ministral 3 3B |
|---|---|---|---|---|---|---|
| Reading comprehension | CoQA (EM) | 75.8 | 66.7 | 43.0 | 53.5 | 67.0 |
| SQuAD (EM) | 84.2 | 74.9 | 85.3 | 77.1 | 80.9 | |
| DROP (EM) | 53.9 | 36.0 | 58.3 | 36.7 | 50.0 | |
| Commonsense | Winogrande | 71.0 | 72.2 | 71.0 | 70.7 | 70.5 |
| HellaSwag | 55.6 | 55.6 | 56.3 | 56.7 | 54.7 | |
| PIQA | 76.0 | 76.1 | 88.1 | 72.1 | 79.5 | |
| SocialIQA | 75.8 | 67.9 | 77.2 | 66.1 | 72.7 | |
| Knowledge | NaturalQuestions (EM) | 20.8 | 20.2 | 19.6 | 21.8 | 19.8 |
| MMLU STEM | 57.2 | 54.4 | 73.5 | 51.4 | 65.6 | |
| MMLU Humanities | 69.3 | 65.8 | 77.3 | 64.7 | 74.2 | |
| MMLU Social sciences | 74.9 | 69.8 | 83.5 | 68.6 | 81.2 | |
| MMLU Other | 66.8 | 64.7 | 77.6 | 66.2 | 74.4 | |
| Reasoning and science | BBH (CoT) | 68.1 | 54.7 | 78.0 | 50.2 | 69.9 |
| AGIEval | 54.9 | 47.4 | 63.0 | 47.7 | 56.3 | |
| ARC-Challenge | 81.2 | 77.9 | 91.3 | 75.7 | 87.5 | |
| ARC-Easy | 89.4 | 87.1 | 93.6 | 86.4 | 92.0 | |
| SciQ | 92.1 | 90.3 | 96.3 | 91.2 | 95.2 | |
| GPQA Diamond | 29.8 | 28.8 | 44.9 | 31.8 | 37.9 | |
| Mathematics | GSM8K | 71.6 | 69.1 | 83.2 | 39.2 | 72.9 |
| Minerva MATH-500 (pass@1) | 39.8 | 36.0 | 51.6 | 24.7 | 39.2 | |
| Minerva MATH-500 (pass@16) | 48.5 | 40.7 | 59.7 | 32.4 | 45.9 | |
| Code | HumanEval (pass@1) | 51.8 | 32.7 | 60.2 | 37.2 | 47.5 |
| MBPP (pass@1) | 50.9 | 50.1 | 51.0 | 47.1 | 46.4 | |
| LBPP (pass@1) | 19.4 | 9.4 | 17.0 | 5.6 | 19.5 | |
| Multilingual | Global-MMLU IT | 59.8 | 53.8 | 71.5 | 57.0 | 66.2 |
| Global-MMLU ES | 58.8 | 59.5 | 70.8 | 55.7 | 67.7 | |
| Global-MMLU FR | 56.5 | 54.3 | 73.3 | 53.2 | 67.5 | |
| Global-MMLU DE | 59.0 | 56.3 | 68.0 | 53.0 | 67.2 | |
| In-house | Basic skills | 86.3 | 87.5 | 91.3 | 86.4 | 87.1 |
Long context
RULER scores by context length.
| Model | 4k | 8k | 16k | 32k | 64k | 128k | 256k | 512k |
|---|---|---|---|---|---|---|---|---|
| Domyn Edge 4B | 92.6 | 89.7 | 87.5 | 85.1 | 81.3 | 78.4 | 74.5 | 60.8 |
| Qwen3.5 4B | 86.5 | 87.3 | 85.7 | 86.9 | 84.4 | 79.8 | n/a | n/a |
Training
Data
The stage 1 pre-training mix is 8.44T tokens drawn from five source families: web, code, STEM, multilingual and synthetic. The full per-source lineage ships with the dataset repository.
Stages. Training ran in six stages over 13T tokens:
| Stage | Ends at | What it adds |
|---|---|---|
| 1. Pre-training | 8.5T | Web, code, STEM, multilingual and synthetic mix at 8k training context |
| 2. Specialized pre-training | 10.1T | Raised weight on reasoning, math and code at 16k training context |
| 3. Context extension | 10.3T | First extension to 64k training context |
| 4. Mid-training | 12.3T | Agentic tool-calling and coding trajectories, 32k training context |
| 5. Context extension | 12.7T | Second extension, reaching a 1M training context |
| 6. Annealing | 13T | Learning-rate decay in two steps: 64k, then 512k training context |
Infrastructure
- Hardware: 128 nodes × 4 custom NVIDIA A100 64GB GPUs (512 in total) on Leonardo at CINECA. NVLink 3.0 (200GB/s) within a node; two dual-port HDR100 interfaces (400Gbps) per node.
- Framework: TorchTitan, with Domyn Forge as the training code on top of it.
- Parallelism: hybrid FSDP intra-node and DDP inter-node. Global batch of 1024 sequences (2 per GPU), about 8M tokens per step at 8k context. For the long-context stages, context parallelism scales up to 16 to reach 1M tokens.
- Precision: bfloat16 FSDP-style mixed precision. Master weights, gradients and optimizer states in float32; live weights and activations in bfloat16; cross-entropy in float32.
Tokenizer
The tokenizer extends the GPT-4 BPE vocabulary with tokens for the five primary European languages. We stopped at 115k tokens, the elbow of the parity-versus-vocabulary-size curve; with 16 special tokens the embedding has 115,272 entries. Parity is the token count of a text relative to its English translation; fertility is tokens per word. Lower is better for both.
| Metric | GPT-4 (100k) | Domyn Edge (115k) | Change |
|---|---|---|---|
| Parity, Italian | 1.64 | 1.58 | −3.7% |
| Parity, Spanish | 1.54 | 1.51 | −1.9% |
| Parity, French | 1.61 | 1.56 | −3.1% |
| Parity, German | 1.59 | 1.48 | −6.9% |
| Parity, average | 1.60 | 1.53 | −4.4% |
| Fertility, code | 2.88 | 2.88 | 0.0% |
| Fertility, English | 1.40 | 1.40 | 0.0% |
| Fertility, multilingual | 2.32 | 2.25 | −3.0% |
| Fertility, average | 2.20 | 2.18 | −0.9% |
The added tokens lower parity in all four non-English primary languages, most of all in German, while leaving English and code fertility unchanged.
Open resources
For an overview of the release, see the launch blog post.
Released in bundle 1 (this release):
- Base model:
domyn/Domyn-Edge-Base - Intermediate checkpoints:
domyn/Domyn-Edge-Base-checkpoints, 14 branches: one per trillion tokens through 8T, the end of stage 1 at 8.5T, and the endpoint of every later stage - Stage 1 pre-training dataset: the 8.44T mix above, with per-source provenance, filtering and token counts
- SFT model:
domyn/Domyn-Edge-SFT, with the recipes that reproduce it - Domyn Forge: the training code, pre-training and fine-tuning recipes
- SDAG: the DAG execution engine with SLURM integration that ran the data pipeline
- Domyn Swarm: a toolkit to spin up vLLM endpoints and run high-throughput inference jobs on SLURM and DGX Cloud Lepton
- Collection: Domyn Edge on Hugging Face
Coming in later bundles: the remaining per-stage pre-training datasets, the SFT post-training mix, the teacher models and their post-training data (bundle 2), and the fully post-trained model (bundle 3). The technical report is expected in November 2026.
Limitations
Recommended context is 256k, not 1M. The model was trained to a 1M-token context, but RULER scores degrade past 256k. Treat 256k as the supported working context and anything beyond it as experimental.
Prompt injection on code inputs. The base model is susceptible to instructions embedded in code passed to it as input. Do not place untrusted code in the context of an application that acts on the model's output.
No alignment and no safety tuning. This is a pre-trained checkpoint. It has had no instruction tuning, no preference optimization and no safety work. It will complete harmful, biased or false text as readily as any other text, and it has no refusal behavior.
Uneven multilingual coverage. English, Italian, Spanish, French and German are the primary languages. The remaining eight (Portuguese, Russian, Chinese, Japanese, Arabic, Dutch, Indonesian, Turkish) each carry between 0.40% and 1.09% of the pre-training mix, so coverage is limited.
License
This model is released under Apache 2.0.
Citation
If you find this work valuable, please consider citing it:
@misc{domynedge2026,
title = {Domyn Edge},
author = {Domyn},
year = {2026},
note = {Technical report, forthcoming in November 2026.},
}
- Downloads last month
- 18