| --- |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| language: |
| - en |
| - it |
| - es |
| - fr |
| - de |
| - pt |
| - ru |
| - zh |
| - ja |
| - ar |
| - nl |
| - id |
| - tr |
| tags: |
| - domyn |
| - domyn-edge |
| - long-context |
| - agentic |
| - fully-open |
| datasets: |
| - domyn/Domyn-Edge-Pretraining-Mix-8.4T |
| --- |
| |
| <p align="center"> |
| <picture> |
| <source media="(prefers-color-scheme: dark)" srcset="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835cb0fb6bd22a7cba7d645_domyn-logo-payoff-primary-white.svg"> |
| <img alt="Domyn" src="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835ca86622d317630fa3861_domyn-logo-primary.svg" width="400"> |
| </picture> |
| </p> |
| |
| # Domyn Edge Base |
|
|
| > [!IMPORTANT] |
| > **Phased release.** Domyn Edge is a rolling release in three bundles. The first, released on |
| > October 20, 2026, includes this base model and |
| > [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT), a supervised fine-tuned |
| > version that chats and calls tools. Further post-training checkpoints follow, up to the fully |
| > post-trained model in the final bundle. See [Open resources](#open-resources) for everything |
| > released so far. |
|
|
| > [!Note] |
| > This repository holds the **pre-trained base model**. It has no instruction tuning or |
| > alignment, so it is not suited to conversational or user-facing applications. Intended uses |
| > are continued pre-training, fine-tuning, in-context learning experiments and research into |
| > how capabilities form during training. |
|
|
| ## Table of contents |
|
|
| 1. [Model summary](#model-summary) |
| 2. [How to use](#how-to-use) |
| 3. [Evaluation](#evaluation) |
| 4. [Training](#training) |
| 5. [Limitations](#limitations) |
| 6. [License](#license) |
| 7. [Citation](#citation) |
|
|
| ## Model summary |
|
|
| Domyn Edge is a 4B-parameter dense transformer trained from scratch on 13 trillion tokens, end |
| to end on European public infrastructure, and built for agentic work at long context. |
|
|
| **Key features** |
|
|
| - **Agentic focus**: the mid-training mix includes multi-turn tool-calling and agentic coding |
| trajectories. |
| - **Fully open**: weights, intermediate checkpoints, pre-training data and training code are |
| all released, so the model can be audited, reproduced and continued rather than only run. |
| - **Multilingual**: five primary European languages (English, Italian, Spanish, French, |
| German), with a further eight languages at lower token counts. |
| - **Long context**: trained natively to a 1M-token context, without YaRN or RoPE scaling; 256k |
| is the recommended working context. |
|
|
| **Architecture at a glance** |
|
|
| - Type: causal language model, dense decoder-only transformer, pre-trained only |
| - Parameters: 4B (3.93B) |
| - Layers: 36, hidden size 2560, arranged as 2 full-attention, 8 × (3 sliding-window + 1 |
| full-attention), 2 full-attention |
| - Vocabulary: 115,272 tokens (115,256 regular, 16 special); input and output embeddings are tied |
| - Feed-forward: SwiGLU, intermediate dimension 9728 |
| - Normalization: sandwich RMSNorm (epsilon 1e-6); QK normalization with RMSNorm, parametrized |
| on keys only |
| - Context length: 1,048,576 trained, 262,144 recommended |
| - Training precision: bfloat16 |
|
|
| | Attention | Layers | Q / KV heads | Head dim | Span | Position encoding | |
| |---|---:|---:|---:|---|---| |
| | Sliding window | 24 | 32 / 8 | 64 | 1024 tokens | RoPE, base 10,000 | |
| | Full | 12 | 32 / 8 | 256 | full context | none (NoPE) | |
|
|
| ## How to use |
|
|
| ### Recommended environment |
|
|
| We tested Domyn Edge with these versions installed together. Other versions may work. |
|
|
| ```bash |
| # CUDA 12 drivers (525+): cu129 or cu126. Drivers 580+: drop --index-url for the CUDA 13 build. |
| pip install "torch==2.13.0" --index-url https://download.pytorch.org/whl/cu129 |
| pip install "transformers==5.16.1" "accelerate==1.14.0" "einops==0.8.2" |
| ``` |
|
|
| ### Transformers |
|
|
| With the recommended environment installed: |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "domyn/Domyn-Edge-Base" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| dtype=torch.bfloat16, |
| device_map="auto", |
| trust_remote_code=True, |
| ) |
| |
| prompt = "The Leonardo supercomputer, hosted by CINECA, is" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=128, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.95, |
| ) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ### vLLM |
|
|
| **Stock vLLM does not yet support Domyn Edge.** Released vLLM versions stop at startup, even with |
| `--trust-remote-code`, with `Model architectures ['DomynEdgeForCausalLM'] are not supported`. |
|
|
| Native support is pending in [vllm-project/vllm#62544](https://github.com/vllm-project/vllm/pull/62544). |
| Until it is merged and released, install vLLM from the PR branch. The branch follows vLLM `main`, |
| which requires PyTorch 2.14 built for CUDA 13, so it needs **NVIDIA driver 580 or newer**: |
|
|
| ```bash |
| git clone https://github.com/vllm-project/vllm.git |
| cd vllm |
| git fetch origin pull/62544/head:domyn-edge |
| git checkout domyn-edge |
| VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto |
| ``` |
|
|
| `VLLM_USE_PRECOMPILED=1` downloads prebuilt CUDA 13 kernels instead of compiling vLLM from source. |
| The install replaces the PyTorch build from the |
| [recommended environment](#recommended-environment) with PyTorch 2.14 for CUDA 13. On older |
| drivers, use [Transformers](#transformers) until a vLLM release with Domyn Edge support is |
| available for CUDA 12. |
|
|
| Then serve the model: |
|
|
| ```bash |
| vllm serve domyn/Domyn-Edge-Base \ |
| --trust-remote-code \ |
| --max-model-len 262144 |
| ``` |
|
|
| The config sets `max_position_embeddings` to 262144, the recommended context, so the command |
| above works as shipped. To serve longer contexts, raise the limit at load time, for example to |
| 512k: |
|
|
| ```bash |
| vllm serve domyn/Domyn-Edge-Base \ |
| --trust-remote-code \ |
| --hf-overrides '{"max_position_embeddings": 1048576}' \ |
| --max-model-len 524288 \ |
| --gpu-memory-utilization 0.95 |
| ``` |
|
|
| ## Evaluation |
|
|
| All numbers below are for the final checkpoint (**13.03T tokens**). |
|
|
| We compare against strong base models of up to 4B parameters. **Bold** marks the best score in |
| each row; <u>underline</u> marks the best fully open model. SmolLM3 3B is the only other fully |
| open model in the set, so it sits next to Domyn Edge. |
|
|
| ### Base suite |
|
|
| | Group | Benchmark | Domyn Edge 4B | SmolLM3 3B | Qwen3.5 4B | Gemma 3 4B | Ministral 3 3B | |
| |---|---|---:|---:|---:|---:|---:| |
| | Reading comprehension | CoQA (EM) | <u>**75.8**</u> | 66.7 | 43.0 | 53.5 | 67.0 | |
| | | SQuAD (EM) | <u>84.2</u> | 74.9 | **85.3** | 77.1 | 80.9 | |
| | | DROP (EM) | <u>53.9</u> | 36.0 | **58.3** | 36.7 | 50.0 | |
| | Commonsense | Winogrande | 71.0 | <u>**72.2**</u> | 71.0 | 70.7 | 70.5 | |
| | | HellaSwag | <u>55.6</u> | <u>55.6</u> | 56.3 | **56.7** | 54.7 | |
| | | PIQA | 76.0 | <u>76.1</u> | **88.1** | 72.1 | 79.5 | |
| | | SocialIQA | <u>75.8</u> | 67.9 | **77.2** | 66.1 | 72.7 | |
| | Knowledge | NaturalQuestions (EM) | <u>20.8</u> | 20.2 | 19.6 | **21.8** | 19.8 | |
| | | MMLU STEM | <u>57.2</u> | 54.4 | **73.5** | 51.4 | 65.6 | |
| | | MMLU Humanities | <u>69.3</u> | 65.8 | **77.3** | 64.7 | 74.2 | |
| | | MMLU Social sciences | <u>74.9</u> | 69.8 | **83.5** | 68.6 | 81.2 | |
| | | MMLU Other | <u>66.8</u> | 64.7 | **77.6** | 66.2 | 74.4 | |
| | Reasoning and science | BBH (CoT) | <u>68.1</u> | 54.7 | **78.0** | 50.2 | 69.9 | |
| | | AGIEval | <u>54.9</u> | 47.4 | **63.0** | 47.7 | 56.3 | |
| | | ARC-Challenge | <u>81.2</u> | 77.9 | **91.3** | 75.7 | 87.5 | |
| | | ARC-Easy | <u>89.4</u> | 87.1 | **93.6** | 86.4 | 92.0 | |
| | | SciQ | <u>92.1</u> | 90.3 | **96.3** | 91.2 | 95.2 | |
| | | GPQA Diamond | <u>29.8</u> | 28.8 | **44.9** | 31.8 | 37.9 | |
| | Mathematics | GSM8K | <u>71.6</u> | 69.1 | **83.2** | 39.2 | 72.9 | |
| | | Minerva MATH-500 (pass@1) | <u>39.8</u> | 36.0 | **51.6** | 24.7 | 39.2 | |
| | | Minerva MATH-500 (pass@16) | <u>48.5</u> | 40.7 | **59.7** | 32.4 | 45.9 | |
| | Code | HumanEval (pass@1) | <u>51.8</u> | 32.7 | **60.2** | 37.2 | 47.5 | |
| | | MBPP (pass@1) | <u>50.9</u> | 50.1 | **51.0** | 47.1 | 46.4 | |
| | | LBPP (pass@1) | <u>19.4</u> | 9.4 | 17.0 | 5.6 | **19.5** | |
| | Multilingual | Global-MMLU IT | <u>59.8</u> | 53.8 | **71.5** | 57.0 | 66.2 | |
| | | Global-MMLU ES | 58.8 | <u>59.5</u> | **70.8** | 55.7 | 67.7 | |
| | | Global-MMLU FR | <u>56.5</u> | 54.3 | **73.3** | 53.2 | 67.5 | |
| | | Global-MMLU DE | <u>59.0</u> | 56.3 | **68.0** | 53.0 | 67.2 | |
| | In-house | Basic skills | 86.3 | <u>87.5</u> | **91.3** | 86.4 | 87.1 | |
|
|
| ### Long context |
|
|
| [RULER](https://github.com/NVIDIA/RULER) scores by context length. |
|
|
| | Model | 4k | 8k | 16k | 32k | 64k | 128k | 256k | 512k | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| |
| | Domyn Edge 4B | **92.6** | **89.7** | **87.5** | 85.1 | 81.3 | 78.4 | 74.5 | 60.8 | |
| | Qwen3.5 4B | 86.5 | 87.3 | 85.7 | **86.9** | **84.4** | **79.8** | n/a | n/a | |
|
|
| ## Training |
|
|
| ### Data |
|
|
| The stage 1 pre-training mix is **8.44T tokens** drawn from five source families: web, code, STEM, |
| multilingual and synthetic. The full |
| per-source lineage ships with the |
| [dataset repository](https://huggingface.co/datasets/domyn/Domyn-Edge-Pretraining-Mix-8.4T). |
|
|
| **Stages.** Training ran in six stages over 13T tokens: |
|
|
| | Stage | Ends at | What it adds | |
| |---|---|---| |
| | 1. Pre-training | 8.5T | Web, code, STEM, multilingual and synthetic mix at 8k training context | |
| | 2. Specialized pre-training | 10.1T | Raised weight on reasoning, math and code at 16k training context | |
| | 3. Context extension | 10.3T | First extension to 64k training context | |
| | 4. Mid-training | 12.3T | Agentic tool-calling and coding trajectories, 32k training context | |
| | 5. Context extension | 12.7T | Second extension, reaching a 1M training context | |
| | 6. Annealing | 13T | Learning-rate decay in two steps: 64k, then 512k training context | |
|
|
| ### Infrastructure |
|
|
| - **Hardware**: 128 nodes × 4 custom NVIDIA A100 64GB GPUs (512 in total) on Leonardo at |
| CINECA. NVLink 3.0 (200GB/s) within a node; two dual-port HDR100 interfaces (400Gbps) per node. |
| - **Framework**: [TorchTitan](https://github.com/pytorch/torchtitan), with |
| [Domyn Forge](#open-resources) as the training code on top of it. |
| - **Parallelism**: hybrid FSDP intra-node and DDP inter-node. Global batch of 1024 |
| sequences (2 per GPU), about 8M tokens per step at 8k context. For the long-context stages, |
| context parallelism scales up to 16 to reach 1M tokens. |
| - **Precision**: bfloat16 FSDP-style mixed precision. Master weights, gradients and optimizer |
| states in float32; live weights and activations in bfloat16; cross-entropy in float32. |
|
|
| ### Tokenizer |
|
|
| The tokenizer extends the GPT-4 BPE vocabulary with tokens for the five primary European |
| languages. We stopped at 115k tokens, the elbow of the parity-versus-vocabulary-size curve; |
| with 16 special tokens the embedding has 115,272 entries. Parity is the token count of a text |
| relative to its English translation; fertility is tokens per word. Lower is better for both. |
|
|
| | Metric | GPT-4 (100k) | Domyn Edge (115k) | Change | |
| |---|---:|---:|---:| |
| | Parity, Italian | 1.64 | **1.58** | −3.7% | |
| | Parity, Spanish | 1.54 | **1.51** | −1.9% | |
| | Parity, French | 1.61 | **1.56** | −3.1% | |
| | Parity, German | 1.59 | **1.48** | −6.9% | |
| | **Parity, average** | 1.60 | **1.53** | **−4.4%** | |
| | Fertility, code | 2.88 | 2.88 | 0.0% | |
| | Fertility, English | 1.40 | 1.40 | 0.0% | |
| | Fertility, multilingual | 2.32 | **2.25** | −3.0% | |
| | **Fertility, average** | 2.20 | **2.18** | **−0.9%** | |
|
|
| The added tokens lower parity in all four non-English primary languages, most of all in |
| German, while leaving English and code fertility unchanged. |
|
|
| ### Open resources |
|
|
| For an overview of the release, see the [launch blog post](https://domyn.webflow.io/blog/domyn-edge-compact-model-open). |
|
|
| Released in bundle 1 (this release): |
|
|
| - **Base model**: [`domyn/Domyn-Edge-Base`](https://huggingface.co/domyn/Domyn-Edge-Base) |
| - **Intermediate checkpoints**: [`domyn/Domyn-Edge-Base-checkpoints`](https://huggingface.co/domyn/Domyn-Edge-Base-checkpoints), 14 branches: one per trillion tokens through 8T, the end of stage 1 at 8.5T, and the endpoint of every later stage |
| - **Stage 1 pre-training dataset**: the 8.44T mix above, with per-source provenance, filtering and token counts |
| - **SFT model**: [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT), with the recipes that reproduce it |
| - **Domyn Forge**: [the training code](https://github.com/DomynAI/domyn-forge), pre-training and fine-tuning recipes |
| - **SDAG**: [the DAG execution engine](https://github.com/DomynAI/sdag) with SLURM integration that ran the data pipeline |
| - **Domyn Swarm**: [a toolkit](https://github.com/DomynAI/domyn-swarm) to spin up vLLM endpoints and run high-throughput inference jobs on SLURM and DGX Cloud Lepton |
| - **Collection**: [Domyn Edge](https://huggingface.co/collections/domyn/domyn-edge) on Hugging Face |
|
|
| Coming in later bundles: the remaining per-stage pre-training datasets, the SFT post-training mix, the teacher models and |
| their post-training data (bundle 2), and the fully post-trained model (bundle 3). The technical |
| report is expected in November 2026. |
|
|
| ## Limitations |
|
|
| **Recommended context is 256k, not 1M.** The model was trained to a 1M-token context, but |
| RULER scores degrade past 256k. Treat 256k as the supported working context and anything |
| beyond it as experimental. |
|
|
| **Prompt injection on code inputs.** The base model is susceptible to instructions embedded in |
| code passed to it as input. Do not place untrusted code in the context of an application that |
| acts on the model's output. |
|
|
| **No alignment and no safety tuning.** This is a pre-trained checkpoint. It has had no |
| instruction tuning, no preference optimization and no safety work. It will complete harmful, |
| biased or false text as readily as any other text, and it has no refusal behavior. |
|
|
| **Uneven multilingual coverage.** English, Italian, Spanish, French and German are the primary |
| languages. The remaining eight (Portuguese, Russian, Chinese, Japanese, Arabic, Dutch, |
| Indonesian, Turkish) each carry between 0.40% and 1.09% of the pre-training mix, so coverage |
| is limited. |
|
|
| ## License |
|
|
| This model is released under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
| ## Citation |
|
|
| If you find this work valuable, please consider citing it: |
|
|
| ```bibtex |
| @misc{domynedge2026, |
| title = {Domyn Edge}, |
| author = {Domyn}, |
| year = {2026}, |
| note = {Technical report, forthcoming in November 2026.}, |
| } |
| ``` |
|
|
| --- |
