README.md

---
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
language:
  - en
  - it
  - es
  - fr
  - de
  - pt
  - ru
  - zh
  - ja
  - ar
  - nl
  - id
  - tr
tags:
  - domyn
  - domyn-edge
  - checkpoints
  - long-context
  - fully-open
datasets:
  - domyn/Domyn-Edge-Pretraining-Mix-8.4T
---

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835cb0fb6bd22a7cba7d645_domyn-logo-payoff-primary-white.svg">
    <img alt="Domyn" src="https://cdn.prod.website-files.com/682dcb35e7fd7a313cae803a/6835ca86622d317630fa3861_domyn-logo-primary.svg" width="400">
  </picture>
</p>

# Domyn Edge Base Checkpoints

> [!IMPORTANT]
> **Phased release.** Domyn Edge is a rolling release in three bundles. The first, released on
> October 20, 2026, includes these checkpoints, the base model and
> [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT), a supervised fine-tuned
> version that chats and calls tools. Further post-training checkpoints follow, up to the fully
> post-trained model in the final bundle. See
> [Open resources](https://huggingface.co/domyn/Domyn-Edge-Base#open-resources) on the base model
> card for everything released so far.

> [!Note]
> This repository holds the **intermediate pre-training checkpoints** of Domyn Edge Base, one per
> branch, so that researchers can study how capabilities form over 13 trillion tokens. Every
> branch is a pre-trained checkpoint with no instruction tuning and no alignment.
>
> **The final checkpoint is not here.** It lives in
> [`domyn/Domyn-Edge-Base`](https://huggingface.co/domyn/Domyn-Edge-Base), together with the
> documentation of the architecture, usage, data and evaluation. This card covers only the
> checkpoint series.

## Table of contents

1. [What this repository contains](#what-this-repository-contains)
2. [How to load a checkpoint](#how-to-load-a-checkpoint)
3. [What changes across training](#what-changes-across-training)
4. [Limitations](#limitations)
5. [License](#license)
6. [Citation](#citation)

## What this repository contains

One branch per checkpoint. Branches are named by stage and by **tokens seen, not by step**.
Through pre-training we release a checkpoint every trillion tokens, plus the end of stage 1 at
8.5T; every later stage releases its endpoint.

| Branch | Phase | Tokens | Training context | Config max context |
|---|---|---|---|---|
| `stage1-pretraining-1T` … `-8T` | pre-training | one every 1T | 8k | 8k |
| `stage1-pretraining-8.5T` | pre-training, end of stage 1 | 8.5T | 8k | 8k |
| `stage2-specialized-10.1T` | specialized pre-training | 10.1T | 16k | 16k |
| `stage3-ctx-ext-10.3T` | first context extension | 10.3T | 64k | 64k |
| `stage4-midtraining-12.3T` | mid-training | 12.3T | 32k | 64k |
| `stage5-ctx-ext-12.7T` | second context extension | 12.7T | 1M | 1M |
| `stage6-annealing-13T` | annealing, end of 64k step | 13T | 64k | 256k |

Fourteen branches in total. The **final checkpoint**, at 13.03T tokens after the 512k annealing
step, is the model published at
[`domyn/Domyn-Edge-Base`](https://huggingface.co/domyn/Domyn-Edge-Base).

The stage 1 branches use a 115,269-token vocabulary; the three fill-in-the-middle tokens were
added at stage 2, giving the 115,272 tokens of every later branch and of the final model.

What each stage adds is described on the
[base model card](https://huggingface.co/domyn/Domyn-Edge-Base#data), together with the 8.44T-token
stage 1 mix that every branch was trained on. The mixes for stages 2 to 6 ship in a later bundle.

## How to load a checkpoint

With the [recommended environment](https://huggingface.co/domyn/Domyn-Edge-Base#recommended-environment)
installed, pass a branch name as `revision`. `main` holds only this card, so `revision` is required:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "domyn/Domyn-Edge-Base-checkpoints"
revision = "stage4-midtraining-12.3T"

tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    revision=revision,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
```

Everything else is the same as for the final model. The recommended environment, the generation
example and the serving instructions are on the
[base model card](https://huggingface.co/domyn/Domyn-Edge-Base#how-to-use), and every caveat there
applies to every branch here, including that **stock vLLM does not yet support this
architecture**.

## What changes across training

The clearest difference between branches is usable context, measured with
[RULER](https://github.com/NVIDIA/RULER). Branches before `stage3-ctx-ext-10.3T` are
short-context models whose scores drop beyond 16k. The first extension lifts every length (64k
from 20.2 to 64.0), mid-training on 32k sequences gives back part of the longest lengths (128k from
46.6 to 28.1), and the second extension and annealing recover them, up to the final model's 74.5
at 256k.

## Limitations

**Early branches are short-context models.** Every branch before `stage3-ctx-ext-10.3T` has a
config limited to its training context and a usable context that ends at about 16k. Do not read a
low long-context score on an early branch as a property of the architecture.

**Later is not uniformly better.** Scores do not move monotonically across the series.
Long-context performance regresses across mid-training, and annealing improves every length up to
256k while 512k gets worse. Pick the branch that matches the question you are asking rather than
assuming the last one dominates.

**Every limitation of the final model applies.** No branch has had alignment or safety tuning,
and all share the prompt-injection and multilingual caveats listed on the
[base model card](https://huggingface.co/domyn/Domyn-Edge-Base#limitations). For a model that
chats and calls tools, see [`domyn/Domyn-Edge-SFT`](https://huggingface.co/domyn/Domyn-Edge-SFT).

## License

This model is released under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Citation

If you find this work valuable, please consider citing it:

```bibtex
@misc{domynedge2026,
  title  = {Domyn Edge},
  author = {Domyn},
  year   = {2026},
  note   = {Technical report, forthcoming in November 2026.},
}
```