Blame for README.md

---
license: apache-2.0
base_model: OpenMOSS-Team/MOSS-Image-Base
base_model_relation: finetune
pipeline_tag: text-to-image
language:
  - sv
tags:
  - moss-image
  - handwriting
  - historical-documents
  - archives
  - htr
  - synthetic-data
---

# diffu-moss-r6b

A full fine-tune of the DiT of [MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base) that draws
whole synthetic archive pages: Swedish handwritten records of the 16th to 19th centuries. It is the drawing model of
[diffu](https://github.com/Borg93/diffu2), which makes training pages for page-level handwriting recognition: a page
layout (a PAGE/ALTO file, or the lines' boxes as JSON), the text to write in it (or the layout's own) and a sample page
go in; a page image and its exact ALTO and PAGE labels come out.

The model reads a MOSS Layout prompt, one text item per line with its exact text and its box, and a sample page given
as extra image tokens, whose paper, ink and look it follows. Only the DiT is in this repository. The text encoder,
tokenizer, VAE and scheduler are MOSS-Image-Base's, unchanged.

**This version (9 Nov 2026)** is run B: trained on the whole ground truth (54 collections, 203 volumes) on the Leonardo
supercomputer, 7,000 steps of 16 samples, the averaged (EMA) weights. It replaces the first version of this repository
(eight collections, trained on one DGX Spark), which stays in the repository's history (revision `726b9fe`).

## Files

| File | What |
|---|---|
| `diffusion_pytorch_model.safetensors` | the DiT's weights, bfloat16 (10.2 GB, 462 tensors, the same names and shapes as Base's DiT): the exponential moving average (decay 0.999) at step 7,000 |
| `config.json` | `diffu_moss.transformer.MossTransformer` (diffu's own implementation of MOSS-Image's DiT) |
| `export.json` | `takes_reference: true`, `sample_size: legible`: the model takes a sample page, sized at the drawn page's writing scale |
| `tensorboard/` | the training curves (loss, gradient norm, learning rate, step time, memory; validation loss by prompt form every 500 steps): the **Training metrics** tab |

## Use

With diffu (see its `docs/INFERENCE.md` for running on a Ray or KubeRay cluster):

```bash
hf download OpenMOSS-Team/MOSS-Image-Base --revision 85975793d8b3d432f1b6e27f352abaa4c2a4c0d3 --local-dir base
hf download Gabriel/diffu-moss-r6b --local-dir dit
git clone https://github.com/Borg93/diffu2 && cd diffu2 && uv sync --all-packages
uv run diffu-moss-init --base ../base --transformer ../dit --out ../moss-diffu
uv run diffu-run --jobs run.jsonl --draw-settings draw.json --model ../moss-diffu --out out --gpus 1
```

`diffu-moss-init` reads `export.json` and writes the sample's size into the model directory, so the sample page is
sized as in training (`legible`: a whole page's sample at the drawn page's pixel count).

### With a sample page

The sample page is an ordinary scan (JPEG or PNG) of a page whose **paper, ink and look** the drawn page should have:
best a page of the same volume as the layout, or any page whose look you want. It does not set the handwriting (see
Limitations). Each page job in `run.jsonl` is one line of JSON: the layout (`template`, a PAGE/ALTO file, or `layout`,
the lines' boxes in pixels), the text (left out: the layout's own), the style, the sample page under `reference`, and
an id:

```json
{"id": "court-0001", "template": "templates/0105.xml", "text": "Anno 1649 den 6 Aprilis Tå laga tingh höltz medh Allmogen ...", "style": {"provenance": {"title": "the court records of Jämtlands domsaga", "years": "1649"}, "look": {"greyscale": false, "paper": "cream", "ink": "black"}}, "reference": {"image": "samples/court-sample.jpg", "caption": "sample"}}
```

`"caption": "sample"` is the prompt form this model was trained on most (the sample page alone, nothing named; the
style's title and look are then left out of the prompt). `"sample_look"` adds the look in words beside the sample.
`draw.json` holds the draw settings, for example `{"steps": 50, "device": "cuda", "reference_guidance_scale": 2}`.
Each page comes out as `<id>.jpg` with its `<id>.alto.xml` and `<id>.page.xml`. `docs/INFERENCE.md` section 5
describes every field, with a worked example of each combination.

One page from the command line, without a run file:

```bash
uv run diffu-generate --model ../moss-diffu --templates templates/0105.xml --text-from text.txt \
    --reference samples/court-sample.jpg --reference-caption sample --reference-guidance 2 --whole-page --out pages
```

Recommended drawing: about 1 MP, 50 steps, guidance 3, shift 5, block 22 skipped in the unconditional pass (MOSS-Image's
own recipe), and **reference guidance 2** (a third pass toward the sample: `--reference-guidance 2` in
`diffu-generate`, `reference_guidance_scale: 2` in the draw settings), which makes grey microfilm samples give grey
pages reliably.

On Blackwell GPUs the DiT can run in MXFP8, quantized from these weights as the model loads: give the draw settings
`{"precision": "mxfp8", "compile_blocks": true}`. For SGLang Diffusion, `diffu-moss-export-hf` writes this DiT in the
official MOSS-Image layout, and diffu's plugin `diffu_sglang_moss` adds the sample page (`docs/INFERENCE.md` section 13).

## Training

- Data: the page pairs of diffu's whole Swedish archival handwriting ground truth at about 1 MP: 143,379 training
  rows (whole pages and two windows of whole lines from each page) from 54 collections and 203 volumes, each paired
  with a sample page from the same volume; 10 % of the volumes held out (11,920 rows). Court, chancery, church, police,
  mining, military and travel records, 16th to 19th century.
- Prompt forms: the sample page alone (60 %), the sample with the look in words (20 %), no sample (20 %); no prompt
  names an archive, years or a hand. A fault in building the set (found 11 Nov 2026, fixed for later runs) tied
  each form to one kind of row: the sample alone was trained on windows only (86,027 windows, no whole page), no
  sample on whole pages only (28,676), the sample with the look in words on 20,197 whole pages and 8,479 windows.
  Measured after: on 20 held-out whole pages the sample-alone form's error is the trained form's within 0.05 % at
  every noise level, so it is drawn with as advised above. Half of the sample-alone rows were restyled the same way
  on window and sample (microfilm grey, colourised, tinted), so the look has to come from the sample.
- Recipe: MOSS-Image's own full fine-tune, from Base at revision `85975793`: Muon on the joint blocks' matrices and
  AdamW elsewhere, learning rate 5e-5 (constant after 50 warmup steps), gradients clipped at 1, float32 weights under
  bfloat16 autocast, an exponential moving average of the weights at 0.999; text encoder and VAE frozen.
- Scale: 16 samples a step over 8 A100 64 GB (two nodes of Leonardo, CINECA; FSDP2), 7,000 steps (112,000 samples,
  about 0.8 of a pass), ~23 hours. Planned for 9,000 steps; stopped at 7,000 as the validation loss had been flat since
  step 6,000.

Validation loss at a fixed noise level and noise, by prompt form (lower is better):

| Step | Sample alone | Sample with look words | No sample |
|---|---|---|---|
| 500 | 0.6709 | 0.6708 | 0.6725 |
| 6,000 | 0.6584 | 0.6583 | 0.6605 |
| 7,000 | 0.6585 | 0.6586 | 0.6606 |

## Evaluation

The swap test: held-out windows from six held-out volumes, each drawn with its own volume's sample and with three
other volumes' samples, from the same noise, judged beside the real scans:

| Check | Step 7,000 (raw weights, no reference guidance) | The first version of this repository |
|---|---|---|
| The page's own sample gives a page of the right kind (colour or grey) | 6 of 6 | 4 of 5 |
| A grey sample makes a colour page grey | 7 of 7 | 3 of 5 |
| A colour sample makes a grey page colour | 3 of 3 | — |
| The sample's handwriting is followed | no: by eye one hand a page, letter for letter, whatever the sample | 1 of 12 |

With these EMA weights and reference guidance 2, on two of the windows with four samples each: every grey sample gives
a grey page and every colour sample a colour page, the text exact. Against step 1,000 of the same run the writing is
thinner, lighter and more slanted, closer to the real pages.

## Limitations

- **The handwriting does not follow the sample page.** The model takes the paper, ink and look from the sample and
  writes its own plausible hand for the page. A specific scribe's hand comes from a hand adapter (a LoRA) trained on
  top of this model: tested for one scribe so far (Överståthållarämbetet's 1715 letter book), whose adapter carries
  his hand while the look still follows the sample; adapters for more scribes are being tested.
- Every line has to be checked: diffu reads each drawn line back and marks a line it cannot confirm as unclear in the
  labels.
- Trained on Swedish archival handwriting; printed forms and other kinds of pages are untested.

## Licence

Apache-2.0, as MOSS-Image-Base.