README.md

---
license: mit
library_name: pytorch
pipeline_tag: image-to-image
tags:
  - handwriting-synthesis
  - vision-transformer
  - generative-adversarial-network
  - pytorch
  - computer-vision
  - iam
  - vietnamese-handwriting
---

# WriteViT: Handwritten Text Generation with Vision Transformers

WriteViT is a one-shot handwritten text synthesis framework that learns a writer's style from a small set of reference images and generates handwritten text in that style. The model combines a ViT-based writer encoder, a multi-scale Transformer generator with conditional positional encoding, and a lightweight ViT recognizer.

<p align="center">
  <a href="https://arxiv.org/abs/2505.13235"><img alt="arXiv" src="https://img.shields.io/badge/arXiv-2505.13235-b31b1b.svg"></a>
  <a href="https://colab.research.google.com/drive/15Lswqr-aQwI-fF6yRoGYt-2pxSlC2L-R"><img alt="Open in Colab" src="https://colab.research.google.com/assets/colab-badge.svg"></a>
  <img alt="License" src="https://img.shields.io/badge/license-MIT-blue.svg">
</p>

<p align="center">
  <img src="./Figures/architecture.png" alt="WriteViT architecture overview" width="92%">
</p>

## Model Summary

| Field | Description |
| --- | --- |
| Task | Handwritten text image generation |
| Framework | PyTorch |
| Architecture | ViT writer encoder, Transformer generator, ViT recognizer |
| Supported Data | IAM English handwriting and Vietnamese handwriting data |
| Image Height | 32px |
| Checkpoints | English and Vietnamese checkpoints included |
| Intended Use | Research, reproducibility, handwriting synthesis, and data augmentation experiments |

## Highlights

- Learns writer style from a small set of reference handwriting images.
- Generates handwritten text conditioned on target text and writer style.
- Supports English IAM-style handwriting and Vietnamese handwriting generation.
- Includes prepared dataset pickles, lexicons, font templates, and released checkpoints.
- Provides training code and qualitative result figures in one reproducible PyTorch release.

## Qualitative Results

### Handwriting Generation

<p align="center">
  <img src="./Figures/Generation.png" alt="WriteViT handwriting generation results" width="92%">
</p>

### Handwriting Reconstruction

<p align="center">
  <img src="./Figures/Reconstruction.png" alt="WriteViT handwriting reconstruction results" width="92%">
</p>

## Released Artifacts

| File | Purpose |
| --- | --- |
| `File/eng_ckpt.pth` | Released English/IAM checkpoint |
| `File/vn_ckpt.pth` | Released Vietnamese checkpoint |
| `File/vgg19.pth` | VGG19 backbone checkpoint/resource used by the project |
| `File/IAM.pickle` | Prepared IAM handwriting dataset pickle |
| `File/VN.pickle` | Prepared Vietnamese handwriting dataset pickle |
| `File/unifont.pickle` | Font/template data for query rendering |
| `File/english_words.txt` | English lexicon |
| `File/vn_words.txt` | Vietnamese lexicon |

## Repository Structure

```text
.
|-- data/                 # Dataset loading and preparation utilities
|-- Figures/              # Architecture and qualitative result figures
|-- File/                 # Datasets, checkpoints, lexicons, and font resources
|-- models/               # Generator, discriminators, recognizer, and writer encoder
|-- util/                 # Shared model and training utilities
|-- params.py             # Experiment and dataset configuration
|-- train.py              # Training entry point
`-- requirements.txt
```

## Installation

Python 3.7 or newer and a CUDA-capable GPU are recommended for training.

Install PyTorch for your CUDA version first, then install the project dependencies:

```bash
pip install -r requirements.txt
```

## Download Checkpoints and Artifacts

The released checkpoints, prepared data files, lexicons, and font resources are hosted in this Hugging Face repository. Clone the full release with Git LFS:

```bash
git lfs install
git clone https://huggingface.co/DAIR-Group/WriteViT
cd WriteViT
```

If you want to download the artifacts into another local copy of this codebase, use `huggingface_hub`:

```bash
pip install -U huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="DAIR-Group/WriteViT",
    repo_type="model",
    local_dir=".",
    allow_patterns=[
        "File/*.pth",
        "File/*.pickle",
        "File/*.txt",
        "config.json",
        "README.md",
    ],
)
PY
```

The default IAM setup expects these files under `File/`:

```text
File/
├── IAM.pickle
├── VN.pickle
├── eng_ckpt.pth
├── vn_ckpt.pth
├── vgg19.pth
├── english_words.txt
├── vn_words.txt
└── unifont.pickle
```

## Configuration

Experiment settings are defined in `params.py`.

The default configuration uses IAM:

```python
DATASET = 'IAM'
DATASET_PATHS = './File/IAM.pickle'
NUM_WRITERS = 339
WORDS_PATH = './File/english_words.txt'
```

To train or evaluate with Vietnamese data, switch to:

```python
DATASET = 'VNDB'
DATASET_PATHS = './File/VN.pickle'
NUM_WRITERS = 106
WORDS_PATH = './File/vn_words.txt'
```

The available recognizer backbones are `resnet18`, `vgg11`, and `vgg19`.

## Training

Review `params.py`, especially `DATASET`, `DATASET_PATHS`, `NUM_WRITERS`, `WORDS_PATH`, `BACKBONE`, learning rates, batch size, and `RESUME`.

Start training:

```bash
CUDA_VISIBLE_DEVICES=0 python train.py
```

Outputs are written to:

```text
saved_models/<EXP_NAME>/
saved_images/<EXP_NAME>/
```

When `RESUME = True`, the training script loads:

```text
saved_models/<EXP_NAME>/model.pth
```

## Data Format

Prepared dataset pickle files contain writer-split handwriting samples:

```python
{
    "train": {
        "writer_id": [
            {"img": PIL.Image.Image, "label": "handwritten text"}
        ]
    },
    "test": {
        "writer_id": [
            {"img": PIL.Image.Image, "label": "handwritten text"}
        ]
    }
}
```

 

 
## Resources

- Paper: https://arxiv.org/abs/2505.13235
- Interactive demo: https://colab.research.google.com/drive/15Lswqr-aQwI-fF6yRoGYt-2pxSlC2L-R
- Original datasets/checkpoints folder: https://drive.google.com/drive/folders/1ZgYS6-6l6fjKY75RJipONBByujIgf-uE

## Citation

If you use WriteViT in your research, please cite:

```bibtex
@article{nam2025writevit,
  title   = {WriteViT: Handwritten Text Generation with Vision Transformer},
  author  = {Dang Hoai Nam and Huynh Tong Dang Khoa and Vo Nguyen Le Duy},
  journal = {arXiv preprint arXiv:2505.13235},
  year    = {2025}
}
```

## Acknowledgements

This repository builds on Handwriting Transformers by Ankan Kumar Bhunia et al. We thank the authors for making their work publicly available.