# DAT Byte Small (200M)
**DAT Byte** is a family of byte-level **D**ifferential-**A**ttention **T**ransformers, trained from scratch on an RTX 5090.
This model is the smallest in the family, with approximately 200 million parameters. It was trained on a set of Discord-style chat content, public-domain books, and English Bible translations. Larger models in the family received a larger and more diverse dataset.
---
## Training Data
As the smallest DAT Byte model, this version was trained on a reduced dataset totaling ~727MB, composed exclusively of the following sources:
- [**Gutenberg English**](https://huggingface.co/datasets/sedthh/gutenberg_english) — English books in the public domain
- [**OpenDiscord**](https://huggingface.co/datasets/hudsongouge/Open-Discord) — Discord dumps in ChatML format
- Proprietary Discord dumps (similar structure and tone to OpenDiscord)
- A diverse set of public domain English Bible translations (~34MB total)
> All listed datasets were used **i…
