File size: 2,179 Bytes
ae73cf0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: apache-2.0
language:
- en
tags:
- byte-level
- hierarchical
- tokenizer-free
datasets:
- HuggingFaceFW/fineweb-edu
inference: false
---
# Flat byte baseline (pretrained)
Flat 22-layer byte-level transformer, no segmentation and no hierarchy. Every byte costs one forward pass.
From **[Dynamic Multi-Byte Prediction With Hierarchical Language Models](https://arxiv.org/abs/2608.15454)**.
Code: **[skai-research/lca-multibyte](https://github.com/skai-research/lca-multibyte)**.
| | |
| --- | --- |
| Parameters | 374M |
| Vocabulary | 261 (256 bytes + `<pad>`/`</s>`/`<unk>` + `<en>`, `<eot>`) |
| Context | 4096 bytes |
| `model_config` | `[22, (0,), 0, 0]` |
| `attn_type` | `None` |
| Precision | fp32 |
## Results
| Metric | Value |
| --- | --- |
| Byte-level BPC (validation) | 0.888 |
| Training steps | 48,186 |
## Usage
This is not a `transformers` architecture — load it with the code from the paper repo:
```bash
git clone https://github.com/skai-research/lca-multibyte && cd lca-multibyte
uv sync && export PYTHONPATH=$(pwd)
```
```python
from huggingface_hub import snapshot_download
from src.eval.model_loader import load_fxt_model
path = snapshot_download("skai-research/bytes-base")
model, tokenizer, config = load_fxt_model(path, device="cuda")
```
Or generate directly:
```bash
python src/eval/generate.py --model_path $(python -c \
"from huggingface_hub import snapshot_download; print(snapshot_download('skai-research/bytes-base'))") \
--prompt "The capital of France is" --mode cached
```
`--show_tokenization` prints the learned segment boundaries.
## Training
Pretrained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-100BT`.
Config: `configs/train/modern_fxt_baseline_btyes_256_scale_bp_dual.yaml`.
## Citation
```bibtex
@article{owodunni2026lca,
title = {Dynamic Multi-Byte Prediction With Hierarchical Language Models},
author = {Owodunni, Abraham Toluwase and Okocha, Chibuzor and Grant, Christan
and Limisiewicz, Tomasz and Kumar, Sachin},
year = {2026},
journal = {arXiv preprint arXiv:2608.15454},
url = {https://arxiv.org/abs/2608.15454}
}
```
|