metadata
license: apache-2.0
language:
- en
tags:
- byte-level
- hierarchical
- tokenizer-free
datasets:
- HuggingFaceFW/fineweb-edu
inference: false
Flat byte baseline (pretrained)
Flat 22-layer byte-level transformer, no segmentation and no hierarchy. Every byte costs one forward pass.
From Dynamic Multi-Byte Prediction With Hierarchical Language Models. Code: skai-research/lca-multibyte.
| Parameters | 374M |
| Vocabulary | 261 (256 bytes + <pad>/</s>/<unk> + <en>, <eot>) |
| Context | 4096 bytes |
model_config |
[22, (0,), 0, 0] |
attn_type |
None |
| Precision | fp32 |
Results
| Metric | Value |
|---|---|
| Byte-level BPC (validation) | 0.888 |
| Training steps | 48,186 |
Usage
This is not a transformers architecture — load it with the code from the paper repo:
git clone https://github.com/skai-research/lca-multibyte && cd lca-multibyte
uv sync && export PYTHONPATH=$(pwd)
from huggingface_hub import snapshot_download
from src.eval.model_loader import load_fxt_model
path = snapshot_download("skai-research/bytes-base")
model, tokenizer, config = load_fxt_model(path, device="cuda")
Or generate directly:
python src/eval/generate.py --model_path $(python -c \
"from huggingface_hub import snapshot_download; print(snapshot_download('skai-research/bytes-base'))") \
--prompt "The capital of France is" --mode cached
--show_tokenization prints the learned segment boundaries.
Training
Pretrained on FineWeb-Edu sample-100BT.
Config: configs/train/modern_fxt_baseline_btyes_256_scale_bp_dual.yaml.
Citation
@article{owodunni2026lca,
title = {Dynamic Multi-Byte Prediction With Hierarchical Language Models},
author = {Owodunni, Abraham Toluwase and Okocha, Chibuzor and Grant, Christan
and Limisiewicz, Tomasz and Kumar, Sachin},
year = {2026},
journal = {arXiv preprint arXiv:2608.15454},
url = {https://arxiv.org/abs/2608.15454}
}