| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - byte-level |
| - hierarchical |
| - tokenizer-free |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| inference: false |
| --- |
| |
| # Flat byte baseline (pretrained) |
|
|
| Flat 22-layer byte-level transformer, no segmentation and no hierarchy. Every byte costs one forward pass. |
|
|
| From **[Dynamic Multi-Byte Prediction With Hierarchical Language Models](https://arxiv.org/abs/2608.15454)**. |
| Code: **[skai-research/lca-multibyte](https://github.com/skai-research/lca-multibyte)**. |
|
|
| | | | |
| | --- | --- | |
| | Parameters | 374M | |
| | Vocabulary | 261 (256 bytes + `<pad>`/`</s>`/`<unk>` + `<en>`, `<eot>`) | |
| | Context | 4096 bytes | |
| | `model_config` | `[22, (0,), 0, 0]` | |
| | `attn_type` | `None` | |
| | Precision | fp32 | |
|
|
| ## Results |
|
|
| | Metric | Value | |
| | --- | --- | |
| | Byte-level BPC (validation) | 0.888 | |
| | Training steps | 48,186 | |
|
|
| ## Usage |
|
|
| This is not a `transformers` architecture — load it with the code from the paper repo: |
|
|
| ```bash |
| git clone https://github.com/skai-research/lca-multibyte && cd lca-multibyte |
| uv sync && export PYTHONPATH=$(pwd) |
| ``` |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from src.eval.model_loader import load_fxt_model |
| |
| path = snapshot_download("skai-research/bytes-base") |
| model, tokenizer, config = load_fxt_model(path, device="cuda") |
| ``` |
|
|
| Or generate directly: |
|
|
| ```bash |
| python src/eval/generate.py --model_path $(python -c \ |
| "from huggingface_hub import snapshot_download; print(snapshot_download('skai-research/bytes-base'))") \ |
| --prompt "The capital of France is" --mode cached |
| ``` |
|
|
| `--show_tokenization` prints the learned segment boundaries. |
|
|
| ## Training |
|
|
| Pretrained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-100BT`. |
| Config: `configs/train/modern_fxt_baseline_btyes_256_scale_bp_dual.yaml`. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{owodunni2026lca, |
| title = {Dynamic Multi-Byte Prediction With Hierarchical Language Models}, |
| author = {Owodunni, Abraham Toluwase and Okocha, Chibuzor and Grant, Christan |
| and Limisiewicz, Tomasz and Kumar, Sachin}, |
| year = {2026}, |
| journal = {arXiv preprint arXiv:2608.15454}, |
| url = {https://arxiv.org/abs/2608.15454} |
| } |
| ``` |
|
|
