---
license: apache-2.0
language:
- en
tags:
- byte-level
- hierarchical
- tokenizer-free
datasets:
- HuggingFaceFW/fineweb-edu
inference: false
---

# Flat byte baseline (pretrained)

Flat 22-layer byte-level transformer, no segmentation and no hierarchy. Every byte costs one forward pass.

From **[Dynamic Multi-Byte Prediction With Hierarchical Language Models](https://arxiv.org/abs/2608.15454)**.
Code: **[skai-research/lca-multibyte](https://github.com/skai-research/lca-multibyte)**.

| | |
| --- | --- |
| Parameters | 374M |
| Vocabulary | 261 (256 bytes + `<pad>`/`</s>`/`<unk>` + `<en>`, `<eot>`) |
| Context | 4096 bytes |
| `model_config` | `[22, (0,), 0, 0]` |
| `attn_type` | `None` |
| Precision | fp32 |

## Results

| Metric | Value |
| --- | --- |
| Byte-level BPC (validation) | 0.888 |
| Training steps | 48,186 |

## Usage

This is not a `transformers` architecture — load it with the code from the paper repo:

```bash
git clone https://github.com/skai-research/lca-multibyte && cd lca-multibyte
uv sync && export PYTHONPATH=$(pwd)
```

```python
from huggingface_hub import snapshot_download
from src.eval.model_loader import load_fxt_model

path = snapshot_download("skai-research/bytes-base")
model, tokenizer, config = load_fxt_model(path, device="cuda")
```

Or generate directly:

```bash
python src/eval/generate.py --model_path $(python -c \
  "from huggingface_hub import snapshot_download; print(snapshot_download('skai-research/bytes-base'))") \
  --prompt "The capital of France is" --mode cached
```

`--show_tokenization` prints the learned segment boundaries.

## Training

Pretrained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-100BT`.
Config: `configs/train/modern_fxt_baseline_btyes_256_scale_bp_dual.yaml`.

## Citation

```bibtex
@article{owodunni2026lca,
  title   = {Dynamic Multi-Byte Prediction With Hierarchical Language Models},
  author  = {Owodunni, Abraham Toluwase and Okocha, Chibuzor and Grant, Christan
             and Limisiewicz, Tomasz and Kumar, Sachin},
  year    = {2026},
  journal = {arXiv preprint arXiv:2608.15454},
  url     = {https://arxiv.org/abs/2608.15454}
}
```