| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: [dynamic-chunking, tokenization, hnet, byte-level, interpretability] |
| --- |
| |
| # H-Net dynamic-chunking pilot models |
|
|
| 31 byte-level H-Net checkpoints (22.5M params, 6000 steps) trained to study **what a |
| learned chunker discovers when it is free to choose its own computational granularity**. |
| Every run shares data, seed schedule and FLOP budget; only the chunker's objective varies. |
|
|
| ## Runs |
|
|
| | family | n | what varies | |
| |---|---|---| |
| | `pilot_A_s{0,1,2}` | 3 | no parity loss — the natural chunker (baseline) | |
| | `pilot_B_s{0,1,2}` | 3 | parity loss, per-content-matched-sentence beta | |
| | `pilot_C_s{0,1,2}` | 3 | parity applied then removed (reversion arm) | |
| | `pilot_D_s{0,1,2}` | 3 | stronger equalisation | |
| | `pilot_E_s{0,1,2}` | 3 | entropy-matched variant | |
| | `pilot_Apc_s{0,1,2}` | 3 | per-codepoint control | |
| | `pilot_Bpc_s{0,1,2}` | 3 | parity loss, per-codepoint beta | |
| | `emactl_noema_s{0,1,2}` | 3 | EMA ablation control (built from A) | |
| | `emactl_genreg_s{0,1,2}` | 3 | generic-regularisation control, entropy-matched to B | |
| | `jitter_s{0,1,2}` | 3 | threshold-jitter training (tau ~ U[0.30, 0.75]) | |
| | `baseline_en` | 1 | English-only baseline | |
|
|
| ## Format |
|
|
| Weights are `model.safetensors`; `config.json` carries `model_config`, the training config, |
| and a `tied_weights` map. `emb.weight` and `head.weight` are tied, so only `emb.weight` is |
| stored — restore with: |
|
|
| ```python |
| from safetensors.torch import load_file |
| import json |
| sd = load_file("model.safetensors") |
| cfg = json.load(open("config.json")) |
| for dst, src in cfg["tied_weights"].items(): |
| sd[dst] = sd[src] # head.weight <- emb.weight |
| ``` |
|
|
| Verified bit-exact against the original training checkpoints (`max|Δ| = 0.0`, 111 tensors). |
| `final.json` and `log.jsonl` are the trainer's own records. |
|
|
| ## What these were used to find |
|
|
| - Segmentation crystallises **after** capability: the router lags BPB descent in 36/36 |
| language-seed pairs at half of each clock's own change. |
| - Learned units are **not linguistic**. Morpheme F1 sits at or below a rate-matched random |
| baseline; gold word+morph features add ≤0.001 held-out AUROC over computational features |
| in 9/9 languages. |
| - Chunks are **not functionally localised**: aligned-minus-misaligned activation patching is |
| at the construction rate (0/28 cells above 2 across-seed SD), while killing the whole chunk |
| pathway costs +0.34 nats/byte. Information is additive and diffuse across chunks. |
| - The chunk-level main network is worth only **~0.07 BPB** at this scale. |
|
|
| Negative results are reported as results. See the companion |
| [results dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-results) |
| and [probe dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-probes). |
|
|
| ## Limitations |
|
|
| 16.2M/22.5M params, 6000 steps, one hierarchy stage, 3 seeds. Only the final step was |
| retained (`keep_ckpt: false`), so no intermediate weights exist. Third-party models used for |
| comparison (`allenai/Bolmo-1B`, BLT, ByT5) are **not** redistributed here. |
|
|
