ZeroFLops enwik8 language models

Checkpoints from the hard-residual LM research line of the ZeroFLops project. The code that produced and evaluates these checkpoints lives in the companion GitHub repository: Ckal/ZeroFLops-paper.

What this model is

A character-level language model over enwik8. Two parts:

  1. A dense residual transformer (86.3 M parameters for e8b_cache) that produces the output distribution in the normal way.
  2. A hard table β€” a count table over byte contexts, words and the current document, holding 81,785,402 integer entries. It is built by counting, not by floating-point arithmetic.

The table is not the output. These checkpoints were trained with --hard_feature, which means each expert's byte-prediction distribution and its evidence counters are fed to the transformer as extra input dimensions: 256 logit values for each of the 5 experts, plus 23 statistics β€” 1303 extra dimensions added alongside the token embedding. The transformer then learns how much to trust each expert.

Naming caveat. The table mechanism is the "zero FLOPs" idea for the lookup side: the memory is integer counting, so no matrix multiply is spent on it. The published checkpoints are not zero-FLOP models overall β€” they still run a full float transformer. A separate table-only mode (--mode hard) exists for the pure-counting configuration.

The 5 experts

expert what it counts
ngram_o2 2-byte contexts
ngram the main expert β€” mid-length byte contexts
ngram_long long contexts (8, 12, 16 bytes)
word lowercase word tokens
cache bytes seen so far in the current document

Files

file what it is size sha256
enwik8_e8b_cache_best.pt e8b_cache β€” the strongest dynamic-evaluation model (val 1.0463, dynamic 0.9756) 345 MB f916c445e448f07478de8d31e3bc707898a0675dfec44f947a17127ae50f594c
enwik8_e8l_cache_best.pt e8l_cache β€” the longest-trained model (400k steps, val 1.0350) 158 MB 8f51f30dbdf5a43f82e7d2d686edd4127434ec878b984381a2662d76ab0cc636
enwik8_e8b_alone_best.pt e8b_alone β€” the no-memory baseline (85.3M, test 1.0750, dynamic 0.9990). Needs no table file, so it stands alone 341 MB ea92db1d8554e0534daf24998dcf162aec1aa37227fa8f8771f5d44b59bccf54
p_e8_tab_hard.pt the hard table (5 experts, 81,785,402 entries) used by the two memory checkpoints 1.18 GB 6ef63a411e96646766fb1bf813053ad698438aa7787ded99f9d7fefa183fd45d
gf256_hard_model_v7.pt the GF(256) Connect Four V7 hard checkpoint β€” a different research line, published here so the artifact is not only on GitHub 118 MB e196957c349be328b50edd975e6bba9a7a47115c49139279ebb85ab882d2d827

p_e8_tab_hard.pt is required by the two memory checkpoints (enwik8_e8b_cache_best.pt, enwik8_e8l_cache_best.pt). They store the table path in their hard_path field (p_e8_tab_hard.pt) and resolve it relative to the checkpoint's own directory, so download the table into the same folder as the checkpoint. Without the table, those two cannot be loaded. The no-memory baseline enwik8_e8b_alone_best.pt carries no table and stands alone.

Measured results (enwik8, standard 5 MB test split)

Bits per byte, lower is better. 1.20 BPC means the text can be stored in 15% of its size.

Static evaluation

params counts the transformer's parameters only. The hard table adds 81.8 M integer entries on top β€” as the paper notes, the table size has to be reported next to the parameter count.

model params context steps val test probe table
e8b_cache 86.3 M 1024 250k 1.0463 1.0270 56.9% yes
e8l_cache 39.5 M 512 400k 1.0350 1.0176 β€” yes
e8b_alone 85.3 M 1024 250k 1.0917 1.0750 69.2% no
e8f_alone 38.8 M 512 200k 1.1193 1.1050 63.1% no

e8b_alone and e8f_alone are the same architecture without the table inputs, so the table's contribution is the gap between the *_cache and *_alone rows.

Dynamic evaluation ("learning while reading")

After scoring each window the model takes one small training step on it, so it only ever learns from text it has already been scored.

model test test, learning while reading gain step size
e8b_cache 1.0270 0.9756 0.0513 rms 5e-06
e8b_alone 1.0750 0.9990 0.0760 rms 1e-05
e8f_alone 1.1050 1.0481 0.0569 rms 5e-06
e8l_cache 1.0176 0.9804 0.0371 rms 5e-06
e8l_cache 1.0176 0.9795 0.0380 rms 1e-05

The two e8l_cache rows are the same checkpoint evaluated at two learning rates (5e-06 and 1e-05); the lower step size wins.

Published reference: Transformer-XL 277 M with dynamic evaluation reaches 0.94 on the same test file (Dai et al. 2019).

What the memory is worth

The ablation in the paper (paper/analysis_e8l/analysis.json) isolates each component. It was run on the e8l_cache checkpoint in a separate harness, so its baseline ("memory as trained", 1.0630) sits above that run's training-time validation (1.0350) β€” the two are measured on different window sets. Read the rows as deltas against their own baseline, not against the table above.

configuration val BPC
memory as trained 1.0630
no long contexts (8+ bytes) 1.1051
no word expert 1.0763
no document cache 1.1612
byte contexts only up to 2 bytes 1.1665
evidence counts all set to 0 1.1135
all tables empty (unigram only) 1.2928
no memory at all (uniform, no counts) 6.5112
memory of a different window 12.8174

The last row is the control that matters: feeding the model memory from a different window is catastrophic, so the mechanism genuinely reads the current document rather than acting as a generic prior.

Reproducing

# data
python download_hf_corpus.py            # enwik8 + web-text into data/

# train (the CFG block at the top of run_all.py holds every setting)
python run_all.py --stages gold
python run_all.py --stages e8final

# static evaluation -> paper/results.md
python run_all.py --report

# dynamic evaluation -> runs/p_dyn__*.json
python run_all.py --stages dyn

--report does not re-run dynamic evaluation, and the component ablation is a separate artifact (paper/analysis_e8l/analysis.json), not part of --report.

The corpora and the trained checkpoints are not in git β€” they are reproducible from the commands above. The full parameter surface and the download commands are documented in RESEARCH_STATUS.md in the GitHub repo.

Loading a published checkpoint

import argparse, torch
from hard_residual_lm import HardModel, build_model   # from the GitHub repo

# weights_only=False is required here: the checkpoint also stores the argparse
# namespace of the training run, which torch cannot load under weights_only=True.
ck = torch.load("enwik8_e8b_cache_best.pt", map_location="cpu", weights_only=False)
args = argparse.Namespace(**ck["args"]); args.device = "cpu"

hm = HardModel.load("p_e8_tab_hard.pt", "cpu").set_runtime(args)
model = build_model(args, hm)
model.load_state_dict(ck["model"])
model.eval()

Caveats

  • The probe is a multiple-choice knowledge probe: 65 fixed questions, each with 4 candidate answers, scored by which answer costs the fewest bits. The first answer is the correct one. It is a diagnostic, not a win rate β€” and it is not a game-playing benchmark.
  • e8l_cache has no alone counterpart checkpoint, so its memory contribution is inferred from the ablation rather than measured directly on that checkpoint.
  • The cross-domain transfer claim (code / wiki_de / wiki_en) was never measured. The domain stage is prepared but has not been run.
  • The dynamic-evaluation rows depend on the learning rate used during evaluation; the numbers above are not reproducible from the checkpoint alone without the step size.

Citation

TBD β€” the paper is in preparation.

Further reading

  • Discussion #1 β€” the release write-up (ablation, mechanism, file list).
  • Discussion #2 β€” "0.9756 BPC on enwik8 β€” trained in 10 hours on one GPU", including the wrong-window control that shows the model reads the document rather than acting as a generic prior.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support