Symbolic Wavelet Language Model — Gutenberg benchmark
This repository contains the notebook and recorded benchmark outputs for a 126,746-parameter non-attention symbolic wavelet language model evaluated against parameter-matched Hugging Face Transformer baselines on the same held-out Project Gutenberg validation set.
Result
Lower is better. In this benchmark, the symbolic wavelet run is lower than every tested Transformer baseline on every directly comparable cross-tokenizer metric.
| Model | Params | NLL / canonical byte | Bits / canonical byte | Canonical-byte PPL | NLL / lexical unit |
|---|---|---|---|---|---|
| Symbolic Wavelet (ours) | 126,746 | 1.6442 | 2.3720 | 5.1767 | 12.1927 |
| Llama | 125,392 | 3.1088 | 4.4850 | 22.3936 | 23.0539 |
| Qwen3 | 123,280 | 2.3097 | 3.3322 | 10.0714 | 17.1281 |
| Gemma 3 | 125,112 | 2.7367 | 3.9482 | 15.4357 | 20.2945 |
| DeepSeek-V3-style | 118,736 | 2.3232 | 3.3516 | 10.2081 | 17.2281 |
Against the strongest baseline, Qwen3, the symbolic wavelet run improves NLL/canonical-byte, bits/canonical-byte, and NLL/lexical-unit by 28.8%, and canonical-byte perplexity by 48.6%.
Shared evaluation
- Held-out windows: 512
- Canonical target bytes: 26,578
- Target lexical units: 3,584
- Hardware used for these runs: NVIDIA L4
- Transformer parameter ceiling: 126,746
- Symbolic Wavelet parameters: 126,746
- Symbolic Wavelet final step: 700
- Symbolic Wavelet training time recorded by the run: 644.4 s
The cross-tokenizer comparison uses canonical byte NLL, bits per canonical byte, canonical-byte perplexity, and NLL per lexical unit. Native tokenizer NLL/PPL is intentionally not used for cross-tokenizer ranking.
Repository contents
Symbolic_Wavelet_L4_Transformers_Benchmarks.ipynb— benchmark notebook.benchmark_summary.json— consolidated comparison.results/— raw JSON result files copied from the completed runs.smoke_test.py— verifies notebook syntax, result-file integrity, shared validation counts, parameter ceiling, and that the symbolic wavelet result beats all four baselines on all directly comparable metrics.
Smoke test
python smoke_test.py
Expected final line:
SMOKE TEST PASSED: symbolic wavelet beats all 4 baselines on all 4 comparable metrics.
Scope of the claim
The claim in this repository is limited to the controlled benchmark recorded here. It is not a claim of general superiority over full-scale pretrained Llama, Qwen, Gemma, or DeepSeek models.