Release TransWave 96.13M C4 checkpoint

CITATION.cff

@@ -0,0 +1,9 @@
+cff-version: 1.2.0
+message: "If you use TransWave, please cite this software repository."
+title: "TransWave"
+type: software
+authors:
+  - family-names: "Majumdar"
+    given-names: "Angshul"
+year: 2026
+version: 1.0.0

README.md

@@ -0,0 +1,166 @@
+---
+license: agpl-3.0
+language:
+- en
+library_name: transformers
+datasets:
+- allenai/c4
+tags:
+- causal-lm
+- byte-level
+- raw-byte
+- attention-free
+- meyer
+- wavelet
+---
+
+
+# TransWave
+
+
+TransWave is a 96.13M-parameter attention-free causal raw-byte language model using learned block-local lifting and learned Meyer-structured multiresolution memory.
+
+
+This repository contains the best checkpoint from the completed C4 English pretraining run.
+
+
+## Checkpoint
+
+
+- Architecture: `causal_transwave_blocklifting_meyer_byte_lm_g4_v2`
+- Parameters: 96,130,040
+- Input vocabulary: 256 raw byte values
+- External BPE/SentencePiece tokenizer: none
+- Trained C4 context: 20,480 model tokens
+- Structural architecture maximum: 32,768 raw-byte tokens
+- Physical training batch: 280
+- Training tokens consumed: 4,499,988,480
+- Optimizer steps: 785
+- Dataset: `allenai/c4:en`
+- Objective: exact causal next-byte NLL
+- Source commit: `5cae31b81e15160379ded79f32bcbb443d7db9f7`
+
+
+One model token is one raw UTF-8 byte.
+
+
+## Validation result
+
+
+The run exhausted its 4.5B-token budget before reaching the PPL=3 early-stop target.
+
+
+- Best/final held-out NLL: 2.228725893
+- Best/final byte perplexity: 9.288024598
+- Validation tokens: 16,773,120
+
+
+## Tokenization
+
+
+There are two distinct layers and they should not be conflated.
+
+
+`TransWaveByteTokenizer` is the Hugging Face input codec. It converts text to raw UTF-8 byte IDs 0...255 and back. It has no BPE merges and no learned external vocabulary.
+
+
+The learned tokenizer is inside the neural network itself. `CTWBlockLiftingTokenizer` contains the trained shared byte embedding, three learned lifting stages and the learned cross-scale mixer. Those learned parameters are contained in `best_model.pt` and are loaded as part of `CTWModel`.
+
+
+## Architecture
+
+
+Eight raw bytes form one block. A shared 256-entry embedding maps each byte to 96 channels. Three learned block-local lifting stages recursively produce eight multiresolution scales, giving a 768-dimensional block state.
+
+
+Each of 12 layers combines a learned 512-block causal Meyer/Bernstein spectral FIR, learned full 8x8 scale mixing, learned spectral gates, a separate strictly causal DWConv7 path, RMS normalization and a SwiGLU feed-forward network of width 2816.
+
+
+The global block representation is shifted by one block before decoding: block b is predicted from h[b-1], never h[b]. Within each target block, an 8-step GRUCell decoder of width 256 predicts bytes autoregressively while reusing the shared byte embedding for the classifier.
+
+
+## Load
+
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+
+repo = "Angshul/TransWave"
+
+
+tokenizer = AutoTokenizer.from_pretrained(
+    repo,
+    trust_remote_code=True,
+    use_fast=False,
+)
+
+
+model = AutoModelForCausalLM.from_pretrained(
+    repo,
+    trust_remote_code=True,
+).eval()
+
+
+ids = tokenizer(
+    "TransWave operates on raw UTF-8 bytes.",
+    return_tensors="pt",
+)["input_ids"]
+
+
+# Exact scoring currently requires complete 8-byte blocks.
+ids = ids[:, : (ids.shape[1] // 8) * 8]
+
+
+with torch.no_grad():
+    out = model(input_ids=ids, labels=ids)
+
+
+print("NLL:", float(out.loss))
+print("PPL:", float(torch.exp(out.loss)))
+```
+
+
+## Reference byte generation
+
+
+The custom model exposes `generate_bytes`, an exact but deliberately simple reference generator:
+
+
+```python
+prompt = tokenizer("The future of", return_tensors="pt")["input_ids"]
+
+
+generated = model.generate_bytes(
+    prompt,
+    max_new_tokens=64,
+    do_sample=True,
+    temperature=0.8,
+    top_k=40,
+)
+
+
+print(tokenizer.decode(generated[0].tolist()))
+```
+
+
+The reference generator recomputes completed-block context as needed; it is intended for correctness and checkpoint inspection, not optimized streaming throughput.
+
+
+## Context note
+
+
+The architecture contains positional parameters for 4,096 eight-byte blocks = 32,768 raw-byte tokens. This checkpoint was trained on 20,480-token C4 contexts, so positions beyond 20,480 were not exercised in this C4 pretraining run.
+
+
+## Provenance
+
+
+Implementation: https://github.com/AngshulMajumdar/TransWaveV1
+
+
+Paper: https://openreview.net/forum?id=3a22WBU5kT
+
+
+The Hugging Face portability layer preserves the trained architecture. The only device-portability change in `transwave_core.py` is that the two explicitly-disabled CUDA autocast regions in the Meyer FFT path use the parameter device type, allowing the checkpoint to load and score on CPU as well as CUDA.

SOURCE_COMMIT.txt

@@ -0,0 +1 @@
+5cae31b81e15160379ded79f32bcbb443d7db9f7

__init__.py

@@ -0,0 +1,6 @@
+from .configuration_transwave import TransWaveConfig
+from .modeling_transwave import TransWaveForCausalLM
+from .tokenization_transwave import TransWaveByteTokenizer
+
+
+__all__ = ["TransWaveConfig", "TransWaveForCausalLM", "TransWaveByteTokenizer"]

best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf398ece0870515225b15e0038cfd6715aa4ab00d5152a7f6164e1ede1724086
+size 384624999

config.json

@@ -0,0 +1,33 @@
+{
+  "model_type": "transwave_meyer",
+  "architectures": ["TransWaveForCausalLM"],
+  "auto_map": {
+    "AutoConfig": "configuration_transwave.TransWaveConfig",
+    "AutoModelForCausalLM": "modeling_transwave.TransWaveForCausalLM"
+  },
+  "vocab_size": 256,
+  "byte_dim": 96,
+  "bytes_per_block": 8,
+  "num_scales": 8,
+  "hidden_size": 768,
+  "max_position_embeddings": 32768,
+  "block_context": 4096,
+  "num_hidden_layers": 12,
+  "num_groups": 12,
+  "meyer_bands": 12,
+  "meyer_controls_per_band": 8,
+  "meyer_kernel_blocks": 512,
+  "local_kernel_blocks": 7,
+  "intermediate_size": 2816,
+  "decoder_hidden_size": 256,
+  "trained_context_length": 20480,
+  "architecture_id": "causal_transwave_blocklifting_meyer_byte_lm_g4_v2",
+  "training_dataset": "allenai/c4:en",
+  "training_tokens": 4499988480,
+  "training_steps": 785,
+  "training_physical_batch": 280,
+  "validation_nll": 2.228725892720205,
+  "validation_ppl": 9.288024597671594,
+  "source_commit": "5cae31b81e15160379ded79f32bcbb443d7db9f7",
+  "tie_word_embeddings": true
+}

configuration_transwave.py

@@ -0,0 +1,93 @@
+from transformers import PretrainedConfig
+
+
+class TransWaveConfig(PretrainedConfig):
+
+    model_type = "transwave_meyer"
+
+
+    def __init__(
+        self,
+        vocab_size=256,
+        byte_dim=96,
+        bytes_per_block=8,
+        num_scales=8,
+        hidden_size=768,
+        max_position_embeddings=32768,
+        block_context=4096,
+        num_hidden_layers=12,
+        num_groups=12,
+        meyer_bands=12,
+        meyer_controls_per_band=8,
+        meyer_kernel_blocks=512,
+        local_kernel_blocks=7,
+        intermediate_size=2816,
+        decoder_hidden_size=256,
+        trained_context_length=20480,
+        architecture_id="causal_transwave_blocklifting_meyer_byte_lm_g4_v2",
+        **kwargs,
+    ):
+
+        # config.json may already supply this.
+        # Never provide it twice.
+
+        kwargs.setdefault(
+            "tie_word_embeddings",
+            True,
+        )
+
+        super().__init__(
+            **kwargs
+        )
+
+        self.vocab_size = vocab_size
+
+        self.byte_dim = byte_dim
+
+        self.bytes_per_block = bytes_per_block
+
+        self.num_scales = num_scales
+
+        self.hidden_size = hidden_size
+
+        self.max_position_embeddings = (
+            max_position_embeddings
+        )
+
+        self.block_context = block_context
+
+        self.num_hidden_layers = (
+            num_hidden_layers
+        )
+
+        self.num_groups = num_groups
+
+        self.meyer_bands = meyer_bands
+
+        self.meyer_controls_per_band = (
+            meyer_controls_per_band
+        )
+
+        self.meyer_kernel_blocks = (
+            meyer_kernel_blocks
+        )
+
+        self.local_kernel_blocks = (
+            local_kernel_blocks
+        )
+
+        self.intermediate_size = (
+            intermediate_size
+        )
+
+        self.decoder_hidden_size = (
+            decoder_hidden_size
+        )
+
+        self.trained_context_length = (
+            trained_context_length
+        )
+
+        self.architecture_id = (
+            architecture_id
+        )

modeling_transwave.py

@@ -0,0 +1,300 @@
+from __future__ import annotations
+
+
+import os
+import torch
+import torch.nn.functional as F
+
+
+from huggingface_hub import hf_hub_download
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import CausalLMOutput
+
+
+try:
+    from .configuration_transwave import TransWaveConfig
+    from .transwave_core import (
+        CTWModel,
+        BYTE_VOCAB,
+        BYTE_DIM,
+        BYTES_PER_BLOCK,
+        BYTE_DECODER_DIM,
+    )
+except ImportError:
+    from configuration_transwave import TransWaveConfig
+    from transwave_core import (
+        CTWModel,
+        BYTE_VOCAB,
+        BYTE_DIM,
+        BYTES_PER_BLOCK,
+        BYTE_DECODER_DIM,
+    )
+
+
+
+
+EXPECTED_PARAMS = 96_130_040
+
+
+
+
+class TransWaveForCausalLM(PreTrainedModel):
+    config_class = TransWaveConfig
+    base_model_prefix = "transwave"
+    main_input_name = "input_ids"
+    _tied_weights_keys = ["transwave.decoder.shared_embedding.weight"]
+
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.transwave = CTWModel()
+        # Do not call post_init(): CTWModel already performs the architecture's
+        # exact initialization. The trained checkpoint is loaded afterwards.
+
+
+    def get_input_embeddings(self):
+        return self.transwave.tokenizer.byte_embedding
+
+
+    def set_input_embeddings(self, value):
+        self.transwave.tokenizer.byte_embedding = value
+        self.transwave.decoder.shared_embedding = value
+
+
+    def get_output_embeddings(self):
+        return self.transwave.decoder.shared_embedding
+
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
+        config = kwargs.pop("config", None)
+        revision = kwargs.pop("revision", None)
+        cache_dir = kwargs.pop("cache_dir", None)
+        token = kwargs.pop("token", None)
+        local_files_only = kwargs.pop("local_files_only", False)
+        torch_dtype = kwargs.pop("torch_dtype", None)
+        kwargs.pop("trust_remote_code", None)
+        kwargs.pop("use_safetensors", None)
+
+
+        if config is None:
+            config = TransWaveConfig.from_pretrained(
+                pretrained_model_name_or_path,
+                revision=revision,
+                cache_dir=cache_dir,
+                token=token,
+                local_files_only=local_files_only,
+            )
+
+
+        model = cls(config)
+
+
+        path = os.fspath(pretrained_model_name_or_path)
+        if os.path.isdir(path):
+            ckpt_path = os.path.join(path, "best_model.pt")
+        else:
+            ckpt_path = hf_hub_download(
+                repo_id=path,
+                filename="best_model.pt",
+                revision=revision,
+                cache_dir=cache_dir,
+                token=token,
+                local_files_only=local_files_only,
+            )
+
+
+        checkpoint = torch.load(
+            ckpt_path,
+            map_location="cpu",
+            weights_only=False,
+        )
+        state = checkpoint["model"] if isinstance(checkpoint, dict) and "model" in checkpoint else checkpoint
+
+
+        incompatible = model.transwave.load_state_dict(state, strict=True)
+        if incompatible.missing_keys or incompatible.unexpected_keys:
+            raise RuntimeError(
+                f"Checkpoint mismatch. Missing={incompatible.missing_keys}, "
+                f"unexpected={incompatible.unexpected_keys}"
+            )
+
+
+        n_params = sum(p.numel() for p in model.transwave.parameters())
+        if n_params != EXPECTED_PARAMS:
+            raise RuntimeError(f"Parameter count {n_params:,} != {EXPECTED_PARAMS:,}")
+
+
+        if torch_dtype not in (None, "auto"):
+            model = model.to(dtype=torch_dtype)
+
+
+        model.eval()
+        return model
+
+
+    def nll(self, input_ids, checkpoint_layers=False):
+        return self.transwave.nll(
+            input_ids,
+            checkpoint_layers=checkpoint_layers,
+        )
+
+
+    def contexts(self, input_ids, checkpoint_layers=False):
+        return self.transwave.contexts(
+            input_ids,
+            checkpoint_layers=checkpoint_layers,
+        )
+
+
+    def teacher_forced_logits(self, input_ids):
+        context, target_blocks = self.transwave.contexts(
+            input_ids,
+            checkpoint_layers=False,
+        )
+        decoder = self.transwave.decoder
+        batch, blocks, _ = context.shape
+        n = batch * blocks
+
+
+        hidden = torch.tanh(
+            decoder.context_proj(context)
+        ).reshape(n, BYTE_DECODER_DIM)
+
+
+        previous = decoder.bos[None, :].expand(n, -1)
+        position_logits = []
+
+
+        for r in range(BYTES_PER_BLOCK):
+            decoder_input = previous + decoder.byte_position[r][None, :]
+            hidden = decoder.gru(decoder_input, hidden)
+            z = decoder.to_byte_space(decoder.norm(hidden))
+            logits = F.linear(z, decoder.shared_embedding.weight)
+            position_logits.append(
+                logits.reshape(batch, blocks, BYTE_VOCAB)
+            )
+            target = target_blocks[:, :, r].reshape(-1).long()
+            previous = decoder.shared_embedding(target)
+
+
+        logits = torch.stack(position_logits, dim=2)
+        return logits.reshape(
+            batch,
+            blocks * BYTES_PER_BLOCK,
+            BYTE_VOCAB,
+        )
+
+
+    def forward(
+        self,
+        input_ids=None,
+        labels=None,
+        return_dict=None,
+        **kwargs,
+    ):
+        if input_ids is None:
+            raise ValueError("input_ids are required")
+        if input_ids.ndim != 2:
+            raise ValueError("input_ids must have shape [batch, tokens]")
+        if input_ids.shape[1] % BYTES_PER_BLOCK != 0:
+            raise ValueError("TransWave scoring length must be divisible by 8")
+
+
+        if labels is not None:
+            if labels.shape != input_ids.shape or not torch.equal(labels, input_ids):
+                raise ValueError(
+                    "TransWave performs the causal shift internally; "
+                    "for exact scoring/training, labels must equal input_ids."
+                )
+
+
+        logits = self.teacher_forced_logits(input_ids)
+        loss = self.transwave.nll(
+            input_ids,
+            checkpoint_layers=False,
+        )
+
+
+        if return_dict is False:
+            return (loss, logits)
+
+
+        return CausalLMOutput(
+            loss=loss,
+            logits=logits,
+        )
+
+
+    @torch.no_grad()
+    def generate_bytes(
+        self,
+        input_ids,
+        max_new_tokens=32,
+        do_sample=False,
+        temperature=1.0,
+        top_k=None,
+    ):
+        if input_ids.ndim != 2 or input_ids.shape[0] != 1:
+            raise ValueError("generate_bytes currently supports batch size 1")
+        out = input_ids.long().clone()
+        device = next(self.parameters()).device
+        out = out.to(device)
+
+
+        for _ in range(int(max_new_tokens)):
+            total = out.shape[1]
+            complete = (total // BYTES_PER_BLOCK) * BYTES_PER_BLOCK
+            r = total - complete
+
+
+            if complete == 0:
+                block_context = self.transwave.bos_context[None, :].to(device)
+            else:
+                _, blocks = self.transwave.encode_completed_blocks(
+                    out[:, :complete],
+                    checkpoint_layers=False,
+                )
+                h, _ = self.transwave.encode_completed_blocks(
+                    out[:, :complete],
+                    checkpoint_layers=False,
+                )
+                block_context = h[:, -1, :]
+
+
+            decoder = self.transwave.decoder
+            hidden = torch.tanh(decoder.context_proj(block_context))
+            previous = decoder.bos[None, :]
+
+
+            current = out[:, complete:]
+            logits = None
+            for j in range(r + 1):
+                decoder_input = previous + decoder.byte_position[j][None, :]
+                hidden = decoder.gru(decoder_input, hidden)
+                z = decoder.to_byte_space(decoder.norm(hidden))
+                logits = F.linear(z, decoder.shared_embedding.weight)
+                if j < r:
+                    previous = decoder.shared_embedding(current[:, j])
+
+
+            scores = logits.float() / max(float(temperature), 1e-6)
+            if top_k is not None and int(top_k) > 0:
+                k = min(int(top_k), BYTE_VOCAB)
+                threshold = torch.topk(scores, k, dim=-1).values[:, -1:]
+                scores = scores.masked_fill(scores < threshold, float("-inf"))
+
+
+            if do_sample:
+                next_id = torch.multinomial(
+                    torch.softmax(scores, dim=-1),
+                    num_samples=1,
+                )
+            else:
+                next_id = scores.argmax(dim=-1, keepdim=True)
+
+
+            out = torch.cat([out, next_id], dim=1)
+
+
+        return out

requirements.txt

@@ -0,0 +1,4 @@
+torch>=2.5
+numpy>=1.26
+transformers>=4.46,<5
+huggingface_hub>=0.26

smoke_test.py

@@ -0,0 +1,77 @@
+from __future__ import annotations
+
+
+import math
+from pathlib import Path
+
+
+import torch
+from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
+
+
+ROOT = Path(__file__).resolve().parent
+EXPECTED_PARAMS = 96_130_040
+
+
+print("1/6 tokenizer load + UTF-8 round trip")
+tok = AutoTokenizer.from_pretrained(
+    ROOT,
+    trust_remote_code=True,
+    use_fast=False,
+)
+sample = "TransWave — raw bytes ✓"
+ids = tok(sample, return_tensors="pt")["input_ids"]
+decoded = tok.decode(ids[0].tolist())
+assert decoded == sample, (decoded, sample)
+assert tok.vocab_size == 256
+
+
+print("2/6 config")
+cfg = AutoConfig.from_pretrained(ROOT, trust_remote_code=True)
+assert cfg.architecture_id == "causal_transwave_blocklifting_meyer_byte_lm_g4_v2"
+assert cfg.trained_context_length == 20480
+assert cfg.vocab_size == 256
+
+
+print("3/6 trained checkpoint load")
+model = AutoModelForCausalLM.from_pretrained(
+    ROOT,
+    trust_remote_code=True,
+).eval()
+n = sum(p.numel() for p in model.transwave.parameters())
+assert n == EXPECTED_PARAMS, n
+
+
+print("4/6 exact finite NLL")
+torch.manual_seed(20261012)
+x = torch.randint(0, 256, (1, 32), dtype=torch.long)
+with torch.no_grad():
+    direct = model.transwave.nll(x, checkpoint_layers=False)
+    wrapped = model(input_ids=x, labels=x).loss
+assert torch.isfinite(direct)
+assert torch.isfinite(wrapped)
+assert abs(float(direct - wrapped)) < 2e-5
+print("NLL:", float(wrapped), "PPL:", math.exp(float(wrapped)))
+
+
+print("5/6 future-perturbation causality")
+a = torch.randint(0, 256, (1, 32), dtype=torch.long)
+b = a.clone()
+b[:, 16:] = torch.randint(0, 256, b[:, 16:].shape, dtype=torch.long)
+with torch.no_grad():
+    ca, _ = model.transwave.contexts(a, checkpoint_layers=False)
+    cb, _ = model.transwave.contexts(b, checkpoint_layers=False)
+diff = float((ca[:, :3, :] - cb[:, :3, :]).abs().max())
+assert diff <= 2e-4, diff
+print("future->past max diff:", diff)
+
+
+print("6/6 reference generation")
+prompt = tok("abcdefgh", return_tensors="pt")["input_ids"]
+with torch.no_grad():
+    y = model.generate_bytes(prompt, max_new_tokens=2, do_sample=False)
+assert y.shape == (1, 10)
+assert int(y.min()) >= 0 and int(y.max()) <= 255
+
+
+print("SMOKE TEST: PASS")

special_tokens_map.json

@@ -0,0 +1 @@
+{}

tokenization_transwave.py

@@ -0,0 +1,178 @@
+from __future__ import annotations
+
+import json
+import os
+
+from transformers import PreTrainedTokenizer
+
+
+class TransWaveByteTokenizer(PreTrainedTokenizer):
+
+    vocab_files_names = {
+        "vocab_file": "vocab.json"
+    }
+
+    model_input_names = [
+        "input_ids"
+    ]
+
+
+    def __init__(
+        self,
+        vocab_file=None,
+        model_max_length=20480,
+        **kwargs,
+    ):
+
+        self.vocab_file = vocab_file
+
+        self._id_to_token = [
+            f"<0x{i:02X}>"
+            for i in range(256)
+        ]
+
+        self._token_to_id = {
+            token: i
+            for i, token
+            in enumerate(self._id_to_token)
+        }
+
+        # Values may already arrive through tokenizer_config.json.
+        # Never pass the same keyword twice.
+
+        kwargs.setdefault(
+            "model_max_length",
+            model_max_length,
+        )
+
+        kwargs.setdefault(
+            "clean_up_tokenization_spaces",
+            False,
+        )
+
+        super().__init__(**kwargs)
+
+
+    @property
+    def vocab_size(self):
+
+        return 256
+
+
+    def get_vocab(self):
+
+        vocab = dict(
+            self._token_to_id
+        )
+
+        vocab.update(
+            self.added_tokens_encoder
+        )
+
+        return vocab
+
+
+    def _tokenize(
+        self,
+        text,
+        **kwargs,
+    ):
+
+        return [
+            self._id_to_token[b]
+            for b in text.encode("utf-8")
+        ]
+
+
+    def _convert_token_to_id(
+        self,
+        token,
+    ):
+
+        if token not in self._token_to_id:
+            raise ValueError(
+                f"Unknown byte token: {token}"
+            )
+
+        return self._token_to_id[token]
+
+
+    def _convert_id_to_token(
+        self,
+        index,
+    ):
+
+        index = int(index)
+
+        if not 0 <= index < 256:
+            raise ValueError(
+                f"Byte id out of range: {index}"
+            )
+
+        return self._id_to_token[index]
+
+
+    def convert_tokens_to_string(
+        self,
+        tokens,
+    ):
+
+        data = bytearray()
+
+        for token in tokens:
+
+            if token in self._token_to_id:
+                data.append(
+                    self._token_to_id[token]
+                )
+
+        return bytes(data).decode(
+            "utf-8",
+            errors="replace",
+        )
+
+
+    def save_vocabulary(
+        self,
+        save_directory,
+        filename_prefix=None,
+    ):
+
+        os.makedirs(
+            save_directory,
+            exist_ok=True,
+        )
+
+        filename = (
+            (
+                filename_prefix + "-"
+            )
+            if filename_prefix
+            else ""
+        ) + "vocab.json"
+
+        path = os.path.join(
+            save_directory,
+            filename,
+        )
+
+        vocab = {
+            token: i
+            for i, token
+            in enumerate(self._id_to_token)
+        }
+
+        with open(
+            path,
+            "w",
+            encoding="utf-8",
+        ) as f:
+
+            json.dump(
+                vocab,
+                f,
+                ensure_ascii=False,
+                indent=2,
+            )
+
+        return (path,)

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "TransWaveByteTokenizer",
+  "model_max_length": 20480,
+  "clean_up_tokenization_spaces": false,
+  "auto_map": {
+    "AutoTokenizer": ["tokenization_transwave.TransWaveByteTokenizer", null]
+  }
+}

training_protocol.json

@@ -0,0 +1,21 @@
+{
+  "architecture": "causal_transwave_blocklifting_meyer_byte_lm_g4_v2",
+  "source_commit": "5cae31b81e15160379ded79f32bcbb443d7db9f7",
+  "parameters": 96130040,
+  "dataset": "allenai/c4:en",
+  "objective": "exact causal next-byte NLL",
+  "context_tokens": 20480,
+  "physical_batch": 280,
+  "tokens_per_full_step": 5734400,
+  "maximum_training_tokens": 4499996672,
+  "usable_training_tokens": 4499988480,
+  "target_ppl": 3.0,
+  "base_lr": 0.0001,
+  "min_lr": 1e-05,
+  "warmup_fraction": 0.01,
+  "grad_clip": 1.0,
+  "model_seed": 20261010,
+  "order_seed": 20261011,
+  "allocator": "expandable_segments:True,garbage_collection_threshold:0.80",
+  "fingerprint": "370d05a6d2672fc7838726e78e8fe267b4b9c480b4f3708b64e7136beea128fd"
+}

training_result.json

@@ -0,0 +1,24 @@
+{
+  "release": "TransWave-Meyer-96M-C4",
+  "architecture": "causal_transwave_blocklifting_meyer_byte_lm_g4_v2",
+  "source_commit": "5cae31b81e15160379ded79f32bcbb443d7db9f7",
+  "parameters": 96130040,
+  "dataset": "allenai/c4:en",
+  "context_tokens": 20480,
+  "physical_batch": 280,
+  "tokens_per_full_step": 5734400,
+  "usable_training_tokens": 4499988480,
+  "tokens_seen": 4499988480,
+  "steps": 785,
+  "target_ppl": 3.0,
+  "best_full_validation_nll": 2.228725892720205,
+  "best_full_validation_ppl": 9.288024597671594,
+  "final_validation": {
+    "nll": 2.228725892720205,
+    "ppl": 9.288024597671594,
+    "tokens": 16773120,
+    "contexts": 819
+  },
+  "stop_reason": "maximum_4p5B_token_budget_exhausted",
+  "wall_hours": 2.553788356317414
+}

transwave_core.py

@@ -0,0 +1,3576 @@
+from __future__ import annotations
+
+
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.checkpoint import checkpoint as torch_checkpoint
+
+
+ARCH = "causal_transwave_blocklifting_meyer_byte_lm_g4_v2"
+BYTE_VOCAB = 256
+BYTE_DIM = 96
+BYTES_PER_BLOCK = 8
+N_SCALES = 8
+MODEL_DIM = 768
+MODEL_CONTEXT = 4096
+RAW_BYTES_PER_SEQUENCE = MODEL_CONTEXT * BYTES_PER_BLOCK
+N_LAYERS = 12
+N_GROUPS = 12
+CHANNELS_PER_GROUP = BYTE_DIM // N_GROUPS
+N_MEYER_BANDS = 12
+N_BAND_CONTROLS = 8
+MEYER_KERNEL = 512
+LOCAL_KERNEL = 7
+FFN_DIM = 2816
+BYTE_DECODER_DIM = 256
+HEAD_CHUNK = 32768
+
+
+assert BYTE_DIM * BYTES_PER_BLOCK == MODEL_DIM
+assert RAW_BYTES_PER_SEQUENCE == 32768
+assert BYTE_DIM % N_GROUPS == 0
+
+
+class CTWLocalBlockDelta(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        dim=BYTE_DIM,
+        groups=12,
+        expansion=2,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        if (
+            dim
+            %
+            groups
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "dim must be divisible by groups"
+            )
+
+
+
+
+        hidden = (
+            dim
+            *
+            expansion
+        )
+
+
+
+
+        if (
+            hidden
+            %
+            groups
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "expanded dimension must be divisible by groups"
+            )
+
+
+
+
+        self.depthwise = nn.Conv1d(
+            dim,
+            dim,
+            kernel_size=3,
+            padding=1,
+            groups=dim,
+            bias=True,
+        )
+
+
+
+
+        self.mix_in = nn.Conv1d(
+            dim,
+            hidden,
+            kernel_size=1,
+            groups=groups,
+            bias=True,
+        )
+
+
+
+
+        self.mix_out = nn.Conv1d(
+            hidden,
+            dim,
+            kernel_size=1,
+            groups=groups,
+            bias=True,
+        )
+
+
+
+
+        # Haar-like initialization:
+        # residual branch starts exactly at zero.
+
+
+        nn.init.zeros_(
+            self.mix_out.weight
+        )
+
+
+
+
+        nn.init.zeros_(
+            self.mix_out.bias
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        y = x.transpose(
+            1,
+            2,
+        )
+
+
+
+
+        y = self.depthwise(
+            y
+        )
+
+
+
+
+        y = F.silu(
+            y
+        )
+
+
+
+
+        y = self.mix_in(
+            y
+        )
+
+
+
+
+        y = F.silu(
+            y
+        )
+
+
+
+
+        y = self.mix_out(
+            y
+        )
+
+
+
+
+        return y.transpose(
+            1,
+            2,
+        )
+
+
+
+
+class CTWLiftingStage(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        dim=BYTE_DIM,
+        groups=12,
+        expansion=2,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.predict_delta = (
+            CTWLocalBlockDelta(
+                dim=dim,
+                groups=groups,
+                expansion=expansion,
+            )
+        )
+
+
+
+
+        self.update_delta = (
+            CTWLocalBlockDelta(
+                dim=dim,
+                groups=groups,
+                expansion=expansion,
+            )
+        )
+
+
+
+
+    def predict(
+        self,
+        even,
+    ):
+
+
+        return (
+            even
+            +
+            self.predict_delta(
+                even
+            )
+        )
+
+
+
+
+    def update(
+        self,
+        detail,
+    ):
+
+
+        return (
+            0.5
+            *
+            detail
+            +
+            self.update_delta(
+                detail
+            )
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        if (
+            x.ndim != 3
+            or
+            x.shape[1] % 2 != 0
+        ):
+
+
+            raise ValueError(
+                "Expected [N,T,C] with even T."
+            )
+
+
+
+
+        even = x[
+            :,
+            0::2,
+            :
+        ]
+
+
+
+
+        odd = x[
+            :,
+            1::2,
+            :
+        ]
+
+
+
+
+        detail = (
+            odd
+            -
+            self.predict(
+                even
+            )
+        )
+
+
+
+
+        coarse = (
+            even
+            +
+            self.update(
+                detail
+            )
+        )
+
+
+
+
+        return (
+            coarse,
+            detail,
+        )
+
+
+
+
+class CTWCrossScaleMixer(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        scales=N_SCALES,
+        dim=BYTE_DIM,
+        groups=N_GROUPS,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        if (
+            dim
+            %
+            groups
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "dim must be divisible by groups"
+            )
+
+
+
+
+        self.scales = int(
+            scales
+        )
+
+
+
+
+        self.dim = int(
+            dim
+        )
+
+
+
+
+        self.groups = int(
+            groups
+        )
+
+
+
+
+        self.channels_per_group = (
+            self.dim
+            //
+            self.groups
+        )
+
+
+
+
+        # M_g = I + Delta_g
+
+
+        self.scale_delta = nn.Parameter(
+
+
+            torch.zeros(
+                self.groups,
+                self.scales,
+                self.scales,
+            )
+        )
+
+
+
+
+        # Content-dependent gain.
+
+
+        self.gate_weight = nn.Parameter(
+
+
+            torch.zeros(
+                self.groups,
+                self.scales,
+                self.channels_per_group,
+            )
+        )
+
+
+
+
+        self.gate_bias = nn.Parameter(
+
+
+            torch.zeros(
+                self.groups,
+                self.scales,
+            )
+        )
+
+
+
+
+    def forward(
+        self,
+        z,
+    ):
+
+
+        if (
+            z.ndim != 4
+            or
+            z.shape[2]
+            !=
+            self.scales
+            or
+            z.shape[3]
+            !=
+            self.dim
+        ):
+
+
+            raise ValueError(
+
+
+                f"Expected "
+                f"[B,L,{self.scales},{self.dim}]"
+            )
+
+
+
+
+        b, l, s, _ = z.shape
+
+
+
+
+        zg = z.reshape(
+            b,
+            l,
+            s,
+            self.groups,
+            self.channels_per_group,
+        )
+
+
+
+
+        gate_logits = torch.einsum(
+            "blsgc,gsc->blsg",
+            zg,
+            self.gate_weight,
+        )
+
+
+
+
+        gate_logits = (
+            gate_logits
+            /
+            math.sqrt(
+                self.channels_per_group
+            )
+            +
+            self.gate_bias.transpose(
+                0,
+                1,
+            )
+        )
+
+
+
+
+        # Zero logits -> gain exactly 1.
+
+
+        gain = (
+            2.0
+            *
+            torch.sigmoid(
+                gate_logits
+            )
+        )
+
+
+
+
+        zg = (
+            zg
+            *
+            gain
+            .to(
+                zg.dtype
+            )
+            .unsqueeze(
+                -1
+            )
+        )
+
+
+
+
+        eye = torch.eye(
+            self.scales,
+            device=z.device,
+            dtype=z.dtype,
+        ).unsqueeze(
+            0
+        )
+
+
+
+
+        mix = (
+            eye
+            +
+            self.scale_delta.to(
+                z.dtype
+            )
+        )
+
+
+
+
+        mixed = torch.einsum(
+            "blsgc,gts->bltgc",
+            zg,
+            mix,
+        )
+
+
+
+
+        return mixed.reshape(
+            b,
+            l,
+            s,
+            self.dim,
+        )
+
+
+
+
+class CTWBlockLiftingTokenizer(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.byte_embedding = nn.Embedding(
+            BYTE_VOCAB,
+            BYTE_DIM,
+        )
+
+
+
+
+        nn.init.normal_(
+
+
+            self.byte_embedding.weight,
+
+
+            mean=0.0,
+
+
+            std=(
+                BYTE_DIM
+                **
+                -0.5
+            ),
+        )
+
+
+
+
+        self.stages = nn.ModuleList(
+
+
+            [
+
+
+                CTWLiftingStage(
+                    dim=BYTE_DIM,
+                    groups=N_GROUPS,
+                    expansion=2,
+                )
+
+
+                for _ in range(
+                    3
+                )
+            ]
+        )
+
+
+
+
+        self.cross_scale = (
+            CTWCrossScaleMixer(
+                scales=N_SCALES,
+                dim=BYTE_DIM,
+                groups=N_GROUPS,
+            )
+        )
+
+
+
+
+    def forward(
+        self,
+        byte_ids,
+    ):
+
+
+        if (
+            byte_ids.ndim
+            !=
+            2
+        ):
+
+
+            raise ValueError(
+                "Expected byte IDs [B,8L]."
+            )
+
+
+
+
+        b, t = byte_ids.shape
+
+
+
+
+        if (
+            t
+            %
+            BYTES_PER_BLOCK
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "Byte length must be divisible by 8."
+            )
+
+
+
+
+        l = (
+            t
+            //
+            BYTES_PER_BLOCK
+        )
+
+
+
+
+        # -------------------------------------------------------------
+        # CRITICAL CAUSAL CHANGE:
+        #
+        # each completed eight-byte block becomes an independent item
+        # before ANY lifting convolution is applied.
+        # -------------------------------------------------------------
+
+
+        blocks = (
+            byte_ids
+            .long()
+            .reshape(
+                b,
+                l,
+                BYTES_PER_BLOCK,
+            )
+        )
+
+
+
+
+        x = self.byte_embedding(
+
+
+            blocks.reshape(
+                b
+                *
+                l,
+                BYTES_PER_BLOCK,
+            )
+        )
+
+
+
+
+        details = []
+
+
+
+
+        coarse = x
+
+
+
+
+        for stage in self.stages:
+
+
+            (
+                coarse,
+                detail,
+            ) = stage(
+                coarse
+            )
+
+
+
+
+            details.append(
+                detail
+            )
+
+
+
+
+        # D0=4, D1=2, D2=1, S3=1.
+
+
+        packed = torch.cat(
+
+
+            [
+                details[0],
+                details[1],
+                details[2],
+                coarse,
+            ],
+
+
+            dim=1,
+        )
+
+
+
+
+        if (
+            packed.shape[1]
+            !=
+            N_SCALES
+        ):
+
+
+            raise RuntimeError(
+                "Lifting scale packing mismatch."
+            )
+
+
+
+
+        z = packed.reshape(
+            b,
+            l,
+            N_SCALES,
+            BYTE_DIM,
+        )
+
+
+
+
+        return self.cross_scale(
+            z
+        )
+
+
+
+
+# =============================================================================
+# RMS NORM / SWIGLU
+# =============================================================================
+
+
+class CTWRMSNorm(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        dim,
+        eps=1e-6,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.weight = nn.Parameter(
+            torch.ones(
+                dim
+            )
+        )
+
+
+
+
+        self.eps = eps
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        xf = x.float()
+
+
+
+
+        y = (
+            xf
+            *
+            torch.rsqrt(
+
+
+                xf.pow(2).mean(
+                    dim=-1,
+                    keepdim=True,
+                )
+                +
+                self.eps
+            )
+        )
+
+
+
+
+        return (
+            y.to(
+                x.dtype
+            )
+            *
+            self.weight.to(
+                x.dtype
+            )
+        )
+
+
+
+
+class CTWSwiGLU(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        dim,
+        hidden,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.gate = nn.Linear(
+            dim,
+            hidden,
+        )
+
+
+
+
+        self.value = nn.Linear(
+            dim,
+            hidden,
+        )
+
+
+
+
+        self.out = nn.Linear(
+            hidden,
+            dim,
+        )
+
+
+
+
+        for m in (
+            self.gate,
+            self.value,
+            self.out,
+        ):
+
+
+            nn.init.xavier_uniform_(
+                m.weight
+            )
+
+
+
+
+            nn.init.zeros_(
+                m.bias
+            )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        return self.out(
+
+
+            F.silu(
+                self.gate(
+                    x
+                )
+            )
+
+
+            *
+
+
+            self.value(
+                x
+            )
+        )
+
+
+
+
+# =============================================================================
+# MEYER BASIS
+# =============================================================================
+
+
+def ctw_meyer_nu(
+    x,
+):
+
+
+    x = x.clamp(
+        0.0,
+        1.0,
+    )
+
+
+
+
+    return (
+        x**4
+        *
+        (
+            35.0
+            -
+            84.0 * x
+            +
+            70.0 * x**2
+            -
+            20.0 * x**3
+        )
+    )
+
+
+
+
+def ctw_build_meyer_basis(
+    kernel_size,
+    bands,
+    controls,
+):
+
+
+    nf = (
+        kernel_size
+        //
+        2
+        +
+        1
+    )
+
+
+
+
+    f = torch.linspace(
+        0.0,
+        1.0,
+        nf,
+        dtype=torch.float32,
+    )
+
+
+
+
+    edges = torch.linspace(
+        0.0,
+        1.0,
+        bands + 1,
+        dtype=torch.float32,
+    )
+
+
+
+
+    windows = []
+
+
+
+
+    for j in range(
+        bands
+    ):
+
+
+        left = float(
+            edges[j]
+        )
+
+
+
+
+        right = float(
+            edges[j + 1]
+        )
+
+
+
+
+        delta = (
+            0.35
+            *
+            (
+                right
+                -
+                left
+            )
+        )
+
+
+
+
+        if j == 0:
+
+
+            left_gate = torch.ones_like(
+                f
+            )
+
+
+        else:
+
+
+            a = (
+                left
+                -
+                delta
+            )
+
+
+
+
+            b = (
+                left
+                +
+                delta
+            )
+
+
+
+
+            u = (
+                f
+                -
+                a
+            ) / max(
+                b - a,
+                1e-8,
+            )
+
+
+
+
+            left_gate = torch.sin(
+
+
+                0.5
+                *
+                math.pi
+                *
+                ctw_meyer_nu(
+                    u
+                )
+            )
+
+
+
+
+            left_gate = torch.where(
+                f <= a,
+                torch.zeros_like(
+                    f
+                ),
+                left_gate,
+            )
+
+
+
+
+            left_gate = torch.where(
+                f >= b,
+                torch.ones_like(
+                    f
+                ),
+                left_gate,
+            )
+
+
+
+
+        if j == bands - 1:
+
+
+            right_gate = torch.ones_like(
+                f
+            )
+
+
+        else:
+
+
+            a = (
+                right
+                -
+                delta
+            )
+
+
+
+
+            b = (
+                right
+                +
+                delta
+            )
+
+
+
+
+            u = (
+                f
+                -
+                a
+            ) / max(
+                b - a,
+                1e-8,
+            )
+
+
+
+
+            right_gate = torch.cos(
+
+
+                0.5
+                *
+                math.pi
+                *
+                ctw_meyer_nu(
+                    u
+                )
+            )
+
+
+
+
+            right_gate = torch.where(
+                f <= a,
+                torch.ones_like(
+                    f
+                ),
+                right_gate,
+            )
+
+
+
+
+            right_gate = torch.where(
+                f >= b,
+                torch.zeros_like(
+                    f
+                ),
+                right_gate,
+            )
+
+
+
+
+        windows.append(
+            left_gate
+            *
+            right_gate
+        )
+
+
+
+
+    windows = torch.stack(
+        windows,
+        dim=0,
+    )
+
+
+
+
+    windows = (
+        windows
+        /
+        torch.sqrt(
+
+
+            windows.pow(2).sum(
+                dim=0,
+                keepdim=True,
+            )
+            +
+            1e-12
+        )
+    )
+
+
+
+
+    degree = (
+        controls
+        -
+        1
+    )
+
+
+
+
+    basis = []
+
+
+
+
+    for j in range(
+        bands
+    ):
+
+
+        left = edges[j]
+
+
+
+
+        right = edges[j + 1]
+
+
+
+
+        u = (
+            (
+                f
+                -
+                left
+            )
+            /
+            (
+                right
+                -
+                left
+                +
+                1e-12
+            )
+        ).clamp(
+            0.0,
+            1.0,
+        )
+
+
+
+
+        row = []
+
+
+
+
+        for m in range(
+            controls
+        ):
+
+
+            bern = (
+
+
+                math.comb(
+                    degree,
+                    m,
+                )
+
+
+                *
+
+
+                u.pow(
+                    m
+                )
+
+
+                *
+
+
+                (
+                    1.0
+                    -
+                    u
+                ).pow(
+                    degree
+                    -
+                    m
+                )
+            )
+
+
+
+
+            row.append(
+                windows[j]
+                *
+                bern
+            )
+
+
+
+
+        basis.append(
+
+
+            torch.stack(
+                row,
+                dim=0,
+            )
+        )
+
+
+
+
+    return torch.stack(
+        basis,
+        dim=0,
+    )
+
+
+
+
+# =============================================================================
+# SAME-TIME FULL 8x8 SCALE MIXING
+# =============================================================================
+
+
+class CTWScaleMixer(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        eye = torch.eye(
+            N_SCALES
+        )
+
+
+
+
+        self.weight = nn.Parameter(
+
+
+            eye
+            .unsqueeze(
+                0
+            )
+            .repeat(
+                N_GROUPS,
+                1,
+                1,
+            )
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        b, l, s, c = x.shape
+
+
+
+
+        xg = x.reshape(
+            b,
+            l,
+            s,
+            N_GROUPS,
+            CHANNELS_PER_GROUP,
+        )
+
+
+
+
+        y = torch.einsum(
+            "blsgc,gts->bltgc",
+            xg,
+            self.weight,
+        )
+
+
+
+
+        return y.reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+# =============================================================================
+# STRICTLY CAUSAL MEYER FIR
+#
+# h[k] is a nonnegative lag:
+#
+# y[t] = SUM_{k=0}^{K-1} h[k] x[t-k]
+#
+# FFT only implements this zero-padded LINEAR convolution efficiently.
+# =============================================================================
+
+
+class CTWCausalMeyerFIR(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        basis = ctw_build_meyer_basis(
+            MEYER_KERNEL,
+            N_MEYER_BANDS,
+            N_BAND_CONTROLS,
+        )
+
+
+
+
+        self.register_buffer(
+            "basis",
+            basis,
+            persistent=False,
+        )
+
+
+
+
+        imag_mask = torch.ones(
+            MEYER_KERNEL
+            //
+            2
+            +
+            1,
+            dtype=torch.float32,
+        )
+
+
+
+
+        imag_mask[0] = 0.0
+
+
+
+
+        if (
+            MEYER_KERNEL
+            %
+            2
+            ==
+            0
+        ):
+
+
+            imag_mask[-1] = 0.0
+
+
+
+
+        self.register_buffer(
+            "imag_mask",
+            imag_mask,
+            persistent=False,
+        )
+
+
+
+
+        shape = (
+            N_GROUPS,
+            N_SCALES,
+            N_MEYER_BANDS,
+            N_BAND_CONTROLS,
+        )
+
+
+
+
+        self.coeff_re = nn.Parameter(
+            torch.empty(
+                shape
+            )
+        )
+
+
+
+
+        self.coeff_im = nn.Parameter(
+            torch.empty(
+                shape
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.coeff_re,
+            mean=0.0,
+            std=1e-3,
+        )
+
+
+
+
+        nn.init.normal_(
+            self.coeff_im,
+            mean=0.0,
+            std=1e-3,
+        )
+
+
+
+
+    def causal_kernel(
+        self,
+    ):
+
+
+        with torch.autocast(
+            device_type=self.coeff_re.device.type,
+            enabled=False,
+        ):
+
+
+            basis = self.basis.float()
+
+
+
+
+            re = torch.einsum(
+                "gsbm,bmf->gsf",
+                self.coeff_re.float(),
+                basis,
+            )
+
+
+
+
+            im = torch.einsum(
+                "gsbm,bmf->gsf",
+                self.coeff_im.float(),
+                basis,
+            )
+
+
+
+
+            im = (
+                im
+                *
+                self.imag_mask[
+                    None,
+                    None,
+                    :
+                ]
+            )
+
+
+
+
+            spectrum = torch.complex(
+                re,
+                im,
+            )
+
+
+
+
+            h = torch.fft.irfft(
+                spectrum,
+                n=MEYER_KERNEL,
+                dim=-1,
+            )
+
+
+
+
+        return h
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        # [B,L,S,96]
+
+
+        b, l, s, c = x.shape
+
+
+
+
+        # [B,G,S,Cg,L]
+
+
+        xg = (
+            x
+            .reshape(
+                b,
+                l,
+                s,
+                N_GROUPS,
+                CHANNELS_PER_GROUP,
+            )
+            .permute(
+                0,
+                3,
+                2,
+                4,
+                1,
+            )
+            .contiguous()
+        )
+
+
+
+
+        linear_length = (
+            l
+            +
+            MEYER_KERNEL
+            -
+            1
+        )
+
+
+
+
+        nfft = (
+            1
+            <<
+            (
+                linear_length
+                -
+                1
+            ).bit_length()
+        )
+
+
+
+
+        with torch.autocast(
+            device_type=self.coeff_re.device.type,
+            enabled=False,
+        ):
+
+
+            xf = xg.float()
+
+
+
+
+            h = self.causal_kernel()
+
+
+
+
+            X = torch.fft.rfft(
+                xf,
+                n=nfft,
+                dim=-1,
+            )
+
+
+
+
+            H = torch.fft.rfft(
+                h,
+                n=nfft,
+                dim=-1,
+            )
+
+
+
+
+            Y = torch.fft.irfft(
+
+
+                X
+                *
+                H[
+                    None,
+                    :,
+                    :,
+                    None,
+                    :
+                ],
+
+
+                n=nfft,
+                dim=-1,
+            )
+
+
+
+
+            # First L outputs of LINEAR convolution.
+
+
+            Y = Y[
+                ...,
+                :l
+            ]
+
+
+
+
+        return (
+
+
+            Y
+            .permute(
+                0,
+                4,
+                2,
+                1,
+                3,
+            )
+            .contiguous()
+            .reshape(
+                b,
+                l,
+                s,
+                c,
+            )
+            .to(
+                x.dtype
+            )
+        )
+
+
+
+
+# =============================================================================
+# STRICTLY LEFT-CAUSAL LOCAL CONVOLUTION
+# =============================================================================
+
+
+class CTWCausalLocalConv(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.conv = nn.Conv1d(
+            MODEL_DIM,
+            MODEL_DIM,
+            kernel_size=LOCAL_KERNEL,
+            groups=MODEL_DIM,
+            bias=True,
+        )
+
+
+
+
+        nn.init.normal_(
+            self.conv.weight,
+            mean=0.0,
+            std=0.02,
+        )
+
+
+
+
+        nn.init.zeros_(
+            self.conv.bias
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        xt = x.transpose(
+            1,
+            2,
+        )
+
+
+
+
+        xt = F.pad(
+            xt,
+            (
+                LOCAL_KERNEL
+                -
+                1,
+                0,
+            ),
+        )
+
+
+
+
+        return (
+            self.conv(
+                xt
+            )
+            .transpose(
+                1,
+                2,
+            )
+        )
+
+
+
+
+# =============================================================================
+# ONE CAUSAL TRANSWAVE LAYER
+# =============================================================================
+
+
+class CTWLayer(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.norm1 = CTWRMSNorm(
+            MODEL_DIM
+        )
+
+
+
+
+        self.input_gate = nn.Linear(
+            MODEL_DIM,
+            MODEL_DIM,
+        )
+
+
+
+
+        self.output_gate = nn.Linear(
+            MODEL_DIM,
+            MODEL_DIM,
+        )
+
+
+
+
+        nn.init.zeros_(
+            self.input_gate.weight
+        )
+
+
+
+
+        nn.init.constant_(
+            self.input_gate.bias,
+            2.0,
+        )
+
+
+
+
+        nn.init.zeros_(
+            self.output_gate.weight
+        )
+
+
+
+
+        nn.init.constant_(
+            self.output_gate.bias,
+            2.0,
+        )
+
+
+
+
+        self.scale_in = (
+            CTWScaleMixer()
+        )
+
+
+
+
+        self.meyer = (
+            CTWCausalMeyerFIR()
+        )
+
+
+
+
+        self.scale_out = (
+            CTWScaleMixer()
+        )
+
+
+
+
+        self.local = (
+            CTWCausalLocalConv()
+        )
+
+
+
+
+        self.alpha_meyer = nn.Parameter(
+
+
+            torch.tensor(
+                0.10,
+                dtype=torch.float32,
+            )
+        )
+
+
+
+
+        self.alpha_local = nn.Parameter(
+
+
+            torch.tensor(
+                0.10,
+                dtype=torch.float32,
+            )
+        )
+
+
+
+
+        self.norm2 = CTWRMSNorm(
+            MODEL_DIM
+        )
+
+
+
+
+        self.ffn = CTWSwiGLU(
+            MODEL_DIM,
+            FFN_DIM,
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        b, l, s, c = x.shape
+
+
+
+
+        flat = x.reshape(
+            b,
+            l,
+            MODEL_DIM,
+        )
+
+
+
+
+        n = self.norm1(
+            flat
+        )
+
+
+
+
+        gate_in = torch.sigmoid(
+
+
+            self.input_gate(
+                n
+            )
+
+
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        gate_out = torch.sigmoid(
+
+
+            self.output_gate(
+                n
+            )
+
+
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        spectral = self.meyer(
+
+
+            self.scale_in(
+                x
+                *
+                gate_in
+            )
+        )
+
+
+
+
+        spectral = self.scale_out(
+            spectral
+        )
+
+
+
+
+        local = self.local(
+            n
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        x = (
+
+
+            x
+
+
+            +
+
+
+            self.alpha_meyer.to(
+                x.dtype
+            )
+            *
+            spectral
+            *
+            gate_out
+
+
+            +
+
+
+            self.alpha_local.to(
+                x.dtype
+            )
+            *
+            local
+        )
+
+
+
+
+        flat = x.reshape(
+            b,
+            l,
+            MODEL_DIM,
+        )
+
+
+
+
+        flat = (
+
+
+            flat
+
+
+            +
+
+
+            self.ffn(
+                self.norm2(
+                    flat
+                )
+            )
+        )
+
+
+
+
+        return flat.reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+# =============================================================================
+# EXACT EIGHT-BYTE AUTOREGRESSIVE DECODER
+# =============================================================================
+
+
+class CTWByteDecoder(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        shared_embedding,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.shared_embedding = (
+            shared_embedding
+        )
+
+
+
+
+        self.context_proj = nn.Linear(
+            MODEL_DIM,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        self.gru = nn.GRUCell(
+            BYTE_DIM,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        self.norm = CTWRMSNorm(
+            BYTE_DECODER_DIM
+        )
+
+
+
+
+        self.to_byte_space = nn.Linear(
+            BYTE_DECODER_DIM,
+            BYTE_DIM,
+        )
+
+
+
+
+        self.bos = nn.Parameter(
+            torch.zeros(
+                BYTE_DIM
+            )
+        )
+
+
+
+
+        self.byte_position = nn.Parameter(
+
+
+            torch.zeros(
+                BYTES_PER_BLOCK,
+                BYTE_DIM,
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.bos,
+            mean=0.0,
+            std=0.02,
+        )
+
+
+
+
+        nn.init.normal_(
+            self.byte_position,
+            mean=0.0,
+            std=0.01,
+        )
+
+
+
+
+    def nll(
+        self,
+        context,
+        target_blocks,
+    ):
+
+
+        b, l, _ = context.shape
+
+
+
+
+        n = (
+            b
+            *
+            l
+        )
+
+
+
+
+        hidden = torch.tanh(
+
+
+            self.context_proj(
+                context
+            )
+
+
+        ).reshape(
+            n,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        previous = (
+            self.bos[
+                None,
+                :
+            ]
+            .expand(
+                n,
+                -1,
+            )
+        )
+
+
+
+
+        total_loss = None
+
+
+
+
+        for r in range(
+            BYTES_PER_BLOCK
+        ):
+
+
+            decoder_input = (
+
+
+                previous
+
+
+                +
+
+
+                self.byte_position[
+                    r
+                ][
+                    None,
+                    :
+                ]
+            )
+
+
+
+
+            hidden = self.gru(
+                decoder_input,
+                hidden,
+            )
+
+
+
+
+            z = self.to_byte_space(
+                self.norm(
+                    hidden
+                )
+            )
+
+
+
+
+            target = (
+                target_blocks[
+                    :,
+                    :,
+                    r
+                ]
+                .reshape(
+                    -1
+                )
+                .long()
+            )
+
+
+
+
+            position_loss = None
+
+
+
+
+            for start in range(
+                0,
+                n,
+                HEAD_CHUNK,
+            ):
+
+
+                end = min(
+                    start
+                    +
+                    HEAD_CHUNK,
+                    n,
+                )
+
+
+
+
+                logits = F.linear(
+                    z[
+                        start:end
+                    ],
+                    self.shared_embedding.weight,
+                )
+
+
+
+
+                piece = F.cross_entropy(
+                    logits.float(),
+                    target[
+                        start:end
+                    ],
+                    reduction="sum",
+                )
+
+
+
+
+                position_loss = (
+                    piece
+                    if position_loss is None
+                    else
+                    position_loss
+                    +
+                    piece
+                )
+
+
+
+
+            total_loss = (
+                position_loss
+                if total_loss is None
+                else
+                total_loss
+                +
+                position_loss
+            )
+
+
+
+
+            # Exact teacher forcing:
+            # byte r+1 may condition on true byte r.
+
+
+            previous = self.shared_embedding(
+                target
+            )
+
+
+
+
+        return (
+            total_loss
+            /
+            float(
+                n
+                *
+                BYTES_PER_BLOCK
+            )
+        )
+
+
+
+
+# =============================================================================
+# COMPLETE CAUSAL MODEL
+# =============================================================================
+
+
+class CTWModel(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.tokenizer = (
+            CTWBlockLiftingTokenizer()
+        )
+
+
+
+
+        self.position = nn.Parameter(
+
+
+            torch.zeros(
+                1,
+                MODEL_CONTEXT,
+                MODEL_DIM,
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.position,
+            mean=0.0,
+            std=0.01,
+        )
+
+
+
+
+        self.layers = nn.ModuleList(
+
+
+            [
+
+
+                CTWLayer()
+
+
+                for _ in range(
+                    N_LAYERS
+                )
+            ]
+        )
+
+
+
+
+        self.final_norm = CTWRMSNorm(
+            MODEL_DIM
+        )
+
+
+
+
+        self.bos_context = nn.Parameter(
+            torch.zeros(
+                MODEL_DIM
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.bos_context,
+            mean=0.0,
+            std=0.02,
+        )
+
+
+
+
+        self.decoder = CTWByteDecoder(
+            self.tokenizer.byte_embedding
+        )
+
+
+
+
+    def encode_completed_blocks(
+        self,
+        raw_bytes,
+        checkpoint_layers,
+    ):
+
+
+        if (
+            raw_bytes.ndim
+            !=
+            2
+        ):
+
+
+            raise ValueError(
+                "Expected raw byte IDs [B,T]."
+            )
+
+
+
+
+        b, t = raw_bytes.shape
+
+
+
+
+        if (
+            t
+            %
+            BYTES_PER_BLOCK
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "Raw byte length must be divisible by 8."
+            )
+
+
+
+
+        l = (
+            t
+            //
+            BYTES_PER_BLOCK
+        )
+
+
+
+
+        self.alpha_meyer = nn.Parameter(
+
+
+            torch.tensor(
+                0.10,
+                dtype=torch.float32,
+            )
+        )
+
+
+
+
+        self.alpha_local = nn.Parameter(
+
+
+            torch.tensor(
+                0.10,
+                dtype=torch.float32,
+            )
+        )
+
+
+
+
+        self.norm2 = CTWRMSNorm(
+            MODEL_DIM
+        )
+
+
+
+
+        self.ffn = CTWSwiGLU(
+            MODEL_DIM,
+            FFN_DIM,
+        )
+
+
+
+
+    def forward(
+        self,
+        x,
+    ):
+
+
+        b, l, s, c = x.shape
+
+
+
+
+        flat = x.reshape(
+            b,
+            l,
+            MODEL_DIM,
+        )
+
+
+
+
+        n = self.norm1(
+            flat
+        )
+
+
+
+
+        gate_in = torch.sigmoid(
+
+
+            self.input_gate(
+                n
+            )
+
+
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        gate_out = torch.sigmoid(
+
+
+            self.output_gate(
+                n
+            )
+
+
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        spectral = self.meyer(
+
+
+            self.scale_in(
+                x
+                *
+                gate_in
+            )
+        )
+
+
+
+
+        spectral = self.scale_out(
+            spectral
+        )
+
+
+
+
+        local = self.local(
+            n
+        ).reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+        x = (
+
+
+            x
+
+
+            +
+
+
+            self.alpha_meyer.to(
+                x.dtype
+            )
+            *
+            spectral
+            *
+            gate_out
+
+
+            +
+
+
+            self.alpha_local.to(
+                x.dtype
+            )
+            *
+            local
+        )
+
+
+
+
+        flat = x.reshape(
+            b,
+            l,
+            MODEL_DIM,
+        )
+
+
+
+
+        flat = (
+
+
+            flat
+
+
+            +
+
+
+            self.ffn(
+                self.norm2(
+                    flat
+                )
+            )
+        )
+
+
+
+
+        return flat.reshape(
+            b,
+            l,
+            s,
+            c,
+        )
+
+
+
+
+# =============================================================================
+# EXACT EIGHT-BYTE AUTOREGRESSIVE DECODER
+# =============================================================================
+
+
+class CTWByteDecoder(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+        shared_embedding,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.shared_embedding = (
+            shared_embedding
+        )
+
+
+
+
+        self.context_proj = nn.Linear(
+            MODEL_DIM,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        self.gru = nn.GRUCell(
+            BYTE_DIM,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        self.norm = CTWRMSNorm(
+            BYTE_DECODER_DIM
+        )
+
+
+
+
+        self.to_byte_space = nn.Linear(
+            BYTE_DECODER_DIM,
+            BYTE_DIM,
+        )
+
+
+
+
+        self.bos = nn.Parameter(
+            torch.zeros(
+                BYTE_DIM
+            )
+        )
+
+
+
+
+        self.byte_position = nn.Parameter(
+
+
+            torch.zeros(
+                BYTES_PER_BLOCK,
+                BYTE_DIM,
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.bos,
+            mean=0.0,
+            std=0.02,
+        )
+
+
+
+
+        nn.init.normal_(
+            self.byte_position,
+            mean=0.0,
+            std=0.01,
+        )
+
+
+
+
+    def nll(
+        self,
+        context,
+        target_blocks,
+    ):
+
+
+        b, l, _ = context.shape
+
+
+
+
+        n = (
+            b
+            *
+            l
+        )
+
+
+
+
+        hidden = torch.tanh(
+
+
+            self.context_proj(
+                context
+            )
+
+
+        ).reshape(
+            n,
+            BYTE_DECODER_DIM,
+        )
+
+
+
+
+        previous = (
+            self.bos[
+                None,
+                :
+            ]
+            .expand(
+                n,
+                -1,
+            )
+        )
+
+
+
+
+        total_loss = None
+
+
+
+
+        for r in range(
+            BYTES_PER_BLOCK
+        ):
+
+
+            decoder_input = (
+
+
+                previous
+
+
+                +
+
+
+                self.byte_position[
+                    r
+                ][
+                    None,
+                    :
+                ]
+            )
+
+
+
+
+            hidden = self.gru(
+                decoder_input,
+                hidden,
+            )
+
+
+
+
+            z = self.to_byte_space(
+                self.norm(
+                    hidden
+                )
+            )
+
+
+
+
+            target = (
+                target_blocks[
+                    :,
+                    :,
+                    r
+                ]
+                .reshape(
+                    -1
+                )
+                .long()
+            )
+
+
+
+
+            position_loss = None
+
+
+
+
+            for start in range(
+                0,
+                n,
+                HEAD_CHUNK,
+            ):
+
+
+                end = min(
+                    start
+                    +
+                    HEAD_CHUNK,
+                    n,
+                )
+
+
+
+
+                logits = F.linear(
+                    z[
+                        start:end
+                    ],
+                    self.shared_embedding.weight,
+                )
+
+
+
+
+                piece = F.cross_entropy(
+                    logits.float(),
+                    target[
+                        start:end
+                    ],
+                    reduction="sum",
+                )
+
+
+
+
+                position_loss = (
+                    piece
+                    if position_loss is None
+                    else
+                    position_loss
+                    +
+                    piece
+                )
+
+
+
+
+            total_loss = (
+                position_loss
+                if total_loss is None
+                else
+                total_loss
+                +
+                position_loss
+            )
+
+
+
+
+            # Exact teacher forcing:
+            # byte r+1 may condition on true byte r.
+
+
+            previous = self.shared_embedding(
+                target
+            )
+
+
+
+
+        return (
+            total_loss
+            /
+            float(
+                n
+                *
+                BYTES_PER_BLOCK
+            )
+        )
+
+
+
+
+# =============================================================================
+# COMPLETE CAUSAL MODEL
+# =============================================================================
+
+
+class CTWModel(
+    nn.Module
+):
+
+
+    def __init__(
+        self,
+    ):
+
+
+        super().__init__()
+
+
+
+
+        self.tokenizer = (
+            CTWBlockLiftingTokenizer()
+        )
+
+
+
+
+        self.position = nn.Parameter(
+
+
+            torch.zeros(
+                1,
+                MODEL_CONTEXT,
+                MODEL_DIM,
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.position,
+            mean=0.0,
+            std=0.01,
+        )
+
+
+
+
+        self.layers = nn.ModuleList(
+
+
+            [
+
+
+                CTWLayer()
+
+
+                for _ in range(
+                    N_LAYERS
+                )
+            ]
+        )
+
+
+
+
+        self.final_norm = CTWRMSNorm(
+            MODEL_DIM
+        )
+
+
+
+
+        self.bos_context = nn.Parameter(
+            torch.zeros(
+                MODEL_DIM
+            )
+        )
+
+
+
+
+        nn.init.normal_(
+            self.bos_context,
+            mean=0.0,
+            std=0.02,
+        )
+
+
+
+
+        self.decoder = CTWByteDecoder(
+            self.tokenizer.byte_embedding
+        )
+
+
+
+
+    def encode_completed_blocks(
+        self,
+        raw_bytes,
+        checkpoint_layers,
+    ):
+
+
+        if (
+            raw_bytes.ndim
+            !=
+            2
+        ):
+
+
+            raise ValueError(
+                "Expected raw byte IDs [B,T]."
+            )
+
+
+
+
+        b, t = raw_bytes.shape
+
+
+
+
+        if (
+            t
+            %
+            BYTES_PER_BLOCK
+            !=
+            0
+        ):
+
+
+            raise ValueError(
+                "Raw byte length must be divisible by 8."
+            )
+
+
+
+
+        l = (
+            t
+            //
+            BYTES_PER_BLOCK
+        )
+
+
+
+
+        if (
+            l
+            >
+            MODEL_CONTEXT
+        ):
+
+
+            raise ValueError(
+
+
+                f"Context {l} exceeds "
+                f"MODEL_CONTEXT={MODEL_CONTEXT}."
+            )
+
+
+
+
+        x = self.tokenizer(
+            raw_bytes.long()
+        )
+
+
+
+
+        pos = (
+            self.position[
+                :,
+                :l,
+                :
+            ]
+            .reshape(
+                1,
+                l,
+                N_SCALES,
+                BYTE_DIM,
+            )
+        )
+
+
+
+
+        x = (
+            x
+            +
+            pos.to(
+                x.dtype
+            )
+        )
+
+
+
+
+        for layer in self.layers:
+
+
+            if (
+                checkpoint_layers
+                and
+                self.training
+            ):
+
+
+                x = torch_checkpoint(
+                    layer,
+                    x,
+                    use_reentrant=False,
+                    preserve_rng_state=False,
+                )
+
+
+            else:
+
+
+                x = layer(
+                    x
+                )
+
+
+
+
+        h = self.final_norm(
+
+
+            x.reshape(
+                b,
+                l,
+                MODEL_DIM,
+            )
+        )
+
+
+
+
+        blocks = (
+            raw_bytes
+            .long()
+            .reshape(
+                b,
+                l,
+                BYTES_PER_BLOCK,
+            )
+        )
+
+
+
+
+        return (
+            h,
+            blocks,
+        )
+
+
+
+
+    def contexts(
+        self,
+        raw_bytes,
+        checkpoint_layers=False,
+    ):
+
+
+        (
+            h,
+            blocks,
+        ) = self.encode_completed_blocks(
+            raw_bytes,
+            checkpoint_layers,
+        )
+
+
+
+
+        b, l, _ = h.shape
+
+
+
+
+        context = torch.empty_like(
+            h
+        )
+
+
+
+
+        # First block has no preceding block.
+
+
+        context[
+            :,
+            0,
+            :
+        ] = self.bos_context.to(
+            h.dtype
+        )
+
+
+
+
+        # -------------------------------------------------------------
+        # CRITICAL CAUSAL SHIFT
+        #
+        # prediction for block b receives h[b-1], never h[b].
+        # -------------------------------------------------------------
+
+
+        if l > 1:
+
+
+            context[
+                :,
+                1:,
+                :
+            ] = h[
+                :,
+                :-1,
+                :
+            ]
+
+
+
+
+        return (
+            context,
+            blocks,
+        )
+
+
+
+
+    def nll(
+        self,
+        raw_bytes,
+        checkpoint_layers=True,
+    ):
+
+
+        (
+            context,
+            blocks,
+        ) = self.contexts(
+            raw_bytes,
+            checkpoint_layers,
+        )
+
+
+
+
+        return self.decoder.nll(
+            context,
+            blocks,
+        )

vocab.json

@@ -0,0 +1,258 @@
+{
+  "<0x00>": 0,
+  "<0x01>": 1,
+  "<0x02>": 2,
+  "<0x03>": 3,
+  "<0x04>": 4,
+  "<0x05>": 5,
+  "<0x06>": 6,
+  "<0x07>": 7,
+  "<0x08>": 8,
+  "<0x09>": 9,
+  "<0x0A>": 10,
+  "<0x0B>": 11,
+  "<0x0C>": 12,
+  "<0x0D>": 13,
+  "<0x0E>": 14,
+  "<0x0F>": 15,
+  "<0x10>": 16,
+  "<0x11>": 17,
+  "<0x12>": 18,
+  "<0x13>": 19,
+  "<0x14>": 20,
+  "<0x15>": 21,
+  "<0x16>": 22,
+  "<0x17>": 23,
+  "<0x18>": 24,
+  "<0x19>": 25,
+  "<0x1A>": 26,
+  "<0x1B>": 27,
+  "<0x1C>": 28,
+  "<0x1D>": 29,
+  "<0x1E>": 30,
+  "<0x1F>": 31,
+  "<0x20>": 32,
+  "<0x21>": 33,
+  "<0x22>": 34,
+  "<0x23>": 35,
+  "<0x24>": 36,
+  "<0x25>": 37,
+  "<0x26>": 38,
+  "<0x27>": 39,
+  "<0x28>": 40,
+  "<0x29>": 41,
+  "<0x2A>": 42,
+  "<0x2B>": 43,
+  "<0x2C>": 44,
+  "<0x2D>": 45,
+  "<0x2E>": 46,
+  "<0x2F>": 47,
+  "<0x30>": 48,
+  "<0x31>": 49,
+  "<0x32>": 50,
+  "<0x33>": 51,
+  "<0x34>": 52,
+  "<0x35>": 53,
+  "<0x36>": 54,
+  "<0x37>": 55,
+  "<0x38>": 56,
+  "<0x39>": 57,
+  "<0x3A>": 58,
+  "<0x3B>": 59,
+  "<0x3C>": 60,
+  "<0x3D>": 61,
+  "<0x3E>": 62,
+  "<0x3F>": 63,
+  "<0x40>": 64,
+  "<0x41>": 65,
+  "<0x42>": 66,
+  "<0x43>": 67,
+  "<0x44>": 68,
+  "<0x45>": 69,
+  "<0x46>": 70,
+  "<0x47>": 71,
+  "<0x48>": 72,
+  "<0x49>": 73,
+  "<0x4A>": 74,
+  "<0x4B>": 75,
+  "<0x4C>": 76,
+  "<0x4D>": 77,
+  "<0x4E>": 78,
+  "<0x4F>": 79,
+  "<0x50>": 80,
+  "<0x51>": 81,
+  "<0x52>": 82,
+  "<0x53>": 83,
+  "<0x54>": 84,
+  "<0x55>": 85,
+  "<0x56>": 86,
+  "<0x57>": 87,
+  "<0x58>": 88,
+  "<0x59>": 89,
+  "<0x5A>": 90,
+  "<0x5B>": 91,
+  "<0x5C>": 92,
+  "<0x5D>": 93,
+  "<0x5E>": 94,
+  "<0x5F>": 95,
+  "<0x60>": 96,
+  "<0x61>": 97,
+  "<0x62>": 98,
+  "<0x63>": 99,
+  "<0x64>": 100,
+  "<0x65>": 101,
+  "<0x66>": 102,
+  "<0x67>": 103,
+  "<0x68>": 104,
+  "<0x69>": 105,
+  "<0x6A>": 106,
+  "<0x6B>": 107,
+  "<0x6C>": 108,
+  "<0x6D>": 109,
+  "<0x6E>": 110,
+  "<0x6F>": 111,
+  "<0x70>": 112,
+  "<0x71>": 113,
+  "<0x72>": 114,
+  "<0x73>": 115,
+  "<0x74>": 116,
+  "<0x75>": 117,
+  "<0x76>": 118,
+  "<0x77>": 119,
+  "<0x78>": 120,
+  "<0x79>": 121,
+  "<0x7A>": 122,
+  "<0x7B>": 123,
+  "<0x7C>": 124,
+  "<0x7D>": 125,
+  "<0x7E>": 126,
+  "<0x7F>": 127,
+  "<0x80>": 128,
+  "<0x81>": 129,
+  "<0x82>": 130,
+  "<0x83>": 131,
+  "<0x84>": 132,
+  "<0x85>": 133,
+  "<0x86>": 134,
+  "<0x87>": 135,
+  "<0x88>": 136,
+  "<0x89>": 137,
+  "<0x8A>": 138,
+  "<0x8B>": 139,
+  "<0x8C>": 140,
+  "<0x8D>": 141,
+  "<0x8E>": 142,
+  "<0x8F>": 143,
+  "<0x90>": 144,
+  "<0x91>": 145,
+  "<0x92>": 146,
+  "<0x93>": 147,
+  "<0x94>": 148,
+  "<0x95>": 149,
+  "<0x96>": 150,
+  "<0x97>": 151,
+  "<0x98>": 152,
+  "<0x99>": 153,
+  "<0x9A>": 154,
+  "<0x9B>": 155,
+  "<0x9C>": 156,
+  "<0x9D>": 157,
+  "<0x9E>": 158,
+  "<0x9F>": 159,
+  "<0xA0>": 160,
+  "<0xA1>": 161,
+  "<0xA2>": 162,
+  "<0xA3>": 163,
+  "<0xA4>": 164,
+  "<0xA5>": 165,
+  "<0xA6>": 166,
+  "<0xA7>": 167,
+  "<0xA8>": 168,
+  "<0xA9>": 169,
+  "<0xAA>": 170,
+  "<0xAB>": 171,
+  "<0xAC>": 172,
+  "<0xAD>": 173,
+  "<0xAE>": 174,
+  "<0xAF>": 175,
+  "<0xB0>": 176,
+  "<0xB1>": 177,
+  "<0xB2>": 178,
+  "<0xB3>": 179,
+  "<0xB4>": 180,
+  "<0xB5>": 181,
+  "<0xB6>": 182,
+  "<0xB7>": 183,
+  "<0xB8>": 184,
+  "<0xB9>": 185,
+  "<0xBA>": 186,
+  "<0xBB>": 187,
+  "<0xBC>": 188,
+  "<0xBD>": 189,
+  "<0xBE>": 190,
+  "<0xBF>": 191,
+  "<0xC0>": 192,
+  "<0xC1>": 193,
+  "<0xC2>": 194,
+  "<0xC3>": 195,
+  "<0xC4>": 196,
+  "<0xC5>": 197,
+  "<0xC6>": 198,
+  "<0xC7>": 199,
+  "<0xC8>": 200,
+  "<0xC9>": 201,
+  "<0xCA>": 202,
+  "<0xCB>": 203,
+  "<0xCC>": 204,
+  "<0xCD>": 205,
+  "<0xCE>": 206,
+  "<0xCF>": 207,
+  "<0xD0>": 208,
+  "<0xD1>": 209,
+  "<0xD2>": 210,
+  "<0xD3>": 211,
+  "<0xD4>": 212,
+  "<0xD5>": 213,
+  "<0xD6>": 214,
+  "<0xD7>": 215,
+  "<0xD8>": 216,
+  "<0xD9>": 217,
+  "<0xDA>": 218,
+  "<0xDB>": 219,
+  "<0xDC>": 220,
+  "<0xDD>": 221,
+  "<0xDE>": 222,
+  "<0xDF>": 223,
+  "<0xE0>": 224,
+  "<0xE1>": 225,
+  "<0xE2>": 226,
+  "<0xE3>": 227,
+  "<0xE4>": 228,
+  "<0xE5>": 229,
+  "<0xE6>": 230,
+  "<0xE7>": 231,
+  "<0xE8>": 232,
+  "<0xE9>": 233,
+  "<0xEA>": 234,
+  "<0xEB>": 235,
+  "<0xEC>": 236,
+  "<0xED>": 237,
+  "<0xEE>": 238,
+  "<0xEF>": 239,
+  "<0xF0>": 240,
+  "<0xF1>": 241,
+  "<0xF2>": 242,
+  "<0xF3>": 243,
+  "<0xF4>": 244,
+  "<0xF5>": 245,
+  "<0xF6>": 246,
+  "<0xF7>": 247,
+  "<0xF8>": 248,
+  "<0xF9>": 249,
+  "<0xFA>": 250,
+  "<0xFB>": 251,
+  "<0xFC>": 252,
+  "<0xFD>": 253,
+  "<0xFE>": 254,
+  "<0xFF>": 255
+}