E6E831728/affine-recoded-minimal-code-table-free

Affine-Recoded Minimal Code Table-Free Model

This is an anonymized research checkpoint for the paper:

Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

Model variant

This repository contains the fully table-free affine-recoded minimal binary-code model.

The model does not use an input embedding table. Instead, token codes are computed directly from token IDs.

For each token ID t, the model computes:

c(t) = bin_16(t)

and then applies a fixed invertible affine recoding over GF(2):

c_tilde(t) = A c(t) xor b

where:

• A is an invertible binary matrix in GL(16, 2)
• b is a fixed binary shift vector

The resulting 16-dimensional binary code is tiled to model width 1024.

The model uses:

0 trainable input-embedding parameters
0 input embedding table

The output projection remains standard and trainable.

Architecture

• decoder-only Transformer
• vocabulary size: 65,536
• model width: 1024
• number of layers: 32
• number of attention heads: 32
• context length: 1024
• rotary positional embeddings
• GELU activations
• untied trainable output projection

Loading example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "E6E831728/affine-recoded-minimal-code-table-free"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of UK?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Standardized base-model evaluation

The checkpoint was evaluated as a base causal language model with EleutherAI LM Evaluation Harness v0.4.10.

Evaluation protocol:

• Hugging Face backend: hf
• maximum context length: 1,024
• add_bos_token=False
• no chat template
• deterministic likelihood-based evaluation
• harness seeds: 0,1234,1234,1234
• base checkpoints only; no SFT or instruction checkpoints

Metric	Learned input table	Fixed Binary-16	Affine GF(2), table-free	SmolLM2-135M	SmolLM2-360M
HellaSwag acc	28.49 ± 0.45	29.04 ± 0.45	29.04 ± 0.45	35.36 ± 0.48	43.05 ± 0.49
HellaSwag acc_norm	31.32 ± 0.46	32.32 ± 0.47	31.80 ± 0.46	43.02 ± 0.49	56.28 ± 0.50
ARC-Easy acc	46.38 ± 1.02	47.90 ± 1.03	47.64 ± 1.02	64.44 ± 0.98	70.24 ± 0.94
ARC-Easy acc_norm	40.70 ± 1.01	40.87 ± 1.01	41.20 ± 1.01	58.75 ± 1.01	68.18 ± 0.96
ARC-Challenge acc	20.39 ± 1.18	19.62 ± 1.16	21.33 ± 1.20	28.07 ± 1.31	36.26 ± 1.40
ARC-Challenge acc_norm	25.85 ± 1.28	26.19 ± 1.28	24.83 ± 1.26	29.61 ± 1.33	38.05 ± 1.42
PIQA acc	62.35 ± 1.13	62.57 ± 1.13	62.68 ± 1.13	68.44 ± 1.08	71.38 ± 1.05
PIQA acc_norm	60.61 ± 1.14	62.08 ± 1.13	60.94 ± 1.14	68.39 ± 1.08	71.82 ± 1.05
WinoGrande acc	50.20 ± 1.41	50.12 ± 1.41	50.43 ± 1.41	52.57 ± 1.40	59.35 ± 1.38
OpenBookQA acc	18.40 ± 1.73	17.20 ± 1.69	17.60 ± 1.70	22.00 ± 1.85	24.80 ± 1.93
OpenBookQA acc_norm	29.20 ± 2.04	31.00 ± 2.07	29.40 ± 2.04	32.60 ± 2.10	37.80 ± 2.17
CommonsenseQA acc	20.31 ± 1.15	19.90 ± 1.14	20.23 ± 1.15	19.90 ± 1.14	21.05 ± 1.17
MMLU 0-shot	24.13 ± 0.36	23.86 ± 0.36	24.11 ± 0.36	24.24 ± 0.36	25.47 ± 0.37
MMLU 5-shot	25.68 ± 0.37	25.60 ± 0.37	25.66 ± 0.37	25.39 ± 0.37	25.05 ± 0.37
LAMBADA accuracy	22.38 ± 0.58	21.23 ± 0.57	21.99 ± 0.58	42.97 ± 0.69	53.31 ± 0.70
LAMBADA perplexity	95.14 ± 4.01	101.74 ± 4.27	100.61 ± 4.17	19.06 ± 0.63	9.38 ± 0.27
WikiText word perplexity	81.04	74.87	76.17	25.53	18.84
WikiText byte perplexity	2.27	2.24	2.25	1.83	1.73
WikiText bits/byte	1.19	1.16	1.17	0.87	0.79

The three paper checkpoints form the controlled architectural comparison. SmolLM2-135M and SmolLM2-360M are external reference models, not matched baselines: they use different architectures, tokenizers, training mixtures, and much larger pretraining budgets. SmolLM2-135M was trained on approximately 2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper checkpoints saw approximately 16–17B tokens. Their scores therefore provide context for absolute capability and must not be interpreted as isolating the effect of the input parameterization.

Perplexity values should be interpreted especially cautiously across different tokenizers. The primary controlled comparison is among the three paper models, which share the same tokenizer, data pipeline, and architecture.

Input-interface audit

This checkpoint has no input embedding table. Token codes are generated algorithmically from token IDs, and the fixed affine matrix and shift are registered as non-trainable buffers.

import torch
from transformers import AutoModelForCausalLM

repo_id = (
    "E6E831728/"
    "affine-recoded-minimal-code-table-free"
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).cpu().eval()

print("get_input_embeddings():", model.get_input_embeddings())
print(
    "input-code parameters:",
    [
        name
        for name, _ in model.named_parameters()
        if name.startswith("input_code.")
    ],
)
print(
    "input-code buffers:",
    [
        name
        for name, _ in model.named_buffers()
        if name.startswith("input_code.")
    ],
)

ids = torch.arange(model.config.vocab_size).unsqueeze(0)

with torch.no_grad():
    codes = model.input_code.encode_bits(ids)[0]

weights = 1 << torch.arange(model.config.code_bits)
packed = (codes.long() * weights).sum(dim=-1)

print("code shape:", tuple(codes.shape))
print("unique values:", torch.unique(codes).tolist())
print("unique codes:", torch.unique(packed).numel())
print("collisions:", model.config.vocab_size - torch.unique(packed).numel())

assert model.get_input_embeddings() is None
assert not hasattr(model, "token_embeddings")
assert not any(
    name.startswith("input_code.")
    for name, _ in model.named_parameters()
)
assert torch.all((codes == 0) | (codes == 1))
assert torch.unique(packed).numel() == model.config.vocab_size

Expected audit properties:

get_input_embeddings(): None
input-code parameters: []
input-code buffers: ['input_code.bit_positions', 'input_code.A_gf2', 'input_code.b_gf2']
code shape: (65536, 16)
unique values: [0.0, 1.0]
unique codes: 65536
collisions: 0

Intended use

This checkpoint is provided for anonymous review and reproducibility. It demonstrates that the fixed minimal-code input interface remains viable even when the canonical token-ID binary code is randomly recoded by an invertible affine transform.

Limitations

This model is a research checkpoint. It is not intended for deployment. It may produce incorrect, biased, unsafe, or nonsensical outputs.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.