README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
tags:
  - causal-lm
  - base-model
  - custom-code
  - safetensors
  - research
  - fixed-token-codes
  - frozen-input-representations
---

# AB-EXT Binary16 — 1.711B parameters, 100B-token target

A **base pretrained decoder-only causal language model** released for
research on trainable input embedding tables and fixed token identities.

It is not an instruction-tuned or preference-optimized assistant.

## Research question

Can a shared contextual network learn useful language-modeling behavior
without independently trainable token-specific input vectors?

The controlled family contains a learned-input model, a canonical
16-bit-code model, and an invertibly recoded GF2 model. They share the
contextual backbone and output-head architecture, but not the same total
trainable parameter count.

**The results support viability, not performance equivalence.**
The fixed-code models retain substantial capability while the learned
model performs better on several informative evaluations.

## Model specification

| Property | Value |
|---|---|
| Input mode | `binary16` |
| Trainable parameters | 1,711,376,384 |
| Trainable input parameters | 0 |
| Trainable body parameters, excluding input/output | 1,610,713,088 |
| Trainable untied output-head parameters | 100,663,296 |
| Persistent input-buffer values | 786,432 |
| Hidden width | 2048 |
| Decoder blocks | 24 |
| Attention heads | 32 |
| FFN intermediate width | 8192 |
| Training context | 2048 tokens |
| Position encoding | RoPE |
| Normalization / activation | RMSNorm / SwiGLU |
| Tokenizer source | `HuggingFaceTB/SmolLM2-1.7B` |
| Exported tokenizer revision | effd688a12921b4cc83e3312b6feb579f70f9c71 |
| Evaluated training runs for this interface | One |

The stored tensor-value count includes buffers and must not be reported
as the trainable parameter count.

## Input representation


Each token ID is represented by its canonical little-endian binary code:

$$
c(t)_j =
\left\lfloor \frac{t}{2^j} \right\rfloor \bmod 2,
\qquad j=0,\ldots,15.
$$

Since the vocabulary contains 49,152 entries, an injective fixed-length
binary code requires 16 bits.

The code is repeated 128 times to width 2048:

$$
x(t)=
\underbrace{c(t)\Vert\cdots\Vert c(t)}_{128\text{ copies}}.
$$

There are **zero trainable input-interface parameters** and no additional
trainable input projection before the standard backbone.
The backbone and the untied output vocabulary projection remain trainable.

The evaluated implementation stores the 49,152-by-16 codebook as a
persistent non-trainable buffer. It is not literally lookup-free.
“Minimal” describes fixed-length binary identity width, not the storage
of the complete model or an entropy-optimal token code.



## Training

- **Training tokens:** Approximately 100B according to the run report; target budget 100,000,000,000 prediction targets. 
- **Training precision:** FP32 parameters with BF16 autocast in the supplied trainer.
- **Reported recipe:** AdamW; peak learning rate 0.00015; minimum
  scheduled learning rate 0.00001; 2000 warmup steps; cosine decay;
  weight decay 0.01; betas 0.9 and 0.95; gradient clipping 1.0.
- **Reported launch geometry:** two GPUs per run, microbatch eight per
  GPU, eight accumulation steps, sequence length 2048.

The launch geometry corresponds to 262,144 prediction targets per
optimizer step. Exact final counts must come from the checkpoint,
not from the requested budget.

The supplied sampler selects within-document windows from eligible
documents of at least 2049 tokens. Sampling can repeat or overlap
windows; 100B processed targets does not imply 100B unique corpus tokens.
The original trainer does not fully restore per-rank sampling state
on resume. A shared recipe alone does not establish identical realized
sample order across interrupted runs.

The model weights were NOT initialized from SmolLM2.
SmolLM2 supplies tokenizer artifacts, not pretrained model weights.

## Evaluation results

These scores are transcribed from the supplied completed evaluation
summary; the generator does not rerun benchmarks. Raw, unrounded
harness outputs remain authoritative.

Accuracy entries are percentages. Their reported `±` values are
evaluation standard errors, **not variation across training seeds**.
Perplexities and bits per byte are not percentages.

| Metric | Shots | Result |
|---|---:|---:|
| HellaSwag acc (%) | 0 | 40.70 ± 0.49 |
| HellaSwag acc_norm (%) | 0 | 52.40 ± 0.50 |
| ARC-Easy acc (%) | 0 | 67.59 ± 0.96 |
| ARC-Easy acc_norm (%) | 0 | 61.53 ± 1.00 |
| ARC-Challenge acc (%) | 0 | 32.34 ± 1.37 |
| ARC-Challenge acc_norm (%) | 0 | 34.04 ± 1.38 |
| PIQA acc (%) | 0 | 70.51 ± 1.06 |
| PIQA acc_norm (%) | 0 | 71.11 ± 1.06 |
| WinoGrande acc (%) | 0 | 55.33 ± 1.40 |
| OpenBookQA acc (%) | 0 | 28.20 ± 2.01 |
| OpenBookQA acc_norm (%) | 0 | 38.00 ± 2.17 |
| CommonsenseQA acc (%) | 0 | 20.56 ± 1.16 |
| MMLU acc (%) | 0 | 25.88 ± 0.37 |
| MMLU acc (%; some prompts truncated) | 5 | 25.57 ± 0.37 |
| LAMBADA accuracy (%) | 0 | 42.75 ± 0.69 |
| LAMBADA perplexity ↓ | 0 | 17.91 ± 0.62 |
| WikiText word perplexity ↓ | — | 18.58 |
| WikiText byte perplexity ↓ | — | 1.73 |
| WikiText bits/byte ↓ | — | 0.79 |

### Audit and coverage limitations

The supplied audit reports identical sample/prompt multisets across
all six models in each completed task group.

For MMLU 5-shot, **1,508 / 56,168 candidate log-likelihood requests**
were marked as truncated for each model, approximately 2.68%.
These are candidate requests, not necessarily distinct questions.
The displayed MMLU 5-shot score therefore includes truncated prompts.

No truncations were reported for the other groups by that audit.
For WikiText rolling likelihood, this does not mean that whole documents
fit into one model context: rolling windowing is part of scoring.

<details>
<summary>Full six-model comparison</summary>

| Metric | AB-EXT Learned | AB-EXT Binary16 | AB-EXT GF2 | SmolLM2-135M | SmolLM2-360M | SmolLM2-1.7B |
|---|---:|---:|---:|---:|---:|---:|
| HellaSwag acc (%); shots=0 | 44.21 ± 0.50 | 40.70 ± 0.49 | 40.24 ± 0.49 | 35.36 ± 0.48 | 43.05 ± 0.49 | 53.38 ± 0.50 |
| HellaSwag acc_norm (%); shots=0 | 57.79 ± 0.49 | 52.40 ± 0.50 | 51.44 ± 0.50 | 43.02 ± 0.49 | 56.28 ± 0.50 | 71.43 ± 0.45 |
| ARC-Easy acc (%); shots=0 | 71.63 ± 0.92 | 67.59 ± 0.96 | 66.84 ± 0.97 | 64.44 ± 0.98 | 70.24 ± 0.94 | 77.86 ± 0.85 |
| ARC-Easy acc_norm (%); shots=0 | 66.04 ± 0.97 | 61.53 ± 1.00 | 60.73 ± 1.00 | 58.75 ± 1.01 | 68.18 ± 0.96 | 73.36 ± 0.91 |
| ARC-Challenge acc (%); shots=0 | 35.92 ± 1.40 | 32.34 ± 1.37 | 30.55 ± 1.35 | 28.07 ± 1.31 | 36.26 ± 1.40 | 44.37 ± 1.45 |
| ARC-Challenge acc_norm (%); shots=0 | 37.63 ± 1.42 | 34.04 ± 1.38 | 34.22 ± 1.39 | 29.61 ± 1.33 | 38.05 ± 1.42 | 47.27 ± 1.46 |
| PIQA acc (%); shots=0 | 72.69 ± 1.04 | 70.51 ± 1.06 | 71.16 ± 1.06 | 68.44 ± 1.08 | 71.38 ± 1.05 | 76.99 ± 0.98 |
| PIQA acc_norm (%); shots=0 | 72.14 ± 1.05 | 71.11 ± 1.06 | 72.14 ± 1.05 | 68.39 ± 1.08 | 71.82 ± 1.05 | 77.20 ± 0.98 |
| WinoGrande acc (%); shots=0 | 58.56 ± 1.38 | 55.33 ± 1.40 | 55.01 ± 1.40 | 52.57 ± 1.40 | 59.35 ± 1.38 | 65.98 ± 1.33 |
| OpenBookQA acc (%); shots=0 | 27.60 ± 2.00 | 28.20 ± 2.01 | 25.20 ± 1.94 | 22.00 ± 1.85 | 24.80 ± 1.93 | 32.20 ± 2.09 |
| OpenBookQA acc_norm (%); shots=0 | 37.80 ± 2.17 | 38.00 ± 2.17 | 36.80 ± 2.16 | 32.60 ± 2.10 | 37.80 ± 2.17 | 44.40 ± 2.22 |
| CommonsenseQA acc (%); shots=0 | 19.82 ± 1.14 | 20.56 ± 1.16 | 19.74 ± 1.14 | 19.90 ± 1.14 | 21.05 ± 1.17 | 41.69 ± 1.41 |
| MMLU acc (%); shots=0 | 25.32 ± 0.37 | 25.88 ± 0.37 | 26.11 ± 0.37 | 24.25 ± 0.36 | 25.47 ± 0.37 | 48.40 ± 0.41 |
| MMLU acc (%; some prompts truncated); shots=5 | 25.48 ± 0.37 | 25.57 ± 0.37 | 24.66 ± 0.36 | 25.15 ± 0.36 | 25.03 ± 0.37 | 50.06 ± 0.41 |
| LAMBADA accuracy (%); shots=0 | 47.72 ± 0.70 | 42.75 ± 0.69 | 42.29 ± 0.69 | 42.97 ± 0.69 | 53.31 ± 0.70 | 67.51 ± 0.65 |
| LAMBADA perplexity ↓; shots=0 | 12.88 ± 0.42 | 17.91 ± 0.62 | 18.47 ± 0.63 | 19.06 ± 0.63 | 9.38 ± 0.27 | 4.44 ± 0.10 |
| WikiText word perplexity ↓; shots=— | 16.50 | 18.58 | 19.03 | 23.14 | 17.12 | 11.62 |
| WikiText byte perplexity ↓; shots=— | 1.69 | 1.73 | 1.73 | 1.80 | 1.70 | 1.58 |
| WikiText bits/byte ↓; shots=— | 0.76 | 0.79 | 0.79 | 0.85 | 0.77 | 0.66 |

</details>

### How to interpret SmolLM2 comparisons

All scores above are from the supplied local evaluation summary, not
copied leaderboard scores.

The SmolLM2 technical report gives approximate training budgets of:

| External reference | Published budget | Relative to 100B |
|---|---:|---:|
| SmolLM2-135M | 2T tokens | 20× |
| SmolLM2-360M | 4T tokens | 40× |
| SmolLM2-1.7B | 11T tokens | 110× |

Source: https://arxiv.org/abs/2502.02737

These models differ in architecture, size, data, training schedule,
and compute. They are quality references, **not matched controls** and
not proof of a sample-efficiency advantage.

## Usage

Review the custom Python files before enabling `trust_remote_code=True`.
Use a tested Transformers version and pin the Hub revision for
reproducible deployment.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'E6E831728/ab_ext_binary16'
# For published Hub use, pin revision to a reviewed commit.
revision = None

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    revision=revision,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer(
    "Gravity is",
    return_tensors="pt",
    add_special_tokens=False,
    return_attention_mask=True,
).to("cuda")

pad_id = tokenizer.pad_token_id
if pad_id is None:
    pad_id = tokenizer.eos_token_id

with torch.inference_mode():
    output = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=32,
        do_sample=False,
        use_cache=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=pad_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The implementation does not provide a KV cache.
The trained context is 2048 tokens; the supplied generation adapter
uses a sliding window when the context grows beyond its limit.
This is not evidence of trained long-context capability.

### Loss API

The original training model consumes already-shifted targets.
The HF runtime is intended to expose the usual causal-LM convention
with an internal label shift. Do not pass already-shifted labels to
such a runtime.

Before fine-tuning, verify the actual runtime's loss implementation.
Forward-logit equivalence does not by itself test label conventions.

## Verification and integrity

The supplied verification logs report:

- successful BF16 loading and generation for all three releases;
- exactly matching original/exported forward logits on four short
  prompts for each model, in the tested verification configuration.

These are smoke and implementation-parity checks, not an exhaustive
test across padding, context lengths, dtypes, or generation modes.

- Weight file: `model.safetensors`
- Weight SHA-256: `c3e78f24f03dff36b6174985dc5189e8ce37a52685d022ade36fa68c3239801b`
- Stored tensor values: 1,712,162,816
- Stored values by dtype: `{"F32": 1712162816}`
- Trainable parameter count: 1,711,376,384
- Persistent input-buffer values: 786,432

This card update does not modify the weights, tokenizer, model code,
or configuration.

## Limitations and intended use

- Research use and text completion; not a validated high-stakes assistant.
- One evaluated training run per input interface at this scale.
- Fixed-code and learned-input models are backbone-matched, not
  total-parameter-matched.
- No measured runtime or energy advantage is established by parameter
  counts alone.
- One GF2 recoding does not establish invariance to arbitrary codes.
- The output vocabulary matrix remains trainable and token-specific.
- Input-code structure is not fitted to the pretraining objective, but
  the tokenizer and its ID assignment can contain corpus-derived structure.
- Benchmark contamination has not been independently certified absent.
- Generated text can be false, biased, or harmful.

- Zero/one coding maps token ID zero to a zero input vector; the
  zero-offset GF2 transform preserves it. In the supplied bias-free
  architecture, a context made entirely of zero-code tokens gives
  uniform logits. This does not apply to arbitrary contexts ending
  in that token.


## Attribution and licensing

Tokenizer artifacts are sourced from `HuggingFaceTB/SmolLM2-1.7B`.