--- license: mit library_name: transformers tags: - emulation - custom_code --- # elephantine: one architecture, many models This repo holds **one** transformers architecture, `ElephantineForCausalLM` (remote code: `configuration_elephantine.py`, `modeling_elephantine.py`), and several **settings** of it. Each subfolder holds a `config.json` and a `model.safetensors`. Every subfolder uses the same config class and the same model class. Only the sizes, the weights and the branch coefficients differ. Each setting reproduces a different public model through that model's own forward pass. This is a small demo of the idea in [The Emulation Frontier](https://alexlitzenberger.com/blog/the_emulation_frontier) (code: [alexlitz/the_emulation_frontier](https://github.com/alexlitz/the_emulation_frontier)). It replaces an earlier August 2026 demo, a stock JambaForCausalLM reproducing one tiny NoPE-Llama. ## What native emulation is Model A *natively emulates* model B when some setting of A's weights makes A's forward pass compute B's, token for token, with no fine-tuning and no approximation beyond floating-point rounding. The "elephantine" is the architecture that emulates everything in a zoo of public decoder LMs. It takes the union of what the models do, and each operation a family may or may not use is a **branch with a coefficient**, off by default. Every layer is x += Attn(norm_a(x)) softmax attention: RoPE (settable frequencies), causal OR full mask, ... x += SSM(norm_s(x)) short conv -> general gated-delta recurrence -> norm / gate -> out x += FFN(norm_f(x)) routed MoE + a gated shared expert; SiLU / GeLU / ReLU / ... branches and the SSM recurrence is the general gated-delta rule `S_t = Diag(a_t) S_(t-1) - gamma_t (S_(t-1) k_t) k_t^T + w_t v_t k_t^T, y_t = S_t q_t`, whose knobs cover Mamba-1/2/3, gated DeltaNet, RWKV-4..7, RG-LRU, LFM2's gated conv, and more. A model is loaded by zeroing the branches it doesn't use and copying its weights into the ones it does (`elephantine.universal.from_*` in the code repo). `modeling_elephantine.py` is a verbatim copy of that architecture (`elephantine/universal.py`, pure torch; no fla / mamba_ssm needed). It computes in float64. ## Settings in this repo | subfolder | original | max \|Δ logits\| vs original, fp32 | ... fp64 (casts lifted) | greedy continuations, 3 prompts | size | |---|---|---|---|---|---| | `smollm2-135m` | HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 0 | 3 / 3 identical | 0.35 GB | | `smollm2-135m-inflated` | HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 2.1e-13 | 3 / 3 identical | 0.58 GB | | `pythia-160m` | EleutherAI/pythia-160m | 6.1e-04 | 2.8e-12 | 3 / 3 identical | 0.32 GB | | `falcon-h1-tiny-90m` | tiiuae/Falcon-H1-Tiny-90M-Base | 4.9e-05 | 1.5e-13 | 3 / 3 identical | 1.41 GB | | `mamba-130m-mic` | huggingface-mic/mamba-130m-hf-mic | 1.6e-06 | 2.2e-15 | 3 / 3 identical | 0.06 GB | | `qwen3-0.6b-diffusion-mdlm` | dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 | 7.0e-05 | 2.0e-13 | 3 / 3 identical | 1.62 GB | "fp32" compares against the original model run by its own transformers class in float32 (its normal precision); the gap is the original's own float32 rounding. "fp64" runs the same original code in float64 with its internal float32 casts lifted (`_lift_fp32` in the code repo's tests). The remaining gap is float64 rounding, which is the precision of the construction itself. All numbers are max |Δ| over every position and vocabulary entry of 3 prompts, measured after reloading each subfolder through `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`. The reload is bit-identical to the host built in memory. **Storage.** Each tensor is stored in the narrowest dtype that holds it exactly. A bf16 checkpoint's copied weights are stored as bf16, Pythia's as fp16, and folded or derived values as fp32 or fp64. Tensors still at the architecture's default (the unused branches) are not stored, and the model class recreates them at load. The model widens everything to float64. Total upload: 4.35 GB. **No KV cache.** `generate()` works, but every step re-runs the whole prefix (a demo, not a server). Padding masks are not supported. Use unpadded, equal-length batches. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m") out = m.generate(tok("The capital of France is", return_tensors="pt").input_ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` Each subfolder's section below has its own snippet. `m.config.host` holds the host sizes; `m.config.branches` summarises the branch coefficients the setting switches on (the weights file is authoritative). ### `smollm2-135m` - **Source:** [`HuggingFaceTB/SmolLM2-135M`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) @ `93efa2f097`, loaded with `elephantine.universal.from_llama`. - **Family:** Llama: softmax attention, RoPE, GQA (tiled to MHA), RMSNorm, SwiGLU. - **Host:** d 576, L 30, 9 heads x 64, FFN 1536, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 218M host parameters, 176M stored (non-default), 352 MB on disk (bfloat16 176.1M, float32 0.0M, float64 0.0M). - **Mask:** causal. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 1.4e-04 (max \|logit\| 36.3) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 0 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m") ids = tok("The capital of France is", return_tensors="pt").input_ids out = m.generate(ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` Greedy continuations from this setting: `The capital of France is` → `' the capital of the country.\n\nThe capital of France is the capital of'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n elif n == '`; `Once upon a time, there was a little` → `' girl named Lily. She lived in a big house with her family, but she'` ### `smollm2-135m-inflated` - **Source:** [`HuggingFaceTB/SmolLM2-135M`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) @ `93efa2f097`, loaded with `elephantine.universal.from_llama`, then inflated (d 768, heads 12, shared_ff 2048, L 40). - **Family:** the same SmolLM2-135M inside a wider and deeper host (extra units zeroed). - **Host:** d 768, L 40, 12 heads x 64, FFN 2048, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 457M host parameters, 288M stored (non-default), 579 MB on disk (bfloat16 287.8M, float32 0.0M, float64 0.3M). - **Mask:** causal. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 1.4e-04 (max \|logit\| 36.3) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.1e-13 | | max \|Δ logits\| vs the un-inflated host | 2.1e-13 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated") ids = tok("The capital of France is", return_tensors="pt").input_ids out = m.generate(ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` Greedy continuations from this setting: `The capital of France is` → `' the capital of the country.\n\nThe capital of France is the capital of'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n elif n == '`; `Once upon a time, there was a little` → `' girl named Lily. She lived in a big house with her family, but she'` ### `pythia-160m` - **Source:** [`EleutherAI/pythia-160m`](https://huggingface.co/EleutherAI/pythia-160m) @ `50f5173d93`, loaded with `elephantine.universal.from_gpt_neox`. - **Family:** GPT-NeoX: LayerNorm, parallel attention + MLP block, partial RoPE (25% of each head), ungated exact (erf) GeLU MLP, biases. - **Host:** d 768, L 12, 12 heads x 64, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 50304; 220M host parameters, 162M stored (non-default), 325 MB on disk (bfloat16 0.0M, float16 162.3M, float32 0.0M, float64 0.0M). - **Mask:** causal. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 6.1e-04 (max \|logit\| 843.1) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.8e-12 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m") ids = tok("The capital of France is", return_tensors="pt").input_ids out = m.generate(ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` Greedy continuations from this setting: `The capital of France is` → `' located in the city of Paris.\n\nThe city is also home to the'`; `def fibonacci(n):` → `'\n """\n Returns the number of Fibonacci numbers.\n '`; `Once upon a time, there was a little` → `' girl who was a little girl. She was a little girl, and she was'` ### `falcon-h1-tiny-90m` - **Source:** [`tiiuae/Falcon-H1-Tiny-90M-Base`](https://huggingface.co/tiiuae/Falcon-H1-Tiny-90M-Base) @ `7994372e93`, loaded with `elephantine.universal.from_falcon_h1`. - **Family:** Falcon-H1 hybrid: a Mamba-2 SSM (through the gated-delta recurrence) and GQA attention in parallel in every layer, then SwiGLU; muP multipliers folded into the weights. - **Host:** d 512, L 24, 8 heads x 64, FFN 768, SSM in 1688 / 24 heads (dk 64, dv 32), vocab 32768; 1456M host parameters, 630M stored (non-default), 1412 MB on disk (bfloat16 582.0M, float32 33.6M, float64 14.2M). - **Mask:** causal. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 4.9e-05 (max \|logit\| 43.5) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 1.5e-13 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m") ids = tok("The capital of France is", return_tensors="pt").input_ids out = m.generate(ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` Greedy continuations from this setting: `The capital of France is` → `' the capital of France is the capital of France is the capital of France is the'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n '`; `Once upon a time, there was a little` → `' girl named Laura who was a little girl. She was a little girl'` ### `mamba-130m-mic` - **Source:** [`huggingface-mic/mamba-130m-hf-mic`](https://huggingface.co/huggingface-mic/mamba-130m-hf-mic) @ `ba38c62c9e`, loaded with `elephantine.universal.from_mamba`. - **Family:** Mamba-1 (selective SSM through the gated-delta recurrence): the code path of state-spaces/mamba-130m-hf. - **Host:** d 64, L 2, 1 heads x 2, FFN 1, SSM in 256 / 128 heads (dk 16, dv 1), vocab 50280; 26M host parameters, 13M stored (non-default), 60 MB on disk (bfloat16 0.3M, float32 10.7M, float64 2.1M). - **Mask:** causal. - **Note:** Random-weight test fixture (seed 17) published by huggingface-mic: the real MambaForCausalLM class and mamba-130m-hf's tokenizer, shrunk to d 64 and 2 layers. It is not a trained model; its continuations are not meaningful text. The trained mamba-130m-hf needs a ~33B-parameter host (the SSM branch's knob projections are dense: (2 x ssm_in) x (ssm_heads x dk) = 6144 x 24576 per projection per layer), ~264 GB in fp64, so it is not part of this demo. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 1.6e-06 (max \|logit\| 6.5) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.2e-15 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic") ids = tok("The capital of France is", return_tensors="pt").input_ids out = m.generate(ids, max_new_tokens=16, do_sample=False) print(tok.decode(out[0])) ``` ### `qwen3-0.6b-diffusion-mdlm` - **Source:** [`dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1`](https://huggingface.co/dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1) @ `c8d24a3f4a`, loaded with `elephantine.universal.from_llama`. - **Family:** masked-diffusion LM (MDLM, from Qwen3-0.6B): bidirectional attention, per-head qk-norm; runs under the full mask (causal=False). - **Host:** d 1024, L 28, 16 heads x 128, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 151936; 1049M host parameters, 810M stored (non-default), 1621 MB on disk (bfloat16 810.5M, float32 0.0M, float64 0.0M). - **Mask:** full (bidirectional): the config sets causal=false. | measured (3 prompts) | value | |---|---| | max \|Δ logits\| vs the original in fp32 | 7.0e-05 (max \|logit\| 25.4) | | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.0e-13 | | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | argmax agreement with the fp32 original (all positions) | 100.0% | | MDLM greedy fill of 8 masks, vs the fp32 original | 3 / 3 identical | | MDLM greedy fill of 8 masks, vs the fp64 original | 3 / 3 identical | ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm") ids = tok("The capital of France is", return_tensors="pt").input_ids masked = torch.cat([ids, torch.full((1, 8), tok.mask_token_id)], 1) # import torch logits = m(input_ids=masked).logits # full mask (config.causal = False) ``` ## Inflation: any size, the active subset `smollm2-135m-inflated` is SmolLM2-135M inside a host that is wider (d 576 → 768, 9 → 12 heads, FFN 1536 → 2048) and deeper (30 → 40 layers). The extra heads, FFN units and residual coordinates are zeroed. The 10 extra layers are left at the architecture's default, with all-zero output projections, so each one adds exactly 0. The wider residual stream needs one weight setting on each RMSNorm, `eps' = eps·d/d'` and `w' = w·sqrt(d/d')`. The output matches the un-inflated host to float64 rounding (table above). The same move inflates a target to any size. A bigger host runs a smaller model on the active subset of its units, and the rest stay zero. `push_hub_demo.inflate` in the code repo does this for heads, FFN width, depth, and (for RMSNorm targets) hidden width. ## How this relates to the ~26.5T elephantine The writeup's elephantine is this same architecture **sized to carry the whole zoo**. It takes the per-axis maximum over all 3,906 classified decoder LMs: hidden 16,384, 192 layers, kv/token 14,336, 128 query heads, ~2.76M FFN units per layer, ~26.5T parameters in total and ~4.52T active. **This demo is not that model.** Each setting here is the same architecture at the size its one target needs (one target per subfolder), plus one inflated example. The ~26.5T model is a size, not a checkpoint. Nothing here was trained. Coverage, from the code repo (`python -m graph.minimal_emulator`, `FRONTIER.md`): the one universal host carries **3,900 of 3,906** classified models. The 6 not carried are in mixer families with no tested branch (Cyrah, whose code is in a gated repo, and unclassified recurrences). Of the carried models, **3,531** have a whole-model `from_*` loader that accepts their model_type and config. For each of these a test runs the loaded host end to end against that type's own transformers code, or a labelled transcription of its remote code. The other 369 are carried by priced slot constructions only. Two things this demo found that the coverage counts don't show. A *trained* SSM is expensive in the universal host: the SSM branch's knob projections are dense, so the trained `state-spaces/mamba-130m-hf` needs a ~33B-parameter host (~264 GB in fp64) and `LiquidAI/LFM2-350M` ~13B. That is why the Mamba-1 setting here is a random-weight fixture of mamba-130m's code path, and the trained SSM setting is Falcon-H1-Tiny-90M (Mamba-2: a 1.46B-parameter host for a 91M-parameter model). RWKV-4 is carried block by block, but universal.py has no whole-model RWKV loader, so it is not a setting here. ## Rebuilding From the code repo: `python -m elephantine.push_hub_demo --out DIR [--push]` rebuilds every setting from the original checkpoints, measures it, and writes this card. `tests/test_hub_demo.py` checks the vendored copy against `elephantine/universal.py` (source and forward), rebuilds two settings, reloads them through `AutoModelForCausalLM` with `trust_remote_code`, and checks them against the originals. ## License MIT (the code and this card). The weights in each subfolder are re-encodings of the original checkpoints and keep those models' terms: SmolLM2-135M, Pythia-160m and the dLLM Qwen3-0.6B diffusion model are Apache-2.0; Falcon-H1-Tiny is under the Falcon-LLM license (https://falconllm.tii.ae/falcon-terms-and-conditions.html); the mamba-130m-hf-mic fixture's card declares no license.