elephantine: one architecture, many models

This repo holds one transformers architecture, ElephantineForCausalLM (remote code: configuration_elephantine.py, modeling_elephantine.py), and several settings of it. Each subfolder holds a config.json and a model.safetensors. Every subfolder uses the same config class and the same model class. Only the sizes, the weights and the branch coefficients differ. Each setting reproduces a different public model through that model's own forward pass.

This is a small demo of the idea in The Emulation Frontier (code: alexlitz/the_emulation_frontier). It replaces an earlier August 2026 demo, a stock JambaForCausalLM reproducing one tiny NoPE-Llama.

What native emulation is

Model A natively emulates model B when some setting of A's weights makes A's forward pass compute B's, token for token, with no fine-tuning and no approximation beyond floating-point rounding. The "elephantine" is the architecture that emulates everything in a zoo of public decoder LMs. It takes the union of what the models do, and each operation a family may or may not use is a branch with a coefficient, off by default. Every layer is

x += Attn(norm_a(x))     softmax attention: RoPE (settable frequencies), causal OR full mask, ...
x += SSM(norm_s(x))      short conv -> general gated-delta recurrence -> norm / gate -> out
x += FFN(norm_f(x))      routed MoE + a gated shared expert; SiLU / GeLU / ReLU / ... branches

and the SSM recurrence is the general gated-delta rule S_t = Diag(a_t) S_(t-1) - gamma_t (S_(t-1) k_t) k_t^T + w_t v_t k_t^T, y_t = S_t q_t, whose knobs cover Mamba-1/2/3, gated DeltaNet, RWKV-4..7, RG-LRU, LFM2's gated conv, and more. A model is loaded by zeroing the branches it doesn't use and copying its weights into the ones it does (elephantine.universal.from_* in the code repo). modeling_elephantine.py is a verbatim copy of that architecture (elephantine/universal.py, pure torch; no fla / mamba_ssm needed). It computes in float64.

Settings in this repo

subfolder original max |Ξ” logits| vs original, fp32 ... fp64 (casts lifted) greedy continuations, 3 prompts size
smollm2-135m HuggingFaceTB/SmolLM2-135M 1.4e-04 0 3 / 3 identical 0.35 GB
smollm2-135m-inflated HuggingFaceTB/SmolLM2-135M 1.4e-04 2.1e-13 3 / 3 identical 0.58 GB
pythia-160m EleutherAI/pythia-160m 6.1e-04 2.8e-12 3 / 3 identical 0.32 GB
falcon-h1-tiny-90m tiiuae/Falcon-H1-Tiny-90M-Base 4.9e-05 1.5e-13 3 / 3 identical 1.41 GB
mamba-130m-mic huggingface-mic/mamba-130m-hf-mic 1.6e-06 2.2e-15 3 / 3 identical 0.06 GB
qwen3-0.6b-diffusion-mdlm dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 7.0e-05 2.0e-13 3 / 3 identical 1.62 GB

"fp32" compares against the original model run by its own transformers class in float32 (its normal precision); the gap is the original's own float32 rounding. "fp64" runs the same original code in float64 with its internal float32 casts lifted (_lift_fp32 in the code repo's tests). The remaining gap is float64 rounding, which is the precision of the construction itself. All numbers are max |Ξ”| over every position and vocabulary entry of 3 prompts, measured after reloading each subfolder through AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True). The reload is bit-identical to the host built in memory.

Storage. Each tensor is stored in the narrowest dtype that holds it exactly. A bf16 checkpoint's copied weights are stored as bf16, Pythia's as fp16, and folded or derived values as fp32 or fp64. Tensors still at the architecture's default (the unused branches) are not stored, and the model class recreates them at load. The model widens everything to float64. Total upload: 4.35 GB.

No KV cache. generate() works, but every step re-runs the whole prefix (a demo, not a server). Padding masks are not supported. Use unpadded, equal-length batches.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m")
out = m.generate(tok("The capital of France is", return_tensors="pt").input_ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

Each subfolder's section below has its own snippet. m.config.host holds the host sizes; m.config.branches summarises the branch coefficients the setting switches on (the weights file is authoritative).

smollm2-135m

  • Source: HuggingFaceTB/SmolLM2-135M @ 93efa2f097, loaded with elephantine.universal.from_llama.
  • Family: Llama: softmax attention, RoPE, GQA (tiled to MHA), RMSNorm, SwiGLU.
  • Host: d 576, L 30, 9 heads x 64, FFN 1536, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 218M host parameters, 176M stored (non-default), 352 MB on disk (bfloat16 176.1M, float32 0.0M, float64 0.0M).
  • Mask: causal.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 1.4e-04 (max |logit| 36.3)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 0
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
greedy 16 new tokens, vs the fp32 original 3 / 3 identical
greedy 16 new tokens, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

Greedy continuations from this setting: The capital of France is β†’ ' the capital of the country.\n\nThe capital of France is the capital of'; def fibonacci(n): β†’ '\n if n == 0:\n return 0\n elif n == '; Once upon a time, there was a little β†’ ' girl named Lily. She lived in a big house with her family, but she'

smollm2-135m-inflated

  • Source: HuggingFaceTB/SmolLM2-135M @ 93efa2f097, loaded with elephantine.universal.from_llama, then inflated (d 768, heads 12, shared_ff 2048, L 40).
  • Family: the same SmolLM2-135M inside a wider and deeper host (extra units zeroed).
  • Host: d 768, L 40, 12 heads x 64, FFN 2048, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 457M host parameters, 288M stored (non-default), 579 MB on disk (bfloat16 287.8M, float32 0.0M, float64 0.3M).
  • Mask: causal.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 1.4e-04 (max |logit| 36.3)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 2.1e-13
max |Ξ” logits| vs the un-inflated host 2.1e-13
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
greedy 16 new tokens, vs the fp32 original 3 / 3 identical
greedy 16 new tokens, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

Greedy continuations from this setting: The capital of France is β†’ ' the capital of the country.\n\nThe capital of France is the capital of'; def fibonacci(n): β†’ '\n if n == 0:\n return 0\n elif n == '; Once upon a time, there was a little β†’ ' girl named Lily. She lived in a big house with her family, but she'

pythia-160m

  • Source: EleutherAI/pythia-160m @ 50f5173d93, loaded with elephantine.universal.from_gpt_neox.
  • Family: GPT-NeoX: LayerNorm, parallel attention + MLP block, partial RoPE (25% of each head), ungated exact (erf) GeLU MLP, biases.
  • Host: d 768, L 12, 12 heads x 64, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 50304; 220M host parameters, 162M stored (non-default), 325 MB on disk (bfloat16 0.0M, float16 162.3M, float32 0.0M, float64 0.0M).
  • Mask: causal.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 6.1e-04 (max |logit| 843.1)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 2.8e-12
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
greedy 16 new tokens, vs the fp32 original 3 / 3 identical
greedy 16 new tokens, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

Greedy continuations from this setting: The capital of France is β†’ ' located in the city of Paris.\n\nThe city is also home to the'; def fibonacci(n): β†’ '\n """\n Returns the number of Fibonacci numbers.\n '; Once upon a time, there was a little β†’ ' girl who was a little girl. She was a little girl, and she was'

falcon-h1-tiny-90m

  • Source: tiiuae/Falcon-H1-Tiny-90M-Base @ 7994372e93, loaded with elephantine.universal.from_falcon_h1.
  • Family: Falcon-H1 hybrid: a Mamba-2 SSM (through the gated-delta recurrence) and GQA attention in parallel in every layer, then SwiGLU; muP multipliers folded into the weights.
  • Host: d 512, L 24, 8 heads x 64, FFN 768, SSM in 1688 / 24 heads (dk 64, dv 32), vocab 32768; 1456M host parameters, 630M stored (non-default), 1412 MB on disk (bfloat16 582.0M, float32 33.6M, float64 14.2M).
  • Mask: causal.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 4.9e-05 (max |logit| 43.5)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 1.5e-13
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
greedy 16 new tokens, vs the fp32 original 3 / 3 identical
greedy 16 new tokens, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

Greedy continuations from this setting: The capital of France is β†’ ' the capital of France is the capital of France is the capital of France is the'; def fibonacci(n): β†’ '\n if n == 0:\n return 0\n '; Once upon a time, there was a little β†’ ' girl named Laura who was a little girl. She was a little girl'

mamba-130m-mic

  • Source: huggingface-mic/mamba-130m-hf-mic @ ba38c62c9e, loaded with elephantine.universal.from_mamba.
  • Family: Mamba-1 (selective SSM through the gated-delta recurrence): the code path of state-spaces/mamba-130m-hf.
  • Host: d 64, L 2, 1 heads x 2, FFN 1, SSM in 256 / 128 heads (dk 16, dv 1), vocab 50280; 26M host parameters, 13M stored (non-default), 60 MB on disk (bfloat16 0.3M, float32 10.7M, float64 2.1M).
  • Mask: causal.
  • Note: Random-weight test fixture (seed 17) published by huggingface-mic: the real MambaForCausalLM class and mamba-130m-hf's tokenizer, shrunk to d 64 and 2 layers. It is not a trained model; its continuations are not meaningful text. The trained mamba-130m-hf needs a ~33B-parameter host (the SSM branch's knob projections are dense: (2 x ssm_in) x (ssm_heads x dk) = 6144 x 24576 per projection per layer), ~264 GB in fp64, so it is not part of this demo.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 1.6e-06 (max |logit| 6.5)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 2.2e-15
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
greedy 16 new tokens, vs the fp32 original 3 / 3 identical
greedy 16 new tokens, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))

qwen3-0.6b-diffusion-mdlm

  • Source: dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 @ c8d24a3f4a, loaded with elephantine.universal.from_llama.
  • Family: masked-diffusion LM (MDLM, from Qwen3-0.6B): bidirectional attention, per-head qk-norm; runs under the full mask (causal=False).
  • Host: d 1024, L 28, 16 heads x 128, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 151936; 1049M host parameters, 810M stored (non-default), 1621 MB on disk (bfloat16 810.5M, float32 0.0M, float64 0.0M).
  • Mask: full (bidirectional): the config sets causal=false.
measured (3 prompts) value
max |Ξ” logits| vs the original in fp32 7.0e-05 (max |logit| 25.4)
max |Ξ” logits| vs the original in fp64 (its fp32 casts lifted) 2.0e-13
reload through AutoModel vs the in-memory host 0 (state bit-identical)
argmax agreement with the fp32 original (all positions) 100.0%
MDLM greedy fill of 8 masks, vs the fp32 original 3 / 3 identical
MDLM greedy fill of 8 masks, vs the fp64 original 3 / 3 identical
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm")
ids = tok("The capital of France is", return_tensors="pt").input_ids
masked = torch.cat([ids, torch.full((1, 8), tok.mask_token_id)], 1)  # import torch
logits = m(input_ids=masked).logits      # full mask (config.causal = False)

Inflation: any size, the active subset

smollm2-135m-inflated is SmolLM2-135M inside a host that is wider (d 576 β†’ 768, 9 β†’ 12 heads, FFN 1536 β†’ 2048) and deeper (30 β†’ 40 layers). The extra heads, FFN units and residual coordinates are zeroed. The 10 extra layers are left at the architecture's default, with all-zero output projections, so each one adds exactly 0. The wider residual stream needs one weight setting on each RMSNorm, eps' = epsΒ·d/d' and w' = wΒ·sqrt(d/d'). The output matches the un-inflated host to float64 rounding (table above). The same move inflates a target to any size. A bigger host runs a smaller model on the active subset of its units, and the rest stay zero. push_hub_demo.inflate in the code repo does this for heads, FFN width, depth, and (for RMSNorm targets) hidden width.

How this relates to the ~26.5T elephantine

The writeup's elephantine is this same architecture sized to carry the whole zoo. It takes the per-axis maximum over all 3,906 classified decoder LMs: hidden 16,384, 192 layers, kv/token 14,336, 128 query heads, ~2.76M FFN units per layer, ~26.5T parameters in total and ~4.52T active. This demo is not that model. Each setting here is the same architecture at the size its one target needs (one target per subfolder), plus one inflated example. The ~26.5T model is a size, not a checkpoint. Nothing here was trained.

Coverage, from the code repo (python -m graph.minimal_emulator, FRONTIER.md): the one universal host carries 3,900 of 3,906 classified models. The 6 not carried are in mixer families with no tested branch (Cyrah, whose code is in a gated repo, and unclassified recurrences). Of the carried models, 3,531 have a whole-model from_* loader that accepts their model_type and config. For each of these a test runs the loaded host end to end against that type's own transformers code, or a labelled transcription of its remote code. The other 369 are carried by priced slot constructions only.

Two things this demo found that the coverage counts don't show. A trained SSM is expensive in the universal host: the SSM branch's knob projections are dense, so the trained state-spaces/mamba-130m-hf needs a 33B-parameter host (264 GB in fp64) and LiquidAI/LFM2-350M ~13B. That is why the Mamba-1 setting here is a random-weight fixture of mamba-130m's code path, and the trained SSM setting is Falcon-H1-Tiny-90M (Mamba-2: a 1.46B-parameter host for a 91M-parameter model). RWKV-4 is carried block by block, but universal.py has no whole-model RWKV loader, so it is not a setting here.

Rebuilding

From the code repo: python -m elephantine.push_hub_demo --out DIR [--push] rebuilds every setting from the original checkpoints, measures it, and writes this card. tests/test_hub_demo.py checks the vendored copy against elephantine/universal.py (source and forward), rebuilds two settings, reloads them through AutoModelForCausalLM with trust_remote_code, and checks them against the originals.

License

MIT (the code and this card). The weights in each subfolder are re-encodings of the original checkpoints and keep those models' terms: SmolLM2-135M, Pythia-160m and the dLLM Qwen3-0.6B diffusion model are Apache-2.0; Falcon-H1-Tiny is under the Falcon-LLM license (https://falconllm.tii.ae/falcon-terms-and-conditions.html); the mamba-130m-hf-mic fixture's card declares no license.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support