Instructions to use alexlitz/elephantine with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use alexlitz/elephantine with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("alexlitz/elephantine", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
elephantine: one architecture, many models
This repo holds one transformers architecture, ElephantineForCausalLM (remote code:
configuration_elephantine.py, modeling_elephantine.py), and several settings of it. Each
subfolder holds a config.json and a model.safetensors. Every subfolder uses the same config class and the same model
class. Only the sizes, the weights and the branch coefficients differ. Each setting reproduces a
different public model through that model's own forward pass.
This is a small demo of the idea in The Emulation Frontier (code: alexlitz/the_emulation_frontier). It replaces an earlier August 2026 demo, a stock JambaForCausalLM reproducing one tiny NoPE-Llama.
What native emulation is
Model A natively emulates model B when some setting of A's weights makes A's forward pass compute B's, token for token, with no fine-tuning and no approximation beyond floating-point rounding. The "elephantine" is the architecture that emulates everything in a zoo of public decoder LMs. It takes the union of what the models do, and each operation a family may or may not use is a branch with a coefficient, off by default. Every layer is
x += Attn(norm_a(x)) softmax attention: RoPE (settable frequencies), causal OR full mask, ...
x += SSM(norm_s(x)) short conv -> general gated-delta recurrence -> norm / gate -> out
x += FFN(norm_f(x)) routed MoE + a gated shared expert; SiLU / GeLU / ReLU / ... branches
and the SSM recurrence is the general gated-delta rule
S_t = Diag(a_t) S_(t-1) - gamma_t (S_(t-1) k_t) k_t^T + w_t v_t k_t^T, y_t = S_t q_t, whose
knobs cover Mamba-1/2/3, gated DeltaNet, RWKV-4..7, RG-LRU, LFM2's gated conv, and more. A model is
loaded by zeroing the branches it doesn't use and copying its weights into the ones it does
(elephantine.universal.from_* in the code repo). modeling_elephantine.py is a verbatim copy of
that architecture (elephantine/universal.py, pure torch; no fla / mamba_ssm needed). It computes
in float64.
Settings in this repo
| subfolder | original | max |Ξ logits| vs original, fp32 | ... fp64 (casts lifted) | greedy continuations, 3 prompts | size |
|---|---|---|---|---|---|
smollm2-135m |
HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 0 | 3 / 3 identical | 0.35 GB |
smollm2-135m-inflated |
HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 2.1e-13 | 3 / 3 identical | 0.58 GB |
pythia-160m |
EleutherAI/pythia-160m | 6.1e-04 | 2.8e-12 | 3 / 3 identical | 0.32 GB |
falcon-h1-tiny-90m |
tiiuae/Falcon-H1-Tiny-90M-Base | 4.9e-05 | 1.5e-13 | 3 / 3 identical | 1.41 GB |
mamba-130m-mic |
huggingface-mic/mamba-130m-hf-mic | 1.6e-06 | 2.2e-15 | 3 / 3 identical | 0.06 GB |
qwen3-0.6b-diffusion-mdlm |
dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 | 7.0e-05 | 2.0e-13 | 3 / 3 identical | 1.62 GB |
"fp32" compares against the original model run by its own transformers class in float32 (its normal
precision); the gap is the original's own float32 rounding. "fp64" runs the same original code in
float64 with its internal float32 casts lifted (_lift_fp32 in the code repo's tests). The
remaining gap is float64 rounding, which is the precision of the construction itself. All numbers are
max |Ξ| over every position and vocabulary entry of 3 prompts, measured after reloading each
subfolder through AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True). The reload
is bit-identical to the host built in memory.
Storage. Each tensor is stored in the narrowest dtype that holds it exactly. A bf16 checkpoint's copied weights are stored as bf16, Pythia's as fp16, and folded or derived values as fp32 or fp64. Tensors still at the architecture's default (the unused branches) are not stored, and the model class recreates them at load. The model widens everything to float64. Total upload: 4.35 GB.
No KV cache. generate() works, but every step re-runs the whole prefix (a demo, not a server).
Padding masks are not supported. Use unpadded, equal-length batches.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m")
out = m.generate(tok("The capital of France is", return_tensors="pt").input_ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
Each subfolder's section below has its own snippet. m.config.host holds the host sizes;
m.config.branches summarises the branch coefficients the setting switches on (the weights file
is authoritative).
smollm2-135m
- Source:
HuggingFaceTB/SmolLM2-135M@93efa2f097, loaded withelephantine.universal.from_llama. - Family: Llama: softmax attention, RoPE, GQA (tiled to MHA), RMSNorm, SwiGLU.
- Host: d 576, L 30, 9 heads x 64, FFN 1536, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 218M host parameters, 176M stored (non-default), 352 MB on disk (bfloat16 176.1M, float32 0.0M, float64 0.0M).
- Mask: causal.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 1.4e-04 (max |logit| 36.3) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 0 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| greedy 16 new tokens, vs the fp32 original | 3 / 3 identical |
| greedy 16 new tokens, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
Greedy continuations from this setting: The capital of France is β ' the capital of the country.\n\nThe capital of France is the capital of'; def fibonacci(n): β '\n if n == 0:\n return 0\n elif n == '; Once upon a time, there was a little β ' girl named Lily. She lived in a big house with her family, but she'
smollm2-135m-inflated
- Source:
HuggingFaceTB/SmolLM2-135M@93efa2f097, loaded withelephantine.universal.from_llama, then inflated (d 768, heads 12, shared_ff 2048, L 40). - Family: the same SmolLM2-135M inside a wider and deeper host (extra units zeroed).
- Host: d 768, L 40, 12 heads x 64, FFN 2048, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 457M host parameters, 288M stored (non-default), 579 MB on disk (bfloat16 287.8M, float32 0.0M, float64 0.3M).
- Mask: causal.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 1.4e-04 (max |logit| 36.3) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 2.1e-13 |
| max |Ξ logits| vs the un-inflated host | 2.1e-13 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| greedy 16 new tokens, vs the fp32 original | 3 / 3 identical |
| greedy 16 new tokens, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
Greedy continuations from this setting: The capital of France is β ' the capital of the country.\n\nThe capital of France is the capital of'; def fibonacci(n): β '\n if n == 0:\n return 0\n elif n == '; Once upon a time, there was a little β ' girl named Lily. She lived in a big house with her family, but she'
pythia-160m
- Source:
EleutherAI/pythia-160m@50f5173d93, loaded withelephantine.universal.from_gpt_neox. - Family: GPT-NeoX: LayerNorm, parallel attention + MLP block, partial RoPE (25% of each head), ungated exact (erf) GeLU MLP, biases.
- Host: d 768, L 12, 12 heads x 64, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 50304; 220M host parameters, 162M stored (non-default), 325 MB on disk (bfloat16 0.0M, float16 162.3M, float32 0.0M, float64 0.0M).
- Mask: causal.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 6.1e-04 (max |logit| 843.1) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 2.8e-12 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| greedy 16 new tokens, vs the fp32 original | 3 / 3 identical |
| greedy 16 new tokens, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
Greedy continuations from this setting: The capital of France is β ' located in the city of Paris.\n\nThe city is also home to the'; def fibonacci(n): β '\n """\n Returns the number of Fibonacci numbers.\n '; Once upon a time, there was a little β ' girl who was a little girl. She was a little girl, and she was'
falcon-h1-tiny-90m
- Source:
tiiuae/Falcon-H1-Tiny-90M-Base@7994372e93, loaded withelephantine.universal.from_falcon_h1. - Family: Falcon-H1 hybrid: a Mamba-2 SSM (through the gated-delta recurrence) and GQA attention in parallel in every layer, then SwiGLU; muP multipliers folded into the weights.
- Host: d 512, L 24, 8 heads x 64, FFN 768, SSM in 1688 / 24 heads (dk 64, dv 32), vocab 32768; 1456M host parameters, 630M stored (non-default), 1412 MB on disk (bfloat16 582.0M, float32 33.6M, float64 14.2M).
- Mask: causal.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 4.9e-05 (max |logit| 43.5) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 1.5e-13 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| greedy 16 new tokens, vs the fp32 original | 3 / 3 identical |
| greedy 16 new tokens, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
Greedy continuations from this setting: The capital of France is β ' the capital of France is the capital of France is the capital of France is the'; def fibonacci(n): β '\n if n == 0:\n return 0\n '; Once upon a time, there was a little β ' girl named Laura who was a little girl. She was a little girl'
mamba-130m-mic
- Source:
huggingface-mic/mamba-130m-hf-mic@ba38c62c9e, loaded withelephantine.universal.from_mamba. - Family: Mamba-1 (selective SSM through the gated-delta recurrence): the code path of state-spaces/mamba-130m-hf.
- Host: d 64, L 2, 1 heads x 2, FFN 1, SSM in 256 / 128 heads (dk 16, dv 1), vocab 50280; 26M host parameters, 13M stored (non-default), 60 MB on disk (bfloat16 0.3M, float32 10.7M, float64 2.1M).
- Mask: causal.
- Note: Random-weight test fixture (seed 17) published by huggingface-mic: the real MambaForCausalLM class and mamba-130m-hf's tokenizer, shrunk to d 64 and 2 layers. It is not a trained model; its continuations are not meaningful text. The trained mamba-130m-hf needs a ~33B-parameter host (the SSM branch's knob projections are dense: (2 x ssm_in) x (ssm_heads x dk) = 6144 x 24576 per projection per layer), ~264 GB in fp64, so it is not part of this demo.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 1.6e-06 (max |logit| 6.5) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 2.2e-15 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| greedy 16 new tokens, vs the fp32 original | 3 / 3 identical |
| greedy 16 new tokens, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0]))
qwen3-0.6b-diffusion-mdlm
- Source:
dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1@c8d24a3f4a, loaded withelephantine.universal.from_llama. - Family: masked-diffusion LM (MDLM, from Qwen3-0.6B): bidirectional attention, per-head qk-norm; runs under the full mask (causal=False).
- Host: d 1024, L 28, 16 heads x 128, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 151936; 1049M host parameters, 810M stored (non-default), 1621 MB on disk (bfloat16 810.5M, float32 0.0M, float64 0.0M).
- Mask: full (bidirectional): the config sets causal=false.
| measured (3 prompts) | value |
|---|---|
| max |Ξ logits| vs the original in fp32 | 7.0e-05 (max |logit| 25.4) |
| max |Ξ logits| vs the original in fp64 (its fp32 casts lifted) | 2.0e-13 |
| reload through AutoModel vs the in-memory host | 0 (state bit-identical) |
| argmax agreement with the fp32 original (all positions) | 100.0% |
| MDLM greedy fill of 8 masks, vs the fp32 original | 3 / 3 identical |
| MDLM greedy fill of 8 masks, vs the fp64 original | 3 / 3 identical |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm")
ids = tok("The capital of France is", return_tensors="pt").input_ids
masked = torch.cat([ids, torch.full((1, 8), tok.mask_token_id)], 1) # import torch
logits = m(input_ids=masked).logits # full mask (config.causal = False)
Inflation: any size, the active subset
smollm2-135m-inflated is SmolLM2-135M inside a host that is wider (d 576 β 768, 9 β 12 heads,
FFN 1536 β 2048) and deeper (30 β 40 layers). The extra heads, FFN units and residual coordinates
are zeroed. The 10 extra layers are left at the architecture's default, with all-zero output projections, so each one adds
exactly 0. The wider residual stream needs one weight setting on each RMSNorm,
eps' = epsΒ·d/d' and w' = wΒ·sqrt(d/d'). The output matches the un-inflated host to float64 rounding (table
above). The same move inflates a target to any size. A bigger host runs a smaller model on the
active subset of its units, and the rest stay zero. push_hub_demo.inflate in the code repo does
this for heads, FFN width, depth, and (for RMSNorm targets) hidden width.
How this relates to the ~26.5T elephantine
The writeup's elephantine is this same architecture sized to carry the whole zoo. It takes the per-axis maximum over all 3,906 classified decoder LMs: hidden 16,384, 192 layers, kv/token 14,336, 128 query heads, ~2.76M FFN units per layer, ~26.5T parameters in total and ~4.52T active. This demo is not that model. Each setting here is the same architecture at the size its one target needs (one target per subfolder), plus one inflated example. The ~26.5T model is a size, not a checkpoint. Nothing here was trained.
Coverage, from the code repo (python -m graph.minimal_emulator, FRONTIER.md): the one universal
host carries 3,900 of 3,906 classified models. The 6 not carried are
in mixer families with no tested branch (Cyrah, whose code is in a gated repo, and unclassified recurrences). Of the carried
models, 3,531 have a whole-model from_* loader that accepts their model_type and config. For each of these a test runs
the loaded host end to end against that type's own transformers code, or a labelled transcription of its remote code. The other
369 are carried by priced slot constructions only.
Two things this demo found that the coverage counts don't show. A trained SSM is expensive in the
universal host: the SSM branch's knob projections are dense, so the trained
state-spaces/mamba-130m-hf needs a 33B-parameter host (264 GB in fp64) and LiquidAI/LFM2-350M
~13B. That is why the Mamba-1 setting here is a random-weight fixture of mamba-130m's code path,
and the trained SSM setting is Falcon-H1-Tiny-90M (Mamba-2: a 1.46B-parameter host for a 91M-parameter model).
RWKV-4 is carried block by block, but universal.py has no whole-model RWKV loader, so it is not a
setting here.
Rebuilding
From the code repo: python -m elephantine.push_hub_demo --out DIR [--push] rebuilds every
setting from the original checkpoints, measures it, and writes this card.
tests/test_hub_demo.py checks the vendored copy against elephantine/universal.py (source and
forward), rebuilds two settings, reloads them through AutoModelForCausalLM with
trust_remote_code, and checks them against the originals.
License
MIT (the code and this card). The weights in each subfolder are re-encodings of the original checkpoints and keep those models' terms: SmolLM2-135M, Pythia-160m and the dLLM Qwen3-0.6B diffusion model are Apache-2.0; Falcon-H1-Tiny is under the Falcon-LLM license (https://falconllm.tii.ae/falcon-terms-and-conditions.html); the mamba-130m-hf-mic fixture's card declares no license.