Instructions to use alexlitz/elephantine with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use alexlitz/elephantine with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("alexlitz/elephantine", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from alexlitz/elephantine: direct link, hf CLI and curl.
- Browser
- Download file 19.2 kB
-
https://huggingface.co/alexlitz/elephantine/resolve/main/README.md
- Command line
-
hf download hf://alexlitz/elephantine/README.md
-
curl -L -o README.md https://huggingface.co/alexlitz/elephantine/resolve/main/README.md
19.2 kB
| license: mit | |
| library_name: transformers | |
| tags: | |
| - emulation | |
| - custom_code | |
| # elephantine: one architecture, many models | |
| This repo holds **one** transformers architecture, `ElephantineForCausalLM` (remote code: | |
| `configuration_elephantine.py`, `modeling_elephantine.py`), and several **settings** of it. Each | |
| subfolder holds a `config.json` and a `model.safetensors`. Every subfolder uses the same config class and the same model | |
| class. Only the sizes, the weights and the branch coefficients differ. Each setting reproduces a | |
| different public model through that model's own forward pass. | |
| This is a small demo of the idea in [The Emulation Frontier](https://alexlitzenberger.com/blog/the_emulation_frontier) (code: | |
| [alexlitz/the_emulation_frontier](https://github.com/alexlitz/the_emulation_frontier)). It replaces an earlier August 2026 demo, a stock | |
| JambaForCausalLM reproducing one tiny NoPE-Llama. | |
| ## What native emulation is | |
| Model A *natively emulates* model B when some setting of A's weights makes A's forward pass compute | |
| B's, token for token, with no fine-tuning and no approximation beyond floating-point rounding. The | |
| "elephantine" is the architecture that emulates everything in a zoo of public decoder LMs. It takes the union of what | |
| the models do, and each operation a family may or may not use is a **branch with a coefficient**, | |
| off by default. Every layer is | |
| x += Attn(norm_a(x)) softmax attention: RoPE (settable frequencies), causal OR full mask, ... | |
| x += SSM(norm_s(x)) short conv -> general gated-delta recurrence -> norm / gate -> out | |
| x += FFN(norm_f(x)) routed MoE + a gated shared expert; SiLU / GeLU / ReLU / ... branches | |
| and the SSM recurrence is the general gated-delta rule | |
| `S_t = Diag(a_t) S_(t-1) - gamma_t (S_(t-1) k_t) k_t^T + w_t v_t k_t^T, y_t = S_t q_t`, whose | |
| knobs cover Mamba-1/2/3, gated DeltaNet, RWKV-4..7, RG-LRU, LFM2's gated conv, and more. A model is | |
| loaded by zeroing the branches it doesn't use and copying its weights into the ones it does | |
| (`elephantine.universal.from_*` in the code repo). `modeling_elephantine.py` is a verbatim copy of | |
| that architecture (`elephantine/universal.py`, pure torch; no fla / mamba_ssm needed). It computes | |
| in float64. | |
| ## Settings in this repo | |
| | subfolder | original | max \|Δ logits\| vs original, fp32 | ... fp64 (casts lifted) | greedy continuations, 3 prompts | size | | |
| |---|---|---|---|---|---| | |
| | `smollm2-135m` | HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 0 | 3 / 3 identical | 0.35 GB | | |
| | `smollm2-135m-inflated` | HuggingFaceTB/SmolLM2-135M | 1.4e-04 | 2.1e-13 | 3 / 3 identical | 0.58 GB | | |
| | `pythia-160m` | EleutherAI/pythia-160m | 6.1e-04 | 2.8e-12 | 3 / 3 identical | 0.32 GB | | |
| | `falcon-h1-tiny-90m` | tiiuae/Falcon-H1-Tiny-90M-Base | 4.9e-05 | 1.5e-13 | 3 / 3 identical | 1.41 GB | | |
| | `mamba-130m-mic` | huggingface-mic/mamba-130m-hf-mic | 1.6e-06 | 2.2e-15 | 3 / 3 identical | 0.06 GB | | |
| | `qwen3-0.6b-diffusion-mdlm` | dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 | 7.0e-05 | 2.0e-13 | 3 / 3 identical | 1.62 GB | | |
| "fp32" compares against the original model run by its own transformers class in float32 (its normal | |
| precision); the gap is the original's own float32 rounding. "fp64" runs the same original code in | |
| float64 with its internal float32 casts lifted (`_lift_fp32` in the code repo's tests). The | |
| remaining gap is float64 rounding, which is the precision of the construction itself. All numbers are | |
| max |Δ| over every position and vocabulary entry of 3 prompts, measured after reloading each | |
| subfolder through `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`. The reload | |
| is bit-identical to the host built in memory. | |
| **Storage.** Each tensor is stored in the narrowest dtype that holds it exactly. A bf16 | |
| checkpoint's copied weights are stored as bf16, Pythia's as fp16, and folded or derived values as fp32 or fp64. Tensors still at | |
| the architecture's default (the unused branches) are not stored, and the model class recreates | |
| them at load. The model widens everything to float64. Total upload: 4.35 GB. | |
| **No KV cache.** `generate()` works, but every step re-runs the whole prefix (a demo, not a server). | |
| Padding masks are not supported. Use unpadded, equal-length batches. | |
| ## Usage | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m") | |
| out = m.generate(tok("The capital of France is", return_tensors="pt").input_ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| Each subfolder's section below has its own snippet. `m.config.host` holds the host sizes; | |
| `m.config.branches` summarises the branch coefficients the setting switches on (the weights file | |
| is authoritative). | |
| ### `smollm2-135m` | |
| - **Source:** [`HuggingFaceTB/SmolLM2-135M`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) @ `93efa2f097`, loaded with `elephantine.universal.from_llama`. | |
| - **Family:** Llama: softmax attention, RoPE, GQA (tiled to MHA), RMSNorm, SwiGLU. | |
| - **Host:** d 576, L 30, 9 heads x 64, FFN 1536, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 218M host parameters, 176M stored (non-default), 352 MB on disk (bfloat16 176.1M, float32 0.0M, float64 0.0M). | |
| - **Mask:** causal. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 1.4e-04 (max \|logit\| 36.3) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 0 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | |
| | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| out = m.generate(ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| Greedy continuations from this setting: `The capital of France is` → `' the capital of the country.\n\nThe capital of France is the capital of'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n elif n == '`; `Once upon a time, there was a little` → `' girl named Lily. She lived in a big house with her family, but she'` | |
| ### `smollm2-135m-inflated` | |
| - **Source:** [`HuggingFaceTB/SmolLM2-135M`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) @ `93efa2f097`, loaded with `elephantine.universal.from_llama`, then inflated (d 768, heads 12, shared_ff 2048, L 40). | |
| - **Family:** the same SmolLM2-135M inside a wider and deeper host (extra units zeroed). | |
| - **Host:** d 768, L 40, 12 heads x 64, FFN 2048, SSM in 1 / 1 heads (dk 1, dv 1), vocab 49152; 457M host parameters, 288M stored (non-default), 579 MB on disk (bfloat16 287.8M, float32 0.0M, float64 0.3M). | |
| - **Mask:** causal. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 1.4e-04 (max \|logit\| 36.3) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.1e-13 | | |
| | max \|Δ logits\| vs the un-inflated host | 2.1e-13 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | |
| | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="smollm2-135m-inflated") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| out = m.generate(ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| Greedy continuations from this setting: `The capital of France is` → `' the capital of the country.\n\nThe capital of France is the capital of'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n elif n == '`; `Once upon a time, there was a little` → `' girl named Lily. She lived in a big house with her family, but she'` | |
| ### `pythia-160m` | |
| - **Source:** [`EleutherAI/pythia-160m`](https://huggingface.co/EleutherAI/pythia-160m) @ `50f5173d93`, loaded with `elephantine.universal.from_gpt_neox`. | |
| - **Family:** GPT-NeoX: LayerNorm, parallel attention + MLP block, partial RoPE (25% of each head), ungated exact (erf) GeLU MLP, biases. | |
| - **Host:** d 768, L 12, 12 heads x 64, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 50304; 220M host parameters, 162M stored (non-default), 325 MB on disk (bfloat16 0.0M, float16 162.3M, float32 0.0M, float64 0.0M). | |
| - **Mask:** causal. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 6.1e-04 (max \|logit\| 843.1) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.8e-12 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | |
| | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| out = m.generate(ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| Greedy continuations from this setting: `The capital of France is` → `' located in the city of Paris.\n\nThe city is also home to the'`; `def fibonacci(n):` → `'\n """\n Returns the number of Fibonacci numbers.\n '`; `Once upon a time, there was a little` → `' girl who was a little girl. She was a little girl, and she was'` | |
| ### `falcon-h1-tiny-90m` | |
| - **Source:** [`tiiuae/Falcon-H1-Tiny-90M-Base`](https://huggingface.co/tiiuae/Falcon-H1-Tiny-90M-Base) @ `7994372e93`, loaded with `elephantine.universal.from_falcon_h1`. | |
| - **Family:** Falcon-H1 hybrid: a Mamba-2 SSM (through the gated-delta recurrence) and GQA attention in parallel in every layer, then SwiGLU; muP multipliers folded into the weights. | |
| - **Host:** d 512, L 24, 8 heads x 64, FFN 768, SSM in 1688 / 24 heads (dk 64, dv 32), vocab 32768; 1456M host parameters, 630M stored (non-default), 1412 MB on disk (bfloat16 582.0M, float32 33.6M, float64 14.2M). | |
| - **Mask:** causal. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 4.9e-05 (max \|logit\| 43.5) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 1.5e-13 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | |
| | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="falcon-h1-tiny-90m") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| out = m.generate(ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| Greedy continuations from this setting: `The capital of France is` → `' the capital of France is the capital of France is the capital of France is the'`; `def fibonacci(n):` → `'\n if n == 0:\n return 0\n '`; `Once upon a time, there was a little` → `' girl named Laura who was a little girl. She was a little girl'` | |
| ### `mamba-130m-mic` | |
| - **Source:** [`huggingface-mic/mamba-130m-hf-mic`](https://huggingface.co/huggingface-mic/mamba-130m-hf-mic) @ `ba38c62c9e`, loaded with `elephantine.universal.from_mamba`. | |
| - **Family:** Mamba-1 (selective SSM through the gated-delta recurrence): the code path of state-spaces/mamba-130m-hf. | |
| - **Host:** d 64, L 2, 1 heads x 2, FFN 1, SSM in 256 / 128 heads (dk 16, dv 1), vocab 50280; 26M host parameters, 13M stored (non-default), 60 MB on disk (bfloat16 0.3M, float32 10.7M, float64 2.1M). | |
| - **Mask:** causal. | |
| - **Note:** Random-weight test fixture (seed 17) published by huggingface-mic: the real MambaForCausalLM class and mamba-130m-hf's tokenizer, shrunk to d 64 and 2 layers. It is not a trained model; its continuations are not meaningful text. The trained mamba-130m-hf needs a ~33B-parameter host (the SSM branch's knob projections are dense: (2 x ssm_in) x (ssm_heads x dk) = 6144 x 24576 per projection per layer), ~264 GB in fp64, so it is not part of this demo. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 1.6e-06 (max \|logit\| 6.5) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.2e-15 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | greedy 16 new tokens, vs the fp32 original | 3 / 3 identical | | |
| | greedy 16 new tokens, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="mamba-130m-mic") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| out = m.generate(ids, max_new_tokens=16, do_sample=False) | |
| print(tok.decode(out[0])) | |
| ``` | |
| ### `qwen3-0.6b-diffusion-mdlm` | |
| - **Source:** [`dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1`](https://huggingface.co/dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1) @ `c8d24a3f4a`, loaded with `elephantine.universal.from_llama`. | |
| - **Family:** masked-diffusion LM (MDLM, from Qwen3-0.6B): bidirectional attention, per-head qk-norm; runs under the full mask (causal=False). | |
| - **Host:** d 1024, L 28, 16 heads x 128, FFN 3072, SSM in 1 / 1 heads (dk 1, dv 1), vocab 151936; 1049M host parameters, 810M stored (non-default), 1621 MB on disk (bfloat16 810.5M, float32 0.0M, float64 0.0M). | |
| - **Mask:** full (bidirectional): the config sets causal=false. | |
| | measured (3 prompts) | value | | |
| |---|---| | |
| | max \|Δ logits\| vs the original in fp32 | 7.0e-05 (max \|logit\| 25.4) | | |
| | max \|Δ logits\| vs the original in fp64 (its fp32 casts lifted) | 2.0e-13 | | |
| | reload through AutoModel vs the in-memory host | 0 (state bit-identical) | | |
| | argmax agreement with the fp32 original (all positions) | 100.0% | | |
| | MDLM greedy fill of 8 masks, vs the fp32 original | 3 / 3 identical | | |
| | MDLM greedy fill of 8 masks, vs the fp64 original | 3 / 3 identical | | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm", trust_remote_code=True) | |
| tok = AutoTokenizer.from_pretrained("alexlitz/elephantine", subfolder="qwen3-0.6b-diffusion-mdlm") | |
| ids = tok("The capital of France is", return_tensors="pt").input_ids | |
| masked = torch.cat([ids, torch.full((1, 8), tok.mask_token_id)], 1) # import torch | |
| logits = m(input_ids=masked).logits # full mask (config.causal = False) | |
| ``` | |
| ## Inflation: any size, the active subset | |
| `smollm2-135m-inflated` is SmolLM2-135M inside a host that is wider (d 576 → 768, 9 → 12 heads, | |
| FFN 1536 → 2048) and deeper (30 → 40 layers). The extra heads, FFN units and residual coordinates | |
| are zeroed. The 10 extra layers are left at the architecture's default, with all-zero output projections, so each one adds | |
| exactly 0. The wider residual stream needs one weight setting on each RMSNorm, | |
| `eps' = eps·d/d'` and `w' = w·sqrt(d/d')`. The output matches the un-inflated host to float64 rounding (table | |
| above). The same move inflates a target to any size. A bigger host runs a smaller model on the | |
| active subset of its units, and the rest stay zero. `push_hub_demo.inflate` in the code repo does | |
| this for heads, FFN width, depth, and (for RMSNorm targets) hidden width. | |
| ## How this relates to the ~26.5T elephantine | |
| The writeup's elephantine is this same architecture **sized to carry the whole zoo**. It takes the | |
| per-axis maximum over all 3,906 classified decoder LMs: hidden 16,384, 192 layers, | |
| kv/token 14,336, 128 query heads, ~2.76M FFN units per layer, ~26.5T | |
| parameters in total and ~4.52T active. **This demo is not that model.** Each setting here is | |
| the same architecture at the size its one target needs (one target per subfolder), plus one | |
| inflated example. The ~26.5T model is a size, not a checkpoint. Nothing here was trained. | |
| Coverage, from the code repo (`python -m graph.minimal_emulator`, `FRONTIER.md`): the one universal | |
| host carries **3,900 of 3,906** classified models. The 6 not carried are | |
| in mixer families with no tested branch (Cyrah, whose code is in a gated repo, and unclassified recurrences). Of the carried | |
| models, **3,531** have a whole-model `from_*` loader that accepts their model_type and config. For each of these a test runs | |
| the loaded host end to end against that type's own transformers code, or a labelled transcription of its remote code. The other | |
| 369 are carried by priced slot constructions only. | |
| Two things this demo found that the coverage counts don't show. A *trained* SSM is expensive in the | |
| universal host: the SSM branch's knob projections are dense, so the trained | |
| `state-spaces/mamba-130m-hf` needs a ~33B-parameter host (~264 GB in fp64) and `LiquidAI/LFM2-350M` | |
| ~13B. That is why the Mamba-1 setting here is a random-weight fixture of mamba-130m's code path, | |
| and the trained SSM setting is Falcon-H1-Tiny-90M (Mamba-2: a 1.46B-parameter host for a 91M-parameter model). | |
| RWKV-4 is carried block by block, but universal.py has no whole-model RWKV loader, so it is not a | |
| setting here. | |
| ## Rebuilding | |
| From the code repo: `python -m elephantine.push_hub_demo --out DIR [--push]` rebuilds every | |
| setting from the original checkpoints, measures it, and writes this card. | |
| `tests/test_hub_demo.py` checks the vendored copy against `elephantine/universal.py` (source and | |
| forward), rebuilds two settings, reloads them through `AutoModelForCausalLM` with | |
| `trust_remote_code`, and checks them against the originals. | |
| ## License | |
| MIT (the code and this card). The weights in each subfolder are re-encodings of the original | |
| checkpoints and keep those models' terms: SmolLM2-135M, Pythia-160m and the dLLM Qwen3-0.6B | |
| diffusion model are Apache-2.0; Falcon-H1-Tiny is under the Falcon-LLM license | |
| (https://falconllm.tii.ae/falcon-terms-and-conditions.html); the mamba-130m-hf-mic fixture's card | |
| declares no license. | |