shrink/Jamba-tiny-dev-mic.html

<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>ai21labs/Jamba-tiny-dev: shrunk architecture</title>
<style>
:root{--bg:#fcfcfb;--fg:#0b0b0b;--muted:#52514e;--line:#e4e3df;--ok:#1d7a43;--warn:#b35c00;--s0:#2a78d6;--s1:#eb6834;--s2:#1baf7a;--s3:#eda100;--s4:#e87ba4;--s5:#008300;--s6:#6250d6;--s7:#e34948;--s8:#8a8984;}
@media (prefers-color-scheme:dark){:root:not([data-theme="light"]){--bg:#1a1a19;--fg:#f0efec;--muted:#c3c2b7;--line:#3a3936;--ok:#2f9e5a;--warn:#d9822b;--s0:#3987e5;--s1:#d95926;--s2:#199e70;--s3:#c98500;--s4:#d55181;--s5:#008300;--s6:#9085e9;--s7:#e66767;--s8:#6f6e69;}}
:root[data-theme="dark"]{--bg:#1a1a19;--fg:#f0efec;--muted:#c3c2b7;--line:#3a3936;--ok:#2f9e5a;--warn:#d9822b;--s0:#3987e5;--s1:#d95926;--s2:#199e70;--s3:#c98500;--s4:#d55181;--s5:#008300;--s6:#9085e9;--s7:#e66767;--s8:#6f6e69;}
body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
main{max-width:1040px;margin:0 auto;padding:24px 16px 64px}
h1{font-size:24px;margin:0 0 4px}h2{font-size:18px;margin:36px 0 8px}h3{font-size:14px;margin:0 0 8px;font-family:ui-monospace,monospace}
.meta,.note{color:var(--muted);margin:0 0 12px;overflow-wrap:anywhere}
a{color:inherit}
.stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin:20px 0}
.stat{border:1px solid var(--line);border-radius:8px;padding:12px;display:flex;flex-direction:column}
.stat .v{font-size:26px;font-weight:600}.stat .k{color:var(--muted);font-size:13px}
.findings{border-radius:8px;padding:4px 16px;border:1px solid var(--line)}
.findings.ok{border-left:4px solid var(--ok)}.findings.warn{border-left:4px solid var(--warn)}
.findings h2{margin:12px 0 4px;font-size:16px}
.stack{border:1px solid var(--line);border-radius:8px;padding:12px;margin:12px 0}
.row{display:flex;align-items:center;gap:12px;margin:6px 0}
.lab{flex:0 0 110px;color:var(--muted);font-size:13px}
.cells{display:flex;flex-wrap:wrap;gap:2px}
.cells i{width:12px;height:12px;border-radius:2px;display:block}
.bar{flex:1;display:flex;gap:2px;height:16px}.bar i{display:block;border-radius:2px;min-width:2px}
table{border-collapse:collapse;width:100%;font-size:13px;margin:8px 0;display:block;overflow-x:auto}
th,td{text-align:left;padding:4px 10px 4px 0;border-bottom:1px solid var(--line);vertical-align:top}
th{color:var(--muted);font-weight:500}
td:first-child{font-family:ui-monospace,monospace;overflow-wrap:anywhere}
td i{display:inline-block;width:10px;height:10px;border-radius:2px;margin-right:6px}
tr.bad td{color:var(--warn)}
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
</style></head><body><main>
<p class="meta"><a href="../index.html">← MIC coverage of Transformers model types</a></p>
<h1>ai21labs/Jamba-tiny-dev</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/Jamba-tiny-dev-mic/tree/d03ef8cc4d1f81b8af2ae68030ecaa9bc1d51a6d" title="huggingface-mic/Jamba-tiny-dev-mic@d03ef8cc4d1f81b8af2ae68030ecaa9bc1d51a6d">huggingface-mic/Jamba-tiny-dev-mic@d03ef8cc4d1f</a> ← <a href="https://huggingface.co/ai21labs/Jamba-tiny-dev/tree/ed303361004ac875426a61675edecf8e9d976882" title="ai21labs/Jamba-tiny-dev@ed303361004ac875426a61675edecf8e9d976882">ai21labs/Jamba-tiny-dev@ed303361004a</a> · JambaForCausalLM · random weights</p>
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=Jamba-tiny-dev-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> · <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">318.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">8.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>✓ Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original · 16</span><div class="cells"><i class="c0" title="layer 0: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 1: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 2: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 3: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c2" title="layer 4: is_causal=True, Attention, AttentionDecoderLayer …"></i><i class="c1" title="layer 5: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 6: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 7: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 8: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 9: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 10: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 11: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c2" title="layer 12: is_causal=True, Attention, AttentionDecoderLayer …"></i><i class="c1" title="layer 13: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 14: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 15: Conv1d, Experts, MambaDecoderLayer …"></i></div></div><div class="row"><span class="lab">Fixture · 5</span><div class="cells"><i class="c0" title="layer 0: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 1: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c0" title="layer 2: Conv1d, MLP, MambaDecoderLayer …"></i><i class="c1" title="layer 3: Conv1d, Experts, MambaDecoderLayer …"></i><i class="c2" title="layer 4: is_causal=True, Attention, AttentionDecoderLayer …"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv1d, MLP, MambaDecoderLayer …</td><td>6</td><td>2</td><td>✓ kept</td></tr><tr><td><i class="c1"></i>Conv1d, Experts, MambaDecoderLayer …</td><td>8</td><td>2</td><td>✓ kept</td></tr><tr><td><i class="c2"></i>is_causal=True, Attention, AttentionDecoderLayer …</td><td>2</td><td>1</td><td>✓ kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.789420" title="model.layers: 251.6M (78.9%)"></i><i class="c1" style="flex:0.105289" title="lm_head: 33.6M (10.5%)"></i><i class="c2" style="flex:0.105289" title="model.embed_tokens: 33.6M (10.5%)"></i><i class="c3" style="flex:0.000002" title="model.final_layernorm: 512 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.050189" title="model.layers: 443.3K (5.0%)"></i><i class="c1" style="flex:0.474902" title="lm_head: 4.2M (47.5%)"></i><i class="c2" style="flex:0.474902" title="model.embed_tokens: 4.2M (47.5%)"></i><i class="c3" style="flex:0.000007" title="model.final_layernorm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>251.6M</td><td>78.9%</td><td>443.3K</td><td>5.0%</td></tr><tr><td><i class="c1"></i>lm_head</td><td>33.6M</td><td>10.5%</td><td>4.2M</td><td>47.5%</td></tr><tr><td><i class="c2"></i>model.embed_tokens</td><td>33.6M</td><td>10.5%</td><td>4.2M</td><td>47.5%</td></tr><tr><td><i class="c3"></i>model.final_layernorm</td><td>512</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (8 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>MoE routing</td><td>top-2 of 8 experts</td><td>top-2 of 4 experts</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>jamba</td><td>jamba</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>65536</td><td>65536</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>262144</td><td>262144</td><td>✓ kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Jamba omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>89</td><td>30</td></tr><tr class=""><td>RMSNorm</td><td>75</td><td>23</td></tr><tr class=""><td>SiLUActivation</td><td>30</td><td>9</td></tr><tr class=""><td>Conv1d</td><td>14</td><td>4</td></tr><tr class=""><td>MambaDecoderLayer</td><td>14</td><td>4</td></tr><tr class=""><td>MambaMixer</td><td>14</td><td>4</td></tr><tr class=""><td>Experts</td><td>8</td><td>2</td></tr><tr class=""><td>MLP</td><td>8</td><td>3</td></tr><tr class=""><td>SparseMoeBlock</td><td>8</td><td>2</td></tr><tr class=""><td>TopKRouter</td><td>8</td><td>2</td></tr><tr class=""><td>Attention</td><td>2</td><td>1</td></tr><tr class=""><td>AttentionDecoderLayer</td><td>2</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 39 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>512</td><td>64</td></tr><tr><td>intermediate_size</td><td>2048</td><td>128</td></tr><tr><td>num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>num_experts</td><td>8</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>16</td><td>5</td></tr></table>
</main></body></html>