Hugging Face's logo

Spaces:
huggingface-mic
/
mic-coverage
Running

App Files Community
mic-coverage / shrink /DeepSeek-R1-mic.html
tarekziade's picture
tarekziade HF Staff
Fleet tags for model types with several MICs; CI results from the MIC failures dashboard
9d3d1d2 verified
Raw History Blame Contribute Delete
13.8 kB
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>deepseek-ai/DeepSeek-R1: shrunk architecture</title>
<style>
:root{--bg:#fcfcfb;--fg:#0b0b0b;--muted:#52514e;--line:#e4e3df;--ok:#1d7a43;--warn:#b35c00;--s0:#2a78d6;--s1:#eb6834;--s2:#1baf7a;--s3:#eda100;--s4:#e87ba4;--s5:#008300;--s6:#6250d6;--s7:#e34948;--s8:#8a8984;}
@media (prefers-color-scheme:dark){:root:not([data-theme="light"]){--bg:#1a1a19;--fg:#f0efec;--muted:#c3c2b7;--line:#3a3936;--ok:#2f9e5a;--warn:#d9822b;--s0:#3987e5;--s1:#d95926;--s2:#199e70;--s3:#c98500;--s4:#d55181;--s5:#008300;--s6:#9085e9;--s7:#e66767;--s8:#6f6e69;}}
:root[data-theme="dark"]{--bg:#1a1a19;--fg:#f0efec;--muted:#c3c2b7;--line:#3a3936;--ok:#2f9e5a;--warn:#d9822b;--s0:#3987e5;--s1:#d95926;--s2:#199e70;--s3:#c98500;--s4:#d55181;--s5:#008300;--s6:#9085e9;--s7:#e66767;--s8:#6f6e69;}
body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
main{max-width:1040px;margin:0 auto;padding:24px 16px 64px}
h1{font-size:24px;margin:0 0 4px}h2{font-size:18px;margin:36px 0 8px}h3{font-size:14px;margin:0 0 8px;font-family:ui-monospace,monospace}
.meta,.note{color:var(--muted);margin:0 0 12px;overflow-wrap:anywhere}
a{color:inherit}
.stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin:20px 0}
.stat{border:1px solid var(--line);border-radius:8px;padding:12px;display:flex;flex-direction:column}
.stat .v{font-size:26px;font-weight:600}.stat .k{color:var(--muted);font-size:13px}
.findings{border-radius:8px;padding:4px 16px;border:1px solid var(--line)}
.findings.ok{border-left:4px solid var(--ok)}.findings.warn{border-left:4px solid var(--warn)}
.findings h2{margin:12px 0 4px;font-size:16px}
.stack{border:1px solid var(--line);border-radius:8px;padding:12px;margin:12px 0}
.row{display:flex;align-items:center;gap:12px;margin:6px 0}
.lab{flex:0 0 110px;color:var(--muted);font-size:13px}
.cells{display:flex;flex-wrap:wrap;gap:2px}
.cells i{width:12px;height:12px;border-radius:2px;display:block}
.bar{flex:1;display:flex;gap:2px;height:16px}.bar i{display:block;border-radius:2px;min-width:2px}
table{border-collapse:collapse;width:100%;font-size:13px;margin:8px 0;display:block;overflow-x:auto}
th,td{text-align:left;padding:4px 10px 4px 0;border-bottom:1px solid var(--line);vertical-align:top}
th{color:var(--muted);font-weight:500}
td:first-child{font-family:ui-monospace,monospace;overflow-wrap:anywhere}
td i{display:inline-block;width:10px;height:10px;border-radius:2px;margin-right:6px}
tr.bad td{color:var(--warn)}
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
</style></head><body><main>
<p class="meta"><a href="../index.html">← MIC coverage of Transformers model types</a></p>
<h1>deepseek-ai/DeepSeek-R1</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/DeepSeek-R1-mic/tree/89eddb3d8f9419e63b667eb7d5244fec8c768ed9" title="huggingface-mic/DeepSeek-R1-mic@89eddb3d8f9419e63b667eb7d5244fec8c768ed9">huggingface-mic/DeepSeek-R1-mic@89eddb3d8f94</a> ← <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad" title="deepseek-ai/DeepSeek-R1@56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad">deepseek-ai/DeepSeek-R1@56d4cbbb4d29</a> · DeepseekV3ForCausalLM · random weights · built before the shrink rule (no recorded recipe)</p>
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=DeepSeek-R1-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> · <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">671.0B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>✓ Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original · 61</span><div class="cells"><i class="c0" title="layer 0: no Experts"></i><i class="c0" title="layer 1: no Experts"></i><i class="c0" title="layer 2: no Experts"></i><i class="c1" title="layer 3: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 4: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 5: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 6: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 7: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 8: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 9: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 10: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 11: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 12: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 13: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 14: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 15: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 16: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 17: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 18: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 19: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 20: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 21: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 22: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 23: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 24: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 25: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 26: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 27: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 28: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 29: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 30: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 31: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 32: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 33: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 34: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 35: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 36: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 37: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 38: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 39: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 40: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 41: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 42: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 43: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 44: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 45: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 46: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 47: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 48: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 49: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 50: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 51: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 52: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 53: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 54: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 55: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 56: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 57: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 58: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 59: Experts, MoE, TopkRouter"></i><i class="c1" title="layer 60: Experts, MoE, TopkRouter"></i></div></div><div class="row"><span class="lab">Fixture · 2</span><div class="cells"><i class="c0" title="layer 0: no Experts"></i><i class="c1" title="layer 1: Experts, MoE, TopkRouter"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no Experts</td><td>3</td><td>1</td><td>✓ kept</td></tr><tr><td><i class="c1"></i>Experts, MoE, TopkRouter</td><td>58</td><td>1</td><td>✓ kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.997238" title="model.layers: 669.2B (99.7%)"></i><i class="c1" style="flex:0.001381" title="model.embed_tokens: 926.7M (0.1%)"></i><i class="c2" style="flex:0.001381" title="lm_head: 926.7M (0.1%)"></i><i class="c3" style="flex:0.000000" title="model.norm: 7.2K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.004593" title="model.layers: 76.4K (0.5%)"></i><i class="c1" style="flex:0.497702" title="model.embed_tokens: 8.3M (49.8%)"></i><i class="c2" style="flex:0.497702" title="lm_head: 8.3M (49.8%)"></i><i class="c3" style="flex:0.000004" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>669.2B</td><td>99.7%</td><td>76.4K</td><td>0.5%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>926.7M</td><td>0.1%</td><td>8.3M</td><td>49.8%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>926.7M</td><td>0.1%</td><td>8.3M</td><td>49.8%</td></tr><tr><td><i class="c3"></i>model.norm</td><td>7.2K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (128 heads, 128 KV)</td><td>MHA (4 heads, 4 KV)</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>MoE routing</td><td>top-8 of 256 experts</td><td>top-2 of 4 experts</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>yarn</td><td>yarn</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>deepseek_v3</td><td>deepseek_v3</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>129280</td><td>129280</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>✓ kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>163840</td><td>163840</td><td>✓ kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix DeepseekV3 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>489</td><td>17</td></tr><tr class=""><td>RMSNorm</td><td>245</td><td>9</td></tr><tr class=""><td>SiLUActivation</td><td>119</td><td>3</td></tr><tr class=""><td>Attention</td><td>61</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>61</td><td>2</td></tr><tr class=""><td>MLP</td><td>61</td><td>2</td></tr><tr class=""><td>Experts</td><td>58</td><td>1</td></tr><tr class=""><td>MoE</td><td>58</td><td>1</td></tr><tr class=""><td>TopkRouter</td><td>58</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">18 fields changed, 35 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>first_k_dense_replace</td><td>3</td><td>1</td></tr><tr><td>head_dim</td><td>64</td><td>8</td></tr><tr><td>hidden_size</td><td>7168</td><td>64</td></tr><tr><td>intermediate_size</td><td>18432</td><td>128</td></tr><tr><td>kv_lora_rank</td><td>512</td><td>16</td></tr><tr><td>moe_intermediate_size</td><td>2048</td><td>32</td></tr><tr><td>n_group</td><td>8</td><td>1</td></tr><tr><td>n_routed_experts</td><td>256</td><td>4</td></tr><tr><td>num_attention_heads</td><td>128</td><td>4</td></tr><tr><td>num_experts_per_tok</td><td>8</td><td>2</td></tr><tr><td>num_hidden_layers</td><td>61</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>128</td><td>4</td></tr><tr><td>q_lora_rank</td><td>1536</td><td>16</td></tr><tr><td>qk_nope_head_dim</td><td>128</td><td>16</td></tr><tr><td>qk_rope_head_dim</td><td>64</td><td>8</td></tr><tr><td>quantization_config</td><td>{&quot;activation_scheme&quot;: &quot;dynamic&quot;, &quot;fmt&quot;: &quot;e4m3&quot;, &quot;quant_method&quot;: &quot;fp8&quot;, &quot;weigh…</td><td>—</td></tr><tr><td>topk_group</td><td>4</td><td>1</td></tr><tr><td>v_head_dim</td><td>128</td><td>16</td></tr></table>
</main></body></html>