Spaces:
Running
Running
Fleet tags for model types with several MICs; CI results from the MIC failures dashboard (part 2)
Browse filesThis view is limited to 50 files because it contains too many changes. Β See raw diff
- shrink/llava-v1.6-mistral-7b-hf-mic.html +1 -0
- shrink/long-t5-tglobal-base-mic.html +1 -0
- shrink/longformer-base-4096-mic.html +1 -0
- shrink/luke-base-mic.html +1 -0
- shrink/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic.html +1 -0
- shrink/mamba-130m-hf-mic.html +1 -0
- shrink/mask2former-swin-tiny-coco-instance-mic.html +1 -0
- shrink/mbart-large-50-mic.html +1 -0
- shrink/mistral3-mic.html +1 -0
- shrink/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic.html +1 -0
- shrink/mms-tts-eng-mic.html +1 -0
- shrink/mobilenet_v2_1.0_224-mic.html +3 -2
- shrink/mobilenet_v2_1.0_224-mic.json +1 -1
- shrink/mobilevit-xx-small-mic.html +1 -0
- shrink/modernbert-mic.html +1 -0
- shrink/mpnet-base-mic.html +1 -0
- shrink/mt5-small-mic.html +1 -0
- shrink/nllb-200-distilled-600M-mic.html +1 -0
- shrink/nomic-embed-text-v1.5-mic.html +1 -0
- shrink/opt-125m-mic.html +1 -0
- shrink/opus-mt-fr-en-mic.html +1 -0
- shrink/owlv2-base-patch16-ensemble-mic.html +1 -0
- shrink/owlvit-base-patch32-mic.html +1 -0
- shrink/paligemma-3b-pt-224-mic.html +1 -0
- shrink/pegasus-xsum-mic.html +1 -0
- shrink/phi-2-mic.html +1 -0
- shrink/privacy-filter-mic.html +1 -0
- shrink/pvt-tiny-224-mic.html +1 -0
- shrink/pythia-160m-mic.html +1 -0
- shrink/qwen2-mic.html +1 -0
- shrink/qwen2_5_omni-mic.html +1 -0
- shrink/qwen2_5_vl-mic.html +1 -0
- shrink/qwen2_audio-mic.html +1 -0
- shrink/recurrentgemma-2b-mic.html +1 -0
- shrink/resnet-18-mic.html +3 -2
- shrink/resnet-18-mic.json +1 -1
- shrink/roberta-base-mic.html +1 -0
- shrink/rtdetr_r50vd-mic.html +1 -0
- shrink/sam2.1-hiera-tiny-mic.html +1 -0
- shrink/seamless-m4t-v2-large-mic.html +1 -0
- shrink/segformer-b0-finetuned-ade-512-512-mic.html +3 -2
- shrink/segformer-b0-finetuned-ade-512-512-mic.json +1 -1
- shrink/siglip-base-patch16-224-mic.html +1 -0
- shrink/smollm-1.7b-mic.html +1 -0
- shrink/smollm-mic.html +1 -0
- shrink/smolvlm-mic.html +1 -0
- shrink/speech_t5-mic.html +1 -0
- shrink/stablelm-2-1_6b-mic.html +1 -0
- shrink/starcoder2-3b-mic.html +1 -0
- shrink/swin-tiny-patch4-window7-224-mic.html +1 -0
shrink/llava-v1.6-mistral-7b-hf-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>llava-hf/llava-v1.6-mistral-7b-hf</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/llava-v1.6-mistral-7b-hf-mic/tree/2575b1c20c7676e7707d37ef750132fd64d1fc03" title="huggingface-mic/llava-v1.6-mistral-7b-hf-mic@2575b1c20c7676e7707d37ef750132fd64d1fc03">huggingface-mic/llava-v1.6-mistral-7b-hf-mic@2575b1c20c76</a> β <a href="https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/tree/2424fdd47412fccc66d91719126b420e9fbd7065" title="llava-hf/llava-v1.6-mistral-7b-hf@2424fdd47412fccc66d91719126b420e9fbd7065">llava-hf/llava-v1.6-mistral-7b-hf@2424fdd47412</a> Β· LlavaNextForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">7.6B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: CLIPEncoderLayer"></i><i class="c0" title="layer 1: CLIPEncoderLayer"></i><i class="c0" title="layer 2: CLIPEncoderLayer"></i><i class="c0" title="layer 3: CLIPEncoderLayer"></i><i class="c0" title="layer 4: CLIPEncoderLayer"></i><i class="c0" title="layer 5: CLIPEncoderLayer"></i><i class="c0" title="layer 6: CLIPEncoderLayer"></i><i class="c0" title="layer 7: CLIPEncoderLayer"></i><i class="c0" title="layer 8: CLIPEncoderLayer"></i><i class="c0" title="layer 9: CLIPEncoderLayer"></i><i class="c0" title="layer 10: CLIPEncoderLayer"></i><i class="c0" title="layer 11: CLIPEncoderLayer"></i><i class="c0" title="layer 12: CLIPEncoderLayer"></i><i class="c0" title="layer 13: CLIPEncoderLayer"></i><i class="c0" title="layer 14: CLIPEncoderLayer"></i><i class="c0" title="layer 15: CLIPEncoderLayer"></i><i class="c0" title="layer 16: CLIPEncoderLayer"></i><i class="c0" title="layer 17: CLIPEncoderLayer"></i><i class="c0" title="layer 18: CLIPEncoderLayer"></i><i class="c0" title="layer 19: CLIPEncoderLayer"></i><i class="c0" title="layer 20: CLIPEncoderLayer"></i><i class="c0" title="layer 21: CLIPEncoderLayer"></i><i class="c0" title="layer 22: CLIPEncoderLayer"></i><i class="c0" title="layer 23: CLIPEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: CLIPEncoderLayer"></i><i class="c0" title="layer 1: CLIPEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>CLIPEncoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i><i class="c0" title="layer 2: MistralDecoderLayer"></i><i class="c0" title="layer 3: MistralDecoderLayer"></i><i class="c0" title="layer 4: MistralDecoderLayer"></i><i class="c0" title="layer 5: MistralDecoderLayer"></i><i class="c0" title="layer 6: MistralDecoderLayer"></i><i class="c0" title="layer 7: MistralDecoderLayer"></i><i class="c0" title="layer 8: MistralDecoderLayer"></i><i class="c0" title="layer 9: MistralDecoderLayer"></i><i class="c0" title="layer 10: MistralDecoderLayer"></i><i class="c0" title="layer 11: MistralDecoderLayer"></i><i class="c0" title="layer 12: MistralDecoderLayer"></i><i class="c0" title="layer 13: MistralDecoderLayer"></i><i class="c0" title="layer 14: MistralDecoderLayer"></i><i class="c0" title="layer 15: MistralDecoderLayer"></i><i class="c0" title="layer 16: MistralDecoderLayer"></i><i class="c0" title="layer 17: MistralDecoderLayer"></i><i class="c0" title="layer 18: MistralDecoderLayer"></i><i class="c0" title="layer 19: MistralDecoderLayer"></i><i class="c0" title="layer 20: MistralDecoderLayer"></i><i class="c0" title="layer 21: MistralDecoderLayer"></i><i class="c0" title="layer 22: MistralDecoderLayer"></i><i class="c0" title="layer 23: MistralDecoderLayer"></i><i class="c0" title="layer 24: MistralDecoderLayer"></i><i class="c0" title="layer 25: MistralDecoderLayer"></i><i class="c0" title="layer 26: MistralDecoderLayer"></i><i class="c0" title="layer 27: MistralDecoderLayer"></i><i class="c0" title="layer 28: MistralDecoderLayer"></i><i class="c0" title="layer 29: MistralDecoderLayer"></i><i class="c0" title="layer 30: MistralDecoderLayer"></i><i class="c0" title="layer 31: MistralDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MistralDecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.922402" title="model.language_model.layers: 7.0B (92.2%)"></i><i class="c1" style="flex:0.039952" title="model.vision_tower.encoder.layers: 302.3M (4.0%)"></i><i class="c2" style="flex:0.017357" title="model.language_model: 131.3M (1.7%)"></i><i class="c3" style="flex:0.017357" title="lm_head: 131.3M (1.7%)"></i><i class="c4" style="flex:0.002773" title="model.multi_modal_projector: 21.0M (0.3%)"></i><i class="c5" style="flex:0.000158" title="model.vision_tower: 1.2M (0.0%)"></i><i class="c6" style="flex:0.000001" title="model: 4.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.017093" title="model.language_model.layers: 74.0K (1.7%)"></i><i class="c1" style="flex:0.015466" title="model.vision_tower.encoder.layers: 66.9K (1.5%)"></i><i class="c2" style="flex:0.474110" title="model.language_model: 2.1M (47.4%)"></i><i class="c3" style="flex:0.474095" title="lm_head: 2.1M (47.4%)"></i><i class="c4" style="flex:0.001922" title="model.multi_modal_projector: 8.3K (0.2%)"></i><i class="c5" style="flex:0.017300" title="model.vision_tower: 74.9K (1.7%)"></i><i class="c6" style="flex:0.000015" title="model: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>7.0B</td><td>92.2%</td><td>74.0K</td><td>1.7%</td></tr><tr><td><i class="c1"></i>model.vision_tower.encoder.layers</td><td>302.3M</td><td>4.0%</td><td>66.9K</td><td>1.5%</td></tr><tr><td><i class="c2"></i>model.language_model</td><td>131.3M</td><td>1.7%</td><td>2.1M</td><td>47.4%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>131.3M</td><td>1.7%</td><td>2.1M</td><td>47.4%</td></tr><tr><td><i class="c4"></i>model.multi_modal_projector</td><td>21.0M</td><td>0.3%</td><td>8.3K</td><td>0.2%</td></tr><tr><td><i class="c5"></i>model.vision_tower</td><td>1.2M</td><td>0.0%</td><td>74.9K</td><td>1.7%</td></tr><tr><td><i class="c6"></i>model</td><td>4.1K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>llava_next</td><td>llava_next</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>32064</td><td>32064</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>clip_vision_model</td><td>clip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>vocab_size</td><td>32000</td><td>32000</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>336</td><td>336</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (32 heads, 8 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>mistral</td><td>mistral</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>32064</td><td>32064</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix LlavaNext omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>371</td><td>29</td></tr><tr class=""><td>MistralRMSNorm</td><td>65</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>50</td><td>6</td></tr><tr class=""><td>MistralAttention</td><td>32</td><td>2</td></tr><tr class=""><td>MistralDecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MistralMLP</td><td>32</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>CLIPAttention</td><td>24</td><td>2</td></tr><tr class=""><td>CLIPEncoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>CLIPMLP</td><td>24</td><td>2</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>CLIPEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>CLIPVisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>CLIPVisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>MistralModel</td><td>1</td><td>1</td></tr><tr class=""><td>MistralRotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">11 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>4096</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>14336</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>8</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4096</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>vision_config.projection_dim</td><td>768</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>llava-hf/llava-v1.6-mistral-7b-hf</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/llava-v1.6-mistral-7b-hf-mic/tree/2575b1c20c7676e7707d37ef750132fd64d1fc03" title="huggingface-mic/llava-v1.6-mistral-7b-hf-mic@2575b1c20c7676e7707d37ef750132fd64d1fc03">huggingface-mic/llava-v1.6-mistral-7b-hf-mic@2575b1c20c76</a> β <a href="https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/tree/2424fdd47412fccc66d91719126b420e9fbd7065" title="llava-hf/llava-v1.6-mistral-7b-hf@2424fdd47412fccc66d91719126b420e9fbd7065">llava-hf/llava-v1.6-mistral-7b-hf@2424fdd47412</a> Β· LlavaNextForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=llava-v1.6-mistral-7b-hf-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">7.6B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: CLIPEncoderLayer"></i><i class="c0" title="layer 1: CLIPEncoderLayer"></i><i class="c0" title="layer 2: CLIPEncoderLayer"></i><i class="c0" title="layer 3: CLIPEncoderLayer"></i><i class="c0" title="layer 4: CLIPEncoderLayer"></i><i class="c0" title="layer 5: CLIPEncoderLayer"></i><i class="c0" title="layer 6: CLIPEncoderLayer"></i><i class="c0" title="layer 7: CLIPEncoderLayer"></i><i class="c0" title="layer 8: CLIPEncoderLayer"></i><i class="c0" title="layer 9: CLIPEncoderLayer"></i><i class="c0" title="layer 10: CLIPEncoderLayer"></i><i class="c0" title="layer 11: CLIPEncoderLayer"></i><i class="c0" title="layer 12: CLIPEncoderLayer"></i><i class="c0" title="layer 13: CLIPEncoderLayer"></i><i class="c0" title="layer 14: CLIPEncoderLayer"></i><i class="c0" title="layer 15: CLIPEncoderLayer"></i><i class="c0" title="layer 16: CLIPEncoderLayer"></i><i class="c0" title="layer 17: CLIPEncoderLayer"></i><i class="c0" title="layer 18: CLIPEncoderLayer"></i><i class="c0" title="layer 19: CLIPEncoderLayer"></i><i class="c0" title="layer 20: CLIPEncoderLayer"></i><i class="c0" title="layer 21: CLIPEncoderLayer"></i><i class="c0" title="layer 22: CLIPEncoderLayer"></i><i class="c0" title="layer 23: CLIPEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: CLIPEncoderLayer"></i><i class="c0" title="layer 1: CLIPEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>CLIPEncoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i><i class="c0" title="layer 2: MistralDecoderLayer"></i><i class="c0" title="layer 3: MistralDecoderLayer"></i><i class="c0" title="layer 4: MistralDecoderLayer"></i><i class="c0" title="layer 5: MistralDecoderLayer"></i><i class="c0" title="layer 6: MistralDecoderLayer"></i><i class="c0" title="layer 7: MistralDecoderLayer"></i><i class="c0" title="layer 8: MistralDecoderLayer"></i><i class="c0" title="layer 9: MistralDecoderLayer"></i><i class="c0" title="layer 10: MistralDecoderLayer"></i><i class="c0" title="layer 11: MistralDecoderLayer"></i><i class="c0" title="layer 12: MistralDecoderLayer"></i><i class="c0" title="layer 13: MistralDecoderLayer"></i><i class="c0" title="layer 14: MistralDecoderLayer"></i><i class="c0" title="layer 15: MistralDecoderLayer"></i><i class="c0" title="layer 16: MistralDecoderLayer"></i><i class="c0" title="layer 17: MistralDecoderLayer"></i><i class="c0" title="layer 18: MistralDecoderLayer"></i><i class="c0" title="layer 19: MistralDecoderLayer"></i><i class="c0" title="layer 20: MistralDecoderLayer"></i><i class="c0" title="layer 21: MistralDecoderLayer"></i><i class="c0" title="layer 22: MistralDecoderLayer"></i><i class="c0" title="layer 23: MistralDecoderLayer"></i><i class="c0" title="layer 24: MistralDecoderLayer"></i><i class="c0" title="layer 25: MistralDecoderLayer"></i><i class="c0" title="layer 26: MistralDecoderLayer"></i><i class="c0" title="layer 27: MistralDecoderLayer"></i><i class="c0" title="layer 28: MistralDecoderLayer"></i><i class="c0" title="layer 29: MistralDecoderLayer"></i><i class="c0" title="layer 30: MistralDecoderLayer"></i><i class="c0" title="layer 31: MistralDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MistralDecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.922402" title="model.language_model.layers: 7.0B (92.2%)"></i><i class="c1" style="flex:0.039952" title="model.vision_tower.encoder.layers: 302.3M (4.0%)"></i><i class="c2" style="flex:0.017357" title="model.language_model: 131.3M (1.7%)"></i><i class="c3" style="flex:0.017357" title="lm_head: 131.3M (1.7%)"></i><i class="c4" style="flex:0.002773" title="model.multi_modal_projector: 21.0M (0.3%)"></i><i class="c5" style="flex:0.000158" title="model.vision_tower: 1.2M (0.0%)"></i><i class="c6" style="flex:0.000001" title="model: 4.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.017093" title="model.language_model.layers: 74.0K (1.7%)"></i><i class="c1" style="flex:0.015466" title="model.vision_tower.encoder.layers: 66.9K (1.5%)"></i><i class="c2" style="flex:0.474110" title="model.language_model: 2.1M (47.4%)"></i><i class="c3" style="flex:0.474095" title="lm_head: 2.1M (47.4%)"></i><i class="c4" style="flex:0.001922" title="model.multi_modal_projector: 8.3K (0.2%)"></i><i class="c5" style="flex:0.017300" title="model.vision_tower: 74.9K (1.7%)"></i><i class="c6" style="flex:0.000015" title="model: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>7.0B</td><td>92.2%</td><td>74.0K</td><td>1.7%</td></tr><tr><td><i class="c1"></i>model.vision_tower.encoder.layers</td><td>302.3M</td><td>4.0%</td><td>66.9K</td><td>1.5%</td></tr><tr><td><i class="c2"></i>model.language_model</td><td>131.3M</td><td>1.7%</td><td>2.1M</td><td>47.4%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>131.3M</td><td>1.7%</td><td>2.1M</td><td>47.4%</td></tr><tr><td><i class="c4"></i>model.multi_modal_projector</td><td>21.0M</td><td>0.3%</td><td>8.3K</td><td>0.2%</td></tr><tr><td><i class="c5"></i>model.vision_tower</td><td>1.2M</td><td>0.0%</td><td>74.9K</td><td>1.7%</td></tr><tr><td><i class="c6"></i>model</td><td>4.1K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>llava_next</td><td>llava_next</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>32064</td><td>32064</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>clip_vision_model</td><td>clip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>vocab_size</td><td>32000</td><td>32000</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>336</td><td>336</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (32 heads, 8 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>mistral</td><td>mistral</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>32064</td><td>32064</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix LlavaNext omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>371</td><td>29</td></tr><tr class=""><td>MistralRMSNorm</td><td>65</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>50</td><td>6</td></tr><tr class=""><td>MistralAttention</td><td>32</td><td>2</td></tr><tr class=""><td>MistralDecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MistralMLP</td><td>32</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>CLIPAttention</td><td>24</td><td>2</td></tr><tr class=""><td>CLIPEncoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>CLIPMLP</td><td>24</td><td>2</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>CLIPEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>CLIPVisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>CLIPVisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>MistralModel</td><td>1</td><td>1</td></tr><tr class=""><td>MistralRotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">11 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>4096</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>14336</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>8</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4096</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>vision_config.projection_dim</td><td>768</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/long-t5-tglobal-base-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/long-t5-tglobal-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/long-t5-tglobal-base-mic/tree/04d69ed32e31d0ce673622fa6832568a878efb1d" title="huggingface-mic/long-t5-tglobal-base-mic@04d69ed32e31d0ce673622fa6832568a878efb1d">huggingface-mic/long-t5-tglobal-base-mic@04d69ed32e31</a> β <a href="https://huggingface.co/google/long-t5-tglobal-base/tree/aecb1376e5bd78db32ebc5c9deb257449b9e2b21" title="google/long-t5-tglobal-base@aecb1376e5bd78db32ebc5c9deb257449b9e2b21">google/long-t5-tglobal-base@aecb1376e5bd</a> Β· LongT5ForConditionalGeneration Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">222.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>encoder.block</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i><i class="c1" title="layer 8: no Embedding"></i><i class="c1" title="layer 9: no Embedding"></i><i class="c1" title="layer 10: no Embedding"></i><i class="c1" title="layer 11: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>11</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.block</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i><i class="c1" title="layer 8: no Embedding"></i><i class="c1" title="layer 9: no Embedding"></i><i class="c1" title="layer 10: no Embedding"></i><i class="c1" title="layer 11: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>11</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.508154" title="decoder.block: 113.3M (50.8%)"></i><i class="c1" style="flex:0.381149" title="encoder.block: 85.0M (38.1%)"></i><i class="c2" style="flex:0.110690" title="shared: 24.7M (11.1%)"></i><i class="c3" style="flex:0.000003" title="decoder.final_layer_norm: 768 (0.0%)"></i><i class="c4" style="flex:0.000003" title="encoder.final_layer_norm: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.051107" title="decoder.block: 115.2K (5.1%)"></i><i class="c1" style="flex:0.036627" title="encoder.block: 82.6K (3.7%)"></i><i class="c2" style="flex:0.912209" title="shared: 2.1M (91.2%)"></i><i class="c3" style="flex:0.000028" title="decoder.final_layer_norm: 64 (0.0%)"></i><i class="c4" style="flex:0.000028" title="encoder.final_layer_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>decoder.block</td><td>113.3M</td><td>50.8%</td><td>115.2K</td><td>5.1%</td></tr><tr><td><i class="c1"></i>encoder.block</td><td>85.0M</td><td>38.1%</td><td>82.6K</td><td>3.7%</td></tr><tr><td><i class="c2"></i>shared</td><td>24.7M</td><td>11.1%</td><td>2.1M</td><td>91.2%</td></tr><tr><td><i class="c3"></i>decoder.final_layer_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c4"></i>encoder.final_layer_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>longt5</td><td>longt5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>32128</td><td>32128</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix LongT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>217</td><td>37</td></tr><tr class=""><td>Dropout</td><td>86</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>74</td><td>14</td></tr><tr class=""><td>ModuleList</td><td>26</td><td>6</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>Block</td><td>24</td><td>4</td></tr><tr class=""><td>DenseGatedActDense</td><td>24</td><td>4</td></tr><tr class=""><td>LayerFF</td><td>24</td><td>4</td></tr><tr class=""><td>NewGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>LayerCrossAttention</td><td>12</td><td>2</td></tr><tr class=""><td>LayerSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>LayerTransientGlobalSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>TransientGlobalAttention</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>Stack</td><td>2</td><td>2</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 31 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_ff</td><td>2048</td><td>128</td></tr><tr><td>d_kv</td><td>64</td><td>16</td></tr><tr><td>d_model</td><td>768</td><td>64</td></tr><tr><td>num_decoder_layers</td><td>12</td><td>2</td></tr><tr><td>num_heads</td><td>12</td><td>4</td></tr><tr><td>num_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/long-t5-tglobal-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/long-t5-tglobal-base-mic/tree/04d69ed32e31d0ce673622fa6832568a878efb1d" title="huggingface-mic/long-t5-tglobal-base-mic@04d69ed32e31d0ce673622fa6832568a878efb1d">huggingface-mic/long-t5-tglobal-base-mic@04d69ed32e31</a> β <a href="https://huggingface.co/google/long-t5-tglobal-base/tree/aecb1376e5bd78db32ebc5c9deb257449b9e2b21" title="google/long-t5-tglobal-base@aecb1376e5bd78db32ebc5c9deb257449b9e2b21">google/long-t5-tglobal-base@aecb1376e5bd</a> Β· LongT5ForConditionalGeneration Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=long-t5-tglobal-base-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">222.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>encoder.block</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i><i class="c1" title="layer 8: no Embedding"></i><i class="c1" title="layer 9: no Embedding"></i><i class="c1" title="layer 10: no Embedding"></i><i class="c1" title="layer 11: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>11</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.block</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i><i class="c1" title="layer 8: no Embedding"></i><i class="c1" title="layer 9: no Embedding"></i><i class="c1" title="layer 10: no Embedding"></i><i class="c1" title="layer 11: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>11</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.508154" title="decoder.block: 113.3M (50.8%)"></i><i class="c1" style="flex:0.381149" title="encoder.block: 85.0M (38.1%)"></i><i class="c2" style="flex:0.110690" title="shared: 24.7M (11.1%)"></i><i class="c3" style="flex:0.000003" title="decoder.final_layer_norm: 768 (0.0%)"></i><i class="c4" style="flex:0.000003" title="encoder.final_layer_norm: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.051107" title="decoder.block: 115.2K (5.1%)"></i><i class="c1" style="flex:0.036627" title="encoder.block: 82.6K (3.7%)"></i><i class="c2" style="flex:0.912209" title="shared: 2.1M (91.2%)"></i><i class="c3" style="flex:0.000028" title="decoder.final_layer_norm: 64 (0.0%)"></i><i class="c4" style="flex:0.000028" title="encoder.final_layer_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>decoder.block</td><td>113.3M</td><td>50.8%</td><td>115.2K</td><td>5.1%</td></tr><tr><td><i class="c1"></i>encoder.block</td><td>85.0M</td><td>38.1%</td><td>82.6K</td><td>3.7%</td></tr><tr><td><i class="c2"></i>shared</td><td>24.7M</td><td>11.1%</td><td>2.1M</td><td>91.2%</td></tr><tr><td><i class="c3"></i>decoder.final_layer_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c4"></i>encoder.final_layer_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>longt5</td><td>longt5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>32128</td><td>32128</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix LongT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>217</td><td>37</td></tr><tr class=""><td>Dropout</td><td>86</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>74</td><td>14</td></tr><tr class=""><td>ModuleList</td><td>26</td><td>6</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>Block</td><td>24</td><td>4</td></tr><tr class=""><td>DenseGatedActDense</td><td>24</td><td>4</td></tr><tr class=""><td>LayerFF</td><td>24</td><td>4</td></tr><tr class=""><td>NewGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>LayerCrossAttention</td><td>12</td><td>2</td></tr><tr class=""><td>LayerSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>LayerTransientGlobalSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>TransientGlobalAttention</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>Stack</td><td>2</td><td>2</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 31 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_ff</td><td>2048</td><td>128</td></tr><tr><td>d_kv</td><td>64</td><td>16</td></tr><tr><td>d_model</td><td>768</td><td>64</td></tr><tr><td>num_decoder_layers</td><td>12</td><td>2</td></tr><tr><td>num_heads</td><td>12</td><td>4</td></tr><tr><td>num_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/longformer-base-4096-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>allenai/longformer-base-4096</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/longformer-base-4096-mic/tree/5e8ae916a995900686949962df7210f7676e90fd" title="huggingface-mic/longformer-base-4096-mic@5e8ae916a995900686949962df7210f7676e90fd">huggingface-mic/longformer-base-4096-mic@5e8ae916a995</a> β <a href="https://huggingface.co/allenai/longformer-base-4096/tree/301e6a42cb0d9976a6d6a26a079fef81c18aa895" title="allenai/longformer-base-4096@301e6a42cb0d9976a6d6a26a079fef81c18aa895">allenai/longformer-base-4096@301e6a42cb0d</a> Β· LongformerForMaskedLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">148.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>longformer.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.714914" title="longformer.encoder.layer: 106.3M (71.5%)"></i><i class="c1" style="flex:0.280766" title="longformer.embeddings: 41.8M (28.1%)"></i><i class="c2" style="flex:0.003971" title="lm_head.dense: 590.6K (0.4%)"></i><i class="c3" style="flex:0.000338" title="lm_head: 50.3K (0.0%)"></i><i class="c4" style="flex:0.000010" title="lm_head.layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.025347" title="longformer.encoder.layer: 91.9K (2.5%)"></i><i class="c1" style="flex:0.959608" title="longformer.embeddings: 3.5M (96.0%)"></i><i class="c2" style="flex:0.001147" title="lm_head.dense: 4.2K (0.1%)"></i><i class="c3" style="flex:0.013863" title="lm_head: 50.3K (1.4%)"></i><i class="c4" style="flex:0.000035" title="lm_head.layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>longformer.encoder.layer</td><td>106.3M</td><td>71.5%</td><td>91.9K</td><td>2.5%</td></tr><tr><td><i class="c1"></i>longformer.embeddings</td><td>41.8M</td><td>28.1%</td><td>3.5M</td><td>96.0%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>50.3K</td><td>0.0%</td><td>50.3K</td><td>1.4%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>longformer</td><td>longformer</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50265</td><td>50265</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4098</td><td>4098</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Longformer omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>110</td><td>20</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>attention_window</td><td>12 entries: 12Γ 512</td><td>[512, 512]</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>allenai/longformer-base-4096</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/longformer-base-4096-mic/tree/5e8ae916a995900686949962df7210f7676e90fd" title="huggingface-mic/longformer-base-4096-mic@5e8ae916a995900686949962df7210f7676e90fd">huggingface-mic/longformer-base-4096-mic@5e8ae916a995</a> β <a href="https://huggingface.co/allenai/longformer-base-4096/tree/301e6a42cb0d9976a6d6a26a079fef81c18aa895" title="allenai/longformer-base-4096@301e6a42cb0d9976a6d6a26a079fef81c18aa895">allenai/longformer-base-4096@301e6a42cb0d</a> Β· LongformerForMaskedLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=longformer-base-4096-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">148.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>longformer.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.714914" title="longformer.encoder.layer: 106.3M (71.5%)"></i><i class="c1" style="flex:0.280766" title="longformer.embeddings: 41.8M (28.1%)"></i><i class="c2" style="flex:0.003971" title="lm_head.dense: 590.6K (0.4%)"></i><i class="c3" style="flex:0.000338" title="lm_head: 50.3K (0.0%)"></i><i class="c4" style="flex:0.000010" title="lm_head.layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.025347" title="longformer.encoder.layer: 91.9K (2.5%)"></i><i class="c1" style="flex:0.959608" title="longformer.embeddings: 3.5M (96.0%)"></i><i class="c2" style="flex:0.001147" title="lm_head.dense: 4.2K (0.1%)"></i><i class="c3" style="flex:0.013863" title="lm_head: 50.3K (1.4%)"></i><i class="c4" style="flex:0.000035" title="lm_head.layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>longformer.encoder.layer</td><td>106.3M</td><td>71.5%</td><td>91.9K</td><td>2.5%</td></tr><tr><td><i class="c1"></i>longformer.embeddings</td><td>41.8M</td><td>28.1%</td><td>3.5M</td><td>96.0%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>50.3K</td><td>0.0%</td><td>50.3K</td><td>1.4%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>longformer</td><td>longformer</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50265</td><td>50265</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4098</td><td>4098</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Longformer omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>110</td><td>20</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>attention_window</td><td>12 entries: 12Γ 512</td><td>[512, 512]</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/luke-base-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>studio-ousia/luke-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/luke-base-mic/tree/39d0a41c62c207695ca9fda87542f953dc1d0b56" title="huggingface-mic/luke-base-mic@39d0a41c62c207695ca9fda87542f953dc1d0b56">huggingface-mic/luke-base-mic@39d0a41c62c2</a> β <a href="https://huggingface.co/studio-ousia/luke-base/tree/7438924defd9f3c2018d63c16073bf4bcb6a70aa" title="studio-ousia/luke-base@7438924defd9f3c2018d63c16073bf4bcb6a70aa">studio-ousia/luke-base@7438924defd9</a> Β· LukeForMaskedLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">314.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">135.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>luke.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.408952" title="luke.entity_embeddings: 128.6M (40.9%)"></i><i class="c1" style="flex:0.338104" title="luke.encoder.layer: 106.3M (33.8%)"></i><i class="c2" style="flex:0.124034" title="luke.embeddings: 39.0M (12.4%)"></i><i class="c3" style="flex:0.122771" title="lm_head.decoder: 38.6M (12.3%)"></i><i class="c4" style="flex:0.001878" title="lm_head.dense: 590.6K (0.2%)"></i><i class="c5" style="flex:0.001878" title="luke.pooler: 590.6K (0.2%)"></i><i class="c6" style="flex:0.001590" title="entity_predictions: 500.0K (0.2%)"></i><i class="c7" style="flex:0.000792" title="3 others: 249.2K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.947220" title="luke.entity_embeddings: 128.0M (94.7%)"></i><i class="c1" style="flex:0.000680" title="luke.encoder.layer: 91.9K (0.1%)"></i><i class="c2" style="flex:0.024043" title="luke.embeddings: 3.3M (2.4%)"></i><i class="c3" style="flex:0.023798" title="lm_head.decoder: 3.2M (2.4%)"></i><i class="c4" style="flex:0.000031" title="lm_head.dense: 4.2K (0.0%)"></i><i class="c5" style="flex:0.000031" title="luke.pooler: 4.2K (0.0%)"></i><i class="c6" style="flex:0.003699" title="entity_predictions: 500.0K (0.4%)"></i><i class="c7" style="flex:0.000500" title="3 others: 67.5K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>luke.entity_embeddings</td><td>128.6M</td><td>40.9%</td><td>128.0M</td><td>94.7%</td></tr><tr><td><i class="c1"></i>luke.encoder.layer</td><td>106.3M</td><td>33.8%</td><td>91.9K</td><td>0.1%</td></tr><tr><td><i class="c2"></i>luke.embeddings</td><td>39.0M</td><td>12.4%</td><td>3.3M</td><td>2.4%</td></tr><tr><td><i class="c3"></i>lm_head.decoder</td><td>38.6M</td><td>12.3%</td><td>3.2M</td><td>2.4%</td></tr><tr><td><i class="c4"></i>lm_head.dense</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c5"></i>luke.pooler</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c6"></i>entity_predictions</td><td>500.0K</td><td>0.2%</td><td>500.0K</td><td>0.4%</td></tr><tr><td><i class="c7"></i>3 others</td><td>249.2K</td><td>0.1%</td><td>67.5K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>luke</td><td>luke</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50267</td><td>50267</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Luke omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>114</td><td>24</td></tr><tr class=""><td>Dropout</td><td>38</td><td>8</td></tr><tr class=""><td>LayerNorm</td><td>28</td><td>8</td></tr><tr class=""><td>GELUActivation</td><td>13</td><td>3</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>CrossEntropyLoss</td><td>1</td><td>1</td></tr><tr class=""><td>EntityPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>EntityPredictionHeadTransform</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>EntityEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>Pooler</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Tanh</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 29 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>studio-ousia/luke-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/luke-base-mic/tree/39d0a41c62c207695ca9fda87542f953dc1d0b56" title="huggingface-mic/luke-base-mic@39d0a41c62c207695ca9fda87542f953dc1d0b56">huggingface-mic/luke-base-mic@39d0a41c62c2</a> β <a href="https://huggingface.co/studio-ousia/luke-base/tree/7438924defd9f3c2018d63c16073bf4bcb6a70aa" title="studio-ousia/luke-base@7438924defd9f3c2018d63c16073bf4bcb6a70aa">studio-ousia/luke-base@7438924defd9</a> Β· LukeForMaskedLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=luke-base-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">314.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">135.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>luke.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.408952" title="luke.entity_embeddings: 128.6M (40.9%)"></i><i class="c1" style="flex:0.338104" title="luke.encoder.layer: 106.3M (33.8%)"></i><i class="c2" style="flex:0.124034" title="luke.embeddings: 39.0M (12.4%)"></i><i class="c3" style="flex:0.122771" title="lm_head.decoder: 38.6M (12.3%)"></i><i class="c4" style="flex:0.001878" title="lm_head.dense: 590.6K (0.2%)"></i><i class="c5" style="flex:0.001878" title="luke.pooler: 590.6K (0.2%)"></i><i class="c6" style="flex:0.001590" title="entity_predictions: 500.0K (0.2%)"></i><i class="c7" style="flex:0.000792" title="3 others: 249.2K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.947220" title="luke.entity_embeddings: 128.0M (94.7%)"></i><i class="c1" style="flex:0.000680" title="luke.encoder.layer: 91.9K (0.1%)"></i><i class="c2" style="flex:0.024043" title="luke.embeddings: 3.3M (2.4%)"></i><i class="c3" style="flex:0.023798" title="lm_head.decoder: 3.2M (2.4%)"></i><i class="c4" style="flex:0.000031" title="lm_head.dense: 4.2K (0.0%)"></i><i class="c5" style="flex:0.000031" title="luke.pooler: 4.2K (0.0%)"></i><i class="c6" style="flex:0.003699" title="entity_predictions: 500.0K (0.4%)"></i><i class="c7" style="flex:0.000500" title="3 others: 67.5K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>luke.entity_embeddings</td><td>128.6M</td><td>40.9%</td><td>128.0M</td><td>94.7%</td></tr><tr><td><i class="c1"></i>luke.encoder.layer</td><td>106.3M</td><td>33.8%</td><td>91.9K</td><td>0.1%</td></tr><tr><td><i class="c2"></i>luke.embeddings</td><td>39.0M</td><td>12.4%</td><td>3.3M</td><td>2.4%</td></tr><tr><td><i class="c3"></i>lm_head.decoder</td><td>38.6M</td><td>12.3%</td><td>3.2M</td><td>2.4%</td></tr><tr><td><i class="c4"></i>lm_head.dense</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c5"></i>luke.pooler</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c6"></i>entity_predictions</td><td>500.0K</td><td>0.2%</td><td>500.0K</td><td>0.4%</td></tr><tr><td><i class="c7"></i>3 others</td><td>249.2K</td><td>0.1%</td><td>67.5K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>luke</td><td>luke</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50267</td><td>50267</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Luke omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>114</td><td>24</td></tr><tr class=""><td>Dropout</td><td>38</td><td>8</td></tr><tr class=""><td>LayerNorm</td><td>28</td><td>8</td></tr><tr class=""><td>GELUActivation</td><td>13</td><td>3</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>CrossEntropyLoss</td><td>1</td><td>1</td></tr><tr class=""><td>EntityPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>EntityPredictionHeadTransform</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>EntityEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>Pooler</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Tanh</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 29 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic/tree/b088660049cbac11aedecd214c7612c0717f466c" title="huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic@b088660049cbac11aedecd214c7612c0717f466c">huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic@b088660049cb</a> β <a href="https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/tree/b5113eb38ab63efdd7f280f8c144ea8b13f978ce" title="MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7@b5113eb38ab63efdd7f280f8c144ea8b13f978ce">MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7@b5113eb38ab6</a> Β· DebertaV2ForSequenceClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">278.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>deberta.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.691397" title="deberta.embeddings: 192.8M (69.1%)"></i><i class="c1" style="flex:0.305061" title="deberta.encoder.layer: 85.1M (30.5%)"></i><i class="c2" style="flex:0.002118" title="pooler.dense: 590.6K (0.2%)"></i><i class="c3" style="flex:0.001416" title="deberta.encoder: 394.8K (0.1%)"></i><i class="c4" style="flex:0.000008" title="classifier: 2.3K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.993556" title="deberta.embeddings: 16.1M (99.4%)"></i><i class="c1" style="flex:0.004140" title="deberta.encoder.layer: 66.9K (0.4%)"></i><i class="c2" style="flex:0.000257" title="pooler.dense: 4.2K (0.0%)"></i><i class="c3" style="flex:0.002035" title="deberta.encoder: 32.9K (0.2%)"></i><i class="c4" style="flex:0.000012" title="classifier: 195 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>deberta.embeddings</td><td>192.8M</td><td>69.1%</td><td>16.1M</td><td>99.4%</td></tr><tr><td><i class="c1"></i>deberta.encoder.layer</td><td>85.1M</td><td>30.5%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c2"></i>pooler.dense</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c3"></i>deberta.encoder</td><td>394.8K</td><td>0.1%</td><td>32.9K</td><td>0.2%</td></tr><tr><td><i class="c4"></i>classifier</td><td>2.3K</td><td>0.0%</td><td>195</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>deberta-v2</td><td>deberta-v2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>251000</td><td>251000</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix DebertaV2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>51</td><td>11</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>DisentangledSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ContextPooler</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForSequenceClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 31 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>pooler_hidden_size</td><td>768</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic/tree/b088660049cbac11aedecd214c7612c0717f466c" title="huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic@b088660049cbac11aedecd214c7612c0717f466c">huggingface-mic/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic@b088660049cb</a> β <a href="https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/tree/b5113eb38ab63efdd7f280f8c144ea8b13f978ce" title="MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7@b5113eb38ab63efdd7f280f8c144ea8b13f978ce">MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7@b5113eb38ab6</a> Β· DebertaV2ForSequenceClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">278.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>deberta.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.691397" title="deberta.embeddings: 192.8M (69.1%)"></i><i class="c1" style="flex:0.305061" title="deberta.encoder.layer: 85.1M (30.5%)"></i><i class="c2" style="flex:0.002118" title="pooler.dense: 590.6K (0.2%)"></i><i class="c3" style="flex:0.001416" title="deberta.encoder: 394.8K (0.1%)"></i><i class="c4" style="flex:0.000008" title="classifier: 2.3K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.993556" title="deberta.embeddings: 16.1M (99.4%)"></i><i class="c1" style="flex:0.004140" title="deberta.encoder.layer: 66.9K (0.4%)"></i><i class="c2" style="flex:0.000257" title="pooler.dense: 4.2K (0.0%)"></i><i class="c3" style="flex:0.002035" title="deberta.encoder: 32.9K (0.2%)"></i><i class="c4" style="flex:0.000012" title="classifier: 195 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>deberta.embeddings</td><td>192.8M</td><td>69.1%</td><td>16.1M</td><td>99.4%</td></tr><tr><td><i class="c1"></i>deberta.encoder.layer</td><td>85.1M</td><td>30.5%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c2"></i>pooler.dense</td><td>590.6K</td><td>0.2%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c3"></i>deberta.encoder</td><td>394.8K</td><td>0.1%</td><td>32.9K</td><td>0.2%</td></tr><tr><td><i class="c4"></i>classifier</td><td>2.3K</td><td>0.0%</td><td>195</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>deberta-v2</td><td>deberta-v2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>251000</td><td>251000</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix DebertaV2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>51</td><td>11</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>DisentangledSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ContextPooler</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForSequenceClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 31 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>pooler_hidden_size</td><td>768</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mamba-130m-hf-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>state-spaces/mamba-130m-hf</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mamba-130m-hf-mic/tree/049c3bff1eb71679fcc5bf5ffcd2c99dc505f11b" title="huggingface-mic/mamba-130m-hf-mic@049c3bff1eb71679fcc5bf5ffcd2c99dc505f11b">huggingface-mic/mamba-130m-hf-mic@049c3bff1eb7</a> β <a href="https://huggingface.co/state-spaces/mamba-130m-hf/tree/1e76775f628fbf1350fbe4dbb3d971ba64af25a1" title="state-spaces/mamba-130m-hf@1e76775f628fbf1350fbe4dbb3d971ba64af25a1">state-spaces/mamba-130m-hf@1e76775f628f</a> Β· MambaForCausalLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">129.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>backbone.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: Block"></i><i class="c0" title="layer 1: Block"></i><i class="c0" title="layer 2: Block"></i><i class="c0" title="layer 3: Block"></i><i class="c0" title="layer 4: Block"></i><i class="c0" title="layer 5: Block"></i><i class="c0" title="layer 6: Block"></i><i class="c0" title="layer 7: Block"></i><i class="c0" title="layer 8: Block"></i><i class="c0" title="layer 9: Block"></i><i class="c0" title="layer 10: Block"></i><i class="c0" title="layer 11: Block"></i><i class="c0" title="layer 12: Block"></i><i class="c0" title="layer 13: Block"></i><i class="c0" title="layer 14: Block"></i><i class="c0" title="layer 15: Block"></i><i class="c0" title="layer 16: Block"></i><i class="c0" title="layer 17: Block"></i><i class="c0" title="layer 18: Block"></i><i class="c0" title="layer 19: Block"></i><i class="c0" title="layer 20: Block"></i><i class="c0" title="layer 21: Block"></i><i class="c0" title="layer 22: Block"></i><i class="c0" title="layer 23: Block"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Block"></i><i class="c0" title="layer 1: Block"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Block</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.700966" title="backbone.layers: 90.5M (70.1%)"></i><i class="c1" style="flex:0.299028" title="backbone.embeddings: 38.6M (29.9%)"></i><i class="c2" style="flex:0.000006" title="backbone.norm_f: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.026600" title="backbone.layers: 87.9K (2.7%)"></i><i class="c1" style="flex:0.973381" title="backbone.embeddings: 3.2M (97.3%)"></i><i class="c2" style="flex:0.000019" title="backbone.norm_f: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>backbone.layers</td><td>90.5M</td><td>70.1%</td><td>87.9K</td><td>2.7%</td></tr><tr><td><i class="c1"></i>backbone.embeddings</td><td>38.6M</td><td>29.9%</td><td>3.2M</td><td>97.3%</td></tr><tr><td><i class="c2"></i>backbone.norm_f</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mamba</td><td>mamba</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50280</td><td>50280</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mamba omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>97</td><td>9</td></tr><tr class=""><td>RMSNorm</td><td>25</td><td>3</td></tr><tr class=""><td>Conv1d</td><td>24</td><td>2</td></tr><tr class=""><td>Block</td><td>24</td><td>2</td></tr><tr class=""><td>Mixer</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>768</td><td>64</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>1536</td><td>128</td></tr><tr><td>n_layer</td><td>24</td><td>2</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>state-spaces/mamba-130m-hf</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mamba-130m-hf-mic/tree/049c3bff1eb71679fcc5bf5ffcd2c99dc505f11b" title="huggingface-mic/mamba-130m-hf-mic@049c3bff1eb71679fcc5bf5ffcd2c99dc505f11b">huggingface-mic/mamba-130m-hf-mic@049c3bff1eb7</a> β <a href="https://huggingface.co/state-spaces/mamba-130m-hf/tree/1e76775f628fbf1350fbe4dbb3d971ba64af25a1" title="state-spaces/mamba-130m-hf@1e76775f628fbf1350fbe4dbb3d971ba64af25a1">state-spaces/mamba-130m-hf@1e76775f628f</a> Β· MambaForCausalLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mamba-130m-hf-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">129.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>backbone.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: Block"></i><i class="c0" title="layer 1: Block"></i><i class="c0" title="layer 2: Block"></i><i class="c0" title="layer 3: Block"></i><i class="c0" title="layer 4: Block"></i><i class="c0" title="layer 5: Block"></i><i class="c0" title="layer 6: Block"></i><i class="c0" title="layer 7: Block"></i><i class="c0" title="layer 8: Block"></i><i class="c0" title="layer 9: Block"></i><i class="c0" title="layer 10: Block"></i><i class="c0" title="layer 11: Block"></i><i class="c0" title="layer 12: Block"></i><i class="c0" title="layer 13: Block"></i><i class="c0" title="layer 14: Block"></i><i class="c0" title="layer 15: Block"></i><i class="c0" title="layer 16: Block"></i><i class="c0" title="layer 17: Block"></i><i class="c0" title="layer 18: Block"></i><i class="c0" title="layer 19: Block"></i><i class="c0" title="layer 20: Block"></i><i class="c0" title="layer 21: Block"></i><i class="c0" title="layer 22: Block"></i><i class="c0" title="layer 23: Block"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Block"></i><i class="c0" title="layer 1: Block"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Block</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.700966" title="backbone.layers: 90.5M (70.1%)"></i><i class="c1" style="flex:0.299028" title="backbone.embeddings: 38.6M (29.9%)"></i><i class="c2" style="flex:0.000006" title="backbone.norm_f: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.026600" title="backbone.layers: 87.9K (2.7%)"></i><i class="c1" style="flex:0.973381" title="backbone.embeddings: 3.2M (97.3%)"></i><i class="c2" style="flex:0.000019" title="backbone.norm_f: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>backbone.layers</td><td>90.5M</td><td>70.1%</td><td>87.9K</td><td>2.7%</td></tr><tr><td><i class="c1"></i>backbone.embeddings</td><td>38.6M</td><td>29.9%</td><td>3.2M</td><td>97.3%</td></tr><tr><td><i class="c2"></i>backbone.norm_f</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mamba</td><td>mamba</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50280</td><td>50280</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mamba omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>97</td><td>9</td></tr><tr class=""><td>RMSNorm</td><td>25</td><td>3</td></tr><tr class=""><td>Conv1d</td><td>24</td><td>2</td></tr><tr class=""><td>Block</td><td>24</td><td>2</td></tr><tr class=""><td>Mixer</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>768</td><td>64</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>1536</td><td>128</td></tr><tr><td>n_layer</td><td>24</td><td>2</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mask2former-swin-tiny-coco-instance-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mask2former-swin-tiny-coco-instance</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mask2former-swin-tiny-coco-instance-mic/tree/851c4b7fb35cc3ef537b77c5ad1164f3f1747f3a" title="huggingface-mic/mask2former-swin-tiny-coco-instance-mic@851c4b7fb35cc3ef537b77c5ad1164f3f1747f3a">huggingface-mic/mask2former-swin-tiny-coco-instance-mic@851c4b7fb35c</a> β <a href="https://huggingface.co/facebook/mask2former-swin-tiny-coco-instance/tree/22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6" title="facebook/mask2former-swin-tiny-coco-instance@22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6">facebook/mask2former-swin-tiny-coco-instance@22c4a2f15dc8</a> Β· Mask2FormerForUniversalSegmentation Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">47.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">8.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">4</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.pixel_level_module.encoder.swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SwinPatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no SwinPatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.pixel_level_module.decoder.input_projections</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.pixel_level_module.decoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 1: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 2: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 3: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 4: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 5: FormerPixelDecoderEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 1: FormerPixelDecoderEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FormerPixelDecoderEncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.transformer_module.decoder.layers</h3><div class="row"><span class="lab">Original Β· 9</span><div class="cells"><i class="c0" title="layer 0: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 1: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 2: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 3: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 4: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 5: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 6: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 7: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 8: FormerMaskedAttentionDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: FormerMaskedAttentionDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FormerMaskedAttentionDecoderLayer</td><td>9</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.580158" title="model.pixel_level_module.encoder.swin.encoder.layers: 27.5M (58.0%)"></i><i class="c1" style="flex:0.299617" title="model.transformer_module.decoder.layers: 14.2M (30.0%)"></i><i class="c2" style="flex:0.092637" title="model.pixel_level_module.decoder.encoder.layers: 4.4M (9.3%)"></i><i class="c3" style="flex:0.014577" title="model.pixel_level_module: 691.3K (1.5%)"></i><i class="c4" style="flex:0.007304" title="model.pixel_level_module.decoder.input_projections: 346.4K (0.7%)"></i><i class="c5" style="flex:0.005269" title="model.transformer_module: 249.9K (0.5%)"></i><i class="c6" style="flex:0.000439" title="class_predictor: 20.8K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.607105" title="model.pixel_level_module.encoder.swin.encoder.layers: 4.9M (60.7%)"></i><i class="c1" style="flex:0.081904" title="model.transformer_module.decoder.layers: 659.3K (8.2%)"></i><i class="c2" style="flex:0.172717" title="model.pixel_level_module.decoder.encoder.layers: 1.4M (17.3%)"></i><i class="c3" style="flex:0.084472" title="model.pixel_level_module: 680.0K (8.4%)"></i><i class="c4" style="flex:0.028780" title="model.pixel_level_module.decoder.input_projections: 231.7K (2.9%)"></i><i class="c5" style="flex:0.023724" title="model.transformer_module: 191.0K (2.4%)"></i><i class="c6" style="flex:0.001298" title="class_predictor: 10.4K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.pixel_level_module.encoder.swin.encoder.layers</td><td>27.5M</td><td>58.0%</td><td>4.9M</td><td>60.7%</td></tr><tr><td><i class="c1"></i>model.transformer_module.decoder.layers</td><td>14.2M</td><td>30.0%</td><td>659.3K</td><td>8.2%</td></tr><tr><td><i class="c2"></i>model.pixel_level_module.decoder.encoder.layers</td><td>4.4M</td><td>9.3%</td><td>1.4M</td><td>17.3%</td></tr><tr><td><i class="c3"></i>model.pixel_level_module</td><td>691.3K</td><td>1.5%</td><td>680.0K</td><td>8.4%</td></tr><tr><td><i class="c4"></i>model.pixel_level_module.decoder.input_projections</td><td>346.4K</td><td>0.7%</td><td>231.7K</td><td>2.9%</td></tr><tr><td><i class="c5"></i>model.transformer_module</td><td>249.9K</td><td>0.5%</td><td>191.0K</td><td>2.4%</td></tr><tr><td><i class="c6"></i>class_predictor</td><td>20.8K</td><td>0.0%</td><td>10.4K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mask2former</td><td>mask2former</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mask2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>49</td></tr><tr class=""><td>LayerNorm</td><td>73</td><td>25</td></tr><tr class=""><td>Dropout</td><td>13</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>4</td></tr><tr class=""><td>ReLU</td><td>12</td><td>4</td></tr><tr class=""><td>SwinAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinLayer</td><td>12</td><td>4</td></tr><tr class=""><td>SwinMLP</td><td>12</td><td>4</td></tr><tr class=""><td>SwinRelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>SwinDropPath</td><td>11</td><td>3</td></tr><tr class=""><td>FormerAttention</td><td>9</td><td>1</td></tr><tr class=""><td>FormerMaskedAttentionDecoderLayer</td><td>9</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>9</td><td>9</td></tr><tr class=""><td>MultiheadAttention</td><td>9</td><td>1</td></tr><tr class=""><td>NonDynamicallyQuantizableLinear</td><td>9</td><td>1</td></tr><tr class=""><td>Sequential</td><td>8</td><td>5</td></tr><tr class=""><td>Conv2d</td><td>7</td><td>10</td></tr><tr class=""><td>FormerPixelDecoderEncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FormerPixelDecoderEncoderMultiscaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>GroupNorm</td><td>5</td><td>5</td></tr><tr class=""><td>SwinStage</td><td>4</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>FormerPredictionBlock</td><td>3</td><td>3</td></tr><tr class=""><td>SwinPatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>Identity</td><td>2</td><td>2</td></tr><tr class=""><td>FormerSinePositionEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>FormerForUniversalSegmentation</td><td>1</td><td>1</td></tr><tr class=""><td>FormerHungarianMatcher</td><td>1</td><td>1</td></tr><tr class=""><td>FormerLoss</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMLPPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMaskPredictor</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMaskedAttentionDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>FormerModel</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelDecoderEncoderOnly</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelLevelModule</td><td>1</td><td>1</td></tr><tr class=""><td>FormerTransformerModule</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>SwinBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SwinModel</td><td>1</td><td>1</td></tr><tr class=""><td>SwinPatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 78 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.embed_dim</td><td>96</td><td>64</td></tr><tr><td>backbone_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>backbone_config.num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>backbone_config.num_layers</td><td>4</td><td>2</td></tr><tr><td>decoder_layers</td><td>10</td><td>2</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr><tr><td>hidden_dim</td><td>256</td><td>128</td></tr><tr><td>num_attention_heads</td><td>8</td><td>4</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mask2former-swin-tiny-coco-instance</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mask2former-swin-tiny-coco-instance-mic/tree/851c4b7fb35cc3ef537b77c5ad1164f3f1747f3a" title="huggingface-mic/mask2former-swin-tiny-coco-instance-mic@851c4b7fb35cc3ef537b77c5ad1164f3f1747f3a">huggingface-mic/mask2former-swin-tiny-coco-instance-mic@851c4b7fb35c</a> β <a href="https://huggingface.co/facebook/mask2former-swin-tiny-coco-instance/tree/22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6" title="facebook/mask2former-swin-tiny-coco-instance@22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6">facebook/mask2former-swin-tiny-coco-instance@22c4a2f15dc8</a> Β· Mask2FormerForUniversalSegmentation Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mask2former-swin-tiny-coco-instance-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">47.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">8.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">4</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.pixel_level_module.encoder.swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SwinPatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no SwinPatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.pixel_level_module.decoder.input_projections</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.pixel_level_module.decoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 1: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 2: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 3: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 4: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 5: FormerPixelDecoderEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: FormerPixelDecoderEncoderLayer"></i><i class="c0" title="layer 1: FormerPixelDecoderEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FormerPixelDecoderEncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.transformer_module.decoder.layers</h3><div class="row"><span class="lab">Original Β· 9</span><div class="cells"><i class="c0" title="layer 0: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 1: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 2: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 3: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 4: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 5: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 6: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 7: FormerMaskedAttentionDecoderLayer"></i><i class="c0" title="layer 8: FormerMaskedAttentionDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: FormerMaskedAttentionDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FormerMaskedAttentionDecoderLayer</td><td>9</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.580158" title="model.pixel_level_module.encoder.swin.encoder.layers: 27.5M (58.0%)"></i><i class="c1" style="flex:0.299617" title="model.transformer_module.decoder.layers: 14.2M (30.0%)"></i><i class="c2" style="flex:0.092637" title="model.pixel_level_module.decoder.encoder.layers: 4.4M (9.3%)"></i><i class="c3" style="flex:0.014577" title="model.pixel_level_module: 691.3K (1.5%)"></i><i class="c4" style="flex:0.007304" title="model.pixel_level_module.decoder.input_projections: 346.4K (0.7%)"></i><i class="c5" style="flex:0.005269" title="model.transformer_module: 249.9K (0.5%)"></i><i class="c6" style="flex:0.000439" title="class_predictor: 20.8K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.607105" title="model.pixel_level_module.encoder.swin.encoder.layers: 4.9M (60.7%)"></i><i class="c1" style="flex:0.081904" title="model.transformer_module.decoder.layers: 659.3K (8.2%)"></i><i class="c2" style="flex:0.172717" title="model.pixel_level_module.decoder.encoder.layers: 1.4M (17.3%)"></i><i class="c3" style="flex:0.084472" title="model.pixel_level_module: 680.0K (8.4%)"></i><i class="c4" style="flex:0.028780" title="model.pixel_level_module.decoder.input_projections: 231.7K (2.9%)"></i><i class="c5" style="flex:0.023724" title="model.transformer_module: 191.0K (2.4%)"></i><i class="c6" style="flex:0.001298" title="class_predictor: 10.4K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.pixel_level_module.encoder.swin.encoder.layers</td><td>27.5M</td><td>58.0%</td><td>4.9M</td><td>60.7%</td></tr><tr><td><i class="c1"></i>model.transformer_module.decoder.layers</td><td>14.2M</td><td>30.0%</td><td>659.3K</td><td>8.2%</td></tr><tr><td><i class="c2"></i>model.pixel_level_module.decoder.encoder.layers</td><td>4.4M</td><td>9.3%</td><td>1.4M</td><td>17.3%</td></tr><tr><td><i class="c3"></i>model.pixel_level_module</td><td>691.3K</td><td>1.5%</td><td>680.0K</td><td>8.4%</td></tr><tr><td><i class="c4"></i>model.pixel_level_module.decoder.input_projections</td><td>346.4K</td><td>0.7%</td><td>231.7K</td><td>2.9%</td></tr><tr><td><i class="c5"></i>model.transformer_module</td><td>249.9K</td><td>0.5%</td><td>191.0K</td><td>2.4%</td></tr><tr><td><i class="c6"></i>class_predictor</td><td>20.8K</td><td>0.0%</td><td>10.4K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mask2former</td><td>mask2former</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mask2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>49</td></tr><tr class=""><td>LayerNorm</td><td>73</td><td>25</td></tr><tr class=""><td>Dropout</td><td>13</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>4</td></tr><tr class=""><td>ReLU</td><td>12</td><td>4</td></tr><tr class=""><td>SwinAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinLayer</td><td>12</td><td>4</td></tr><tr class=""><td>SwinMLP</td><td>12</td><td>4</td></tr><tr class=""><td>SwinRelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>SwinDropPath</td><td>11</td><td>3</td></tr><tr class=""><td>FormerAttention</td><td>9</td><td>1</td></tr><tr class=""><td>FormerMaskedAttentionDecoderLayer</td><td>9</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>9</td><td>9</td></tr><tr class=""><td>MultiheadAttention</td><td>9</td><td>1</td></tr><tr class=""><td>NonDynamicallyQuantizableLinear</td><td>9</td><td>1</td></tr><tr class=""><td>Sequential</td><td>8</td><td>5</td></tr><tr class=""><td>Conv2d</td><td>7</td><td>10</td></tr><tr class=""><td>FormerPixelDecoderEncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FormerPixelDecoderEncoderMultiscaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>GroupNorm</td><td>5</td><td>5</td></tr><tr class=""><td>SwinStage</td><td>4</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>FormerPredictionBlock</td><td>3</td><td>3</td></tr><tr class=""><td>SwinPatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>Identity</td><td>2</td><td>2</td></tr><tr class=""><td>FormerSinePositionEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>FormerForUniversalSegmentation</td><td>1</td><td>1</td></tr><tr class=""><td>FormerHungarianMatcher</td><td>1</td><td>1</td></tr><tr class=""><td>FormerLoss</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMLPPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMaskPredictor</td><td>1</td><td>1</td></tr><tr class=""><td>FormerMaskedAttentionDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>FormerModel</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelDecoderEncoderOnly</td><td>1</td><td>1</td></tr><tr class=""><td>FormerPixelLevelModule</td><td>1</td><td>1</td></tr><tr class=""><td>FormerTransformerModule</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>SwinBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SwinModel</td><td>1</td><td>1</td></tr><tr class=""><td>SwinPatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 78 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.embed_dim</td><td>96</td><td>64</td></tr><tr><td>backbone_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>backbone_config.num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>backbone_config.num_layers</td><td>4</td><td>2</td></tr><tr><td>decoder_layers</td><td>10</td><td>2</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr><tr><td>hidden_dim</td><td>256</td><td>128</td></tr><tr><td>num_attention_heads</td><td>8</td><td>4</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mbart-large-50-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mbart-large-50</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mbart-large-50-mic/tree/ef21bcc00bb0aac630f0218f55b5c88b59a01fb5" title="huggingface-mic/mbart-large-50-mic@ef21bcc00bb0aac630f0218f55b5c88b59a01fb5">huggingface-mic/mbart-large-50-mic@ef21bcc00bb0</a> β <a href="https://huggingface.co/facebook/mbart-large-50/tree/4ef55a20b36c6903b832e38f0604ab4bdf22c7d6" title="facebook/mbart-large-50@4ef55a20b36c6903b832e38f0604ab4bdf22c7d6">facebook/mbart-large-50@4ef55a20b36c</a> Β· MBartForConditionalGeneration Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">610.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.419158" title="model.shared: 256.1M (41.9%)"></i><i class="c1" style="flex:0.329951" title="model.decoder.layers: 201.6M (33.0%)"></i><i class="c2" style="flex:0.247438" title="model.encoder.layers: 151.2M (24.7%)"></i><i class="c3" style="flex:0.001727" title="model.encoder: 1.1M (0.2%)"></i><i class="c4" style="flex:0.001727" title="model.decoder: 1.1M (0.2%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.981643" title="model.shared: 16.0M (98.2%)"></i><i class="c1" style="flex:0.006163" title="model.decoder.layers: 100.5K (0.6%)"></i><i class="c2" style="flex:0.004106" title="model.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.004043" title="model.encoder: 65.9K (0.4%)"></i><i class="c4" style="flex:0.004043" title="model.decoder: 65.9K (0.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>256.1M</td><td>41.9%</td><td>16.0M</td><td>98.2%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>201.6M</td><td>33.0%</td><td>100.5K</td><td>0.6%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>151.2M</td><td>24.7%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>1.1M</td><td>0.2%</td><td>65.9K</td><td>0.4%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>1.1M</td><td>0.2%</td><td>65.9K</td><td>0.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mbart</td><td>mbart</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>250054</td><td>250054</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>1024</td><td>1024</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MBart omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>64</td><td>14</td></tr><tr class=""><td>Attention</td><td>36</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>3</td><td>3</td></tr><tr class=""><td>LearnedPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>12</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mbart-large-50</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mbart-large-50-mic/tree/ef21bcc00bb0aac630f0218f55b5c88b59a01fb5" title="huggingface-mic/mbart-large-50-mic@ef21bcc00bb0aac630f0218f55b5c88b59a01fb5">huggingface-mic/mbart-large-50-mic@ef21bcc00bb0</a> β <a href="https://huggingface.co/facebook/mbart-large-50/tree/4ef55a20b36c6903b832e38f0604ab4bdf22c7d6" title="facebook/mbart-large-50@4ef55a20b36c6903b832e38f0604ab4bdf22c7d6">facebook/mbart-large-50@4ef55a20b36c</a> Β· MBartForConditionalGeneration Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mbart-large-50-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">610.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.419158" title="model.shared: 256.1M (41.9%)"></i><i class="c1" style="flex:0.329951" title="model.decoder.layers: 201.6M (33.0%)"></i><i class="c2" style="flex:0.247438" title="model.encoder.layers: 151.2M (24.7%)"></i><i class="c3" style="flex:0.001727" title="model.encoder: 1.1M (0.2%)"></i><i class="c4" style="flex:0.001727" title="model.decoder: 1.1M (0.2%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.981643" title="model.shared: 16.0M (98.2%)"></i><i class="c1" style="flex:0.006163" title="model.decoder.layers: 100.5K (0.6%)"></i><i class="c2" style="flex:0.004106" title="model.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.004043" title="model.encoder: 65.9K (0.4%)"></i><i class="c4" style="flex:0.004043" title="model.decoder: 65.9K (0.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>256.1M</td><td>41.9%</td><td>16.0M</td><td>98.2%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>201.6M</td><td>33.0%</td><td>100.5K</td><td>0.6%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>151.2M</td><td>24.7%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>1.1M</td><td>0.2%</td><td>65.9K</td><td>0.4%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>1.1M</td><td>0.2%</td><td>65.9K</td><td>0.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mbart</td><td>mbart</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>250054</td><td>250054</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>1024</td><td>1024</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MBart omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>64</td><td>14</td></tr><tr class=""><td>Attention</td><td>36</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>3</td><td>3</td></tr><tr class=""><td>LearnedPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>12</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mistral3-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>mistralai/Mistral-Small-3.1-24B-Instruct-2503</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mistral3-mic/tree/ad30ba063d8bf5a68e71a54668e8eb4a8ebd2a94" title="huggingface-mic/mistral3-mic@ad30ba063d8bf5a68e71a54668e8eb4a8ebd2a94">huggingface-mic/mistral3-mic@ad30ba063d8b</a> β <a href="https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/tree/68faf511d618ef198fef186659617cfd2eb8e33a" title="mistralai/Mistral-Small-3.1-24B-Instruct-2503@68faf511d618ef198fef186659617cfd2eb8e33a">mistralai/Mistral-Small-3.1-24B-Instruct-2503@68faf511d618</a> Β· Mistral3ForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">23.3B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">8.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.transformer.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: PixtralAttentionLayer"></i><i class="c0" title="layer 1: PixtralAttentionLayer"></i><i class="c0" title="layer 2: PixtralAttentionLayer"></i><i class="c0" title="layer 3: PixtralAttentionLayer"></i><i class="c0" title="layer 4: PixtralAttentionLayer"></i><i class="c0" title="layer 5: PixtralAttentionLayer"></i><i class="c0" title="layer 6: PixtralAttentionLayer"></i><i class="c0" title="layer 7: PixtralAttentionLayer"></i><i class="c0" title="layer 8: PixtralAttentionLayer"></i><i class="c0" title="layer 9: PixtralAttentionLayer"></i><i class="c0" title="layer 10: PixtralAttentionLayer"></i><i class="c0" title="layer 11: PixtralAttentionLayer"></i><i class="c0" title="layer 12: PixtralAttentionLayer"></i><i class="c0" title="layer 13: PixtralAttentionLayer"></i><i class="c0" title="layer 14: PixtralAttentionLayer"></i><i class="c0" title="layer 15: PixtralAttentionLayer"></i><i class="c0" title="layer 16: PixtralAttentionLayer"></i><i class="c0" title="layer 17: PixtralAttentionLayer"></i><i class="c0" title="layer 18: PixtralAttentionLayer"></i><i class="c0" title="layer 19: PixtralAttentionLayer"></i><i class="c0" title="layer 20: PixtralAttentionLayer"></i><i class="c0" title="layer 21: PixtralAttentionLayer"></i><i class="c0" title="layer 22: PixtralAttentionLayer"></i><i class="c0" title="layer 23: PixtralAttentionLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: PixtralAttentionLayer"></i><i class="c0" title="layer 1: PixtralAttentionLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>PixtralAttentionLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 40</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i><i class="c0" title="layer 2: MistralDecoderLayer"></i><i class="c0" title="layer 3: MistralDecoderLayer"></i><i class="c0" title="layer 4: MistralDecoderLayer"></i><i class="c0" title="layer 5: MistralDecoderLayer"></i><i class="c0" title="layer 6: MistralDecoderLayer"></i><i class="c0" title="layer 7: MistralDecoderLayer"></i><i class="c0" title="layer 8: MistralDecoderLayer"></i><i class="c0" title="layer 9: MistralDecoderLayer"></i><i class="c0" title="layer 10: MistralDecoderLayer"></i><i class="c0" title="layer 11: MistralDecoderLayer"></i><i class="c0" title="layer 12: MistralDecoderLayer"></i><i class="c0" title="layer 13: MistralDecoderLayer"></i><i class="c0" title="layer 14: MistralDecoderLayer"></i><i class="c0" title="layer 15: MistralDecoderLayer"></i><i class="c0" title="layer 16: MistralDecoderLayer"></i><i class="c0" title="layer 17: MistralDecoderLayer"></i><i class="c0" title="layer 18: MistralDecoderLayer"></i><i class="c0" title="layer 19: MistralDecoderLayer"></i><i class="c0" title="layer 20: MistralDecoderLayer"></i><i class="c0" title="layer 21: MistralDecoderLayer"></i><i class="c0" title="layer 22: MistralDecoderLayer"></i><i class="c0" title="layer 23: MistralDecoderLayer"></i><i class="c0" title="layer 24: MistralDecoderLayer"></i><i class="c0" title="layer 25: MistralDecoderLayer"></i><i class="c0" title="layer 26: MistralDecoderLayer"></i><i class="c0" title="layer 27: MistralDecoderLayer"></i><i class="c0" title="layer 28: MistralDecoderLayer"></i><i class="c0" title="layer 29: MistralDecoderLayer"></i><i class="c0" title="layer 30: MistralDecoderLayer"></i><i class="c0" title="layer 31: MistralDecoderLayer"></i><i class="c0" title="layer 32: MistralDecoderLayer"></i><i class="c0" title="layer 33: MistralDecoderLayer"></i><i class="c0" title="layer 34: MistralDecoderLayer"></i><i class="c0" title="layer 35: MistralDecoderLayer"></i><i class="c0" title="layer 36: MistralDecoderLayer"></i><i class="c0" title="layer 37: MistralDecoderLayer"></i><i class="c0" title="layer 38: MistralDecoderLayer"></i><i class="c0" title="layer 39: MistralDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MistralDecoderLayer</td><td>40</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.952440" title="model.language_model.layers: 22.2B (95.2%)"></i><i class="c1" style="flex:0.028753" title="model.language_model: 671.1M (2.9%)"></i><i class="c2" style="flex:0.017254" title="model.vision_tower.transformer.layers: 402.7M (1.7%)"></i><i class="c3" style="flex:0.001528" title="model.multi_modal_projector: 35.7M (0.2%)"></i><i class="c4" style="flex:0.000026" title="model.vision_tower: 603.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.008596" title="model.language_model.layers: 74.0K (0.9%)"></i><i class="c1" style="flex:0.974615" title="model.language_model: 8.4M (97.5%)"></i><i class="c2" style="flex:0.009547" title="model.vision_tower.transformer.layers: 82.2K (1.0%)"></i><i class="c3" style="flex:0.002863" title="model.multi_modal_projector: 24.6K (0.3%)"></i><i class="c4" style="flex:0.004380" title="model.vision_tower: 37.7K (0.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>22.2B</td><td>95.2%</td><td>74.0K</td><td>0.9%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>671.1M</td><td>2.9%</td><td>8.4M</td><td>97.5%</td></tr><tr><td><i class="c2"></i>model.vision_tower.transformer.layers</td><td>402.7M</td><td>1.7%</td><td>82.2K</td><td>1.0%</td></tr><tr><td><i class="c3"></i>model.multi_modal_projector</td><td>35.7M</td><td>0.2%</td><td>24.6K</td><td>0.3%</td></tr><tr><td><i class="c4"></i>model.vision_tower</td><td>603.1K</td><td>0.0%</td><td>37.7K</td><td>0.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mistral3</td><td>mistral3</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>pixtral</td><td>pixtral</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>1540</td><td>1540</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (32 heads, 8 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>mistral</td><td>mistral</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>131072</td><td>131072</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mistral3 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>452</td><td>32</td></tr><tr class=""><td>MistralRMSNorm</td><td>81</td><td>5</td></tr><tr class=""><td>SiLUActivation</td><td>64</td><td>4</td></tr><tr class=""><td>PixtralRMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>MistralAttention</td><td>40</td><td>2</td></tr><tr class=""><td>MistralDecoderLayer</td><td>40</td><td>2</td></tr><tr class=""><td>MistralMLP</td><td>40</td><td>2</td></tr><tr class=""><td>PixtralAttention</td><td>24</td><td>2</td></tr><tr class=""><td>PixtralAttentionLayer</td><td>24</td><td>2</td></tr><tr class=""><td>PixtralMLP</td><td>24</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>RMSNorm</td><td>1</td><td>1</td></tr><tr class=""><td>MistralModel</td><td>1</td><td>1</td></tr><tr class=""><td>MistralRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralVisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralVisionRotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">11 fields changed, 53 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>5120</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>32768</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>40</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>8</td><td>2</td></tr><tr><td>vision_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>vision_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4096</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>mistralai/Mistral-Small-3.1-24B-Instruct-2503</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mistral3-mic/tree/ad30ba063d8bf5a68e71a54668e8eb4a8ebd2a94" title="huggingface-mic/mistral3-mic@ad30ba063d8bf5a68e71a54668e8eb4a8ebd2a94">huggingface-mic/mistral3-mic@ad30ba063d8b</a> β <a href="https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/tree/68faf511d618ef198fef186659617cfd2eb8e33a" title="mistralai/Mistral-Small-3.1-24B-Instruct-2503@68faf511d618ef198fef186659617cfd2eb8e33a">mistralai/Mistral-Small-3.1-24B-Instruct-2503@68faf511d618</a> Β· Mistral3ForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mistral3-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 24 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">23.3B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">8.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.transformer.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: PixtralAttentionLayer"></i><i class="c0" title="layer 1: PixtralAttentionLayer"></i><i class="c0" title="layer 2: PixtralAttentionLayer"></i><i class="c0" title="layer 3: PixtralAttentionLayer"></i><i class="c0" title="layer 4: PixtralAttentionLayer"></i><i class="c0" title="layer 5: PixtralAttentionLayer"></i><i class="c0" title="layer 6: PixtralAttentionLayer"></i><i class="c0" title="layer 7: PixtralAttentionLayer"></i><i class="c0" title="layer 8: PixtralAttentionLayer"></i><i class="c0" title="layer 9: PixtralAttentionLayer"></i><i class="c0" title="layer 10: PixtralAttentionLayer"></i><i class="c0" title="layer 11: PixtralAttentionLayer"></i><i class="c0" title="layer 12: PixtralAttentionLayer"></i><i class="c0" title="layer 13: PixtralAttentionLayer"></i><i class="c0" title="layer 14: PixtralAttentionLayer"></i><i class="c0" title="layer 15: PixtralAttentionLayer"></i><i class="c0" title="layer 16: PixtralAttentionLayer"></i><i class="c0" title="layer 17: PixtralAttentionLayer"></i><i class="c0" title="layer 18: PixtralAttentionLayer"></i><i class="c0" title="layer 19: PixtralAttentionLayer"></i><i class="c0" title="layer 20: PixtralAttentionLayer"></i><i class="c0" title="layer 21: PixtralAttentionLayer"></i><i class="c0" title="layer 22: PixtralAttentionLayer"></i><i class="c0" title="layer 23: PixtralAttentionLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: PixtralAttentionLayer"></i><i class="c0" title="layer 1: PixtralAttentionLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>PixtralAttentionLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 40</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i><i class="c0" title="layer 2: MistralDecoderLayer"></i><i class="c0" title="layer 3: MistralDecoderLayer"></i><i class="c0" title="layer 4: MistralDecoderLayer"></i><i class="c0" title="layer 5: MistralDecoderLayer"></i><i class="c0" title="layer 6: MistralDecoderLayer"></i><i class="c0" title="layer 7: MistralDecoderLayer"></i><i class="c0" title="layer 8: MistralDecoderLayer"></i><i class="c0" title="layer 9: MistralDecoderLayer"></i><i class="c0" title="layer 10: MistralDecoderLayer"></i><i class="c0" title="layer 11: MistralDecoderLayer"></i><i class="c0" title="layer 12: MistralDecoderLayer"></i><i class="c0" title="layer 13: MistralDecoderLayer"></i><i class="c0" title="layer 14: MistralDecoderLayer"></i><i class="c0" title="layer 15: MistralDecoderLayer"></i><i class="c0" title="layer 16: MistralDecoderLayer"></i><i class="c0" title="layer 17: MistralDecoderLayer"></i><i class="c0" title="layer 18: MistralDecoderLayer"></i><i class="c0" title="layer 19: MistralDecoderLayer"></i><i class="c0" title="layer 20: MistralDecoderLayer"></i><i class="c0" title="layer 21: MistralDecoderLayer"></i><i class="c0" title="layer 22: MistralDecoderLayer"></i><i class="c0" title="layer 23: MistralDecoderLayer"></i><i class="c0" title="layer 24: MistralDecoderLayer"></i><i class="c0" title="layer 25: MistralDecoderLayer"></i><i class="c0" title="layer 26: MistralDecoderLayer"></i><i class="c0" title="layer 27: MistralDecoderLayer"></i><i class="c0" title="layer 28: MistralDecoderLayer"></i><i class="c0" title="layer 29: MistralDecoderLayer"></i><i class="c0" title="layer 30: MistralDecoderLayer"></i><i class="c0" title="layer 31: MistralDecoderLayer"></i><i class="c0" title="layer 32: MistralDecoderLayer"></i><i class="c0" title="layer 33: MistralDecoderLayer"></i><i class="c0" title="layer 34: MistralDecoderLayer"></i><i class="c0" title="layer 35: MistralDecoderLayer"></i><i class="c0" title="layer 36: MistralDecoderLayer"></i><i class="c0" title="layer 37: MistralDecoderLayer"></i><i class="c0" title="layer 38: MistralDecoderLayer"></i><i class="c0" title="layer 39: MistralDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MistralDecoderLayer"></i><i class="c0" title="layer 1: MistralDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MistralDecoderLayer</td><td>40</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.952440" title="model.language_model.layers: 22.2B (95.2%)"></i><i class="c1" style="flex:0.028753" title="model.language_model: 671.1M (2.9%)"></i><i class="c2" style="flex:0.017254" title="model.vision_tower.transformer.layers: 402.7M (1.7%)"></i><i class="c3" style="flex:0.001528" title="model.multi_modal_projector: 35.7M (0.2%)"></i><i class="c4" style="flex:0.000026" title="model.vision_tower: 603.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.008596" title="model.language_model.layers: 74.0K (0.9%)"></i><i class="c1" style="flex:0.974615" title="model.language_model: 8.4M (97.5%)"></i><i class="c2" style="flex:0.009547" title="model.vision_tower.transformer.layers: 82.2K (1.0%)"></i><i class="c3" style="flex:0.002863" title="model.multi_modal_projector: 24.6K (0.3%)"></i><i class="c4" style="flex:0.004380" title="model.vision_tower: 37.7K (0.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>22.2B</td><td>95.2%</td><td>74.0K</td><td>0.9%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>671.1M</td><td>2.9%</td><td>8.4M</td><td>97.5%</td></tr><tr><td><i class="c2"></i>model.vision_tower.transformer.layers</td><td>402.7M</td><td>1.7%</td><td>82.2K</td><td>1.0%</td></tr><tr><td><i class="c3"></i>model.multi_modal_projector</td><td>35.7M</td><td>0.2%</td><td>24.6K</td><td>0.3%</td></tr><tr><td><i class="c4"></i>model.vision_tower</td><td>603.1K</td><td>0.0%</td><td>37.7K</td><td>0.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mistral3</td><td>mistral3</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>pixtral</td><td>pixtral</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>1540</td><td>1540</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (32 heads, 8 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>mistral</td><td>mistral</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>131072</td><td>131072</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Mistral3 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>452</td><td>32</td></tr><tr class=""><td>MistralRMSNorm</td><td>81</td><td>5</td></tr><tr class=""><td>SiLUActivation</td><td>64</td><td>4</td></tr><tr class=""><td>PixtralRMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>MistralAttention</td><td>40</td><td>2</td></tr><tr class=""><td>MistralDecoderLayer</td><td>40</td><td>2</td></tr><tr class=""><td>MistralMLP</td><td>40</td><td>2</td></tr><tr class=""><td>PixtralAttention</td><td>24</td><td>2</td></tr><tr class=""><td>PixtralAttentionLayer</td><td>24</td><td>2</td></tr><tr class=""><td>PixtralMLP</td><td>24</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>RMSNorm</td><td>1</td><td>1</td></tr><tr class=""><td>MistralModel</td><td>1</td><td>1</td></tr><tr class=""><td>MistralRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralVisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>PixtralVisionRotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">11 fields changed, 53 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>5120</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>32768</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>40</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>8</td><td>2</td></tr><tr><td>vision_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>vision_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4096</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic/tree/3b6c70187a273832016dad164af1af9a796a0bbd" title="huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic@3b6c70187a273832016dad164af1af9a796a0bbd">huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic@3b6c70187a27</a> β <a href="https://huggingface.co/openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det/tree/ef2053aa1e0cc9d9cee1df31ca919ee9aae4fc8e" title="openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det@ef2053aa1e0cc9d9cee1df31ca919ee9aae4fc8e">openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det@ef2053aa1e0c</a> Β· MMGroundingDinoForObjectDetection Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">172.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">7.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">7</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.backbone.conv_encoder.model.swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SwinPatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no SwinPatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.input_proj_vision</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.text_backbone.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: BertLayer"></i><i class="c0" title="layer 1: BertLayer"></i><i class="c0" title="layer 2: BertLayer"></i><i class="c0" title="layer 3: BertLayer"></i><i class="c0" title="layer 4: BertLayer"></i><i class="c0" title="layer 5: BertLayer"></i><i class="c0" title="layer 6: BertLayer"></i><i class="c0" title="layer 7: BertLayer"></i><i class="c0" title="layer 8: BertLayer"></i><i class="c0" title="layer 9: BertLayer"></i><i class="c0" title="layer 10: BertLayer"></i><i class="c0" title="layer 11: BertLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: BertLayer"></i><i class="c0" title="layer 1: BertLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>BertLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: FusionLayer"></i><i class="c0" title="layer 1: FusionLayer"></i><i class="c0" title="layer 2: FusionLayer"></i><i class="c0" title="layer 3: FusionLayer"></i><i class="c0" title="layer 4: FusionLayer"></i><i class="c0" title="layer 5: FusionLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: FusionLayer"></i><i class="c0" title="layer 1: FusionLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FusionLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.491890" title="model.text_backbone.encoder.layer: 85.1M (49.2%)"></i><i class="c1" style="flex:0.159114" title="model.backbone.conv_encoder.model.swin.encoder.layers: 27.5M (15.9%)"></i><i class="c2" style="flex:0.137856" title="model.text_backbone: 23.8M (13.8%)"></i><i class="c3" style="flex:0.126686" title="model.encoder.layers: 21.9M (12.7%)"></i><i class="c4" style="flex:0.062790" title="model.decoder.layers: 10.9M (6.3%)"></i><i class="c5" style="flex:0.012241" title="model.input_proj_vision: 2.1M (1.2%)"></i><i class="c6" style="flex:0.004602" title="model.decoder.bbox_embed: 795.7K (0.5%)"></i><i class="c7" style="flex:0.004823" title="9 others: 834.0K (0.5%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.008668" title="model.text_backbone.encoder.layer: 66.9K (0.9%)"></i><i class="c1" style="flex:0.632811" title="model.backbone.conv_encoder.model.swin.encoder.layers: 4.9M (63.3%)"></i><i class="c2" style="flex:0.257211" title="model.text_backbone: 2.0M (25.7%)"></i><i class="c3" style="flex:0.022839" title="model.encoder.layers: 176.4K (2.3%)"></i><i class="c4" style="flex:0.018430" title="model.decoder.layers: 142.3K (1.8%)"></i><i class="c5" style="flex:0.045711" title="model.input_proj_vision: 353.0K (4.6%)"></i><i class="c6" style="flex:0.002222" title="model.decoder.bbox_embed: 17.2K (0.2%)"></i><i class="c7" style="flex:0.012108" title="9 others: 93.5K (1.2%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.text_backbone.encoder.layer</td><td>85.1M</td><td>49.2%</td><td>66.9K</td><td>0.9%</td></tr><tr><td><i class="c1"></i>model.backbone.conv_encoder.model.swin.encoder.layers</td><td>27.5M</td><td>15.9%</td><td>4.9M</td><td>63.3%</td></tr><tr><td><i class="c2"></i>model.text_backbone</td><td>23.8M</td><td>13.8%</td><td>2.0M</td><td>25.7%</td></tr><tr><td><i class="c3"></i>model.encoder.layers</td><td>21.9M</td><td>12.7%</td><td>176.4K</td><td>2.3%</td></tr><tr><td><i class="c4"></i>model.decoder.layers</td><td>10.9M</td><td>6.3%</td><td>142.3K</td><td>1.8%</td></tr><tr><td><i class="c5"></i>model.input_proj_vision</td><td>2.1M</td><td>1.2%</td><td>353.0K</td><td>4.6%</td></tr><tr><td><i class="c6"></i>model.decoder.bbox_embed</td><td>795.7K</td><td>0.5%</td><td>17.2K</td><td>0.2%</td></tr><tr><td><i class="c7"></i>9 others</td><td>834.0K</td><td>0.5%</td><td>93.5K</td><td>1.2%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mm-grounding-dino</td><td>mm-grounding-dino</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>bert</td><td>bert</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>30522</td><td>30522</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MMGroundingDino omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>382</td><td>122</td></tr><tr class=""><td>LayerNorm</td><td>119</td><td>43</td></tr><tr class=""><td>Dropout</td><td>68</td><td>18</td></tr><tr class=""><td>ModuleList</td><td>27</td><td>19</td></tr><tr class=""><td>GELUActivation</td><td>24</td><td>6</td></tr><tr class=""><td>MultiheadAttention</td><td>18</td><td>6</td></tr><tr class=""><td>ReLU</td><td>18</td><td>6</td></tr><tr class=""><td>MLPPredictionHead</td><td>14</td><td>6</td></tr><tr class=""><td>ContrastiveEmbedding</td><td>13</td><td>5</td></tr><tr class=""><td>BertAttention</td><td>12</td><td>2</td></tr><tr class=""><td>BertIntermediate</td><td>12</td><td>2</td></tr><tr class=""><td>BertLayer</td><td>12</td><td>2</td></tr><tr class=""><td>BertOutput</td><td>12</td><td>2</td></tr><tr class=""><td>BertSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>BertSelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>MultiscaleDeformableAttention</td><td>12</td><td>4</td></tr><tr class=""><td>MultiScaleDeformableAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinLayer</td><td>12</td><td>4</td></tr><tr class=""><td>SwinMLP</td><td>12</td><td>4</td></tr><tr class=""><td>SwinRelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>SwinDropPath</td><td>11</td><td>3</td></tr><tr class=""><td>BiMultiHeadAttention</td><td>6</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>DeformableLayer</td><td>6</td><td>2</td></tr><tr class=""><td>DropPath</td><td>6</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FusionLayer</td><td>6</td><td>2</td></tr><tr class=""><td>TextEnhancerLayer</td><td>6</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>5</td><td>5</td></tr><tr class=""><td>Embedding</td><td>4</td><td>4</td></tr><tr class=""><td>GroupNorm</td><td>4</td><td>4</td></tr><tr class=""><td>Sequential</td><td>4</td><td>4</td></tr><tr class=""><td>SwinStage</td><td>4</td><td>4</td></tr><tr class=""><td>SwinPatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>BertEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>BertEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>BertModel</td><td>1</td><td>1</td></tr><tr class=""><td>Identity</td><td>1</td><td>1</td></tr><tr class=""><td>ConvEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ConvModel</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>SwinBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SwinModel</td><td>1</td><td>1</td></tr><tr class=""><td>SwinPatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">16 fields changed, 93 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.embed_dim</td><td>96</td><td>64</td></tr><tr><td>backbone_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>backbone_config.num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>backbone_config.num_layers</td><td>4</td><td>2</td></tr><tr><td>d_model</td><td>256</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr><tr><td>text_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic/tree/3b6c70187a273832016dad164af1af9a796a0bbd" title="huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic@3b6c70187a273832016dad164af1af9a796a0bbd">huggingface-mic/mm_grounding_dino_tiny_o365v1_goldg_v3det-mic@3b6c70187a27</a> β <a href="https://huggingface.co/openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det/tree/ef2053aa1e0cc9d9cee1df31ca919ee9aae4fc8e" title="openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det@ef2053aa1e0cc9d9cee1df31ca919ee9aae4fc8e">openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det@ef2053aa1e0c</a> Β· MMGroundingDinoForObjectDetection Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mm_grounding_dino_tiny_o365v1_goldg_v3det-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">172.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">7.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">7</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.backbone.conv_encoder.model.swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: SwinPatchMerging"></i><i class="c0" title="layer 1: SwinPatchMerging"></i><i class="c0" title="layer 2: SwinPatchMerging"></i><i class="c1" title="layer 3: no SwinPatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SwinPatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no SwinPatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.input_proj_vision</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.text_backbone.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: BertLayer"></i><i class="c0" title="layer 1: BertLayer"></i><i class="c0" title="layer 2: BertLayer"></i><i class="c0" title="layer 3: BertLayer"></i><i class="c0" title="layer 4: BertLayer"></i><i class="c0" title="layer 5: BertLayer"></i><i class="c0" title="layer 6: BertLayer"></i><i class="c0" title="layer 7: BertLayer"></i><i class="c0" title="layer 8: BertLayer"></i><i class="c0" title="layer 9: BertLayer"></i><i class="c0" title="layer 10: BertLayer"></i><i class="c0" title="layer 11: BertLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: BertLayer"></i><i class="c0" title="layer 1: BertLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>BertLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: FusionLayer"></i><i class="c0" title="layer 1: FusionLayer"></i><i class="c0" title="layer 2: FusionLayer"></i><i class="c0" title="layer 3: FusionLayer"></i><i class="c0" title="layer 4: FusionLayer"></i><i class="c0" title="layer 5: FusionLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: FusionLayer"></i><i class="c0" title="layer 1: FusionLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>FusionLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.491890" title="model.text_backbone.encoder.layer: 85.1M (49.2%)"></i><i class="c1" style="flex:0.159114" title="model.backbone.conv_encoder.model.swin.encoder.layers: 27.5M (15.9%)"></i><i class="c2" style="flex:0.137856" title="model.text_backbone: 23.8M (13.8%)"></i><i class="c3" style="flex:0.126686" title="model.encoder.layers: 21.9M (12.7%)"></i><i class="c4" style="flex:0.062790" title="model.decoder.layers: 10.9M (6.3%)"></i><i class="c5" style="flex:0.012241" title="model.input_proj_vision: 2.1M (1.2%)"></i><i class="c6" style="flex:0.004602" title="model.decoder.bbox_embed: 795.7K (0.5%)"></i><i class="c7" style="flex:0.004823" title="9 others: 834.0K (0.5%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.008668" title="model.text_backbone.encoder.layer: 66.9K (0.9%)"></i><i class="c1" style="flex:0.632811" title="model.backbone.conv_encoder.model.swin.encoder.layers: 4.9M (63.3%)"></i><i class="c2" style="flex:0.257211" title="model.text_backbone: 2.0M (25.7%)"></i><i class="c3" style="flex:0.022839" title="model.encoder.layers: 176.4K (2.3%)"></i><i class="c4" style="flex:0.018430" title="model.decoder.layers: 142.3K (1.8%)"></i><i class="c5" style="flex:0.045711" title="model.input_proj_vision: 353.0K (4.6%)"></i><i class="c6" style="flex:0.002222" title="model.decoder.bbox_embed: 17.2K (0.2%)"></i><i class="c7" style="flex:0.012108" title="9 others: 93.5K (1.2%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.text_backbone.encoder.layer</td><td>85.1M</td><td>49.2%</td><td>66.9K</td><td>0.9%</td></tr><tr><td><i class="c1"></i>model.backbone.conv_encoder.model.swin.encoder.layers</td><td>27.5M</td><td>15.9%</td><td>4.9M</td><td>63.3%</td></tr><tr><td><i class="c2"></i>model.text_backbone</td><td>23.8M</td><td>13.8%</td><td>2.0M</td><td>25.7%</td></tr><tr><td><i class="c3"></i>model.encoder.layers</td><td>21.9M</td><td>12.7%</td><td>176.4K</td><td>2.3%</td></tr><tr><td><i class="c4"></i>model.decoder.layers</td><td>10.9M</td><td>6.3%</td><td>142.3K</td><td>1.8%</td></tr><tr><td><i class="c5"></i>model.input_proj_vision</td><td>2.1M</td><td>1.2%</td><td>353.0K</td><td>4.6%</td></tr><tr><td><i class="c6"></i>model.decoder.bbox_embed</td><td>795.7K</td><td>0.5%</td><td>17.2K</td><td>0.2%</td></tr><tr><td><i class="c7"></i>9 others</td><td>834.0K</td><td>0.5%</td><td>93.5K</td><td>1.2%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mm-grounding-dino</td><td>mm-grounding-dino</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>bert</td><td>bert</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>30522</td><td>30522</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MMGroundingDino omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>382</td><td>122</td></tr><tr class=""><td>LayerNorm</td><td>119</td><td>43</td></tr><tr class=""><td>Dropout</td><td>68</td><td>18</td></tr><tr class=""><td>ModuleList</td><td>27</td><td>19</td></tr><tr class=""><td>GELUActivation</td><td>24</td><td>6</td></tr><tr class=""><td>MultiheadAttention</td><td>18</td><td>6</td></tr><tr class=""><td>ReLU</td><td>18</td><td>6</td></tr><tr class=""><td>MLPPredictionHead</td><td>14</td><td>6</td></tr><tr class=""><td>ContrastiveEmbedding</td><td>13</td><td>5</td></tr><tr class=""><td>BertAttention</td><td>12</td><td>2</td></tr><tr class=""><td>BertIntermediate</td><td>12</td><td>2</td></tr><tr class=""><td>BertLayer</td><td>12</td><td>2</td></tr><tr class=""><td>BertOutput</td><td>12</td><td>2</td></tr><tr class=""><td>BertSelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>BertSelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>MultiscaleDeformableAttention</td><td>12</td><td>4</td></tr><tr class=""><td>MultiScaleDeformableAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinAttention</td><td>12</td><td>4</td></tr><tr class=""><td>SwinLayer</td><td>12</td><td>4</td></tr><tr class=""><td>SwinMLP</td><td>12</td><td>4</td></tr><tr class=""><td>SwinRelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>SwinDropPath</td><td>11</td><td>3</td></tr><tr class=""><td>BiMultiHeadAttention</td><td>6</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>DeformableLayer</td><td>6</td><td>2</td></tr><tr class=""><td>DropPath</td><td>6</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FusionLayer</td><td>6</td><td>2</td></tr><tr class=""><td>TextEnhancerLayer</td><td>6</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>5</td><td>5</td></tr><tr class=""><td>Embedding</td><td>4</td><td>4</td></tr><tr class=""><td>GroupNorm</td><td>4</td><td>4</td></tr><tr class=""><td>Sequential</td><td>4</td><td>4</td></tr><tr class=""><td>SwinStage</td><td>4</td><td>4</td></tr><tr class=""><td>SwinPatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>BertEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>BertEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>BertModel</td><td>1</td><td>1</td></tr><tr class=""><td>Identity</td><td>1</td><td>1</td></tr><tr class=""><td>ConvEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ConvModel</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>SwinBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SwinEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SwinModel</td><td>1</td><td>1</td></tr><tr class=""><td>SwinPatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">16 fields changed, 93 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.embed_dim</td><td>96</td><td>64</td></tr><tr><td>backbone_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>backbone_config.num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>backbone_config.num_layers</td><td>4</td><td>2</td></tr><tr><td>d_model</td><td>256</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr><tr><td>text_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mms-tts-eng-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mms-tts-eng</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mms-tts-eng-mic/tree/9594dd28deb20d0502c494434c3c5f55b4833451" title="huggingface-mic/mms-tts-eng-mic@9594dd28deb20d0502c494434c3c5f55b4833451">huggingface-mic/mms-tts-eng-mic@9594dd28deb2</a> β <a href="https://huggingface.co/facebook/mms-tts-eng/tree/c71de0fe7204c83f1c10820a7d696d0b450048ba" title="facebook/mms-tts-eng@c71de0fe7204c83f1c10820a7d696d0b450048ba">facebook/mms-tts-eng@c71de0fe7204</a> Β· VitsModel Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">36.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">5</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_encoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>flow.flows</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: ResidualCouplingLayer"></i><i class="c0" title="layer 1: ResidualCouplingLayer"></i><i class="c0" title="layer 2: ResidualCouplingLayer"></i><i class="c0" title="layer 3: ResidualCouplingLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: ResidualCouplingLayer"></i><i class="c0" title="layer 1: ResidualCouplingLayer"></i><i class="c0" title="layer 2: ResidualCouplingLayer"></i><i class="c0" title="layer 3: ResidualCouplingLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ResidualCouplingLayer</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.resblocks</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 12</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>HifiGanResidualBlock</td><td>12</td><td>12</td><td>β kept</td></tr></table></div><div class="stack"><h3>posterior_encoder.wavenet.in_layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><div class="row"><span class="lab">Fixture Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleDict</td><td>16</td><td>16</td><td>β kept</td></tr></table></div><div class="stack"><h3>posterior_encoder.wavenet.res_skip_layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><div class="row"><span class="lab">Fixture Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleDict</td><td>16</td><td>16</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.302489" title="decoder.resblocks: 11.0M (30.2%)"></i><i class="c1" style="flex:0.195733" title="flow.flows: 7.1M (19.6%)"></i><i class="c2" style="flex:0.171381" title="text_encoder.encoder.layers: 6.2M (17.1%)"></i><i class="c3" style="flex:0.162894" title="posterior_encoder.wavenet.in_layers: 5.9M (16.3%)"></i><i class="c4" style="flex:0.073389" title="decoder.upsampler: 2.7M (7.3%)"></i><i class="c5" style="flex:0.031823" title="posterior_encoder.wavenet.res_skip_layers: 1.2M (3.2%)"></i><i class="c6" style="flex:0.018979" title="decoder.conv_pre: 688.6K (1.9%)"></i><i class="c7" style="flex:0.043313" title="13 others: 1.6M (4.3%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.672014" title="decoder.resblocks: 11.0M (67.2%)"></i><i class="c1" style="flex:0.050667" title="flow.flows: 827.5K (5.1%)"></i><i class="c2" style="flex:0.008147" title="text_encoder.encoder.layers: 133.1K (0.8%)"></i><i class="c3" style="flex:0.040377" title="posterior_encoder.wavenet.in_layers: 659.5K (4.0%)"></i><i class="c4" style="flex:0.163042" title="decoder.upsampler: 2.7M (16.3%)"></i><i class="c5" style="flex:0.008017" title="posterior_encoder.wavenet.res_skip_layers: 130.9K (0.8%)"></i><i class="c6" style="flex:0.042164" title="decoder.conv_pre: 688.6K (4.2%)"></i><i class="c7" style="flex:0.015573" title="13 others: 254.4K (1.6%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>decoder.resblocks</td><td>11.0M</td><td>30.2%</td><td>11.0M</td><td>67.2%</td></tr><tr><td><i class="c1"></i>flow.flows</td><td>7.1M</td><td>19.6%</td><td>827.5K</td><td>5.1%</td></tr><tr><td><i class="c2"></i>text_encoder.encoder.layers</td><td>6.2M</td><td>17.1%</td><td>133.1K</td><td>0.8%</td></tr><tr><td><i class="c3"></i>posterior_encoder.wavenet.in_layers</td><td>5.9M</td><td>16.3%</td><td>659.5K</td><td>4.0%</td></tr><tr><td><i class="c4"></i>decoder.upsampler</td><td>2.7M</td><td>7.3%</td><td>2.7M</td><td>16.3%</td></tr><tr><td><i class="c5"></i>posterior_encoder.wavenet.res_skip_layers</td><td>1.2M</td><td>3.2%</td><td>130.9K</td><td>0.8%</td></tr><tr><td><i class="c6"></i>decoder.conv_pre</td><td>688.6K</td><td>1.9%</td><td>688.6K</td><td>4.2%</td></tr><tr><td><i class="c7"></i>13 others</td><td>1.6M</td><td>4.3%</td><td>254.4K</td><td>1.6%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>vits</td><td>vits</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>38</td><td>38</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>relu</td><td>relu</td><td>β kept</td></tr><tr class=""><td>model</td><td>sampling_rate</td><td>16000</td><td>16000</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Vits omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Conv1d</td><td>177</td><td>169</td></tr><tr class=""><td>ModuleList</td><td>80</td><td>80</td></tr><tr class=""><td>LayerNorm</td><td>72</td><td>64</td></tr><tr class=""><td>ModuleDict</td><td>64</td><td>64</td></tr><tr class=""><td>ParametrizationList</td><td>64</td><td>64</td></tr><tr class=""><td>ParametrizedConv1d</td><td>64</td><td>64</td></tr><tr class=""><td>_WeightNorm</td><td>64</td><td>64</td></tr><tr class=""><td>Dropout</td><td>27</td><td>19</td></tr><tr class=""><td>Linear</td><td>24</td><td>8</td></tr><tr class=""><td>HifiGanResidualBlock</td><td>12</td><td>12</td></tr><tr class=""><td>DilatedDepthSeparableConv</td><td>10</td><td>10</td></tr><tr class=""><td>ConvFlow</td><td>8</td><td>8</td></tr><tr class=""><td>ReLU</td><td>6</td><td>2</td></tr><tr class=""><td>Attention</td><td>6</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FeedForward</td><td>6</td><td>2</td></tr><tr class=""><td>WaveNet</td><td>5</td><td>5</td></tr><tr class=""><td>ConvTranspose1d</td><td>4</td><td>4</td></tr><tr class=""><td>ResidualCouplingLayer</td><td>4</td><td>4</td></tr><tr class=""><td>ElementwiseAffine</td><td>2</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>HifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PosteriorEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ResidualCouplingBlock</td><td>1</td><td>1</td></tr><tr class=""><td>StochasticDurationPredictor</td><td>1</td><td>1</td></tr><tr class=""><td>TextEncoder</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 50 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>ffn_dim</td><td>768</td><td>128</td></tr><tr><td>hidden_size</td><td>192</td><td>64</td></tr><tr><td>num_attention_heads</td><td>2</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>6</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/mms-tts-eng</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mms-tts-eng-mic/tree/9594dd28deb20d0502c494434c3c5f55b4833451" title="huggingface-mic/mms-tts-eng-mic@9594dd28deb20d0502c494434c3c5f55b4833451">huggingface-mic/mms-tts-eng-mic@9594dd28deb2</a> β <a href="https://huggingface.co/facebook/mms-tts-eng/tree/c71de0fe7204c83f1c10820a7d696d0b450048ba" title="facebook/mms-tts-eng@c71de0fe7204c83f1c10820a7d696d0b450048ba">facebook/mms-tts-eng@c71de0fe7204</a> Β· VitsModel Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mms-tts-eng-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">36.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">5</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_encoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>flow.flows</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: ResidualCouplingLayer"></i><i class="c0" title="layer 1: ResidualCouplingLayer"></i><i class="c0" title="layer 2: ResidualCouplingLayer"></i><i class="c0" title="layer 3: ResidualCouplingLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: ResidualCouplingLayer"></i><i class="c0" title="layer 1: ResidualCouplingLayer"></i><i class="c0" title="layer 2: ResidualCouplingLayer"></i><i class="c0" title="layer 3: ResidualCouplingLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ResidualCouplingLayer</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.resblocks</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 12</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>HifiGanResidualBlock</td><td>12</td><td>12</td><td>β kept</td></tr></table></div><div class="stack"><h3>posterior_encoder.wavenet.in_layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><div class="row"><span class="lab">Fixture Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleDict</td><td>16</td><td>16</td><td>β kept</td></tr></table></div><div class="stack"><h3>posterior_encoder.wavenet.res_skip_layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><div class="row"><span class="lab">Fixture Β· 16</span><div class="cells"><i class="c0" title="layer 0: ModuleDict"></i><i class="c0" title="layer 1: ModuleDict"></i><i class="c0" title="layer 2: ModuleDict"></i><i class="c0" title="layer 3: ModuleDict"></i><i class="c0" title="layer 4: ModuleDict"></i><i class="c0" title="layer 5: ModuleDict"></i><i class="c0" title="layer 6: ModuleDict"></i><i class="c0" title="layer 7: ModuleDict"></i><i class="c0" title="layer 8: ModuleDict"></i><i class="c0" title="layer 9: ModuleDict"></i><i class="c0" title="layer 10: ModuleDict"></i><i class="c0" title="layer 11: ModuleDict"></i><i class="c0" title="layer 12: ModuleDict"></i><i class="c0" title="layer 13: ModuleDict"></i><i class="c0" title="layer 14: ModuleDict"></i><i class="c0" title="layer 15: ModuleDict"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleDict</td><td>16</td><td>16</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.302489" title="decoder.resblocks: 11.0M (30.2%)"></i><i class="c1" style="flex:0.195733" title="flow.flows: 7.1M (19.6%)"></i><i class="c2" style="flex:0.171381" title="text_encoder.encoder.layers: 6.2M (17.1%)"></i><i class="c3" style="flex:0.162894" title="posterior_encoder.wavenet.in_layers: 5.9M (16.3%)"></i><i class="c4" style="flex:0.073389" title="decoder.upsampler: 2.7M (7.3%)"></i><i class="c5" style="flex:0.031823" title="posterior_encoder.wavenet.res_skip_layers: 1.2M (3.2%)"></i><i class="c6" style="flex:0.018979" title="decoder.conv_pre: 688.6K (1.9%)"></i><i class="c7" style="flex:0.043313" title="13 others: 1.6M (4.3%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.672014" title="decoder.resblocks: 11.0M (67.2%)"></i><i class="c1" style="flex:0.050667" title="flow.flows: 827.5K (5.1%)"></i><i class="c2" style="flex:0.008147" title="text_encoder.encoder.layers: 133.1K (0.8%)"></i><i class="c3" style="flex:0.040377" title="posterior_encoder.wavenet.in_layers: 659.5K (4.0%)"></i><i class="c4" style="flex:0.163042" title="decoder.upsampler: 2.7M (16.3%)"></i><i class="c5" style="flex:0.008017" title="posterior_encoder.wavenet.res_skip_layers: 130.9K (0.8%)"></i><i class="c6" style="flex:0.042164" title="decoder.conv_pre: 688.6K (4.2%)"></i><i class="c7" style="flex:0.015573" title="13 others: 254.4K (1.6%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>decoder.resblocks</td><td>11.0M</td><td>30.2%</td><td>11.0M</td><td>67.2%</td></tr><tr><td><i class="c1"></i>flow.flows</td><td>7.1M</td><td>19.6%</td><td>827.5K</td><td>5.1%</td></tr><tr><td><i class="c2"></i>text_encoder.encoder.layers</td><td>6.2M</td><td>17.1%</td><td>133.1K</td><td>0.8%</td></tr><tr><td><i class="c3"></i>posterior_encoder.wavenet.in_layers</td><td>5.9M</td><td>16.3%</td><td>659.5K</td><td>4.0%</td></tr><tr><td><i class="c4"></i>decoder.upsampler</td><td>2.7M</td><td>7.3%</td><td>2.7M</td><td>16.3%</td></tr><tr><td><i class="c5"></i>posterior_encoder.wavenet.res_skip_layers</td><td>1.2M</td><td>3.2%</td><td>130.9K</td><td>0.8%</td></tr><tr><td><i class="c6"></i>decoder.conv_pre</td><td>688.6K</td><td>1.9%</td><td>688.6K</td><td>4.2%</td></tr><tr><td><i class="c7"></i>13 others</td><td>1.6M</td><td>4.3%</td><td>254.4K</td><td>1.6%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>vits</td><td>vits</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>38</td><td>38</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>relu</td><td>relu</td><td>β kept</td></tr><tr class=""><td>model</td><td>sampling_rate</td><td>16000</td><td>16000</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Vits omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Conv1d</td><td>177</td><td>169</td></tr><tr class=""><td>ModuleList</td><td>80</td><td>80</td></tr><tr class=""><td>LayerNorm</td><td>72</td><td>64</td></tr><tr class=""><td>ModuleDict</td><td>64</td><td>64</td></tr><tr class=""><td>ParametrizationList</td><td>64</td><td>64</td></tr><tr class=""><td>ParametrizedConv1d</td><td>64</td><td>64</td></tr><tr class=""><td>_WeightNorm</td><td>64</td><td>64</td></tr><tr class=""><td>Dropout</td><td>27</td><td>19</td></tr><tr class=""><td>Linear</td><td>24</td><td>8</td></tr><tr class=""><td>HifiGanResidualBlock</td><td>12</td><td>12</td></tr><tr class=""><td>DilatedDepthSeparableConv</td><td>10</td><td>10</td></tr><tr class=""><td>ConvFlow</td><td>8</td><td>8</td></tr><tr class=""><td>ReLU</td><td>6</td><td>2</td></tr><tr class=""><td>Attention</td><td>6</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>FeedForward</td><td>6</td><td>2</td></tr><tr class=""><td>WaveNet</td><td>5</td><td>5</td></tr><tr class=""><td>ConvTranspose1d</td><td>4</td><td>4</td></tr><tr class=""><td>ResidualCouplingLayer</td><td>4</td><td>4</td></tr><tr class=""><td>ElementwiseAffine</td><td>2</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>HifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PosteriorEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ResidualCouplingBlock</td><td>1</td><td>1</td></tr><tr class=""><td>StochasticDurationPredictor</td><td>1</td><td>1</td></tr><tr class=""><td>TextEncoder</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 50 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>ffn_dim</td><td>768</td><td>128</td></tr><tr><td>hidden_size</td><td>192</td><td>64</td></tr><tr><td>num_attention_heads</td><td>2</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>6</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mobilenet_v2_1.0_224-mic.html
CHANGED
|
@@ -31,6 +31,7 @@ tr.bad td{color:var(--warn)}
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
-
<h1>google/mobilenet_v2_1.0_224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mobilenet_v2_1.0_224-mic/tree/
|
| 35 |
-
<
|
|
|
|
| 36 |
</main></body></html>
|
|
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
+
<h1>google/mobilenet_v2_1.0_224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mobilenet_v2_1.0_224-mic/tree/6938bb7a591f10a6640c4ce27183bb9762f016b6" title="huggingface-mic/mobilenet_v2_1.0_224-mic@6938bb7a591f10a6640c4ce27183bb9762f016b6">huggingface-mic/mobilenet_v2_1.0_224-mic@6938bb7a591f</a> β <a href="https://huggingface.co/google/mobilenet_v2_1.0_224/tree/75e607b00aeae1297cc89d026a118bce012f5c5a" title="google/mobilenet_v2_1.0_224@75e607b00aeae1297cc89d026a118bce012f5c5a">google/mobilenet_v2_1.0_224@75e607b00aea</a> Β· MobileNetV2ForImageClassification Β· real weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mobilenet_v2_1.0_224-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
+
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>mobilenet_v2.layer</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: ConvLayer"></i><i class="c0" title="layer 1: ConvLayer"></i><i class="c0" title="layer 2: ConvLayer"></i><i class="c0" title="layer 3: ConvLayer"></i><i class="c0" title="layer 4: ConvLayer"></i><i class="c0" title="layer 5: ConvLayer"></i><i class="c0" title="layer 6: ConvLayer"></i><i class="c0" title="layer 7: ConvLayer"></i><i class="c0" title="layer 8: ConvLayer"></i><i class="c0" title="layer 9: ConvLayer"></i><i class="c0" title="layer 10: ConvLayer"></i><i class="c0" title="layer 11: ConvLayer"></i><i class="c0" title="layer 12: ConvLayer"></i><i class="c0" title="layer 13: ConvLayer"></i><i class="c0" title="layer 14: ConvLayer"></i><i class="c0" title="layer 15: ConvLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 16</span><div class="cells"><i class="c0" title="layer 0: ConvLayer"></i><i class="c0" title="layer 1: ConvLayer"></i><i class="c0" title="layer 2: ConvLayer"></i><i class="c0" title="layer 3: ConvLayer"></i><i class="c0" title="layer 4: ConvLayer"></i><i class="c0" title="layer 5: ConvLayer"></i><i class="c0" title="layer 6: ConvLayer"></i><i class="c0" title="layer 7: ConvLayer"></i><i class="c0" title="layer 8: ConvLayer"></i><i class="c0" title="layer 9: ConvLayer"></i><i class="c0" title="layer 10: ConvLayer"></i><i class="c0" title="layer 11: ConvLayer"></i><i class="c0" title="layer 12: ConvLayer"></i><i class="c0" title="layer 13: ConvLayer"></i><i class="c0" title="layer 14: ConvLayer"></i><i class="c0" title="layer 15: ConvLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConvLayer</td><td>16</td><td>16</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.516203" title="mobilenet_v2.layer: 1.8M (51.6%)"></i><i class="c1" style="flex:0.365723" title="classifier: 1.3M (36.6%)"></i><i class="c2" style="flex:0.117553" title="mobilenet_v2.conv_1x1: 412.2K (11.8%)"></i><i class="c3" style="flex:0.000520" title="mobilenet_v2.conv_stem: 1.8K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.516203" title="mobilenet_v2.layer: 1.8M (51.6%)"></i><i class="c1" style="flex:0.365723" title="classifier: 1.3M (36.6%)"></i><i class="c2" style="flex:0.117553" title="mobilenet_v2.conv_1x1: 412.2K (11.8%)"></i><i class="c3" style="flex:0.000520" title="mobilenet_v2.conv_stem: 1.8K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>mobilenet_v2.layer</td><td>1.8M</td><td>51.6%</td><td>1.8M</td><td>51.6%</td></tr><tr><td><i class="c1"></i>classifier</td><td>1.3M</td><td>36.6%</td><td>1.3M</td><td>36.6%</td></tr><tr><td><i class="c2"></i>mobilenet_v2.conv_1x1</td><td>412.2K</td><td>11.8%</td><td>412.2K</td><td>11.8%</td></tr><tr><td><i class="c3"></i>mobilenet_v2.conv_stem</td><td>1.8K</td><td>0.1%</td><td>1.8K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mobilenet_v2</td><td>mobilenet_v2</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>relu6</td><td>relu6</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MobileNetV2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>BatchNorm2d</td><td>52</td><td>52</td></tr><tr class=""><td>Conv2d</td><td>52</td><td>52</td></tr><tr class=""><td>ConvLayer</td><td>52</td><td>52</td></tr><tr class=""><td>ReLU6</td><td>35</td><td>35</td></tr><tr class=""><td>InvertedResidual</td><td>16</td><td>16</td></tr><tr class=""><td>AdaptiveAvgPool2d</td><td>1</td><td>1</td></tr><tr class=""><td>Dropout</td><td>1</td><td>1</td></tr><tr class=""><td>Linear</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>Stem</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">0 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mobilenet_v2_1.0_224-mic.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
{"fixture": "huggingface-mic/mobilenet_v2_1.0_224-mic@
|
|
|
|
| 1 |
+
{"fixture": "huggingface-mic/mobilenet_v2_1.0_224-mic@6938bb7a591f10a6640c4ce27183bb9762f016b6", "findings": []}
|
shrink/mobilevit-xx-small-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>apple/mobilevit-xx-small</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mobilevit-xx-small-mic/tree/1d4aa4d88fb85d89dc12a8e52ffcd0c66cf855ec" title="huggingface-mic/mobilevit-xx-small-mic@1d4aa4d88fb85d89dc12a8e52ffcd0c66cf855ec">huggingface-mic/mobilevit-xx-small-mic@1d4aa4d88fb8</a> β <a href="https://huggingface.co/apple/mobilevit-xx-small/tree/6703997f9e940e312a04b8b7c26463d631c0d1f8" title="apple/mobilevit-xx-small@6703997f9e940e312a04b8b7c26463d631c0d1f8">apple/mobilevit-xx-small@6703997f9e94</a> Β· MobileViTForImageClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">895.3K</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">1</span><span class="k">to review</span></div></div><section class="findings warn"><h2>β To review</h2><ul><li>the fixture keeps 70% of the original's 1.3M parameters</li></ul></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>mobilevit.encoder.layer</h3><div class="row"><span class="lab">Original Β· 5</span><div class="cells"><i class="c0" title="layer 0: MobileNetLayer"></i><i class="c0" title="layer 1: MobileNetLayer"></i><i class="c1" title="layer 2: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 3: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 4: LayerNorm, Linear, Attention β¦"></i></div></div><div class="row"><span class="lab">Fixture Β· 5</span><div class="cells"><i class="c0" title="layer 0: MobileNetLayer"></i><i class="c0" title="layer 1: MobileNetLayer"></i><i class="c1" title="layer 2: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 3: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 4: LayerNorm, Linear, Attention β¦"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MobileNetLayer</td><td>2</td><td>2</td><td>β kept</td></tr><tr><td><i class="c1"></i>LayerNorm, Linear, Attention β¦</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.726653" title="mobilevit.encoder.layer: 924.3K (72.7%)"></i><i class="c1" style="flex:0.252354" title="classifier: 321.0K (25.2%)"></i><i class="c2" style="flex:0.020629" title="mobilevit.conv_1x1_exp: 26.2K (2.1%)"></i><i class="c3" style="flex:0.000365" title="mobilevit.conv_stem: 464 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.611643" title="mobilevit.encoder.layer: 547.6K (61.2%)"></i><i class="c1" style="flex:0.358531" title="classifier: 321.0K (35.9%)"></i><i class="c2" style="flex:0.029308" title="mobilevit.conv_1x1_exp: 26.2K (2.9%)"></i><i class="c3" style="flex:0.000518" title="mobilevit.conv_stem: 464 (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>mobilevit.encoder.layer</td><td>924.3K</td><td>72.7%</td><td>547.6K</td><td>61.2%</td></tr><tr><td><i class="c1"></i>classifier</td><td>321.0K</td><td>25.2%</td><td>321.0K</td><td>35.9%</td></tr><tr><td><i class="c2"></i>mobilevit.conv_1x1_exp</td><td>26.2K</td><td>2.1%</td><td>26.2K</td><td>2.9%</td></tr><tr><td><i class="c3"></i>mobilevit.conv_stem</td><td>464</td><td>0.0%</td><td>464</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mobilevit</td><td>mobilevit</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>256</td><td>256</td><td>β kept</td></tr><tr class=""><td>model</td><td>patch_size</td><td>2</td><td>2</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MobileViT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>55</td><td>55</td></tr><tr class=""><td>Conv2d</td><td>35</td><td>35</td></tr><tr class=""><td>ConvLayer</td><td>35</td><td>35</td></tr><tr class=""><td>SiLUActivation</td><td>34</td><td>34</td></tr><tr class=""><td>BatchNorm2d</td><td>32</td><td>32</td></tr><tr class=""><td>Dropout</td><td>28</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>21</td><td>21</td></tr><tr class=""><td>Attention</td><td>9</td><td>9</td></tr><tr class=""><td>Intermediate</td><td>9</td><td>9</td></tr><tr class=""><td>Output</td><td>9</td><td>9</td></tr><tr class=""><td>SelfAttention</td><td>9</td><td>9</td></tr><tr class=""><td>SelfOutput</td><td>9</td><td>9</td></tr><tr class=""><td>TransformerLayer</td><td>9</td><td>9</td></tr><tr class=""><td>InvertedResidual</td><td>7</td><td>7</td></tr><tr class=""><td>ModuleList</td><td>6</td><td>6</td></tr><tr class=""><td>Layer</td><td>3</td><td>3</td></tr><tr class=""><td>Transformer</td><td>3</td><td>3</td></tr><tr class=""><td>MobileNetLayer</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">2 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_sizes</td><td>[64, 80, 96]</td><td>[16, 32, 64]</td></tr><tr><td>problem_type</td><td>null</td><td>single_label_classification</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>apple/mobilevit-xx-small</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mobilevit-xx-small-mic/tree/1d4aa4d88fb85d89dc12a8e52ffcd0c66cf855ec" title="huggingface-mic/mobilevit-xx-small-mic@1d4aa4d88fb85d89dc12a8e52ffcd0c66cf855ec">huggingface-mic/mobilevit-xx-small-mic@1d4aa4d88fb8</a> β <a href="https://huggingface.co/apple/mobilevit-xx-small/tree/6703997f9e940e312a04b8b7c26463d631c0d1f8" title="apple/mobilevit-xx-small@6703997f9e940e312a04b8b7c26463d631c0d1f8">apple/mobilevit-xx-small@6703997f9e94</a> Β· MobileViTForImageClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mobilevit-xx-small-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 11 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">895.3K</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">1</span><span class="k">to review</span></div></div><section class="findings warn"><h2>β To review</h2><ul><li>the fixture keeps 70% of the original's 1.3M parameters</li></ul></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>mobilevit.encoder.layer</h3><div class="row"><span class="lab">Original Β· 5</span><div class="cells"><i class="c0" title="layer 0: MobileNetLayer"></i><i class="c0" title="layer 1: MobileNetLayer"></i><i class="c1" title="layer 2: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 3: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 4: LayerNorm, Linear, Attention β¦"></i></div></div><div class="row"><span class="lab">Fixture Β· 5</span><div class="cells"><i class="c0" title="layer 0: MobileNetLayer"></i><i class="c0" title="layer 1: MobileNetLayer"></i><i class="c1" title="layer 2: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 3: LayerNorm, Linear, Attention β¦"></i><i class="c1" title="layer 4: LayerNorm, Linear, Attention β¦"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MobileNetLayer</td><td>2</td><td>2</td><td>β kept</td></tr><tr><td><i class="c1"></i>LayerNorm, Linear, Attention β¦</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.726653" title="mobilevit.encoder.layer: 924.3K (72.7%)"></i><i class="c1" style="flex:0.252354" title="classifier: 321.0K (25.2%)"></i><i class="c2" style="flex:0.020629" title="mobilevit.conv_1x1_exp: 26.2K (2.1%)"></i><i class="c3" style="flex:0.000365" title="mobilevit.conv_stem: 464 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.611643" title="mobilevit.encoder.layer: 547.6K (61.2%)"></i><i class="c1" style="flex:0.358531" title="classifier: 321.0K (35.9%)"></i><i class="c2" style="flex:0.029308" title="mobilevit.conv_1x1_exp: 26.2K (2.9%)"></i><i class="c3" style="flex:0.000518" title="mobilevit.conv_stem: 464 (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>mobilevit.encoder.layer</td><td>924.3K</td><td>72.7%</td><td>547.6K</td><td>61.2%</td></tr><tr><td><i class="c1"></i>classifier</td><td>321.0K</td><td>25.2%</td><td>321.0K</td><td>35.9%</td></tr><tr><td><i class="c2"></i>mobilevit.conv_1x1_exp</td><td>26.2K</td><td>2.1%</td><td>26.2K</td><td>2.9%</td></tr><tr><td><i class="c3"></i>mobilevit.conv_stem</td><td>464</td><td>0.0%</td><td>464</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mobilevit</td><td>mobilevit</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>256</td><td>256</td><td>β kept</td></tr><tr class=""><td>model</td><td>patch_size</td><td>2</td><td>2</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MobileViT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>55</td><td>55</td></tr><tr class=""><td>Conv2d</td><td>35</td><td>35</td></tr><tr class=""><td>ConvLayer</td><td>35</td><td>35</td></tr><tr class=""><td>SiLUActivation</td><td>34</td><td>34</td></tr><tr class=""><td>BatchNorm2d</td><td>32</td><td>32</td></tr><tr class=""><td>Dropout</td><td>28</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>21</td><td>21</td></tr><tr class=""><td>Attention</td><td>9</td><td>9</td></tr><tr class=""><td>Intermediate</td><td>9</td><td>9</td></tr><tr class=""><td>Output</td><td>9</td><td>9</td></tr><tr class=""><td>SelfAttention</td><td>9</td><td>9</td></tr><tr class=""><td>SelfOutput</td><td>9</td><td>9</td></tr><tr class=""><td>TransformerLayer</td><td>9</td><td>9</td></tr><tr class=""><td>InvertedResidual</td><td>7</td><td>7</td></tr><tr class=""><td>ModuleList</td><td>6</td><td>6</td></tr><tr class=""><td>Layer</td><td>3</td><td>3</td></tr><tr class=""><td>Transformer</td><td>3</td><td>3</td></tr><tr class=""><td>MobileNetLayer</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">2 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_sizes</td><td>[64, 80, 96]</td><td>[16, 32, 64]</td></tr><tr><td>problem_type</td><td>null</td><td>single_label_classification</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/modernbert-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>answerdotai/ModernBERT-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/modernbert-mic/tree/59688bef85e5803f4ebc8a02227907e465355a30" title="huggingface-mic/modernbert-mic@59688bef85e5803f4ebc8a02227907e465355a30">huggingface-mic/modernbert-mic@59688bef85e5</a> β <a href="https://huggingface.co/answerdotai/ModernBERT-base/tree/8949b909ec900327062f0ebf497f51aef5e6f0c8" title="answerdotai/ModernBERT-base@8949b909ec900327062f0ebf497f51aef5e6f0c8">answerdotai/ModernBERT-base@8949b909ec90</a> Β· ModernBertForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">149.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 22</span><div class="cells"><i class="c0" title="layer 0: full_attention"></i><i class="c1" title="layer 1: sliding_attention"></i><i class="c1" title="layer 2: sliding_attention"></i><i class="c0" title="layer 3: full_attention"></i><i class="c1" title="layer 4: sliding_attention"></i><i class="c1" title="layer 5: sliding_attention"></i><i class="c0" title="layer 6: full_attention"></i><i class="c1" title="layer 7: sliding_attention"></i><i class="c1" title="layer 8: sliding_attention"></i><i class="c0" title="layer 9: full_attention"></i><i class="c1" title="layer 10: sliding_attention"></i><i class="c1" title="layer 11: sliding_attention"></i><i class="c0" title="layer 12: full_attention"></i><i class="c1" title="layer 13: sliding_attention"></i><i class="c1" title="layer 14: sliding_attention"></i><i class="c0" title="layer 15: full_attention"></i><i class="c1" title="layer 16: sliding_attention"></i><i class="c1" title="layer 17: sliding_attention"></i><i class="c0" title="layer 18: full_attention"></i><i class="c1" title="layer 19: sliding_attention"></i><i class="c1" title="layer 20: sliding_attention"></i><i class="c0" title="layer 21: full_attention"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: full_attention"></i><i class="c1" title="layer 1: sliding_attention"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>full_attention</td><td>8</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>sliding_attention</td><td>14</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.737229" title="model.layers: 110.3M (73.7%)"></i><i class="c1" style="flex:0.258483" title="model.embeddings: 38.7M (25.8%)"></i><i class="c2" style="flex:0.003941" title="head.dense: 589.8K (0.4%)"></i><i class="c3" style="flex:0.000337" title="decoder: 50.4K (0.0%)"></i><i class="c4" style="flex:0.000005" title="model.final_norm: 768 (0.0%)"></i><i class="c5" style="flex:0.000005" title="head.norm: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.024436" title="model.layers: 82.1K (2.4%)"></i><i class="c1" style="flex:0.959318" title="model.embeddings: 3.2M (95.9%)"></i><i class="c2" style="flex:0.001219" title="head.dense: 4.1K (0.1%)"></i><i class="c3" style="flex:0.014989" title="decoder: 50.4K (1.5%)"></i><i class="c4" style="flex:0.000019" title="model.final_norm: 64 (0.0%)"></i><i class="c5" style="flex:0.000019" title="head.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>110.3M</td><td>73.7%</td><td>82.1K</td><td>2.4%</td></tr><tr><td><i class="c1"></i>model.embeddings</td><td>38.7M</td><td>25.8%</td><td>3.2M</td><td>95.9%</td></tr><tr><td><i class="c2"></i>head.dense</td><td>589.8K</td><td>0.4%</td><td>4.1K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>decoder</td><td>50.4K</td><td>0.0%</td><td>50.4K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>model.final_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c5"></i>head.norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>modernbert</td><td>modernbert</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50368</td><td>50368</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix ModernBert omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>90</td><td>10</td></tr><tr class=""><td>LayerNorm</td><td>46</td><td>6</td></tr><tr class=""><td>Dropout</td><td>23</td><td>3</td></tr><tr class=""><td>GELUActivation</td><td>23</td><td>3</td></tr><tr class=""><td>Identity</td><td>23</td><td>3</td></tr><tr class=""><td>Attention</td><td>22</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>22</td><td>2</td></tr><tr class=""><td>MLP</td><td>22</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 41 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>1152</td><td>128</td></tr><tr><td>layer_types</td><td>22 entries: 14Γ sliding_attention, 8Γ full_attention</td><td>["full_attention", "sliding_attention"]</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>22</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>answerdotai/ModernBERT-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/modernbert-mic/tree/59688bef85e5803f4ebc8a02227907e465355a30" title="huggingface-mic/modernbert-mic@59688bef85e5803f4ebc8a02227907e465355a30">huggingface-mic/modernbert-mic@59688bef85e5</a> β <a href="https://huggingface.co/answerdotai/ModernBERT-base/tree/8949b909ec900327062f0ebf497f51aef5e6f0c8" title="answerdotai/ModernBERT-base@8949b909ec900327062f0ebf497f51aef5e6f0c8">answerdotai/ModernBERT-base@8949b909ec90</a> Β· ModernBertForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=modernbert-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 3 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">149.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 22</span><div class="cells"><i class="c0" title="layer 0: full_attention"></i><i class="c1" title="layer 1: sliding_attention"></i><i class="c1" title="layer 2: sliding_attention"></i><i class="c0" title="layer 3: full_attention"></i><i class="c1" title="layer 4: sliding_attention"></i><i class="c1" title="layer 5: sliding_attention"></i><i class="c0" title="layer 6: full_attention"></i><i class="c1" title="layer 7: sliding_attention"></i><i class="c1" title="layer 8: sliding_attention"></i><i class="c0" title="layer 9: full_attention"></i><i class="c1" title="layer 10: sliding_attention"></i><i class="c1" title="layer 11: sliding_attention"></i><i class="c0" title="layer 12: full_attention"></i><i class="c1" title="layer 13: sliding_attention"></i><i class="c1" title="layer 14: sliding_attention"></i><i class="c0" title="layer 15: full_attention"></i><i class="c1" title="layer 16: sliding_attention"></i><i class="c1" title="layer 17: sliding_attention"></i><i class="c0" title="layer 18: full_attention"></i><i class="c1" title="layer 19: sliding_attention"></i><i class="c1" title="layer 20: sliding_attention"></i><i class="c0" title="layer 21: full_attention"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: full_attention"></i><i class="c1" title="layer 1: sliding_attention"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>full_attention</td><td>8</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>sliding_attention</td><td>14</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.737229" title="model.layers: 110.3M (73.7%)"></i><i class="c1" style="flex:0.258483" title="model.embeddings: 38.7M (25.8%)"></i><i class="c2" style="flex:0.003941" title="head.dense: 589.8K (0.4%)"></i><i class="c3" style="flex:0.000337" title="decoder: 50.4K (0.0%)"></i><i class="c4" style="flex:0.000005" title="model.final_norm: 768 (0.0%)"></i><i class="c5" style="flex:0.000005" title="head.norm: 768 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.024436" title="model.layers: 82.1K (2.4%)"></i><i class="c1" style="flex:0.959318" title="model.embeddings: 3.2M (95.9%)"></i><i class="c2" style="flex:0.001219" title="head.dense: 4.1K (0.1%)"></i><i class="c3" style="flex:0.014989" title="decoder: 50.4K (1.5%)"></i><i class="c4" style="flex:0.000019" title="model.final_norm: 64 (0.0%)"></i><i class="c5" style="flex:0.000019" title="head.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>110.3M</td><td>73.7%</td><td>82.1K</td><td>2.4%</td></tr><tr><td><i class="c1"></i>model.embeddings</td><td>38.7M</td><td>25.8%</td><td>3.2M</td><td>95.9%</td></tr><tr><td><i class="c2"></i>head.dense</td><td>589.8K</td><td>0.4%</td><td>4.1K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>decoder</td><td>50.4K</td><td>0.0%</td><td>50.4K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>model.final_norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c5"></i>head.norm</td><td>768</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>modernbert</td><td>modernbert</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50368</td><td>50368</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix ModernBert omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>90</td><td>10</td></tr><tr class=""><td>LayerNorm</td><td>46</td><td>6</td></tr><tr class=""><td>Dropout</td><td>23</td><td>3</td></tr><tr class=""><td>GELUActivation</td><td>23</td><td>3</td></tr><tr class=""><td>Identity</td><td>23</td><td>3</td></tr><tr class=""><td>Attention</td><td>22</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>22</td><td>2</td></tr><tr class=""><td>MLP</td><td>22</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 41 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>1152</td><td>128</td></tr><tr><td>layer_types</td><td>22 entries: 14Γ sliding_attention, 8Γ full_attention</td><td>["full_attention", "sliding_attention"]</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>22</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mpnet-base-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/mpnet-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mpnet-base-mic/tree/9d98c3d72a6cae57a9f3ed7f52cf9144628a5f6e" title="huggingface-mic/mpnet-base-mic@9d98c3d72a6cae57a9f3ed7f52cf9144628a5f6e">huggingface-mic/mpnet-base-mic@9d98c3d72a6c</a> β <a href="https://huggingface.co/microsoft/mpnet-base/tree/6996ce1e91bd2a9c7d7f61daec37463394f73f09" title="microsoft/mpnet-base@6996ce1e91bd2a9c7d7f61daec37463394f73f09">microsoft/mpnet-base@6996ce1e91bd</a> Β· MPNetForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">109.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>mpnet.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.776622" title="mpnet.encoder.layer: 85.1M (77.7%)"></i><i class="c1" style="flex:0.217689" title="mpnet.embeddings: 23.8M (21.8%)"></i><i class="c2" style="flex:0.005393" title="lm_head.dense: 590.6K (0.5%)"></i><i class="c3" style="flex:0.000279" title="lm_head: 30.5K (0.0%)"></i><i class="c4" style="flex:0.000014" title="lm_head.layer_norm: 1.5K (0.0%)"></i><i class="c5" style="flex:0.000004" title="mpnet.encoder: 384 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.032051" title="mpnet.encoder.layer: 66.9K (3.2%)"></i><i class="c1" style="flex:0.951218" title="mpnet.embeddings: 2.0M (95.1%)"></i><i class="c2" style="flex:0.001992" title="lm_head.dense: 4.2K (0.2%)"></i><i class="c3" style="flex:0.014616" title="lm_head: 30.5K (1.5%)"></i><i class="c4" style="flex:0.000061" title="lm_head.layer_norm: 128 (0.0%)"></i><i class="c5" style="flex:0.000061" title="mpnet.encoder: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>mpnet.encoder.layer</td><td>85.1M</td><td>77.7%</td><td>66.9K</td><td>3.2%</td></tr><tr><td><i class="c1"></i>mpnet.embeddings</td><td>23.8M</td><td>21.8%</td><td>2.0M</td><td>95.1%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.5%</td><td>4.2K</td><td>0.2%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>30.5K</td><td>0.0%</td><td>30.5K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c5"></i>mpnet.encoder</td><td>384</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mpnet</td><td>mpnet</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>30527</td><td>30527</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MPNet omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>37</td><td>7</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 21 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/mpnet-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mpnet-base-mic/tree/9d98c3d72a6cae57a9f3ed7f52cf9144628a5f6e" title="huggingface-mic/mpnet-base-mic@9d98c3d72a6cae57a9f3ed7f52cf9144628a5f6e">huggingface-mic/mpnet-base-mic@9d98c3d72a6c</a> β <a href="https://huggingface.co/microsoft/mpnet-base/tree/6996ce1e91bd2a9c7d7f61daec37463394f73f09" title="microsoft/mpnet-base@6996ce1e91bd2a9c7d7f61daec37463394f73f09">microsoft/mpnet-base@6996ce1e91bd</a> Β· MPNetForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mpnet-base-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">109.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>mpnet.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.776622" title="mpnet.encoder.layer: 85.1M (77.7%)"></i><i class="c1" style="flex:0.217689" title="mpnet.embeddings: 23.8M (21.8%)"></i><i class="c2" style="flex:0.005393" title="lm_head.dense: 590.6K (0.5%)"></i><i class="c3" style="flex:0.000279" title="lm_head: 30.5K (0.0%)"></i><i class="c4" style="flex:0.000014" title="lm_head.layer_norm: 1.5K (0.0%)"></i><i class="c5" style="flex:0.000004" title="mpnet.encoder: 384 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.032051" title="mpnet.encoder.layer: 66.9K (3.2%)"></i><i class="c1" style="flex:0.951218" title="mpnet.embeddings: 2.0M (95.1%)"></i><i class="c2" style="flex:0.001992" title="lm_head.dense: 4.2K (0.2%)"></i><i class="c3" style="flex:0.014616" title="lm_head: 30.5K (1.5%)"></i><i class="c4" style="flex:0.000061" title="lm_head.layer_norm: 128 (0.0%)"></i><i class="c5" style="flex:0.000061" title="mpnet.encoder: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>mpnet.encoder.layer</td><td>85.1M</td><td>77.7%</td><td>66.9K</td><td>3.2%</td></tr><tr><td><i class="c1"></i>mpnet.embeddings</td><td>23.8M</td><td>21.8%</td><td>2.0M</td><td>95.1%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.5%</td><td>4.2K</td><td>0.2%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>30.5K</td><td>0.0%</td><td>30.5K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c5"></i>mpnet.encoder</td><td>384</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mpnet</td><td>mpnet</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>30527</td><td>30527</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MPNet omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>37</td><td>7</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 21 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/mt5-small-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/mt5-small</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mt5-small-mic/tree/13bc3e6cb33e75d32f533822c690e2115f739038" title="huggingface-mic/mt5-small-mic@13bc3e6cb33e75d32f533822c690e2115f739038">huggingface-mic/mt5-small-mic@13bc3e6cb33e</a> β <a href="https://huggingface.co/google/mt5-small/tree/73fb5dbe4756edadc8fbe8c769b0a109493acf7a" title="google/mt5-small@73fb5dbe4756edadc8fbe8c769b0a109493acf7a">google/mt5-small@73fb5dbe4756</a> Β· MT5ForConditionalGeneration Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">172.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>encoder.block</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>7</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.block</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>7</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.744003" title="shared: 128.1M (74.4%)"></i><i class="c1" style="flex:0.146284" title="decoder.block: 25.2M (14.6%)"></i><i class="c2" style="flex:0.109707" title="encoder.block: 18.9M (11.0%)"></i><i class="c3" style="flex:0.000003" title="decoder.final_layer_norm: 512 (0.0%)"></i><i class="c4" style="flex:0.000003" title="encoder.final_layer_norm: 512 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.987804" title="shared: 16.0M (98.8%)"></i><i class="c1" style="flex:0.007109" title="decoder.block: 115.2K (0.7%)"></i><i class="c2" style="flex:0.005079" title="encoder.block: 82.3K (0.5%)"></i><i class="c3" style="flex:0.000004" title="decoder.final_layer_norm: 64 (0.0%)"></i><i class="c4" style="flex:0.000004" title="encoder.final_layer_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>shared</td><td>128.1M</td><td>74.4%</td><td>16.0M</td><td>98.8%</td></tr><tr><td><i class="c1"></i>decoder.block</td><td>25.2M</td><td>14.6%</td><td>115.2K</td><td>0.7%</td></tr><tr><td><i class="c2"></i>encoder.block</td><td>18.9M</td><td>11.0%</td><td>82.3K</td><td>0.5%</td></tr><tr><td><i class="c3"></i>decoder.final_layer_norm</td><td>512</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c4"></i>encoder.final_layer_norm</td><td>512</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mt5</td><td>mt5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>250112</td><td>250112</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>145</td><td>37</td></tr><tr class=""><td>Dropout</td><td>58</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>42</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>6</td></tr><tr class=""><td>ModuleList</td><td>18</td><td>6</td></tr><tr class=""><td>Block</td><td>16</td><td>4</td></tr><tr class=""><td>DenseGatedActDense</td><td>16</td><td>4</td></tr><tr class=""><td>LayerFF</td><td>16</td><td>4</td></tr><tr class=""><td>LayerSelfAttention</td><td>16</td><td>4</td></tr><tr class=""><td>NewGELUActivation</td><td>16</td><td>4</td></tr><tr class=""><td>LayerCrossAttention</td><td>8</td><td>2</td></tr><tr class=""><td>Embedding</td><td>5</td><td>5</td></tr><tr class=""><td>Stack</td><td>2</td><td>2</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 27 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_ff</td><td>1024</td><td>128</td></tr><tr><td>d_kv</td><td>64</td><td>16</td></tr><tr><td>d_model</td><td>512</td><td>64</td></tr><tr><td>num_decoder_layers</td><td>8</td><td>2</td></tr><tr><td>num_heads</td><td>6</td><td>4</td></tr><tr><td>num_layers</td><td>8</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/mt5-small</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/mt5-small-mic/tree/13bc3e6cb33e75d32f533822c690e2115f739038" title="huggingface-mic/mt5-small-mic@13bc3e6cb33e75d32f533822c690e2115f739038">huggingface-mic/mt5-small-mic@13bc3e6cb33e</a> β <a href="https://huggingface.co/google/mt5-small/tree/73fb5dbe4756edadc8fbe8c769b0a109493acf7a" title="google/mt5-small@73fb5dbe4756edadc8fbe8c769b0a109493acf7a">google/mt5-small@73fb5dbe4756</a> Β· MT5ForConditionalGeneration Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=mt5-small-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">172.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>encoder.block</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>7</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>decoder.block</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i><i class="c1" title="layer 2: no Embedding"></i><i class="c1" title="layer 3: no Embedding"></i><i class="c1" title="layer 4: no Embedding"></i><i class="c1" title="layer 5: no Embedding"></i><i class="c1" title="layer 6: no Embedding"></i><i class="c1" title="layer 7: no Embedding"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Embedding"></i><i class="c1" title="layer 1: no Embedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Embedding</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Embedding</td><td>7</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.744003" title="shared: 128.1M (74.4%)"></i><i class="c1" style="flex:0.146284" title="decoder.block: 25.2M (14.6%)"></i><i class="c2" style="flex:0.109707" title="encoder.block: 18.9M (11.0%)"></i><i class="c3" style="flex:0.000003" title="decoder.final_layer_norm: 512 (0.0%)"></i><i class="c4" style="flex:0.000003" title="encoder.final_layer_norm: 512 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.987804" title="shared: 16.0M (98.8%)"></i><i class="c1" style="flex:0.007109" title="decoder.block: 115.2K (0.7%)"></i><i class="c2" style="flex:0.005079" title="encoder.block: 82.3K (0.5%)"></i><i class="c3" style="flex:0.000004" title="decoder.final_layer_norm: 64 (0.0%)"></i><i class="c4" style="flex:0.000004" title="encoder.final_layer_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>shared</td><td>128.1M</td><td>74.4%</td><td>16.0M</td><td>98.8%</td></tr><tr><td><i class="c1"></i>decoder.block</td><td>25.2M</td><td>14.6%</td><td>115.2K</td><td>0.7%</td></tr><tr><td><i class="c2"></i>encoder.block</td><td>18.9M</td><td>11.0%</td><td>82.3K</td><td>0.5%</td></tr><tr><td><i class="c3"></i>decoder.final_layer_norm</td><td>512</td><td>0.0%</td><td>64</td><td>0.0%</td></tr><tr><td><i class="c4"></i>encoder.final_layer_norm</td><td>512</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>mt5</td><td>mt5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>250112</td><td>250112</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>145</td><td>37</td></tr><tr class=""><td>Dropout</td><td>58</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>42</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>6</td></tr><tr class=""><td>ModuleList</td><td>18</td><td>6</td></tr><tr class=""><td>Block</td><td>16</td><td>4</td></tr><tr class=""><td>DenseGatedActDense</td><td>16</td><td>4</td></tr><tr class=""><td>LayerFF</td><td>16</td><td>4</td></tr><tr class=""><td>LayerSelfAttention</td><td>16</td><td>4</td></tr><tr class=""><td>NewGELUActivation</td><td>16</td><td>4</td></tr><tr class=""><td>LayerCrossAttention</td><td>8</td><td>2</td></tr><tr class=""><td>Embedding</td><td>5</td><td>5</td></tr><tr class=""><td>Stack</td><td>2</td><td>2</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 27 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_ff</td><td>1024</td><td>128</td></tr><tr><td>d_kv</td><td>64</td><td>16</td></tr><tr><td>d_model</td><td>512</td><td>64</td></tr><tr><td>num_decoder_layers</td><td>8</td><td>2</td></tr><tr><td>num_heads</td><td>6</td><td>4</td></tr><tr><td>num_layers</td><td>8</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/nllb-200-distilled-600M-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/nllb-200-distilled-600M</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/nllb-200-distilled-600M-mic/tree/fcaaa8d89cb3bc751afd9c7511e8f2975319ea14" title="huggingface-mic/nllb-200-distilled-600M-mic@fcaaa8d89cb3bc751afd9c7511e8f2975319ea14">huggingface-mic/nllb-200-distilled-600M-mic@fcaaa8d89cb3</a> β <a href="https://huggingface.co/facebook/nllb-200-distilled-600M/tree/f8d333a098d19b4fd9a8b18f94170487ad3f821d" title="facebook/nllb-200-distilled-600M@f8d333a098d19b4fd9a8b18f94170487ad3f821d">facebook/nllb-200-distilled-600M@f8d333a098d1</a> Β· M2M100ForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">615.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.426542" title="model.shared: 262.4M (42.7%)"></i><i class="c1" style="flex:0.327701" title="model.decoder.layers: 201.6M (32.8%)"></i><i class="c2" style="flex:0.245750" title="model.encoder.layers: 151.2M (24.6%)"></i><i class="c3" style="flex:0.000003" title="model.encoder: 2.0K (0.0%)"></i><i class="c4" style="flex:0.000003" title="model.decoder: 2.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.989877" title="model.shared: 16.4M (99.0%)"></i><i class="c1" style="flex:0.006066" title="model.decoder.layers: 100.5K (0.6%)"></i><i class="c2" style="flex:0.004041" title="model.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.000008" title="model.encoder: 128 (0.0%)"></i><i class="c4" style="flex:0.000008" title="model.decoder: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>262.4M</td><td>42.7%</td><td>16.4M</td><td>99.0%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>201.6M</td><td>32.8%</td><td>100.5K</td><td>0.6%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>151.2M</td><td>24.6%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>2.0K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>2.0K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>m2m_100</td><td>m2m_100</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256206</td><td>256206</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>1024</td><td>1024</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix M2M100 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>62</td><td>12</td></tr><tr class=""><td>Attention</td><td>36</td><td>6</td></tr><tr class=""><td>ReLU</td><td>24</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>3</td><td>3</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>12</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/nllb-200-distilled-600M</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/nllb-200-distilled-600M-mic/tree/fcaaa8d89cb3bc751afd9c7511e8f2975319ea14" title="huggingface-mic/nllb-200-distilled-600M-mic@fcaaa8d89cb3bc751afd9c7511e8f2975319ea14">huggingface-mic/nllb-200-distilled-600M-mic@fcaaa8d89cb3</a> β <a href="https://huggingface.co/facebook/nllb-200-distilled-600M/tree/f8d333a098d19b4fd9a8b18f94170487ad3f821d" title="facebook/nllb-200-distilled-600M@f8d333a098d19b4fd9a8b18f94170487ad3f821d">facebook/nllb-200-distilled-600M@f8d333a098d1</a> Β· M2M100ForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=nllb-200-distilled-600M-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">615.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.426542" title="model.shared: 262.4M (42.7%)"></i><i class="c1" style="flex:0.327701" title="model.decoder.layers: 201.6M (32.8%)"></i><i class="c2" style="flex:0.245750" title="model.encoder.layers: 151.2M (24.6%)"></i><i class="c3" style="flex:0.000003" title="model.encoder: 2.0K (0.0%)"></i><i class="c4" style="flex:0.000003" title="model.decoder: 2.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.989877" title="model.shared: 16.4M (99.0%)"></i><i class="c1" style="flex:0.006066" title="model.decoder.layers: 100.5K (0.6%)"></i><i class="c2" style="flex:0.004041" title="model.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.000008" title="model.encoder: 128 (0.0%)"></i><i class="c4" style="flex:0.000008" title="model.decoder: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>262.4M</td><td>42.7%</td><td>16.4M</td><td>99.0%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>201.6M</td><td>32.8%</td><td>100.5K</td><td>0.6%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>151.2M</td><td>24.6%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>2.0K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>2.0K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>m2m_100</td><td>m2m_100</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256206</td><td>256206</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>1024</td><td>1024</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix M2M100 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>62</td><td>12</td></tr><tr class=""><td>Attention</td><td>36</td><td>6</td></tr><tr class=""><td>ReLU</td><td>24</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>3</td><td>3</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>12</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/nomic-embed-text-v1.5-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>nomic-ai/nomic-embed-text-v1.5</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/nomic-embed-text-v1.5-mic/tree/2cbaf2971908b9ad612ba5d9b74e819440dc5344" title="huggingface-mic/nomic-embed-text-v1.5-mic@2cbaf2971908b9ad612ba5d9b74e819440dc5344">huggingface-mic/nomic-embed-text-v1.5-mic@2cbaf2971908</a> β <a href="https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/tree/e9b6763023c676ca8431644204f50c2b100d9aab" title="nomic-ai/nomic-embed-text-v1.5@e9b6763023c676ca8431644204f50c2b100d9aab">nomic-ai/nomic-embed-text-v1.5@e9b6763023c6</a> Β· NomicBertModel Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">136.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.828507" title="layers: 113.3M (82.9%)"></i><i class="c1" style="flex:0.171471" title="embeddings.word_embeddings: 23.4M (17.1%)"></i><i class="c2" style="flex:0.000011" title="embeddings.LayerNorm: 1.5K (0.0%)"></i><i class="c3" style="flex:0.000011" title="embeddings.token_type_embeddings: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.040478" title="layers: 82.4K (4.0%)"></i><i class="c1" style="flex:0.959397" title="embeddings.word_embeddings: 2.0M (95.9%)"></i><i class="c2" style="flex:0.000063" title="embeddings.LayerNorm: 128 (0.0%)"></i><i class="c3" style="flex:0.000063" title="embeddings.token_type_embeddings: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>layers</td><td>113.3M</td><td>82.9%</td><td>82.4K</td><td>4.0%</td></tr><tr><td><i class="c1"></i>embeddings.word_embeddings</td><td>23.4M</td><td>17.1%</td><td>2.0M</td><td>95.9%</td></tr><tr><td><i class="c2"></i>embeddings.LayerNorm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c3"></i>embeddings.token_type_embeddings</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>nomic_bert</td><td>nomic_bert</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>30528</td><td>30528</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix NomicBert omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>84</td><td>14</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>MLP</td><td>12</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 57 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>n_embd</td><td>768</td><td>64</td></tr><tr><td>n_head</td><td>12</td><td>4</td></tr><tr><td>n_inner</td><td>3072</td><td>128</td></tr><tr><td>n_layer</td><td>12</td><td>2</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>nomic-ai/nomic-embed-text-v1.5</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/nomic-embed-text-v1.5-mic/tree/2cbaf2971908b9ad612ba5d9b74e819440dc5344" title="huggingface-mic/nomic-embed-text-v1.5-mic@2cbaf2971908b9ad612ba5d9b74e819440dc5344">huggingface-mic/nomic-embed-text-v1.5-mic@2cbaf2971908</a> β <a href="https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/tree/e9b6763023c676ca8431644204f50c2b100d9aab" title="nomic-ai/nomic-embed-text-v1.5@e9b6763023c676ca8431644204f50c2b100d9aab">nomic-ai/nomic-embed-text-v1.5@e9b6763023c6</a> Β· NomicBertModel Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=nomic-embed-text-v1.5-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">136.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.828507" title="layers: 113.3M (82.9%)"></i><i class="c1" style="flex:0.171471" title="embeddings.word_embeddings: 23.4M (17.1%)"></i><i class="c2" style="flex:0.000011" title="embeddings.LayerNorm: 1.5K (0.0%)"></i><i class="c3" style="flex:0.000011" title="embeddings.token_type_embeddings: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.040478" title="layers: 82.4K (4.0%)"></i><i class="c1" style="flex:0.959397" title="embeddings.word_embeddings: 2.0M (95.9%)"></i><i class="c2" style="flex:0.000063" title="embeddings.LayerNorm: 128 (0.0%)"></i><i class="c3" style="flex:0.000063" title="embeddings.token_type_embeddings: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>layers</td><td>113.3M</td><td>82.9%</td><td>82.4K</td><td>4.0%</td></tr><tr><td><i class="c1"></i>embeddings.word_embeddings</td><td>23.4M</td><td>17.1%</td><td>2.0M</td><td>95.9%</td></tr><tr><td><i class="c2"></i>embeddings.LayerNorm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c3"></i>embeddings.token_type_embeddings</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>nomic_bert</td><td>nomic_bert</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>30528</td><td>30528</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix NomicBert omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>84</td><td>14</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>MLP</td><td>12</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 57 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>n_embd</td><td>768</td><td>64</td></tr><tr><td>n_head</td><td>12</td><td>4</td></tr><tr><td>n_inner</td><td>3072</td><td>128</td></tr><tr><td>n_layer</td><td>12</td><td>2</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/opt-125m-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/opt-125m</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/opt-125m-mic/tree/30dda77b685d9eb9cbf9efebdfdb2a5f14028349" title="huggingface-mic/opt-125m-mic@30dda77b685d9eb9cbf9efebdfdb2a5f14028349">huggingface-mic/opt-125m-mic@30dda77b685d</a> β <a href="https://huggingface.co/facebook/opt-125m/tree/27dcfa74d334bc871f3234de431e71c6eeba5dd6" title="facebook/opt-125m@27dcfa74d334bc871f3234de431e71c6eeba5dd6">facebook/opt-125m@27dcfa74d334</a> Β· OPTForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">125.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">38.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.679136" title="model.decoder.layers: 85.1M (67.9%)"></i><i class="c1" style="flex:0.320864" title="model.decoder: 40.2M (32.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.001721" title="model.decoder.layers: 66.9K (0.2%)"></i><i class="c1" style="flex:0.998279" title="model.decoder: 38.8M (99.8%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.decoder.layers</td><td>85.1M</td><td>67.9%</td><td>66.9K</td><td>0.2%</td></tr><tr><td><i class="c1"></i>model.decoder</td><td>40.2M</td><td>32.1%</td><td>38.8M</td><td>99.8%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>opt</td><td>opt</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50272</td><td>50272</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OPT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>73</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ReLU</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>LearnedPositionalEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/opt-125m</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/opt-125m-mic/tree/30dda77b685d9eb9cbf9efebdfdb2a5f14028349" title="huggingface-mic/opt-125m-mic@30dda77b685d9eb9cbf9efebdfdb2a5f14028349">huggingface-mic/opt-125m-mic@30dda77b685d</a> β <a href="https://huggingface.co/facebook/opt-125m/tree/27dcfa74d334bc871f3234de431e71c6eeba5dd6" title="facebook/opt-125m@27dcfa74d334bc871f3234de431e71c6eeba5dd6">facebook/opt-125m@27dcfa74d334</a> Β· OPTForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=opt-125m-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">125.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">38.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.679136" title="model.decoder.layers: 85.1M (67.9%)"></i><i class="c1" style="flex:0.320864" title="model.decoder: 40.2M (32.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.001721" title="model.decoder.layers: 66.9K (0.2%)"></i><i class="c1" style="flex:0.998279" title="model.decoder: 38.8M (99.8%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.decoder.layers</td><td>85.1M</td><td>67.9%</td><td>66.9K</td><td>0.2%</td></tr><tr><td><i class="c1"></i>model.decoder</td><td>40.2M</td><td>32.1%</td><td>38.8M</td><td>99.8%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>opt</td><td>opt</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50272</td><td>50272</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OPT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>73</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>ReLU</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>LearnedPositionalEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/opus-mt-fr-en-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Helsinki-NLP/opus-mt-fr-en</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/opus-mt-fr-en-mic/tree/d09105cda74d27a8ab952466f1810a83cc412893" title="huggingface-mic/opus-mt-fr-en-mic@d09105cda74d27a8ab952466f1810a83cc412893">huggingface-mic/opus-mt-fr-en-mic@d09105cda74d</a> β <a href="https://huggingface.co/Helsinki-NLP/opus-mt-fr-en/tree/c4aed37b318c763fd177aa449b44e3b783cc6c02" title="Helsinki-NLP/opus-mt-fr-en@c4aed37b318c763fd177aa449b44e3b783cc6c02">Helsinki-NLP/opus-mt-fr-en@c4aed37b318c</a> Β· MarianMTModel Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">75.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: MarianEncoderLayer"></i><i class="c0" title="layer 1: MarianEncoderLayer"></i><i class="c0" title="layer 2: MarianEncoderLayer"></i><i class="c0" title="layer 3: MarianEncoderLayer"></i><i class="c0" title="layer 4: MarianEncoderLayer"></i><i class="c0" title="layer 5: MarianEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MarianEncoderLayer"></i><i class="c0" title="layer 1: MarianEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MarianEncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: MarianDecoderLayer"></i><i class="c0" title="layer 1: MarianDecoderLayer"></i><i class="c0" title="layer 2: MarianDecoderLayer"></i><i class="c0" title="layer 3: MarianDecoderLayer"></i><i class="c0" title="layer 4: MarianDecoderLayer"></i><i class="c0" title="layer 5: MarianDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MarianDecoderLayer"></i><i class="c0" title="layer 1: MarianDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MarianDecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.405558" title="model.shared: 30.5M (40.6%)"></i><i class="c1" style="flex:0.335723" title="model.decoder.layers: 25.2M (33.6%)"></i><i class="c2" style="flex:0.251741" title="model.encoder.layers: 18.9M (25.2%)"></i><i class="c3" style="flex:0.003489" title="model.encoder: 262.1K (0.3%)"></i><i class="c4" style="flex:0.003489" title="model.decoder: 262.1K (0.3%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.942363" title="model.shared: 3.8M (94.2%)"></i><i class="c1" style="flex:0.024860" title="model.decoder.layers: 100.5K (2.5%)"></i><i class="c2" style="flex:0.016563" title="model.encoder.layers: 66.9K (1.7%)"></i><i class="c3" style="flex:0.008107" title="model.encoder: 32.8K (0.8%)"></i><i class="c4" style="flex:0.008107" title="model.decoder: 32.8K (0.8%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>30.5M</td><td>40.6%</td><td>3.8M</td><td>94.2%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>25.2M</td><td>33.6%</td><td>100.5K</td><td>2.5%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>18.9M</td><td>25.2%</td><td>66.9K</td><td>1.7%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>262.1K</td><td>0.3%</td><td>32.8K</td><td>0.8%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>262.1K</td><td>0.3%</td><td>32.8K</td><td>0.8%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>marian</td><td>marian</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>59514</td><td>59514</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MarianMT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>97</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>30</td><td>10</td></tr><tr class=""><td>MarianAttention</td><td>18</td><td>6</td></tr><tr class=""><td>SiLU</td><td>12</td><td>4</td></tr><tr class=""><td>MarianDecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>MarianEncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>MarianSinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>MarianDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>MarianEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MarianModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>512</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Helsinki-NLP/opus-mt-fr-en</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/opus-mt-fr-en-mic/tree/d09105cda74d27a8ab952466f1810a83cc412893" title="huggingface-mic/opus-mt-fr-en-mic@d09105cda74d27a8ab952466f1810a83cc412893">huggingface-mic/opus-mt-fr-en-mic@d09105cda74d</a> β <a href="https://huggingface.co/Helsinki-NLP/opus-mt-fr-en/tree/c4aed37b318c763fd177aa449b44e3b783cc6c02" title="Helsinki-NLP/opus-mt-fr-en@c4aed37b318c763fd177aa449b44e3b783cc6c02">Helsinki-NLP/opus-mt-fr-en@c4aed37b318c</a> Β· MarianMTModel Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=opus-mt-fr-en-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">75.1M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: MarianEncoderLayer"></i><i class="c0" title="layer 1: MarianEncoderLayer"></i><i class="c0" title="layer 2: MarianEncoderLayer"></i><i class="c0" title="layer 3: MarianEncoderLayer"></i><i class="c0" title="layer 4: MarianEncoderLayer"></i><i class="c0" title="layer 5: MarianEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MarianEncoderLayer"></i><i class="c0" title="layer 1: MarianEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MarianEncoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: MarianDecoderLayer"></i><i class="c0" title="layer 1: MarianDecoderLayer"></i><i class="c0" title="layer 2: MarianDecoderLayer"></i><i class="c0" title="layer 3: MarianDecoderLayer"></i><i class="c0" title="layer 4: MarianDecoderLayer"></i><i class="c0" title="layer 5: MarianDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: MarianDecoderLayer"></i><i class="c0" title="layer 1: MarianDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MarianDecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.405558" title="model.shared: 30.5M (40.6%)"></i><i class="c1" style="flex:0.335723" title="model.decoder.layers: 25.2M (33.6%)"></i><i class="c2" style="flex:0.251741" title="model.encoder.layers: 18.9M (25.2%)"></i><i class="c3" style="flex:0.003489" title="model.encoder: 262.1K (0.3%)"></i><i class="c4" style="flex:0.003489" title="model.decoder: 262.1K (0.3%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.942363" title="model.shared: 3.8M (94.2%)"></i><i class="c1" style="flex:0.024860" title="model.decoder.layers: 100.5K (2.5%)"></i><i class="c2" style="flex:0.016563" title="model.encoder.layers: 66.9K (1.7%)"></i><i class="c3" style="flex:0.008107" title="model.encoder: 32.8K (0.8%)"></i><i class="c4" style="flex:0.008107" title="model.decoder: 32.8K (0.8%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.shared</td><td>30.5M</td><td>40.6%</td><td>3.8M</td><td>94.2%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>25.2M</td><td>33.6%</td><td>100.5K</td><td>2.5%</td></tr><tr><td><i class="c2"></i>model.encoder.layers</td><td>18.9M</td><td>25.2%</td><td>66.9K</td><td>1.7%</td></tr><tr><td><i class="c3"></i>model.encoder</td><td>262.1K</td><td>0.3%</td><td>32.8K</td><td>0.8%</td></tr><tr><td><i class="c4"></i>model.decoder</td><td>262.1K</td><td>0.3%</td><td>32.8K</td><td>0.8%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>marian</td><td>marian</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>59514</td><td>59514</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix MarianMT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>97</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>30</td><td>10</td></tr><tr class=""><td>MarianAttention</td><td>18</td><td>6</td></tr><tr class=""><td>SiLU</td><td>12</td><td>4</td></tr><tr class=""><td>MarianDecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>MarianEncoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>MarianSinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>MarianDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>MarianEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MarianModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>512</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>2048</td><td>128</td></tr><tr><td>encoder_layers</td><td>6</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/owlv2-base-patch16-ensemble-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/owlv2-base-patch16-ensemble</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/owlv2-base-patch16-ensemble-mic/tree/de5f6eb0e6f68273b883bf046cd0b1dc76bce3ad" title="huggingface-mic/owlv2-base-patch16-ensemble-mic@de5f6eb0e6f68273b883bf046cd0b1dc76bce3ad">huggingface-mic/owlv2-base-patch16-ensemble-mic@de5f6eb0e6f6</a> β <a href="https://huggingface.co/google/owlv2-base-patch16-ensemble/tree/cfd3195ba4ea9592eec887ded089f4c08eff231d" title="google/owlv2-base-patch16-ensemble@cfd3195ba4ea9592eec887ded089f4c08eff231d">google/owlv2-base-patch16-ensemble@cfd3195ba4ea</a> Β· Owlv2ForObjectDetection Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">155.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>owlv2.text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>owlv2.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.548856" title="owlv2.vision_model.encoder.layers: 85.1M (54.9%)"></i><i class="c1" style="flex:0.244108" title="owlv2.text_model.encoder.layers: 37.8M (24.4%)"></i><i class="c2" style="flex:0.163300" title="owlv2.text_model: 25.3M (16.3%)"></i><i class="c3" style="flex:0.021677" title="owlv2.vision_model: 3.4M (2.2%)"></i><i class="c4" style="flex:0.003811" title="objectness_head.dense0: 590.6K (0.4%)"></i><i class="c5" style="flex:0.003811" title="box_head.dense0: 590.6K (0.4%)"></i><i class="c6" style="flex:0.003811" title="objectness_head.dense1: 590.6K (0.4%)"></i><i class="c7" style="flex:0.010626" title="10 others: 1.6M (1.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018561" title="owlv2.vision_model.encoder.layers: 66.9K (1.9%)"></i><i class="c1" style="flex:0.018561" title="owlv2.text_model.encoder.layers: 66.9K (1.9%)"></i><i class="c2" style="flex:0.877061" title="owlv2.text_model: 3.2M (87.7%)"></i><i class="c3" style="flex:0.077616" title="owlv2.vision_model: 279.9K (7.8%)"></i><i class="c4" style="flex:0.001153" title="objectness_head.dense0: 4.2K (0.1%)"></i><i class="c5" style="flex:0.001153" title="box_head.dense0: 4.2K (0.1%)"></i><i class="c6" style="flex:0.001153" title="objectness_head.dense1: 4.2K (0.1%)"></i><i class="c7" style="flex:0.004740" title="10 others: 17.1K (0.5%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>owlv2.vision_model.encoder.layers</td><td>85.1M</td><td>54.9%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c1"></i>owlv2.text_model.encoder.layers</td><td>37.8M</td><td>24.4%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c2"></i>owlv2.text_model</td><td>25.3M</td><td>16.3%</td><td>3.2M</td><td>87.7%</td></tr><tr><td><i class="c3"></i>owlv2.vision_model</td><td>3.4M</td><td>2.2%</td><td>279.9K</td><td>7.8%</td></tr><tr><td><i class="c4"></i>objectness_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c5"></i>box_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c6"></i>objectness_head.dense1</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>10 others</td><td>1.6M</td><td>1.1%</td><td>17.1K</td><td>0.5%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>owlv2</td><td>owlv2</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>owlv2_text_model</td><td>owlv2_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49408</td><td>49408</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>owlv2_vision_model</td><td>owlv2_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>960</td><td>960</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Owlv2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>155</td><td>35</td></tr><tr class=""><td>LayerNorm</td><td>52</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>MLP</td><td>24</td><td>4</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>GELU</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>BoxPredictionHead</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>ELU</td><td>1</td><td>1</td></tr><tr class=""><td>ClassPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>Sigmoid</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 47 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>projection_dim</td><td>512</td><td>64</td></tr><tr><td>text_config.hidden_size</td><td>512</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>2048</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/owlv2-base-patch16-ensemble</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/owlv2-base-patch16-ensemble-mic/tree/de5f6eb0e6f68273b883bf046cd0b1dc76bce3ad" title="huggingface-mic/owlv2-base-patch16-ensemble-mic@de5f6eb0e6f68273b883bf046cd0b1dc76bce3ad">huggingface-mic/owlv2-base-patch16-ensemble-mic@de5f6eb0e6f6</a> β <a href="https://huggingface.co/google/owlv2-base-patch16-ensemble/tree/cfd3195ba4ea9592eec887ded089f4c08eff231d" title="google/owlv2-base-patch16-ensemble@cfd3195ba4ea9592eec887ded089f4c08eff231d">google/owlv2-base-patch16-ensemble@cfd3195ba4ea</a> Β· Owlv2ForObjectDetection Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=owlv2-base-patch16-ensemble-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">155.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>owlv2.text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>owlv2.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.548856" title="owlv2.vision_model.encoder.layers: 85.1M (54.9%)"></i><i class="c1" style="flex:0.244108" title="owlv2.text_model.encoder.layers: 37.8M (24.4%)"></i><i class="c2" style="flex:0.163300" title="owlv2.text_model: 25.3M (16.3%)"></i><i class="c3" style="flex:0.021677" title="owlv2.vision_model: 3.4M (2.2%)"></i><i class="c4" style="flex:0.003811" title="objectness_head.dense0: 590.6K (0.4%)"></i><i class="c5" style="flex:0.003811" title="box_head.dense0: 590.6K (0.4%)"></i><i class="c6" style="flex:0.003811" title="objectness_head.dense1: 590.6K (0.4%)"></i><i class="c7" style="flex:0.010626" title="10 others: 1.6M (1.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018561" title="owlv2.vision_model.encoder.layers: 66.9K (1.9%)"></i><i class="c1" style="flex:0.018561" title="owlv2.text_model.encoder.layers: 66.9K (1.9%)"></i><i class="c2" style="flex:0.877061" title="owlv2.text_model: 3.2M (87.7%)"></i><i class="c3" style="flex:0.077616" title="owlv2.vision_model: 279.9K (7.8%)"></i><i class="c4" style="flex:0.001153" title="objectness_head.dense0: 4.2K (0.1%)"></i><i class="c5" style="flex:0.001153" title="box_head.dense0: 4.2K (0.1%)"></i><i class="c6" style="flex:0.001153" title="objectness_head.dense1: 4.2K (0.1%)"></i><i class="c7" style="flex:0.004740" title="10 others: 17.1K (0.5%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>owlv2.vision_model.encoder.layers</td><td>85.1M</td><td>54.9%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c1"></i>owlv2.text_model.encoder.layers</td><td>37.8M</td><td>24.4%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c2"></i>owlv2.text_model</td><td>25.3M</td><td>16.3%</td><td>3.2M</td><td>87.7%</td></tr><tr><td><i class="c3"></i>owlv2.vision_model</td><td>3.4M</td><td>2.2%</td><td>279.9K</td><td>7.8%</td></tr><tr><td><i class="c4"></i>objectness_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c5"></i>box_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c6"></i>objectness_head.dense1</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>10 others</td><td>1.6M</td><td>1.1%</td><td>17.1K</td><td>0.5%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>owlv2</td><td>owlv2</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>owlv2_text_model</td><td>owlv2_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49408</td><td>49408</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>owlv2_vision_model</td><td>owlv2_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>960</td><td>960</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Owlv2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>155</td><td>35</td></tr><tr class=""><td>LayerNorm</td><td>52</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>MLP</td><td>24</td><td>4</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>GELU</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>BoxPredictionHead</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>ELU</td><td>1</td><td>1</td></tr><tr class=""><td>ClassPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>Sigmoid</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 47 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>projection_dim</td><td>512</td><td>64</td></tr><tr><td>text_config.hidden_size</td><td>512</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>2048</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/owlvit-base-patch32-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/owlvit-base-patch32</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/owlvit-base-patch32-mic/tree/09d714a0dd1f21dcf214cb144777eb72042a08d0" title="huggingface-mic/owlvit-base-patch32-mic@09d714a0dd1f21dcf214cb144777eb72042a08d0">huggingface-mic/owlvit-base-patch32-mic@09d714a0dd1f</a> β <a href="https://huggingface.co/google/owlvit-base-patch32/tree/cbc355fb364588351c5d51c7f74465e8e7ec6f72" title="google/owlvit-base-patch32@cbc355fb364588351c5d51c7f74465e8e7ec6f72">google/owlvit-base-patch32@cbc355fb3645</a> Β· OwlViTForObjectDetection Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">153.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>owlvit.text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>owlvit.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.555070" title="owlvit.vision_model.encoder.layers: 85.1M (55.5%)"></i><i class="c1" style="flex:0.246872" title="owlvit.text_model.encoder.layers: 37.8M (24.7%)"></i><i class="c2" style="flex:0.165149" title="owlvit.text_model: 25.3M (16.5%)"></i><i class="c3" style="flex:0.018314" title="owlvit.vision_model: 2.8M (1.8%)"></i><i class="c4" style="flex:0.003854" title="box_head.dense1: 590.6K (0.4%)"></i><i class="c5" style="flex:0.003854" title="box_head.dense0: 590.6K (0.4%)"></i><i class="c6" style="flex:0.002569" title="class_head.dense0: 393.7K (0.3%)"></i><i class="c7" style="flex:0.004317" title="7 others: 661.5K (0.4%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018846" title="owlvit.vision_model.encoder.layers: 66.9K (1.9%)"></i><i class="c1" style="flex:0.018846" title="owlvit.text_model.encoder.layers: 66.9K (1.9%)"></i><i class="c2" style="flex:0.890509" title="owlvit.text_model: 3.2M (89.1%)"></i><i class="c3" style="flex:0.065834" title="owlvit.vision_model: 233.9K (6.6%)"></i><i class="c4" style="flex:0.001171" title="box_head.dense1: 4.2K (0.1%)"></i><i class="c5" style="flex:0.001171" title="box_head.dense0: 4.2K (0.1%)"></i><i class="c6" style="flex:0.001171" title="class_head.dense0: 4.2K (0.1%)"></i><i class="c7" style="flex:0.002452" title="7 others: 8.7K (0.2%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>owlvit.vision_model.encoder.layers</td><td>85.1M</td><td>55.5%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c1"></i>owlvit.text_model.encoder.layers</td><td>37.8M</td><td>24.7%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c2"></i>owlvit.text_model</td><td>25.3M</td><td>16.5%</td><td>3.2M</td><td>89.1%</td></tr><tr><td><i class="c3"></i>owlvit.vision_model</td><td>2.8M</td><td>1.8%</td><td>233.9K</td><td>6.6%</td></tr><tr><td><i class="c4"></i>box_head.dense1</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c5"></i>box_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c6"></i>class_head.dense0</td><td>393.7K</td><td>0.3%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>7 others</td><td>661.5K</td><td>0.4%</td><td>8.7K</td><td>0.2%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>owlvit</td><td>owlvit</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>owlvit_text_model</td><td>owlvit_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49408</td><td>49408</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>owlvit_vision_model</td><td>owlvit_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>768</td><td>768</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>32</td><td>32</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OwlViT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>152</td><td>32</td></tr><tr class=""><td>LayerNorm</td><td>52</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>MLP</td><td>24</td><td>4</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>ELU</td><td>1</td><td>1</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>BoxPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ClassPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>Sigmoid</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 84 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>projection_dim</td><td>512</td><td>64</td></tr><tr><td>text_config.hidden_size</td><td>512</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>2048</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/owlvit-base-patch32</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/owlvit-base-patch32-mic/tree/09d714a0dd1f21dcf214cb144777eb72042a08d0" title="huggingface-mic/owlvit-base-patch32-mic@09d714a0dd1f21dcf214cb144777eb72042a08d0">huggingface-mic/owlvit-base-patch32-mic@09d714a0dd1f</a> β <a href="https://huggingface.co/google/owlvit-base-patch32/tree/cbc355fb364588351c5d51c7f74465e8e7ec6f72" title="google/owlvit-base-patch32@cbc355fb364588351c5d51c7f74465e8e7ec6f72">google/owlvit-base-patch32@cbc355fb3645</a> Β· OwlViTForObjectDetection Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=owlvit-base-patch32-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">153.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>owlvit.text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>owlvit.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.555070" title="owlvit.vision_model.encoder.layers: 85.1M (55.5%)"></i><i class="c1" style="flex:0.246872" title="owlvit.text_model.encoder.layers: 37.8M (24.7%)"></i><i class="c2" style="flex:0.165149" title="owlvit.text_model: 25.3M (16.5%)"></i><i class="c3" style="flex:0.018314" title="owlvit.vision_model: 2.8M (1.8%)"></i><i class="c4" style="flex:0.003854" title="box_head.dense1: 590.6K (0.4%)"></i><i class="c5" style="flex:0.003854" title="box_head.dense0: 590.6K (0.4%)"></i><i class="c6" style="flex:0.002569" title="class_head.dense0: 393.7K (0.3%)"></i><i class="c7" style="flex:0.004317" title="7 others: 661.5K (0.4%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018846" title="owlvit.vision_model.encoder.layers: 66.9K (1.9%)"></i><i class="c1" style="flex:0.018846" title="owlvit.text_model.encoder.layers: 66.9K (1.9%)"></i><i class="c2" style="flex:0.890509" title="owlvit.text_model: 3.2M (89.1%)"></i><i class="c3" style="flex:0.065834" title="owlvit.vision_model: 233.9K (6.6%)"></i><i class="c4" style="flex:0.001171" title="box_head.dense1: 4.2K (0.1%)"></i><i class="c5" style="flex:0.001171" title="box_head.dense0: 4.2K (0.1%)"></i><i class="c6" style="flex:0.001171" title="class_head.dense0: 4.2K (0.1%)"></i><i class="c7" style="flex:0.002452" title="7 others: 8.7K (0.2%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>owlvit.vision_model.encoder.layers</td><td>85.1M</td><td>55.5%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c1"></i>owlvit.text_model.encoder.layers</td><td>37.8M</td><td>24.7%</td><td>66.9K</td><td>1.9%</td></tr><tr><td><i class="c2"></i>owlvit.text_model</td><td>25.3M</td><td>16.5%</td><td>3.2M</td><td>89.1%</td></tr><tr><td><i class="c3"></i>owlvit.vision_model</td><td>2.8M</td><td>1.8%</td><td>233.9K</td><td>6.6%</td></tr><tr><td><i class="c4"></i>box_head.dense1</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c5"></i>box_head.dense0</td><td>590.6K</td><td>0.4%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c6"></i>class_head.dense0</td><td>393.7K</td><td>0.3%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>7 others</td><td>661.5K</td><td>0.4%</td><td>8.7K</td><td>0.2%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>owlvit</td><td>owlvit</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>owlvit_text_model</td><td>owlvit_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49408</td><td>49408</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>owlvit_vision_model</td><td>owlvit_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>quick_gelu</td><td>quick_gelu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>768</td><td>768</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>32</td><td>32</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OwlViT omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>152</td><td>32</td></tr><tr class=""><td>LayerNorm</td><td>52</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>MLP</td><td>24</td><td>4</td></tr><tr class=""><td>QuickGELUActivation</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>ELU</td><td>1</td><td>1</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>BoxPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ClassPredictionHead</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>Sigmoid</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 84 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>projection_dim</td><td>512</td><td>64</td></tr><tr><td>text_config.hidden_size</td><td>512</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>2048</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/paligemma-3b-pt-224-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>leo009/paligemma-3b-pt-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/paligemma-3b-pt-224-mic/tree/adfd7e09d295b39ddf4dbeb88ffdb645139b72d8" title="huggingface-mic/paligemma-3b-pt-224-mic@adfd7e09d295b39ddf4dbeb88ffdb645139b72d8">huggingface-mic/paligemma-3b-pt-224-mic@adfd7e09d295</a> β <a href="https://huggingface.co/leo009/paligemma-3b-pt-224/tree/39996beb6fb17c5d16a50d3ef8f7a96ad9d03986" title="leo009/paligemma-3b-pt-224@39996beb6fb17c5d16a50d3ef8f7a96ad9d03986">leo009/paligemma-3b-pt-224@39996beb6fb1</a> Β· PaliGemmaForConditionalGeneration Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.9B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.encoder.layers</h3><div class="row"><span class="lab">Original Β· 27</span><div class="cells"><i class="c0" title="layer 0: SiglipEncoderLayer"></i><i class="c0" title="layer 1: SiglipEncoderLayer"></i><i class="c0" title="layer 2: SiglipEncoderLayer"></i><i class="c0" title="layer 3: SiglipEncoderLayer"></i><i class="c0" title="layer 4: SiglipEncoderLayer"></i><i class="c0" title="layer 5: SiglipEncoderLayer"></i><i class="c0" title="layer 6: SiglipEncoderLayer"></i><i class="c0" title="layer 7: SiglipEncoderLayer"></i><i class="c0" title="layer 8: SiglipEncoderLayer"></i><i class="c0" title="layer 9: SiglipEncoderLayer"></i><i class="c0" title="layer 10: SiglipEncoderLayer"></i><i class="c0" title="layer 11: SiglipEncoderLayer"></i><i class="c0" title="layer 12: SiglipEncoderLayer"></i><i class="c0" title="layer 13: SiglipEncoderLayer"></i><i class="c0" title="layer 14: SiglipEncoderLayer"></i><i class="c0" title="layer 15: SiglipEncoderLayer"></i><i class="c0" title="layer 16: SiglipEncoderLayer"></i><i class="c0" title="layer 17: SiglipEncoderLayer"></i><i class="c0" title="layer 18: SiglipEncoderLayer"></i><i class="c0" title="layer 19: SiglipEncoderLayer"></i><i class="c0" title="layer 20: SiglipEncoderLayer"></i><i class="c0" title="layer 21: SiglipEncoderLayer"></i><i class="c0" title="layer 22: SiglipEncoderLayer"></i><i class="c0" title="layer 23: SiglipEncoderLayer"></i><i class="c0" title="layer 24: SiglipEncoderLayer"></i><i class="c0" title="layer 25: SiglipEncoderLayer"></i><i class="c0" title="layer 26: SiglipEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: SiglipEncoderLayer"></i><i class="c0" title="layer 1: SiglipEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SiglipEncoderLayer</td><td>27</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 18</span><div class="cells"><i class="c0" title="layer 0: GemmaDecoderLayer"></i><i class="c0" title="layer 1: GemmaDecoderLayer"></i><i class="c0" title="layer 2: GemmaDecoderLayer"></i><i class="c0" title="layer 3: GemmaDecoderLayer"></i><i class="c0" title="layer 4: GemmaDecoderLayer"></i><i class="c0" title="layer 5: GemmaDecoderLayer"></i><i class="c0" title="layer 6: GemmaDecoderLayer"></i><i class="c0" title="layer 7: GemmaDecoderLayer"></i><i class="c0" title="layer 8: GemmaDecoderLayer"></i><i class="c0" title="layer 9: GemmaDecoderLayer"></i><i class="c0" title="layer 10: GemmaDecoderLayer"></i><i class="c0" title="layer 11: GemmaDecoderLayer"></i><i class="c0" title="layer 12: GemmaDecoderLayer"></i><i class="c0" title="layer 13: GemmaDecoderLayer"></i><i class="c0" title="layer 14: GemmaDecoderLayer"></i><i class="c0" title="layer 15: GemmaDecoderLayer"></i><i class="c0" title="layer 16: GemmaDecoderLayer"></i><i class="c0" title="layer 17: GemmaDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: GemmaDecoderLayer"></i><i class="c0" title="layer 1: GemmaDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>GemmaDecoderLayer</td><td>18</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.677922" title="model.language_model.layers: 2.0B (67.8%)"></i><i class="c1" style="flex:0.180190" title="model.language_model: 526.8M (18.0%)"></i><i class="c2" style="flex:0.140746" title="model.vision_tower.encoder.layers: 411.5M (14.1%)"></i><i class="c3" style="flex:0.000808" title="model.multi_modal_projector: 2.4M (0.1%)"></i><i class="c4" style="flex:0.000334" title="model.vision_tower: 975.7K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.004196" title="model.language_model.layers: 69.9K (0.4%)"></i><i class="c1" style="flex:0.988281" title="model.language_model: 16.5M (98.8%)"></i><i class="c2" style="flex:0.004019" title="model.vision_tower.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.000250" title="model.multi_modal_projector: 4.2K (0.0%)"></i><i class="c4" style="flex:0.003254" title="model.vision_tower: 54.2K (0.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>2.0B</td><td>67.8%</td><td>69.9K</td><td>0.4%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>526.8M</td><td>18.0%</td><td>16.5M</td><td>98.8%</td></tr><tr><td><i class="c2"></i>model.vision_tower.encoder.layers</td><td>411.5M</td><td>14.1%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.multi_modal_projector</td><td>2.4M</td><td>0.1%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c4"></i>model.vision_tower</td><td>975.7K</td><td>0.0%</td><td>54.2K</td><td>0.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>paligemma</td><td>paligemma</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>257216</td><td>257216</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>siglip_vision_model</td><td>siglip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MQA (8 heads, 1 KV)</td><td>MQA (4 heads, 1 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>gemma</td><td>gemma</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>257216</td><td>257216</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix PaliGemma omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>290</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>55</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>45</td><td>4</td></tr><tr class=""><td>GemmaRMSNorm</td><td>37</td><td>5</td></tr><tr class=""><td>SiglipAttention</td><td>27</td><td>2</td></tr><tr class=""><td>SiglipEncoderLayer</td><td>27</td><td>2</td></tr><tr class=""><td>SiglipMLP</td><td>27</td><td>2</td></tr><tr class=""><td>GemmaAttention</td><td>18</td><td>2</td></tr><tr class=""><td>GemmaDecoderLayer</td><td>18</td><td>2</td></tr><tr class=""><td>GemmaMLP</td><td>18</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaModel</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaTextScaledWordEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipVisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipVisionModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">12 fields changed, 59 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>projection_dim</td><td>2048</td><td>64</td></tr><tr><td>text_config.head_dim</td><td>256</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>16384</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>18</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>1152</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4304</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>27</td><td>2</td></tr><tr><td>vision_config.projection_dim</td><td>2048</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>leo009/paligemma-3b-pt-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/paligemma-3b-pt-224-mic/tree/adfd7e09d295b39ddf4dbeb88ffdb645139b72d8" title="huggingface-mic/paligemma-3b-pt-224-mic@adfd7e09d295b39ddf4dbeb88ffdb645139b72d8">huggingface-mic/paligemma-3b-pt-224-mic@adfd7e09d295</a> β <a href="https://huggingface.co/leo009/paligemma-3b-pt-224/tree/39996beb6fb17c5d16a50d3ef8f7a96ad9d03986" title="leo009/paligemma-3b-pt-224@39996beb6fb17c5d16a50d3ef8f7a96ad9d03986">leo009/paligemma-3b-pt-224@39996beb6fb1</a> Β· PaliGemmaForConditionalGeneration Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=paligemma-3b-pt-224-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.9B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">16.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_tower.encoder.layers</h3><div class="row"><span class="lab">Original Β· 27</span><div class="cells"><i class="c0" title="layer 0: SiglipEncoderLayer"></i><i class="c0" title="layer 1: SiglipEncoderLayer"></i><i class="c0" title="layer 2: SiglipEncoderLayer"></i><i class="c0" title="layer 3: SiglipEncoderLayer"></i><i class="c0" title="layer 4: SiglipEncoderLayer"></i><i class="c0" title="layer 5: SiglipEncoderLayer"></i><i class="c0" title="layer 6: SiglipEncoderLayer"></i><i class="c0" title="layer 7: SiglipEncoderLayer"></i><i class="c0" title="layer 8: SiglipEncoderLayer"></i><i class="c0" title="layer 9: SiglipEncoderLayer"></i><i class="c0" title="layer 10: SiglipEncoderLayer"></i><i class="c0" title="layer 11: SiglipEncoderLayer"></i><i class="c0" title="layer 12: SiglipEncoderLayer"></i><i class="c0" title="layer 13: SiglipEncoderLayer"></i><i class="c0" title="layer 14: SiglipEncoderLayer"></i><i class="c0" title="layer 15: SiglipEncoderLayer"></i><i class="c0" title="layer 16: SiglipEncoderLayer"></i><i class="c0" title="layer 17: SiglipEncoderLayer"></i><i class="c0" title="layer 18: SiglipEncoderLayer"></i><i class="c0" title="layer 19: SiglipEncoderLayer"></i><i class="c0" title="layer 20: SiglipEncoderLayer"></i><i class="c0" title="layer 21: SiglipEncoderLayer"></i><i class="c0" title="layer 22: SiglipEncoderLayer"></i><i class="c0" title="layer 23: SiglipEncoderLayer"></i><i class="c0" title="layer 24: SiglipEncoderLayer"></i><i class="c0" title="layer 25: SiglipEncoderLayer"></i><i class="c0" title="layer 26: SiglipEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: SiglipEncoderLayer"></i><i class="c0" title="layer 1: SiglipEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SiglipEncoderLayer</td><td>27</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 18</span><div class="cells"><i class="c0" title="layer 0: GemmaDecoderLayer"></i><i class="c0" title="layer 1: GemmaDecoderLayer"></i><i class="c0" title="layer 2: GemmaDecoderLayer"></i><i class="c0" title="layer 3: GemmaDecoderLayer"></i><i class="c0" title="layer 4: GemmaDecoderLayer"></i><i class="c0" title="layer 5: GemmaDecoderLayer"></i><i class="c0" title="layer 6: GemmaDecoderLayer"></i><i class="c0" title="layer 7: GemmaDecoderLayer"></i><i class="c0" title="layer 8: GemmaDecoderLayer"></i><i class="c0" title="layer 9: GemmaDecoderLayer"></i><i class="c0" title="layer 10: GemmaDecoderLayer"></i><i class="c0" title="layer 11: GemmaDecoderLayer"></i><i class="c0" title="layer 12: GemmaDecoderLayer"></i><i class="c0" title="layer 13: GemmaDecoderLayer"></i><i class="c0" title="layer 14: GemmaDecoderLayer"></i><i class="c0" title="layer 15: GemmaDecoderLayer"></i><i class="c0" title="layer 16: GemmaDecoderLayer"></i><i class="c0" title="layer 17: GemmaDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: GemmaDecoderLayer"></i><i class="c0" title="layer 1: GemmaDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>GemmaDecoderLayer</td><td>18</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.677922" title="model.language_model.layers: 2.0B (67.8%)"></i><i class="c1" style="flex:0.180190" title="model.language_model: 526.8M (18.0%)"></i><i class="c2" style="flex:0.140746" title="model.vision_tower.encoder.layers: 411.5M (14.1%)"></i><i class="c3" style="flex:0.000808" title="model.multi_modal_projector: 2.4M (0.1%)"></i><i class="c4" style="flex:0.000334" title="model.vision_tower: 975.7K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.004196" title="model.language_model.layers: 69.9K (0.4%)"></i><i class="c1" style="flex:0.988281" title="model.language_model: 16.5M (98.8%)"></i><i class="c2" style="flex:0.004019" title="model.vision_tower.encoder.layers: 66.9K (0.4%)"></i><i class="c3" style="flex:0.000250" title="model.multi_modal_projector: 4.2K (0.0%)"></i><i class="c4" style="flex:0.003254" title="model.vision_tower: 54.2K (0.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>2.0B</td><td>67.8%</td><td>69.9K</td><td>0.4%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>526.8M</td><td>18.0%</td><td>16.5M</td><td>98.8%</td></tr><tr><td><i class="c2"></i>model.vision_tower.encoder.layers</td><td>411.5M</td><td>14.1%</td><td>66.9K</td><td>0.4%</td></tr><tr><td><i class="c3"></i>model.multi_modal_projector</td><td>2.4M</td><td>0.1%</td><td>4.2K</td><td>0.0%</td></tr><tr><td><i class="c4"></i>model.vision_tower</td><td>975.7K</td><td>0.0%</td><td>54.2K</td><td>0.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>paligemma</td><td>paligemma</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>257216</td><td>257216</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>siglip_vision_model</td><td>siglip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MQA (8 heads, 1 KV)</td><td>MQA (4 heads, 1 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>gemma</td><td>gemma</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>257216</td><td>257216</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix PaliGemma omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>290</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>55</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>45</td><td>4</td></tr><tr class=""><td>GemmaRMSNorm</td><td>37</td><td>5</td></tr><tr class=""><td>SiglipAttention</td><td>27</td><td>2</td></tr><tr class=""><td>SiglipEncoderLayer</td><td>27</td><td>2</td></tr><tr class=""><td>SiglipMLP</td><td>27</td><td>2</td></tr><tr class=""><td>GemmaAttention</td><td>18</td><td>2</td></tr><tr class=""><td>GemmaDecoderLayer</td><td>18</td><td>2</td></tr><tr class=""><td>GemmaMLP</td><td>18</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaModel</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>GemmaTextScaledWordEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipVisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>SiglipVisionModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">12 fields changed, 59 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>projection_dim</td><td>2048</td><td>64</td></tr><tr><td>text_config.head_dim</td><td>256</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>16384</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>18</td><td>2</td></tr><tr><td>vision_config.hidden_size</td><td>1152</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4304</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>27</td><td>2</td></tr><tr><td>vision_config.projection_dim</td><td>2048</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/pegasus-xsum-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/pegasus-xsum</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pegasus-xsum-mic/tree/3703172718d029fd3c19c93391ed0e84937d6eb6" title="huggingface-mic/pegasus-xsum-mic@3703172718d029fd3c19c93391ed0e84937d6eb6">huggingface-mic/pegasus-xsum-mic@3703172718d0</a> β <a href="https://huggingface.co/google/pegasus-xsum/tree/8d8ffc158a3bee9fbb03afacdfc347c823c5ec8b" title="google/pegasus-xsum@8d8ffc158a3bee9fbb03afacdfc347c823c5ec8b">google/pegasus-xsum@8d8ffc158a3b</a> Β· PegasusForConditionalGeneration Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">569.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>16</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>16</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.471694" title="model.decoder.layers: 268.7M (47.2%)"></i><i class="c1" style="flex:0.353734" title="model.encoder.layers: 201.5M (35.4%)"></i><i class="c2" style="flex:0.172724" title="model.shared: 98.4M (17.3%)"></i><i class="c3" style="flex:0.000924" title="model.decoder: 526.3K (0.1%)"></i><i class="c4" style="flex:0.000924" title="model.encoder: 526.3K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.015740" title="model.decoder.layers: 100.5K (1.6%)"></i><i class="c1" style="flex:0.010487" title="model.encoder.layers: 66.9K (1.0%)"></i><i class="c2" style="flex:0.963468" title="model.shared: 6.2M (96.3%)"></i><i class="c3" style="flex:0.005153" title="model.decoder: 32.9K (0.5%)"></i><i class="c4" style="flex:0.005153" title="model.encoder: 32.9K (0.5%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.decoder.layers</td><td>268.7M</td><td>47.2%</td><td>100.5K</td><td>1.6%</td></tr><tr><td><i class="c1"></i>model.encoder.layers</td><td>201.5M</td><td>35.4%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c2"></i>model.shared</td><td>98.4M</td><td>17.3%</td><td>6.2M</td><td>96.3%</td></tr><tr><td><i class="c3"></i>model.decoder</td><td>526.3K</td><td>0.1%</td><td>32.9K</td><td>0.5%</td></tr><tr><td><i class="c4"></i>model.encoder</td><td>526.3K</td><td>0.1%</td><td>32.9K</td><td>0.5%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>pegasus</td><td>pegasus</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>96103</td><td>96103</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Pegasus omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>257</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>82</td><td>12</td></tr><tr class=""><td>Attention</td><td>48</td><td>6</td></tr><tr class=""><td>ReLU</td><td>32</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>16</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>16</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>16</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>16</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/pegasus-xsum</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pegasus-xsum-mic/tree/3703172718d029fd3c19c93391ed0e84937d6eb6" title="huggingface-mic/pegasus-xsum-mic@3703172718d029fd3c19c93391ed0e84937d6eb6">huggingface-mic/pegasus-xsum-mic@3703172718d0</a> β <a href="https://huggingface.co/google/pegasus-xsum/tree/8d8ffc158a3bee9fbb03afacdfc347c823c5ec8b" title="google/pegasus-xsum@8d8ffc158a3bee9fbb03afacdfc347c823c5ec8b">google/pegasus-xsum@8d8ffc158a3b</a> Β· PegasusForConditionalGeneration Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=pegasus-xsum-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">569.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>16</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 16</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>16</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.471694" title="model.decoder.layers: 268.7M (47.2%)"></i><i class="c1" style="flex:0.353734" title="model.encoder.layers: 201.5M (35.4%)"></i><i class="c2" style="flex:0.172724" title="model.shared: 98.4M (17.3%)"></i><i class="c3" style="flex:0.000924" title="model.decoder: 526.3K (0.1%)"></i><i class="c4" style="flex:0.000924" title="model.encoder: 526.3K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.015740" title="model.decoder.layers: 100.5K (1.6%)"></i><i class="c1" style="flex:0.010487" title="model.encoder.layers: 66.9K (1.0%)"></i><i class="c2" style="flex:0.963468" title="model.shared: 6.2M (96.3%)"></i><i class="c3" style="flex:0.005153" title="model.decoder: 32.9K (0.5%)"></i><i class="c4" style="flex:0.005153" title="model.encoder: 32.9K (0.5%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.decoder.layers</td><td>268.7M</td><td>47.2%</td><td>100.5K</td><td>1.6%</td></tr><tr><td><i class="c1"></i>model.encoder.layers</td><td>201.5M</td><td>35.4%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c2"></i>model.shared</td><td>98.4M</td><td>17.3%</td><td>6.2M</td><td>96.3%</td></tr><tr><td><i class="c3"></i>model.decoder</td><td>526.3K</td><td>0.1%</td><td>32.9K</td><td>0.5%</td></tr><tr><td><i class="c4"></i>model.encoder</td><td>526.3K</td><td>0.1%</td><td>32.9K</td><td>0.5%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>pegasus</td><td>pegasus</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>96103</td><td>96103</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>512</td><td>512</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Pegasus omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>257</td><td>33</td></tr><tr class=""><td>LayerNorm</td><td>82</td><td>12</td></tr><tr class=""><td>Attention</td><td>48</td><td>6</td></tr><tr class=""><td>ReLU</td><td>32</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>16</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>16</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 38 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>d_model</td><td>1024</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>decoder_layers</td><td>16</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>4096</td><td>128</td></tr><tr><td>encoder_layers</td><td>16</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/phi-2-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/phi-2</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/phi-2-mic/tree/af26f3c429a4d873dfe9c7fdc51fc78a845768fc" title="huggingface-mic/phi-2-mic@af26f3c429a4d873dfe9c7fdc51fc78a845768fc">huggingface-mic/phi-2-mic@af26f3c429a4</a> β <a href="https://huggingface.co/microsoft/phi-2/tree/810d367871c1d460086d9f82db8696f2e0a0fcd0" title="microsoft/phi-2@810d367871c1d460086d9f82db8696f2e0a0fcd0">microsoft/phi-2@810d367871c1</a> Β· PhiForCausalLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.8B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i><i class="c0" title="layer 30: DecoderLayer"></i><i class="c0" title="layer 31: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.905673" title="model.layers: 2.5B (90.6%)"></i><i class="c1" style="flex:0.047172" title="lm_head: 131.1M (4.7%)"></i><i class="c2" style="flex:0.047154" title="model.embed_tokens: 131.1M (4.7%)"></i><i class="c3" style="flex:0.000002" title="model.final_layernorm: 5.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.009996" title="model.layers: 66.7K (1.0%)"></i><i class="c1" style="flex:0.498830" title="lm_head: 3.3M (49.9%)"></i><i class="c2" style="flex:0.491155" title="model.embed_tokens: 3.3M (49.1%)"></i><i class="c3" style="flex:0.000019" title="model.final_layernorm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.5B</td><td>90.6%</td><td>66.7K</td><td>1.0%</td></tr><tr><td><i class="c1"></i>lm_head</td><td>131.1M</td><td>4.7%</td><td>3.3M</td><td>49.9%</td></tr><tr><td><i class="c2"></i>model.embed_tokens</td><td>131.1M</td><td>4.7%</td><td>3.3M</td><td>49.1%</td></tr><tr><td><i class="c3"></i>model.final_layernorm</td><td>5.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>phi</td><td>phi</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>51200</td><td>51200</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu_new</td><td>gelu_new</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Phi omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>13</td></tr><tr class=""><td>Dropout</td><td>33</td><td>3</td></tr><tr class=""><td>LayerNorm</td><td>33</td><td>3</td></tr><tr class=""><td>NewGELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Attention</td><td>32</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2560</td><td>64</td></tr><tr><td>intermediate_size</td><td>10240</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/phi-2</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/phi-2-mic/tree/af26f3c429a4d873dfe9c7fdc51fc78a845768fc" title="huggingface-mic/phi-2-mic@af26f3c429a4d873dfe9c7fdc51fc78a845768fc">huggingface-mic/phi-2-mic@af26f3c429a4</a> β <a href="https://huggingface.co/microsoft/phi-2/tree/810d367871c1d460086d9f82db8696f2e0a0fcd0" title="microsoft/phi-2@810d367871c1d460086d9f82db8696f2e0a0fcd0">microsoft/phi-2@810d367871c1</a> Β· PhiForCausalLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=phi-2-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.8B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i><i class="c0" title="layer 30: DecoderLayer"></i><i class="c0" title="layer 31: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.905673" title="model.layers: 2.5B (90.6%)"></i><i class="c1" style="flex:0.047172" title="lm_head: 131.1M (4.7%)"></i><i class="c2" style="flex:0.047154" title="model.embed_tokens: 131.1M (4.7%)"></i><i class="c3" style="flex:0.000002" title="model.final_layernorm: 5.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.009996" title="model.layers: 66.7K (1.0%)"></i><i class="c1" style="flex:0.498830" title="lm_head: 3.3M (49.9%)"></i><i class="c2" style="flex:0.491155" title="model.embed_tokens: 3.3M (49.1%)"></i><i class="c3" style="flex:0.000019" title="model.final_layernorm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.5B</td><td>90.6%</td><td>66.7K</td><td>1.0%</td></tr><tr><td><i class="c1"></i>lm_head</td><td>131.1M</td><td>4.7%</td><td>3.3M</td><td>49.9%</td></tr><tr><td><i class="c2"></i>model.embed_tokens</td><td>131.1M</td><td>4.7%</td><td>3.3M</td><td>49.1%</td></tr><tr><td><i class="c3"></i>model.final_layernorm</td><td>5.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>phi</td><td>phi</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>51200</td><td>51200</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu_new</td><td>gelu_new</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Phi omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>193</td><td>13</td></tr><tr class=""><td>Dropout</td><td>33</td><td>3</td></tr><tr class=""><td>LayerNorm</td><td>33</td><td>3</td></tr><tr class=""><td>NewGELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Attention</td><td>32</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2560</td><td>64</td></tr><tr><td>intermediate_size</td><td>10240</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/privacy-filter-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>openai/privacy-filter</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/privacy-filter-mic/tree/ad3dba8d001d1fe12c033288bf38745f9772c1b7" title="huggingface-mic/privacy-filter-mic@ad3dba8d001d1fe12c033288bf38745f9772c1b7">huggingface-mic/privacy-filter-mic@ad3dba8d001d</a> β <a href="https://huggingface.co/openai/privacy-filter/tree/7ffa9a043d54d1be65afb281eddf0ffbe629385b" title="openai/privacy-filter@7ffa9a043d54d1be65afb281eddf0ffbe629385b">openai/privacy-filter@7ffa9a043d54</a> Β· OpenAIPrivacyFilterForTokenClassification Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.4B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">13.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>8</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.908493" title="model.layers: 1.3B (90.8%)"></i><i class="c1" style="flex:0.091491" title="model.embed_tokens: 128.0M (9.1%)"></i><i class="c2" style="flex:0.000015" title="score: 21.2K (0.0%)"></i><i class="c3" style="flex:0.000000" title="model.norm: 640 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.017259" title="model.layers: 224.9K (1.7%)"></i><i class="c1" style="flex:0.982571" title="model.embed_tokens: 12.8M (98.3%)"></i><i class="c2" style="flex:0.000165" title="score: 2.1K (0.0%)"></i><i class="c3" style="flex:0.000005" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.3B</td><td>90.8%</td><td>224.9K</td><td>1.7%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>128.0M</td><td>9.1%</td><td>12.8M</td><td>98.3%</td></tr><tr><td><i class="c2"></i>score</td><td>21.2K</td><td>0.0%</td><td>2.1K</td><td>0.0%</td></tr><tr><td><i class="c3"></i>model.norm</td><td>640</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (14 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>MoE routing</td><td>top-4 of 128 experts</td><td>top-2 of 4 experts</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>yarn</td><td>yarn</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>openai_privacy_filter</td><td>openai_privacy_filter</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>200064</td><td>200064</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OpenAIPrivacyFilter omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>33</td><td>9</td></tr><tr class=""><td>RMSNorm</td><td>17</td><td>5</td></tr><tr class=""><td>Attention</td><td>8</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>8</td><td>2</td></tr><tr class=""><td>Experts</td><td>8</td><td>2</td></tr><tr class=""><td>MLP</td><td>8</td><td>2</td></tr><tr class=""><td>TopKRouter</td><td>8</td><td>2</td></tr><tr class=""><td>Dropout</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForTokenClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 32 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>640</td><td>64</td></tr><tr><td>intermediate_size</td><td>640</td><td>128</td></tr><tr><td>num_attention_heads</td><td>14</td><td>4</td></tr><tr><td>num_experts_per_tok</td><td>4</td><td>2</td></tr><tr><td>num_hidden_layers</td><td>8</td><td>2</td></tr><tr><td>num_local_experts</td><td>128</td><td>4</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>openai/privacy-filter</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/privacy-filter-mic/tree/ad3dba8d001d1fe12c033288bf38745f9772c1b7" title="huggingface-mic/privacy-filter-mic@ad3dba8d001d1fe12c033288bf38745f9772c1b7">huggingface-mic/privacy-filter-mic@ad3dba8d001d</a> β <a href="https://huggingface.co/openai/privacy-filter/tree/7ffa9a043d54d1be65afb281eddf0ffbe629385b" title="openai/privacy-filter@7ffa9a043d54d1be65afb281eddf0ffbe629385b">openai/privacy-filter@7ffa9a043d54</a> Β· OpenAIPrivacyFilterForTokenClassification Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=privacy-filter-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.4B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">13.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 8</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>8</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.908493" title="model.layers: 1.3B (90.8%)"></i><i class="c1" style="flex:0.091491" title="model.embed_tokens: 128.0M (9.1%)"></i><i class="c2" style="flex:0.000015" title="score: 21.2K (0.0%)"></i><i class="c3" style="flex:0.000000" title="model.norm: 640 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.017259" title="model.layers: 224.9K (1.7%)"></i><i class="c1" style="flex:0.982571" title="model.embed_tokens: 12.8M (98.3%)"></i><i class="c2" style="flex:0.000165" title="score: 2.1K (0.0%)"></i><i class="c3" style="flex:0.000005" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.3B</td><td>90.8%</td><td>224.9K</td><td>1.7%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>128.0M</td><td>9.1%</td><td>12.8M</td><td>98.3%</td></tr><tr><td><i class="c2"></i>score</td><td>21.2K</td><td>0.0%</td><td>2.1K</td><td>0.0%</td></tr><tr><td><i class="c3"></i>model.norm</td><td>640</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (14 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>MoE routing</td><td>top-4 of 128 experts</td><td>top-2 of 4 experts</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>yarn</td><td>yarn</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>openai_privacy_filter</td><td>openai_privacy_filter</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>200064</td><td>200064</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix OpenAIPrivacyFilter omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>33</td><td>9</td></tr><tr class=""><td>RMSNorm</td><td>17</td><td>5</td></tr><tr class=""><td>Attention</td><td>8</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>8</td><td>2</td></tr><tr class=""><td>Experts</td><td>8</td><td>2</td></tr><tr class=""><td>MLP</td><td>8</td><td>2</td></tr><tr class=""><td>TopKRouter</td><td>8</td><td>2</td></tr><tr class=""><td>Dropout</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForTokenClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 32 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>640</td><td>64</td></tr><tr><td>intermediate_size</td><td>640</td><td>128</td></tr><tr><td>num_attention_heads</td><td>14</td><td>4</td></tr><tr><td>num_experts_per_tok</td><td>4</td><td>2</td></tr><tr><td>num_hidden_layers</td><td>8</td><td>2</td></tr><tr><td>num_local_experts</td><td>128</td><td>4</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/pvt-tiny-224-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Zetatech/pvt-tiny-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pvt-tiny-224-mic/tree/b1c239c03ea2b52e24e2e72cea4498c1932cc76b" title="huggingface-mic/pvt-tiny-224-mic@b1c239c03ea2b52e24e2e72cea4498c1932cc76b">huggingface-mic/pvt-tiny-224-mic@b1c239c03ea2</a> β <a href="https://huggingface.co/Zetatech/pvt-tiny-224/tree/6a44ba96dda6a6f3e2b6913682acfa0071482e83" title="Zetatech/pvt-tiny-224@6a44ba96dda6a6f3e2b6913682acfa0071482e83">Zetatech/pvt-tiny-224@6a44ba96dda6</a> Β· PvtForImageClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">13.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">592.2K</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>pvt.encoder.patch_embeddings</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>pvt.encoder.block</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c1" title="layer 3: no Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c1" title="layer 3: no Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Conv2d</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.866809" title="pvt.encoder.block: 11.5M (86.7%)"></i><i class="c1" style="flex:0.094336" title="pvt.encoder.patch_embeddings: 1.2M (9.4%)"></i><i class="c2" style="flex:0.038778" title="classifier: 513.0K (3.9%)"></i><i class="c3" style="flex:0.000077" title="pvt.encoder: 1.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.547338" title="pvt.encoder.block: 324.2K (54.7%)"></i><i class="c1" style="flex:0.234415" title="pvt.encoder.patch_embeddings: 138.8K (23.4%)"></i><i class="c2" style="flex:0.217814" title="classifier: 129.0K (21.8%)"></i><i class="c3" style="flex:0.000432" title="pvt.encoder: 256 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>pvt.encoder.block</td><td>11.5M</td><td>86.7%</td><td>324.2K</td><td>54.7%</td></tr><tr><td><i class="c1"></i>pvt.encoder.patch_embeddings</td><td>1.2M</td><td>9.4%</td><td>138.8K</td><td>23.4%</td></tr><tr><td><i class="c2"></i>classifier</td><td>513.0K</td><td>3.9%</td><td>129.0K</td><td>21.8%</td></tr><tr><td><i class="c3"></i>pvt.encoder</td><td>1.0K</td><td>0.0%</td><td>256</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>pvt</td><td>pvt</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Pvt omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>49</td><td>25</td></tr><tr class=""><td>Dropout</td><td>28</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>27</td><td>16</td></tr><tr class=""><td>Conv2d</td><td>10</td><td>7</td></tr><tr class=""><td>GELUActivation</td><td>8</td><td>4</td></tr><tr class=""><td>Identity</td><td>8</td><td>4</td></tr><tr class=""><td>Attention</td><td>8</td><td>4</td></tr><tr class=""><td>EfficientSelfAttention</td><td>8</td><td>4</td></tr><tr class=""><td>FFN</td><td>8</td><td>4</td></tr><tr class=""><td>Layer</td><td>8</td><td>4</td></tr><tr class=""><td>SelfOutput</td><td>8</td><td>4</td></tr><tr class=""><td>ModuleList</td><td>6</td><td>6</td></tr><tr class=""><td>PatchEmbeddings</td><td>4</td><td>4</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>depths</td><td>[2, 2, 2, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>hidden_sizes</td><td>[64, 128, 320, 512]</td><td>[16, 32, 64, 128]</td></tr><tr><td>num_attention_heads</td><td>[1, 2, 5, 8]</td><td>[1, 2, 4, 8]</td></tr><tr><td>problem_type</td><td>null</td><td>single_label_classification</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Zetatech/pvt-tiny-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pvt-tiny-224-mic/tree/b1c239c03ea2b52e24e2e72cea4498c1932cc76b" title="huggingface-mic/pvt-tiny-224-mic@b1c239c03ea2b52e24e2e72cea4498c1932cc76b">huggingface-mic/pvt-tiny-224-mic@b1c239c03ea2</a> β <a href="https://huggingface.co/Zetatech/pvt-tiny-224/tree/6a44ba96dda6a6f3e2b6913682acfa0071482e83" title="Zetatech/pvt-tiny-224@6a44ba96dda6a6f3e2b6913682acfa0071482e83">Zetatech/pvt-tiny-224@6a44ba96dda6</a> Β· PvtForImageClassification Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=pvt-tiny-224-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">13.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">592.2K</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>pvt.encoder.patch_embeddings</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c0" title="layer 3: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><div class="stack"><h3>pvt.encoder.block</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c1" title="layer 3: no Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i><i class="c1" title="layer 3: no Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no Conv2d</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.866809" title="pvt.encoder.block: 11.5M (86.7%)"></i><i class="c1" style="flex:0.094336" title="pvt.encoder.patch_embeddings: 1.2M (9.4%)"></i><i class="c2" style="flex:0.038778" title="classifier: 513.0K (3.9%)"></i><i class="c3" style="flex:0.000077" title="pvt.encoder: 1.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.547338" title="pvt.encoder.block: 324.2K (54.7%)"></i><i class="c1" style="flex:0.234415" title="pvt.encoder.patch_embeddings: 138.8K (23.4%)"></i><i class="c2" style="flex:0.217814" title="classifier: 129.0K (21.8%)"></i><i class="c3" style="flex:0.000432" title="pvt.encoder: 256 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>pvt.encoder.block</td><td>11.5M</td><td>86.7%</td><td>324.2K</td><td>54.7%</td></tr><tr><td><i class="c1"></i>pvt.encoder.patch_embeddings</td><td>1.2M</td><td>9.4%</td><td>138.8K</td><td>23.4%</td></tr><tr><td><i class="c2"></i>classifier</td><td>513.0K</td><td>3.9%</td><td>129.0K</td><td>21.8%</td></tr><tr><td><i class="c3"></i>pvt.encoder</td><td>1.0K</td><td>0.0%</td><td>256</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>pvt</td><td>pvt</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Pvt omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>49</td><td>25</td></tr><tr class=""><td>Dropout</td><td>28</td><td>16</td></tr><tr class=""><td>LayerNorm</td><td>27</td><td>16</td></tr><tr class=""><td>Conv2d</td><td>10</td><td>7</td></tr><tr class=""><td>GELUActivation</td><td>8</td><td>4</td></tr><tr class=""><td>Identity</td><td>8</td><td>4</td></tr><tr class=""><td>Attention</td><td>8</td><td>4</td></tr><tr class=""><td>EfficientSelfAttention</td><td>8</td><td>4</td></tr><tr class=""><td>FFN</td><td>8</td><td>4</td></tr><tr class=""><td>Layer</td><td>8</td><td>4</td></tr><tr class=""><td>SelfOutput</td><td>8</td><td>4</td></tr><tr class=""><td>ModuleList</td><td>6</td><td>6</td></tr><tr class=""><td>PatchEmbeddings</td><td>4</td><td>4</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>depths</td><td>[2, 2, 2, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>hidden_sizes</td><td>[64, 128, 320, 512]</td><td>[16, 32, 64, 128]</td></tr><tr><td>num_attention_heads</td><td>[1, 2, 5, 8]</td><td>[1, 2, 4, 8]</td></tr><tr><td>problem_type</td><td>null</td><td>single_label_classification</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/pythia-160m-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>EleutherAI/pythia-160m</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pythia-160m-mic/tree/f21c9ac8c4e817ebcf9cb2963749d2ff55b756e3" title="huggingface-mic/pythia-160m-mic@f21c9ac8c4e817ebcf9cb2963749d2ff55b756e3">huggingface-mic/pythia-160m-mic@f21c9ac8c4e8</a> β <a href="https://huggingface.co/EleutherAI/pythia-160m/tree/50f5173d932e8e61f858120bcb800b97af589f46" title="EleutherAI/pythia-160m@50f5173d932e8e61f858120bcb800b97af589f46">EleutherAI/pythia-160m@50f5173d932e</a> Β· GPTNeoXForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">162.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>gpt_neox.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.523983" title="gpt_neox.layers: 85.1M (52.4%)"></i><i class="c1" style="flex:0.238004" title="gpt_neox.embed_in: 38.6M (23.8%)"></i><i class="c2" style="flex:0.238004" title="lm_head: 38.6M (23.8%)"></i><i class="c3" style="flex:0.000009" title="gpt_neox.final_layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.010290" title="gpt_neox.layers: 66.9K (1.0%)"></i><i class="c1" style="flex:0.494845" title="gpt_neox.embed_in: 3.2M (49.5%)"></i><i class="c2" style="flex:0.494845" title="lm_head: 3.2M (49.5%)"></i><i class="c3" style="flex:0.000020" title="gpt_neox.final_layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>gpt_neox.layers</td><td>85.1M</td><td>52.4%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c1"></i>gpt_neox.embed_in</td><td>38.6M</td><td>23.8%</td><td>3.2M</td><td>49.5%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>38.6M</td><td>23.8%</td><td>3.2M</td><td>49.5%</td></tr><tr><td><i class="c3"></i>gpt_neox.final_layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>gpt_neox</td><td>gpt_neox</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50304</td><td>50304</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix GPTNeoX omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>49</td><td>9</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>MLP</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>EleutherAI/pythia-160m</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/pythia-160m-mic/tree/f21c9ac8c4e817ebcf9cb2963749d2ff55b756e3" title="huggingface-mic/pythia-160m-mic@f21c9ac8c4e817ebcf9cb2963749d2ff55b756e3">huggingface-mic/pythia-160m-mic@f21c9ac8c4e8</a> β <a href="https://huggingface.co/EleutherAI/pythia-160m/tree/50f5173d932e8e61f858120bcb800b97af589f46" title="EleutherAI/pythia-160m@50f5173d932e8e61f858120bcb800b97af589f46">EleutherAI/pythia-160m@50f5173d932e</a> Β· GPTNeoXForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=pythia-160m-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">162.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>gpt_neox.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.523983" title="gpt_neox.layers: 85.1M (52.4%)"></i><i class="c1" style="flex:0.238004" title="gpt_neox.embed_in: 38.6M (23.8%)"></i><i class="c2" style="flex:0.238004" title="lm_head: 38.6M (23.8%)"></i><i class="c3" style="flex:0.000009" title="gpt_neox.final_layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.010290" title="gpt_neox.layers: 66.9K (1.0%)"></i><i class="c1" style="flex:0.494845" title="gpt_neox.embed_in: 3.2M (49.5%)"></i><i class="c2" style="flex:0.494845" title="lm_head: 3.2M (49.5%)"></i><i class="c3" style="flex:0.000020" title="gpt_neox.final_layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>gpt_neox.layers</td><td>85.1M</td><td>52.4%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c1"></i>gpt_neox.embed_in</td><td>38.6M</td><td>23.8%</td><td>3.2M</td><td>49.5%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>38.6M</td><td>23.8%</td><td>3.2M</td><td>49.5%</td></tr><tr><td><i class="c3"></i>gpt_neox.final_layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>gpt_neox</td><td>gpt_neox</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50304</td><td>50304</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix GPTNeoX omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>49</td><td>9</td></tr><tr class=""><td>Dropout</td><td>25</td><td>5</td></tr><tr class=""><td>LayerNorm</td><td>25</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>MLP</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 26 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/qwen2-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2-0.5B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2-mic/tree/f9c74f62c62500474b51b9ae8b086be6bcff3621" title="huggingface-mic/qwen2-mic@f9c74f62c62500474b51b9ae8b086be6bcff3621">huggingface-mic/qwen2-mic@f9c74f62c625</a> β <a href="https://huggingface.co/Qwen/Qwen2-0.5B/tree/91d2aff3f957f99e4c74c962f2f408dcc88a18d8" title="Qwen/Qwen2-0.5B@91d2aff3f957f99e4c74c962f2f408dcc88a18d8">Qwen/Qwen2-0.5B@91d2aff3f957</a> Β· Qwen2ForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">494.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">9.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.724440" title="model.layers: 357.9M (72.4%)"></i><i class="c1" style="flex:0.275558" title="model.embed_tokens: 136.1M (27.6%)"></i><i class="c2" style="flex:0.000002" title="model.norm: 896 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.007577" title="model.layers: 74.2K (0.8%)"></i><i class="c1" style="flex:0.992417" title="model.embed_tokens: 9.7M (99.2%)"></i><i class="c2" style="flex:0.000007" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>357.9M</td><td>72.4%</td><td>74.2K</td><td>0.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>136.1M</td><td>27.6%</td><td>9.7M</td><td>99.2%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>896</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (14 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2</td><td>qwen2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>151936</td><td>151936</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>896</td><td>64</td></tr><tr><td>intermediate_size</td><td>4864</td><td>128</td></tr><tr><td>layer_types</td><td>24 entries: 24Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>num_attention_heads</td><td>14</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2-0.5B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2-mic/tree/f9c74f62c62500474b51b9ae8b086be6bcff3621" title="huggingface-mic/qwen2-mic@f9c74f62c62500474b51b9ae8b086be6bcff3621">huggingface-mic/qwen2-mic@f9c74f62c625</a> β <a href="https://huggingface.co/Qwen/Qwen2-0.5B/tree/91d2aff3f957f99e4c74c962f2f408dcc88a18d8" title="Qwen/Qwen2-0.5B@91d2aff3f957f99e4c74c962f2f408dcc88a18d8">Qwen/Qwen2-0.5B@91d2aff3f957</a> Β· Qwen2ForCausalLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=qwen2-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 2 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">494.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">9.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.724440" title="model.layers: 357.9M (72.4%)"></i><i class="c1" style="flex:0.275558" title="model.embed_tokens: 136.1M (27.6%)"></i><i class="c2" style="flex:0.000002" title="model.norm: 896 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.007577" title="model.layers: 74.2K (0.8%)"></i><i class="c1" style="flex:0.992417" title="model.embed_tokens: 9.7M (99.2%)"></i><i class="c2" style="flex:0.000007" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>357.9M</td><td>72.4%</td><td>74.2K</td><td>0.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>136.1M</td><td>27.6%</td><td>9.7M</td><td>99.2%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>896</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (14 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2</td><td>qwen2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>151936</td><td>151936</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>131072</td><td>131072</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>896</td><td>64</td></tr><tr><td>intermediate_size</td><td>4864</td><td>128</td></tr><tr><td>layer_types</td><td>24 entries: 24Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>num_attention_heads</td><td>14</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/qwen2_5_omni-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2.5-Omni-7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_5_omni-mic/tree/72e87592273c7e7662ffb5e249400c12a3488549" title="huggingface-mic/qwen2_5_omni-mic@72e87592273c7e7662ffb5e249400c12a3488549">huggingface-mic/qwen2_5_omni-mic@72e87592273c</a> β <a href="https://huggingface.co/Qwen/Qwen2.5-Omni-7B/tree/ae9e1690543ffd5c0221dc27f79834d0294cba00" title="Qwen/Qwen2.5-Omni-7B@ae9e1690543ffd5c0221dc27f79834d0294cba00">Qwen/Qwen2.5-Omni-7B@ae9e1690543f</a> Β· Qwen2_5OmniForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">10.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">170.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">8</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>thinker.model.audio_tower.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: AudioEncoderLayer"></i><i class="c0" title="layer 1: AudioEncoderLayer"></i><i class="c0" title="layer 2: AudioEncoderLayer"></i><i class="c0" title="layer 3: AudioEncoderLayer"></i><i class="c0" title="layer 4: AudioEncoderLayer"></i><i class="c0" title="layer 5: AudioEncoderLayer"></i><i class="c0" title="layer 6: AudioEncoderLayer"></i><i class="c0" title="layer 7: AudioEncoderLayer"></i><i class="c0" title="layer 8: AudioEncoderLayer"></i><i class="c0" title="layer 9: AudioEncoderLayer"></i><i class="c0" title="layer 10: AudioEncoderLayer"></i><i class="c0" title="layer 11: AudioEncoderLayer"></i><i class="c0" title="layer 12: AudioEncoderLayer"></i><i class="c0" title="layer 13: AudioEncoderLayer"></i><i class="c0" title="layer 14: AudioEncoderLayer"></i><i class="c0" title="layer 15: AudioEncoderLayer"></i><i class="c0" title="layer 16: AudioEncoderLayer"></i><i class="c0" title="layer 17: AudioEncoderLayer"></i><i class="c0" title="layer 18: AudioEncoderLayer"></i><i class="c0" title="layer 19: AudioEncoderLayer"></i><i class="c0" title="layer 20: AudioEncoderLayer"></i><i class="c0" title="layer 21: AudioEncoderLayer"></i><i class="c0" title="layer 22: AudioEncoderLayer"></i><i class="c0" title="layer 23: AudioEncoderLayer"></i><i class="c0" title="layer 24: AudioEncoderLayer"></i><i class="c0" title="layer 25: AudioEncoderLayer"></i><i class="c0" title="layer 26: AudioEncoderLayer"></i><i class="c0" title="layer 27: AudioEncoderLayer"></i><i class="c0" title="layer 28: AudioEncoderLayer"></i><i class="c0" title="layer 29: AudioEncoderLayer"></i><i class="c0" title="layer 30: AudioEncoderLayer"></i><i class="c0" title="layer 31: AudioEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: AudioEncoderLayer"></i><i class="c0" title="layer 1: AudioEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AudioEncoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>thinker.model.visual.blocks</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>VisionBlock</td><td>32</td><td>32</td><td>β kept</td></tr></table></div><div class="stack"><h3>thinker.model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 28</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>28</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>talker.model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_dit_model.input_embed.spk_encoder.blocks</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ModuleList"></i><i class="c1" title="layer 1: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 2: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 3: ModuleList, Res2NetBlock, Sigmoid β¦"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ModuleList"></i><i class="c1" title="layer 1: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 2: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 3: ModuleList, Res2NetBlock, Sigmoid β¦"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no ModuleList</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>ModuleList, Res2NetBlock, Sigmoid β¦</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_dit_model.transformer_blocks</h3><div class="row"><span class="lab">Original Β· 22</span><div class="cells"><i class="c0" title="layer 0: DiTDecoderLayer"></i><i class="c0" title="layer 1: DiTDecoderLayer"></i><i class="c0" title="layer 2: DiTDecoderLayer"></i><i class="c0" title="layer 3: DiTDecoderLayer"></i><i class="c0" title="layer 4: DiTDecoderLayer"></i><i class="c0" title="layer 5: DiTDecoderLayer"></i><i class="c0" title="layer 6: DiTDecoderLayer"></i><i class="c0" title="layer 7: DiTDecoderLayer"></i><i class="c0" title="layer 8: DiTDecoderLayer"></i><i class="c0" title="layer 9: DiTDecoderLayer"></i><i class="c0" title="layer 10: DiTDecoderLayer"></i><i class="c0" title="layer 11: DiTDecoderLayer"></i><i class="c0" title="layer 12: DiTDecoderLayer"></i><i class="c0" title="layer 13: DiTDecoderLayer"></i><i class="c0" title="layer 14: DiTDecoderLayer"></i><i class="c0" title="layer 15: DiTDecoderLayer"></i><i class="c0" title="layer 16: DiTDecoderLayer"></i><i class="c0" title="layer 17: DiTDecoderLayer"></i><i class="c0" title="layer 18: DiTDecoderLayer"></i><i class="c0" title="layer 19: DiTDecoderLayer"></i><i class="c0" title="layer 20: DiTDecoderLayer"></i><i class="c0" title="layer 21: DiTDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DiTDecoderLayer"></i><i class="c0" title="layer 1: DiTDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DiTDecoderLayer</td><td>22</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_bigvgan_model.ups</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: ModuleList"></i><i class="c0" title="layer 1: ModuleList"></i><i class="c0" title="layer 2: ModuleList"></i><i class="c0" title="layer 3: ModuleList"></i><i class="c0" title="layer 4: ModuleList"></i><i class="c0" title="layer 5: ModuleList"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: ModuleList"></i><i class="c0" title="layer 1: ModuleList"></i><i class="c0" title="layer 2: ModuleList"></i><i class="c0" title="layer 3: ModuleList"></i><i class="c0" title="layer 4: ModuleList"></i><i class="c0" title="layer 5: ModuleList"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleList</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_bigvgan_model.resblocks</h3><div class="row"><span class="lab">Original Β· 18</span><div class="cells"><i class="c0" title="layer 0: AMPBlock"></i><i class="c0" title="layer 1: AMPBlock"></i><i class="c0" title="layer 2: AMPBlock"></i><i class="c0" title="layer 3: AMPBlock"></i><i class="c0" title="layer 4: AMPBlock"></i><i class="c0" title="layer 5: AMPBlock"></i><i class="c0" title="layer 6: AMPBlock"></i><i class="c0" title="layer 7: AMPBlock"></i><i class="c0" title="layer 8: AMPBlock"></i><i class="c0" title="layer 9: AMPBlock"></i><i class="c0" title="layer 10: AMPBlock"></i><i class="c0" title="layer 11: AMPBlock"></i><i class="c0" title="layer 12: AMPBlock"></i><i class="c0" title="layer 13: AMPBlock"></i><i class="c0" title="layer 14: AMPBlock"></i><i class="c0" title="layer 15: AMPBlock"></i><i class="c0" title="layer 16: AMPBlock"></i><i class="c0" title="layer 17: AMPBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 18</span><div class="cells"><i class="c0" title="layer 0: AMPBlock"></i><i class="c0" title="layer 1: AMPBlock"></i><i class="c0" title="layer 2: AMPBlock"></i><i class="c0" title="layer 3: AMPBlock"></i><i class="c0" title="layer 4: AMPBlock"></i><i class="c0" title="layer 5: AMPBlock"></i><i class="c0" title="layer 6: AMPBlock"></i><i class="c0" title="layer 7: AMPBlock"></i><i class="c0" title="layer 8: AMPBlock"></i><i class="c0" title="layer 9: AMPBlock"></i><i class="c0" title="layer 10: AMPBlock"></i><i class="c0" title="layer 11: AMPBlock"></i><i class="c0" title="layer 12: AMPBlock"></i><i class="c0" title="layer 13: AMPBlock"></i><i class="c0" title="layer 14: AMPBlock"></i><i class="c0" title="layer 15: AMPBlock"></i><i class="c0" title="layer 16: AMPBlock"></i><i class="c0" title="layer 17: AMPBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AMPBlock</td><td>18</td><td>18</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.608040" title="thinker.model.language_model.layers: 6.5B (60.8%)"></i><i class="c1" style="flex:0.122090" title="talker.model.layers: 1.3B (12.2%)"></i><i class="c2" style="flex:0.058746" title="thinker.model.visual.blocks: 630.5M (5.9%)"></i><i class="c3" style="flex:0.058668" title="thinker.model.audio_tower.layers: 629.6M (5.9%)"></i><i class="c4" style="flex:0.056008" title="thinker.model: 601.1M (5.6%)"></i><i class="c5" style="flex:0.050781" title="thinker.lm_head: 545.0M (5.1%)"></i><i class="c6" style="flex:0.030120" title="token2wav.code2wav_dit_model.transformer_blocks: 323.3M (3.0%)"></i><i class="c7" style="flex:0.015547" title="8 others: 166.9M (1.6%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.000436" title="thinker.model.language_model.layers: 74.2K (0.0%)"></i><i class="c1" style="flex:0.000436" title="talker.model.layers: 74.2K (0.0%)"></i><i class="c2" style="flex:0.007825" title="thinker.model.visual.blocks: 1.3M (0.8%)"></i><i class="c3" style="flex:0.000392" title="thinker.model.audio_tower.layers: 66.8K (0.0%)"></i><i class="c4" style="flex:0.059675" title="thinker.model: 10.2M (6.0%)"></i><i class="c5" style="flex:0.057122" title="thinker.lm_head: 9.7M (5.7%)"></i><i class="c6" style="flex:0.000683" title="token2wav.code2wav_dit_model.transformer_blocks: 116.4K (0.1%)"></i><i class="c7" style="flex:0.873431" title="8 others: 148.8M (87.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>thinker.model.language_model.layers</td><td>6.5B</td><td>60.8%</td><td>74.2K</td><td>0.0%</td></tr><tr><td><i class="c1"></i>talker.model.layers</td><td>1.3B</td><td>12.2%</td><td>74.2K</td><td>0.0%</td></tr><tr><td><i class="c2"></i>thinker.model.visual.blocks</td><td>630.5M</td><td>5.9%</td><td>1.3M</td><td>0.8%</td></tr><tr><td><i class="c3"></i>thinker.model.audio_tower.layers</td><td>629.6M</td><td>5.9%</td><td>66.8K</td><td>0.0%</td></tr><tr><td><i class="c4"></i>thinker.model</td><td>601.1M</td><td>5.6%</td><td>10.2M</td><td>6.0%</td></tr><tr><td><i class="c5"></i>thinker.lm_head</td><td>545.0M</td><td>5.1%</td><td>9.7M</td><td>5.7%</td></tr><tr><td><i class="c6"></i>token2wav.code2wav_dit_model.transformer_blocks</td><td>323.3M</td><td>3.0%</td><td>116.4K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>8 others</td><td>166.9M</td><td>1.6%</td><td>148.8M</td><td>87.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_5_omni</td><td>qwen2_5_omni</td><td>β kept</td></tr><tr class=""><td>thinker_config</td><td>model_type</td><td>qwen2_5_omni_thinker</td><td>qwen2_5_omni_thinker</td><td>β kept</td></tr><tr class=""><td>thinker_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>model_type</td><td>qwen2_5_omni_audio_encoder</td><td>qwen2_5_omni_audio_encoder</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>num_mel_bins</td><td>128</td><td>128</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>model_type</td><td>qwen2_5_omni_vision_encoder</td><td>qwen2_5_omni_vision_encoder</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>attention style</td><td>GQA (28 heads, 4 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>model_type</td><td>qwen2_5_omni_text</td><td>qwen2_5_omni_text</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>vocab_size</td><td>152064</td><td>152064</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>attention style</td><td>GQA (12 heads, 4 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>model_type</td><td>qwen2_5_omni_talker</td><td>qwen2_5_omni_talker</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>vocab_size</td><td>8448</td><td>8448</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>token2wav_config</td><td>model_type</td><td>qwen2_5_omni_token2wav</td><td>qwen2_5_omni_token2wav</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>model_type</td><td>qwen2_5_omni_dit</td><td>qwen2_5_omni_dit</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>token2wav_config.bigvgan_config</td><td>model_type</td><td>qwen2_5_omni_bigvgan</td><td>qwen2_5_omni_bigvgan</td><td>β kept</td></tr><tr class=""><td>token2wav_config.bigvgan_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2_5Omni omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>945</td><td>289</td></tr><tr class=""><td>RMSNorm</td><td>171</td><td>75</td></tr><tr class=""><td>Conv1d</td><td>132</td><td>132</td></tr><tr class=""><td>ModuleList</td><td>116</td><td>76</td></tr><tr class=""><td>LayerNorm</td><td>110</td><td>10</td></tr><tr class=""><td>AntiAliasedActivation1d</td><td>109</td><td>109</td></tr><tr class=""><td>DownSample1d</td><td>109</td><td>109</td></tr><tr class=""><td>SnakeBeta</td><td>109</td><td>109</td></tr><tr class=""><td>UpSample1d</td><td>109</td><td>109</td></tr><tr class=""><td>SiLUActivation</td><td>84</td><td>36</td></tr><tr class=""><td>Qwen2MLP</td><td>52</td><td>4</td></tr><tr class=""><td>Attention</td><td>52</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>52</td><td>4</td></tr><tr class=""><td>Dropout</td><td>44</td><td>4</td></tr><tr class=""><td>GELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>AudioAttention</td><td>32</td><td>2</td></tr><tr class=""><td>AudioEncoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>32</td></tr><tr class=""><td>VisionAttention</td><td>32</td><td>32</td></tr><tr class=""><td>VisionBlock</td><td>32</td><td>32</td></tr><tr class=""><td>SiLU</td><td>24</td><td>4</td></tr><tr class=""><td>GELU</td><td>23</td><td>3</td></tr><tr class=""><td>DiTAttention</td><td>22</td><td>2</td></tr><tr class=""><td>DiTDecoderLayer</td><td>22</td><td>2</td></tr><tr class=""><td>DiTMLP</td><td>22</td><td>2</td></tr><tr class=""><td>Qwen2_5_OmniAdaLayerNormZero</td><td>22</td><td>2</td></tr><tr class=""><td>AMPBlock</td><td>18</td><td>18</td></tr><tr class=""><td>ReLU</td><td>15</td><td>15</td></tr><tr class=""><td>TimeDelayNetBlock</td><td>12</td><td>12</td></tr><tr class=""><td>ConvTranspose1d</td><td>6</td><td>6</td></tr><tr class=""><td>Embedding</td><td>4</td><td>4</td></tr><tr class=""><td>Res2NetBlock</td><td>3</td><td>3</td></tr><tr class=""><td>Sigmoid</td><td>3</td><td>3</td></tr><tr class=""><td>SqueezeExcitationBlock</td><td>3</td><td>3</td></tr><tr class=""><td>SqueezeExcitationRes2NetBlock</td><td>3</td><td>3</td></tr><tr class=""><td>RotaryEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>AttentiveStatisticsPooling</td><td>1</td><td>1</td></tr><tr class=""><td>AvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Conv3d</td><td>1</td><td>1</td></tr><tr class=""><td>DiTCodecEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>DiTInputEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>DiTTimestepEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ECAPA_TimeDelayNet</td><td>1</td><td>1</td></tr><tr class=""><td>AudioEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>DiTRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>TalkerForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>TalkerModel</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerModel</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerTextModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavBigVGANModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavDiTModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>VisionRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_OmniAdaLayerNormZero_Final</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionPatchEmbed</td><td>1</td><td>1</td></tr><tr class=""><td>Sequential</td><td>1</td><td>1</td></tr><tr class=""><td>SinusPositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>SinusoidsPositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Tanh</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">30 fields changed, 281 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>talker_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>talker_config.hidden_size</td><td>896</td><td>64</td></tr><tr><td>talker_config.intermediate_size</td><td>18944</td><td>128</td></tr><tr><td>talker_config.layer_types</td><td>24 entries: 24Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>talker_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>talker_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>talker_config.num_key_value_heads</td><td>4</td><td>2</td></tr><tr><td>talker_config.rope_parameters</td><td>{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "rβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>thinker_config.audio_config.d_model</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.audio_config.encoder_attention_heads</td><td>20</td><td>4</td></tr><tr><td>thinker_config.audio_config.encoder_ffn_dim</td><td>5120</td><td>128</td></tr><tr><td>thinker_config.audio_config.encoder_layers</td><td>32</td><td>2</td></tr><tr><td>thinker_config.text_config.hidden_size</td><td>3584</td><td>64</td></tr><tr><td>thinker_config.text_config.intermediate_size</td><td>18944</td><td>128</td></tr><tr><td>thinker_config.text_config.layer_types</td><td>28 entries: 28Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>thinker_config.text_config.num_attention_heads</td><td>28</td><td>4</td></tr><tr><td>thinker_config.text_config.num_hidden_layers</td><td>28</td><td>2</td></tr><tr><td>thinker_config.text_config.num_key_value_heads</td><td>4</td><td>2</td></tr><tr><td>thinker_config.text_config.rope_parameters</td><td>{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "rβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>thinker_config.vision_config.embed_dim</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.vision_config.hidden_size</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.vision_config.intermediate_size</td><td>3420</td><td>128</td></tr><tr><td>thinker_config.vision_config.num_heads</td><td>16</td><td>4</td></tr><tr><td>thinker_config.vision_config.out_hidden_size</td><td>3584</td><td>64</td></tr><tr><td>token2wav_config.dit_config.dim</td><td>1024</td><td>64</td></tr><tr><td>token2wav_config.dit_config.emb_dim</td><td>512</td><td>64</td></tr><tr><td>token2wav_config.dit_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>token2wav_config.dit_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>token2wav_config.dit_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>token2wav_config.dit_config.num_hidden_layers</td><td>22</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2.5-Omni-7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_5_omni-mic/tree/72e87592273c7e7662ffb5e249400c12a3488549" title="huggingface-mic/qwen2_5_omni-mic@72e87592273c7e7662ffb5e249400c12a3488549">huggingface-mic/qwen2_5_omni-mic@72e87592273c</a> β <a href="https://huggingface.co/Qwen/Qwen2.5-Omni-7B/tree/ae9e1690543ffd5c0221dc27f79834d0294cba00" title="Qwen/Qwen2.5-Omni-7B@ae9e1690543ffd5c0221dc27f79834d0294cba00">Qwen/Qwen2.5-Omni-7B@ae9e1690543f</a> Β· Qwen2_5OmniForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=qwen2_5_omni-mic&days=30&sticky_only=false" target="_blank" rel="noopener">failing now on main (26 failed checks in 30 days)</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">10.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">170.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">8</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>thinker.model.audio_tower.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: AudioEncoderLayer"></i><i class="c0" title="layer 1: AudioEncoderLayer"></i><i class="c0" title="layer 2: AudioEncoderLayer"></i><i class="c0" title="layer 3: AudioEncoderLayer"></i><i class="c0" title="layer 4: AudioEncoderLayer"></i><i class="c0" title="layer 5: AudioEncoderLayer"></i><i class="c0" title="layer 6: AudioEncoderLayer"></i><i class="c0" title="layer 7: AudioEncoderLayer"></i><i class="c0" title="layer 8: AudioEncoderLayer"></i><i class="c0" title="layer 9: AudioEncoderLayer"></i><i class="c0" title="layer 10: AudioEncoderLayer"></i><i class="c0" title="layer 11: AudioEncoderLayer"></i><i class="c0" title="layer 12: AudioEncoderLayer"></i><i class="c0" title="layer 13: AudioEncoderLayer"></i><i class="c0" title="layer 14: AudioEncoderLayer"></i><i class="c0" title="layer 15: AudioEncoderLayer"></i><i class="c0" title="layer 16: AudioEncoderLayer"></i><i class="c0" title="layer 17: AudioEncoderLayer"></i><i class="c0" title="layer 18: AudioEncoderLayer"></i><i class="c0" title="layer 19: AudioEncoderLayer"></i><i class="c0" title="layer 20: AudioEncoderLayer"></i><i class="c0" title="layer 21: AudioEncoderLayer"></i><i class="c0" title="layer 22: AudioEncoderLayer"></i><i class="c0" title="layer 23: AudioEncoderLayer"></i><i class="c0" title="layer 24: AudioEncoderLayer"></i><i class="c0" title="layer 25: AudioEncoderLayer"></i><i class="c0" title="layer 26: AudioEncoderLayer"></i><i class="c0" title="layer 27: AudioEncoderLayer"></i><i class="c0" title="layer 28: AudioEncoderLayer"></i><i class="c0" title="layer 29: AudioEncoderLayer"></i><i class="c0" title="layer 30: AudioEncoderLayer"></i><i class="c0" title="layer 31: AudioEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: AudioEncoderLayer"></i><i class="c0" title="layer 1: AudioEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AudioEncoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>thinker.model.visual.blocks</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>VisionBlock</td><td>32</td><td>32</td><td>β kept</td></tr></table></div><div class="stack"><h3>thinker.model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 28</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>28</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>talker.model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_dit_model.input_embed.spk_encoder.blocks</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ModuleList"></i><i class="c1" title="layer 1: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 2: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 3: ModuleList, Res2NetBlock, Sigmoid β¦"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ModuleList"></i><i class="c1" title="layer 1: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 2: ModuleList, Res2NetBlock, Sigmoid β¦"></i><i class="c1" title="layer 3: ModuleList, Res2NetBlock, Sigmoid β¦"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no ModuleList</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>ModuleList, Res2NetBlock, Sigmoid β¦</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_dit_model.transformer_blocks</h3><div class="row"><span class="lab">Original Β· 22</span><div class="cells"><i class="c0" title="layer 0: DiTDecoderLayer"></i><i class="c0" title="layer 1: DiTDecoderLayer"></i><i class="c0" title="layer 2: DiTDecoderLayer"></i><i class="c0" title="layer 3: DiTDecoderLayer"></i><i class="c0" title="layer 4: DiTDecoderLayer"></i><i class="c0" title="layer 5: DiTDecoderLayer"></i><i class="c0" title="layer 6: DiTDecoderLayer"></i><i class="c0" title="layer 7: DiTDecoderLayer"></i><i class="c0" title="layer 8: DiTDecoderLayer"></i><i class="c0" title="layer 9: DiTDecoderLayer"></i><i class="c0" title="layer 10: DiTDecoderLayer"></i><i class="c0" title="layer 11: DiTDecoderLayer"></i><i class="c0" title="layer 12: DiTDecoderLayer"></i><i class="c0" title="layer 13: DiTDecoderLayer"></i><i class="c0" title="layer 14: DiTDecoderLayer"></i><i class="c0" title="layer 15: DiTDecoderLayer"></i><i class="c0" title="layer 16: DiTDecoderLayer"></i><i class="c0" title="layer 17: DiTDecoderLayer"></i><i class="c0" title="layer 18: DiTDecoderLayer"></i><i class="c0" title="layer 19: DiTDecoderLayer"></i><i class="c0" title="layer 20: DiTDecoderLayer"></i><i class="c0" title="layer 21: DiTDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DiTDecoderLayer"></i><i class="c0" title="layer 1: DiTDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DiTDecoderLayer</td><td>22</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_bigvgan_model.ups</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: ModuleList"></i><i class="c0" title="layer 1: ModuleList"></i><i class="c0" title="layer 2: ModuleList"></i><i class="c0" title="layer 3: ModuleList"></i><i class="c0" title="layer 4: ModuleList"></i><i class="c0" title="layer 5: ModuleList"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: ModuleList"></i><i class="c0" title="layer 1: ModuleList"></i><i class="c0" title="layer 2: ModuleList"></i><i class="c0" title="layer 3: ModuleList"></i><i class="c0" title="layer 4: ModuleList"></i><i class="c0" title="layer 5: ModuleList"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ModuleList</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>token2wav.code2wav_bigvgan_model.resblocks</h3><div class="row"><span class="lab">Original Β· 18</span><div class="cells"><i class="c0" title="layer 0: AMPBlock"></i><i class="c0" title="layer 1: AMPBlock"></i><i class="c0" title="layer 2: AMPBlock"></i><i class="c0" title="layer 3: AMPBlock"></i><i class="c0" title="layer 4: AMPBlock"></i><i class="c0" title="layer 5: AMPBlock"></i><i class="c0" title="layer 6: AMPBlock"></i><i class="c0" title="layer 7: AMPBlock"></i><i class="c0" title="layer 8: AMPBlock"></i><i class="c0" title="layer 9: AMPBlock"></i><i class="c0" title="layer 10: AMPBlock"></i><i class="c0" title="layer 11: AMPBlock"></i><i class="c0" title="layer 12: AMPBlock"></i><i class="c0" title="layer 13: AMPBlock"></i><i class="c0" title="layer 14: AMPBlock"></i><i class="c0" title="layer 15: AMPBlock"></i><i class="c0" title="layer 16: AMPBlock"></i><i class="c0" title="layer 17: AMPBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 18</span><div class="cells"><i class="c0" title="layer 0: AMPBlock"></i><i class="c0" title="layer 1: AMPBlock"></i><i class="c0" title="layer 2: AMPBlock"></i><i class="c0" title="layer 3: AMPBlock"></i><i class="c0" title="layer 4: AMPBlock"></i><i class="c0" title="layer 5: AMPBlock"></i><i class="c0" title="layer 6: AMPBlock"></i><i class="c0" title="layer 7: AMPBlock"></i><i class="c0" title="layer 8: AMPBlock"></i><i class="c0" title="layer 9: AMPBlock"></i><i class="c0" title="layer 10: AMPBlock"></i><i class="c0" title="layer 11: AMPBlock"></i><i class="c0" title="layer 12: AMPBlock"></i><i class="c0" title="layer 13: AMPBlock"></i><i class="c0" title="layer 14: AMPBlock"></i><i class="c0" title="layer 15: AMPBlock"></i><i class="c0" title="layer 16: AMPBlock"></i><i class="c0" title="layer 17: AMPBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AMPBlock</td><td>18</td><td>18</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.608040" title="thinker.model.language_model.layers: 6.5B (60.8%)"></i><i class="c1" style="flex:0.122090" title="talker.model.layers: 1.3B (12.2%)"></i><i class="c2" style="flex:0.058746" title="thinker.model.visual.blocks: 630.5M (5.9%)"></i><i class="c3" style="flex:0.058668" title="thinker.model.audio_tower.layers: 629.6M (5.9%)"></i><i class="c4" style="flex:0.056008" title="thinker.model: 601.1M (5.6%)"></i><i class="c5" style="flex:0.050781" title="thinker.lm_head: 545.0M (5.1%)"></i><i class="c6" style="flex:0.030120" title="token2wav.code2wav_dit_model.transformer_blocks: 323.3M (3.0%)"></i><i class="c7" style="flex:0.015547" title="8 others: 166.9M (1.6%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.000436" title="thinker.model.language_model.layers: 74.2K (0.0%)"></i><i class="c1" style="flex:0.000436" title="talker.model.layers: 74.2K (0.0%)"></i><i class="c2" style="flex:0.007825" title="thinker.model.visual.blocks: 1.3M (0.8%)"></i><i class="c3" style="flex:0.000392" title="thinker.model.audio_tower.layers: 66.8K (0.0%)"></i><i class="c4" style="flex:0.059675" title="thinker.model: 10.2M (6.0%)"></i><i class="c5" style="flex:0.057122" title="thinker.lm_head: 9.7M (5.7%)"></i><i class="c6" style="flex:0.000683" title="token2wav.code2wav_dit_model.transformer_blocks: 116.4K (0.1%)"></i><i class="c7" style="flex:0.873431" title="8 others: 148.8M (87.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>thinker.model.language_model.layers</td><td>6.5B</td><td>60.8%</td><td>74.2K</td><td>0.0%</td></tr><tr><td><i class="c1"></i>talker.model.layers</td><td>1.3B</td><td>12.2%</td><td>74.2K</td><td>0.0%</td></tr><tr><td><i class="c2"></i>thinker.model.visual.blocks</td><td>630.5M</td><td>5.9%</td><td>1.3M</td><td>0.8%</td></tr><tr><td><i class="c3"></i>thinker.model.audio_tower.layers</td><td>629.6M</td><td>5.9%</td><td>66.8K</td><td>0.0%</td></tr><tr><td><i class="c4"></i>thinker.model</td><td>601.1M</td><td>5.6%</td><td>10.2M</td><td>6.0%</td></tr><tr><td><i class="c5"></i>thinker.lm_head</td><td>545.0M</td><td>5.1%</td><td>9.7M</td><td>5.7%</td></tr><tr><td><i class="c6"></i>token2wav.code2wav_dit_model.transformer_blocks</td><td>323.3M</td><td>3.0%</td><td>116.4K</td><td>0.1%</td></tr><tr><td><i class="c7"></i>8 others</td><td>166.9M</td><td>1.6%</td><td>148.8M</td><td>87.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_5_omni</td><td>qwen2_5_omni</td><td>β kept</td></tr><tr class=""><td>thinker_config</td><td>model_type</td><td>qwen2_5_omni_thinker</td><td>qwen2_5_omni_thinker</td><td>β kept</td></tr><tr class=""><td>thinker_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>model_type</td><td>qwen2_5_omni_audio_encoder</td><td>qwen2_5_omni_audio_encoder</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>thinker_config.audio_config</td><td>num_mel_bins</td><td>128</td><td>128</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>model_type</td><td>qwen2_5_omni_vision_encoder</td><td>qwen2_5_omni_vision_encoder</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>thinker_config.vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>attention style</td><td>GQA (28 heads, 4 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>model_type</td><td>qwen2_5_omni_text</td><td>qwen2_5_omni_text</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>vocab_size</td><td>152064</td><td>152064</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>thinker_config.text_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>attention style</td><td>GQA (12 heads, 4 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>model_type</td><td>qwen2_5_omni_talker</td><td>qwen2_5_omni_talker</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>vocab_size</td><td>8448</td><td>8448</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>talker_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>token2wav_config</td><td>model_type</td><td>qwen2_5_omni_token2wav</td><td>qwen2_5_omni_token2wav</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>model_type</td><td>qwen2_5_omni_dit</td><td>qwen2_5_omni_dit</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>token2wav_config.dit_config</td><td>max_position_embeddings</td><td>32768</td><td>32768</td><td>β kept</td></tr><tr class=""><td>token2wav_config.bigvgan_config</td><td>model_type</td><td>qwen2_5_omni_bigvgan</td><td>qwen2_5_omni_bigvgan</td><td>β kept</td></tr><tr class=""><td>token2wav_config.bigvgan_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2_5Omni omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>945</td><td>289</td></tr><tr class=""><td>RMSNorm</td><td>171</td><td>75</td></tr><tr class=""><td>Conv1d</td><td>132</td><td>132</td></tr><tr class=""><td>ModuleList</td><td>116</td><td>76</td></tr><tr class=""><td>LayerNorm</td><td>110</td><td>10</td></tr><tr class=""><td>AntiAliasedActivation1d</td><td>109</td><td>109</td></tr><tr class=""><td>DownSample1d</td><td>109</td><td>109</td></tr><tr class=""><td>SnakeBeta</td><td>109</td><td>109</td></tr><tr class=""><td>UpSample1d</td><td>109</td><td>109</td></tr><tr class=""><td>SiLUActivation</td><td>84</td><td>36</td></tr><tr class=""><td>Qwen2MLP</td><td>52</td><td>4</td></tr><tr class=""><td>Attention</td><td>52</td><td>4</td></tr><tr class=""><td>DecoderLayer</td><td>52</td><td>4</td></tr><tr class=""><td>Dropout</td><td>44</td><td>4</td></tr><tr class=""><td>GELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>AudioAttention</td><td>32</td><td>2</td></tr><tr class=""><td>AudioEncoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>32</td></tr><tr class=""><td>VisionAttention</td><td>32</td><td>32</td></tr><tr class=""><td>VisionBlock</td><td>32</td><td>32</td></tr><tr class=""><td>SiLU</td><td>24</td><td>4</td></tr><tr class=""><td>GELU</td><td>23</td><td>3</td></tr><tr class=""><td>DiTAttention</td><td>22</td><td>2</td></tr><tr class=""><td>DiTDecoderLayer</td><td>22</td><td>2</td></tr><tr class=""><td>DiTMLP</td><td>22</td><td>2</td></tr><tr class=""><td>Qwen2_5_OmniAdaLayerNormZero</td><td>22</td><td>2</td></tr><tr class=""><td>AMPBlock</td><td>18</td><td>18</td></tr><tr class=""><td>ReLU</td><td>15</td><td>15</td></tr><tr class=""><td>TimeDelayNetBlock</td><td>12</td><td>12</td></tr><tr class=""><td>ConvTranspose1d</td><td>6</td><td>6</td></tr><tr class=""><td>Embedding</td><td>4</td><td>4</td></tr><tr class=""><td>Res2NetBlock</td><td>3</td><td>3</td></tr><tr class=""><td>Sigmoid</td><td>3</td><td>3</td></tr><tr class=""><td>SqueezeExcitationBlock</td><td>3</td><td>3</td></tr><tr class=""><td>SqueezeExcitationRes2NetBlock</td><td>3</td><td>3</td></tr><tr class=""><td>RotaryEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>AttentiveStatisticsPooling</td><td>1</td><td>1</td></tr><tr class=""><td>AvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Conv3d</td><td>1</td><td>1</td></tr><tr class=""><td>DiTCodecEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>DiTInputEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>DiTTimestepEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ECAPA_TimeDelayNet</td><td>1</td><td>1</td></tr><tr class=""><td>AudioEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>DiTRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>TalkerForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>TalkerModel</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerModel</td><td>1</td><td>1</td></tr><tr class=""><td>ThinkerTextModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavBigVGANModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavDiTModel</td><td>1</td><td>1</td></tr><tr class=""><td>Token2WavModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>VisionRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_OmniAdaLayerNormZero_Final</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionPatchEmbed</td><td>1</td><td>1</td></tr><tr class=""><td>Sequential</td><td>1</td><td>1</td></tr><tr class=""><td>SinusPositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>SinusoidsPositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Tanh</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">30 fields changed, 281 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>talker_config.head_dim</td><td>128</td><td>16</td></tr><tr><td>talker_config.hidden_size</td><td>896</td><td>64</td></tr><tr><td>talker_config.intermediate_size</td><td>18944</td><td>128</td></tr><tr><td>talker_config.layer_types</td><td>24 entries: 24Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>talker_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>talker_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>talker_config.num_key_value_heads</td><td>4</td><td>2</td></tr><tr><td>talker_config.rope_parameters</td><td>{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "rβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>thinker_config.audio_config.d_model</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.audio_config.encoder_attention_heads</td><td>20</td><td>4</td></tr><tr><td>thinker_config.audio_config.encoder_ffn_dim</td><td>5120</td><td>128</td></tr><tr><td>thinker_config.audio_config.encoder_layers</td><td>32</td><td>2</td></tr><tr><td>thinker_config.text_config.hidden_size</td><td>3584</td><td>64</td></tr><tr><td>thinker_config.text_config.intermediate_size</td><td>18944</td><td>128</td></tr><tr><td>thinker_config.text_config.layer_types</td><td>28 entries: 28Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>thinker_config.text_config.num_attention_heads</td><td>28</td><td>4</td></tr><tr><td>thinker_config.text_config.num_hidden_layers</td><td>28</td><td>2</td></tr><tr><td>thinker_config.text_config.num_key_value_heads</td><td>4</td><td>2</td></tr><tr><td>thinker_config.text_config.rope_parameters</td><td>{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "rβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>thinker_config.vision_config.embed_dim</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.vision_config.hidden_size</td><td>1280</td><td>64</td></tr><tr><td>thinker_config.vision_config.intermediate_size</td><td>3420</td><td>128</td></tr><tr><td>thinker_config.vision_config.num_heads</td><td>16</td><td>4</td></tr><tr><td>thinker_config.vision_config.out_hidden_size</td><td>3584</td><td>64</td></tr><tr><td>token2wav_config.dit_config.dim</td><td>1024</td><td>64</td></tr><tr><td>token2wav_config.dit_config.emb_dim</td><td>512</td><td>64</td></tr><tr><td>token2wav_config.dit_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>token2wav_config.dit_config.hidden_size</td><td>1024</td><td>64</td></tr><tr><td>token2wav_config.dit_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>token2wav_config.dit_config.num_hidden_layers</td><td>22</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/qwen2_5_vl-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2.5-VL-3B-Instruct</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_5_vl-mic/tree/32d6627e4ea6fc3ba2fe84c2f3f06e7f5116245a" title="huggingface-mic/qwen2_5_vl-mic@32d6627e4ea6fc3ba2fe84c2f3f06e7f5116245a">huggingface-mic/qwen2_5_vl-mic@32d6627e4ea6</a> β <a href="https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/tree/66285546d2b821cf421d4f5eb2576359d3770cd3" title="Qwen/Qwen2.5-VL-3B-Instruct@66285546d2b821cf421d4f5eb2576359d3770cd3">Qwen/Qwen2.5-VL-3B-Instruct@66285546d2b8</a> Β· Qwen2_5_VLForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.8B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">11.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.visual.blocks</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>VisionBlock</td><td>32</td><td>32</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 36</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i><i class="c0" title="layer 30: DecoderLayer"></i><i class="c0" title="layer 31: DecoderLayer"></i><i class="c0" title="layer 32: DecoderLayer"></i><i class="c0" title="layer 33: DecoderLayer"></i><i class="c0" title="layer 34: DecoderLayer"></i><i class="c0" title="layer 35: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>36</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.739028" title="model.language_model.layers: 2.8B (73.9%)"></i><i class="c1" style="flex:0.167918" title="model.visual.blocks: 630.5M (16.8%)"></i><i class="c2" style="flex:0.082876" title="model.language_model: 311.2M (8.3%)"></i><i class="c3" style="flex:0.010178" title="model.visual: 38.2M (1.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.006576" title="model.language_model.layers: 74.2K (0.7%)"></i><i class="c1" style="flex:0.118101" title="model.visual.blocks: 1.3M (11.8%)"></i><i class="c2" style="flex:0.861365" title="model.language_model: 9.7M (86.1%)"></i><i class="c3" style="flex:0.013958" title="model.visual: 157.6K (1.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>2.8B</td><td>73.9%</td><td>74.2K</td><td>0.7%</td></tr><tr><td><i class="c1"></i>model.visual.blocks</td><td>630.5M</td><td>16.8%</td><td>1.3M</td><td>11.8%</td></tr><tr><td><i class="c2"></i>model.language_model</td><td>311.2M</td><td>8.3%</td><td>9.7M</td><td>86.1%</td></tr><tr><td><i class="c3"></i>model.visual</td><td>38.2M</td><td>1.0%</td><td>157.6K</td><td>1.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_5_vl</td><td>qwen2_5_vl</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (16 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>qwen2_5_vl_text</td><td>qwen2_5_vl_text</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>151936</td><td>151936</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>128000</td><td>128000</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>qwen2_5_vl_vision</td><td>qwen2_5_vl_vision</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2_5_VL omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>415</td><td>177</td></tr><tr class=""><td>RMSNorm</td><td>138</td><td>70</td></tr><tr class=""><td>SiLUActivation</td><td>68</td><td>34</td></tr><tr class=""><td>Qwen2MLP</td><td>36</td><td>2</td></tr><tr class=""><td>Attention</td><td>36</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>36</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>32</td></tr><tr class=""><td>VisionAttention</td><td>32</td><td>32</td></tr><tr class=""><td>VisionBlock</td><td>32</td><td>32</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv3d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>TextModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionPatchEmbed</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionTransformerPretrainedModel</td><td>1</td><td>1</td></tr><tr class=""><td>Sequential</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>11008</td><td>128</td></tr><tr><td>text_config.layer_types</td><td>36 entries: 36Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>text_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>36</td><td>2</td></tr><tr><td>text_config.rope_parameters</td><td>{"type": "mrope", "mrope_section": [16, 24, 24], "rope_theta": 1000000.0, "roβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>vision_config.hidden_size</td><td>1280</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3420</td><td>128</td></tr><tr><td>vision_config.num_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.out_hidden_size</td><td>2048</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2.5-VL-3B-Instruct</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_5_vl-mic/tree/32d6627e4ea6fc3ba2fe84c2f3f06e7f5116245a" title="huggingface-mic/qwen2_5_vl-mic@32d6627e4ea6fc3ba2fe84c2f3f06e7f5116245a">huggingface-mic/qwen2_5_vl-mic@32d6627e4ea6</a> β <a href="https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/tree/66285546d2b821cf421d4f5eb2576359d3770cd3" title="Qwen/Qwen2.5-VL-3B-Instruct@66285546d2b821cf421d4f5eb2576359d3770cd3">Qwen/Qwen2.5-VL-3B-Instruct@66285546d2b8</a> Β· Qwen2_5_VLForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=qwen2_5_vl-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 2 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.8B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">11.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.visual.blocks</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 32</span><div class="cells"><i class="c0" title="layer 0: VisionBlock"></i><i class="c0" title="layer 1: VisionBlock"></i><i class="c0" title="layer 2: VisionBlock"></i><i class="c0" title="layer 3: VisionBlock"></i><i class="c0" title="layer 4: VisionBlock"></i><i class="c0" title="layer 5: VisionBlock"></i><i class="c0" title="layer 6: VisionBlock"></i><i class="c0" title="layer 7: VisionBlock"></i><i class="c0" title="layer 8: VisionBlock"></i><i class="c0" title="layer 9: VisionBlock"></i><i class="c0" title="layer 10: VisionBlock"></i><i class="c0" title="layer 11: VisionBlock"></i><i class="c0" title="layer 12: VisionBlock"></i><i class="c0" title="layer 13: VisionBlock"></i><i class="c0" title="layer 14: VisionBlock"></i><i class="c0" title="layer 15: VisionBlock"></i><i class="c0" title="layer 16: VisionBlock"></i><i class="c0" title="layer 17: VisionBlock"></i><i class="c0" title="layer 18: VisionBlock"></i><i class="c0" title="layer 19: VisionBlock"></i><i class="c0" title="layer 20: VisionBlock"></i><i class="c0" title="layer 21: VisionBlock"></i><i class="c0" title="layer 22: VisionBlock"></i><i class="c0" title="layer 23: VisionBlock"></i><i class="c0" title="layer 24: VisionBlock"></i><i class="c0" title="layer 25: VisionBlock"></i><i class="c0" title="layer 26: VisionBlock"></i><i class="c0" title="layer 27: VisionBlock"></i><i class="c0" title="layer 28: VisionBlock"></i><i class="c0" title="layer 29: VisionBlock"></i><i class="c0" title="layer 30: VisionBlock"></i><i class="c0" title="layer 31: VisionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>VisionBlock</td><td>32</td><td>32</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 36</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i><i class="c0" title="layer 30: DecoderLayer"></i><i class="c0" title="layer 31: DecoderLayer"></i><i class="c0" title="layer 32: DecoderLayer"></i><i class="c0" title="layer 33: DecoderLayer"></i><i class="c0" title="layer 34: DecoderLayer"></i><i class="c0" title="layer 35: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>36</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.739028" title="model.language_model.layers: 2.8B (73.9%)"></i><i class="c1" style="flex:0.167918" title="model.visual.blocks: 630.5M (16.8%)"></i><i class="c2" style="flex:0.082876" title="model.language_model: 311.2M (8.3%)"></i><i class="c3" style="flex:0.010178" title="model.visual: 38.2M (1.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.006576" title="model.language_model.layers: 74.2K (0.7%)"></i><i class="c1" style="flex:0.118101" title="model.visual.blocks: 1.3M (11.8%)"></i><i class="c2" style="flex:0.861365" title="model.language_model: 9.7M (86.1%)"></i><i class="c3" style="flex:0.013958" title="model.visual: 157.6K (1.4%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>2.8B</td><td>73.9%</td><td>74.2K</td><td>0.7%</td></tr><tr><td><i class="c1"></i>model.visual.blocks</td><td>630.5M</td><td>16.8%</td><td>1.3M</td><td>11.8%</td></tr><tr><td><i class="c2"></i>model.language_model</td><td>311.2M</td><td>8.3%</td><td>9.7M</td><td>86.1%</td></tr><tr><td><i class="c3"></i>model.visual</td><td>38.2M</td><td>1.0%</td><td>157.6K</td><td>1.4%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_5_vl</td><td>qwen2_5_vl</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>GQA (16 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>qwen2_5_vl_text</td><td>qwen2_5_vl_text</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>151936</td><td>151936</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>128000</td><td>128000</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>RoPE type</td><td>axial</td><td>axial</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>qwen2_5_vl_vision</td><td>qwen2_5_vl_vision</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2_5_VL omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>415</td><td>177</td></tr><tr class=""><td>RMSNorm</td><td>138</td><td>70</td></tr><tr class=""><td>SiLUActivation</td><td>68</td><td>34</td></tr><tr class=""><td>Qwen2MLP</td><td>36</td><td>2</td></tr><tr class=""><td>Attention</td><td>36</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>36</td><td>2</td></tr><tr class=""><td>MLP</td><td>32</td><td>32</td></tr><tr class=""><td>VisionAttention</td><td>32</td><td>32</td></tr><tr class=""><td>VisionBlock</td><td>32</td><td>32</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv3d</td><td>1</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchMerger</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>TextModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionPatchEmbed</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2_5_VisionTransformerPretrainedModel</td><td>1</td><td>1</td></tr><tr class=""><td>Sequential</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>11008</td><td>128</td></tr><tr><td>text_config.layer_types</td><td>36 entries: 36Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>text_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>36</td><td>2</td></tr><tr><td>text_config.rope_parameters</td><td>{"type": "mrope", "mrope_section": [16, 24, 24], "rope_theta": 1000000.0, "roβ¦</td><td>{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",β¦</td></tr><tr><td>vision_config.hidden_size</td><td>1280</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3420</td><td>128</td></tr><tr><td>vision_config.num_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.out_hidden_size</td><td>2048</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/qwen2_audio-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2-Audio-7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_audio-mic/tree/c426549756a18dc16495239883efd38d9a66002b" title="huggingface-mic/qwen2_audio-mic@c426549756a18dc16495239883efd38d9a66002b">huggingface-mic/qwen2_audio-mic@c426549756a1</a> β <a href="https://huggingface.co/Qwen/Qwen2-Audio-7B/tree/dd84470756e6277a71d4d7188773a43cde92696e" title="Qwen/Qwen2-Audio-7B@dd84470756e6277a71d4d7188773a43cde92696e">Qwen/Qwen2-Audio-7B@dd84470756e6</a> Β· Qwen2AudioForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">8.4B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">20.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.audio_tower.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i><i class="c0" title="layer 24: EncoderLayer"></i><i class="c0" title="layer 25: EncoderLayer"></i><i class="c0" title="layer 26: EncoderLayer"></i><i class="c0" title="layer 27: EncoderLayer"></i><i class="c0" title="layer 28: EncoderLayer"></i><i class="c0" title="layer 29: EncoderLayer"></i><i class="c0" title="layer 30: EncoderLayer"></i><i class="c0" title="layer 31: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: Qwen2DecoderLayer"></i><i class="c0" title="layer 1: Qwen2DecoderLayer"></i><i class="c0" title="layer 2: Qwen2DecoderLayer"></i><i class="c0" title="layer 3: Qwen2DecoderLayer"></i><i class="c0" title="layer 4: Qwen2DecoderLayer"></i><i class="c0" title="layer 5: Qwen2DecoderLayer"></i><i class="c0" title="layer 6: Qwen2DecoderLayer"></i><i class="c0" title="layer 7: Qwen2DecoderLayer"></i><i class="c0" title="layer 8: Qwen2DecoderLayer"></i><i class="c0" title="layer 9: Qwen2DecoderLayer"></i><i class="c0" title="layer 10: Qwen2DecoderLayer"></i><i class="c0" title="layer 11: Qwen2DecoderLayer"></i><i class="c0" title="layer 12: Qwen2DecoderLayer"></i><i class="c0" title="layer 13: Qwen2DecoderLayer"></i><i class="c0" title="layer 14: Qwen2DecoderLayer"></i><i class="c0" title="layer 15: Qwen2DecoderLayer"></i><i class="c0" title="layer 16: Qwen2DecoderLayer"></i><i class="c0" title="layer 17: Qwen2DecoderLayer"></i><i class="c0" title="layer 18: Qwen2DecoderLayer"></i><i class="c0" title="layer 19: Qwen2DecoderLayer"></i><i class="c0" title="layer 20: Qwen2DecoderLayer"></i><i class="c0" title="layer 21: Qwen2DecoderLayer"></i><i class="c0" title="layer 22: Qwen2DecoderLayer"></i><i class="c0" title="layer 23: Qwen2DecoderLayer"></i><i class="c0" title="layer 24: Qwen2DecoderLayer"></i><i class="c0" title="layer 25: Qwen2DecoderLayer"></i><i class="c0" title="layer 26: Qwen2DecoderLayer"></i><i class="c0" title="layer 27: Qwen2DecoderLayer"></i><i class="c0" title="layer 28: Qwen2DecoderLayer"></i><i class="c0" title="layer 29: Qwen2DecoderLayer"></i><i class="c0" title="layer 30: Qwen2DecoderLayer"></i><i class="c0" title="layer 31: Qwen2DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Qwen2DecoderLayer"></i><i class="c0" title="layer 1: Qwen2DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Qwen2DecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.771298" title="model.language_model.layers: 6.5B (77.1%)"></i><i class="c1" style="flex:0.076111" title="model.language_model: 639.1M (7.6%)"></i><i class="c2" style="flex:0.076110" title="lm_head: 639.1M (7.6%)"></i><i class="c3" style="flex:0.074983" title="model.audio_tower.layers: 629.6M (7.5%)"></i><i class="c4" style="flex:0.000873" title="model.audio_tower: 7.3M (0.1%)"></i><i class="c5" style="flex:0.000625" title="model.multi_modal_projector: 5.2M (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.004075" title="model.language_model.layers: 82.6K (0.4%)"></i><i class="c1" style="flex:0.492927" title="model.language_model: 10.0M (49.3%)"></i><i class="c2" style="flex:0.492924" title="lm_head: 10.0M (49.3%)"></i><i class="c3" style="flex:0.003298" title="model.audio_tower.layers: 66.8K (0.3%)"></i><i class="c4" style="flex:0.006571" title="model.audio_tower: 133.1K (0.7%)"></i><i class="c5" style="flex:0.000205" title="model.multi_modal_projector: 4.2K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>6.5B</td><td>77.1%</td><td>82.6K</td><td>0.4%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>639.1M</td><td>7.6%</td><td>10.0M</td><td>49.3%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>639.1M</td><td>7.6%</td><td>10.0M</td><td>49.3%</td></tr><tr><td><i class="c3"></i>model.audio_tower.layers</td><td>629.6M</td><td>7.5%</td><td>66.8K</td><td>0.3%</td></tr><tr><td><i class="c4"></i>model.audio_tower</td><td>7.3M</td><td>0.1%</td><td>133.1K</td><td>0.7%</td></tr><tr><td><i class="c5"></i>model.multi_modal_projector</td><td>5.2M</td><td>0.1%</td><td>4.2K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_audio</td><td>qwen2_audio</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>156032</td><td>156032</td><td>β kept</td></tr><tr class=""><td>audio_config</td><td>model_type</td><td>qwen2_audio_encoder</td><td>qwen2_audio_encoder</td><td>β kept</td></tr><tr class=""><td>audio_config</td><td>num_mel_bins</td><td>128</td><td>128</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>qwen2</td><td>qwen2</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>156032</td><td>156032</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2Audio omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>418</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>65</td><td>5</td></tr><tr class=""><td>Qwen2RMSNorm</td><td>65</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2Attention</td><td>32</td><td>2</td></tr><tr class=""><td>Attention</td><td>32</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2DecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2MLP</td><td>32</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Conv1d</td><td>2</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>AvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2Model</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 55 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>audio_config.d_model</td><td>1280</td><td>64</td></tr><tr><td>audio_config.encoder_attention_heads</td><td>20</td><td>4</td></tr><tr><td>audio_config.encoder_ffn_dim</td><td>5120</td><td>128</td></tr><tr><td>audio_config.encoder_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.hidden_size</td><td>4096</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>11008</td><td>128</td></tr><tr><td>text_config.layer_types</td><td>32 entries: 32Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>Qwen/Qwen2-Audio-7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/qwen2_audio-mic/tree/c426549756a18dc16495239883efd38d9a66002b" title="huggingface-mic/qwen2_audio-mic@c426549756a18dc16495239883efd38d9a66002b">huggingface-mic/qwen2_audio-mic@c426549756a1</a> β <a href="https://huggingface.co/Qwen/Qwen2-Audio-7B/tree/dd84470756e6277a71d4d7188773a43cde92696e" title="Qwen/Qwen2-Audio-7B@dd84470756e6277a71d4d7188773a43cde92696e">Qwen/Qwen2-Audio-7B@dd84470756e6</a> Β· Qwen2AudioForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=qwen2_audio-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 3 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">8.4B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">20.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.audio_tower.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i><i class="c0" title="layer 24: EncoderLayer"></i><i class="c0" title="layer 25: EncoderLayer"></i><i class="c0" title="layer 26: EncoderLayer"></i><i class="c0" title="layer 27: EncoderLayer"></i><i class="c0" title="layer 28: EncoderLayer"></i><i class="c0" title="layer 29: EncoderLayer"></i><i class="c0" title="layer 30: EncoderLayer"></i><i class="c0" title="layer 31: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.language_model.layers</h3><div class="row"><span class="lab">Original Β· 32</span><div class="cells"><i class="c0" title="layer 0: Qwen2DecoderLayer"></i><i class="c0" title="layer 1: Qwen2DecoderLayer"></i><i class="c0" title="layer 2: Qwen2DecoderLayer"></i><i class="c0" title="layer 3: Qwen2DecoderLayer"></i><i class="c0" title="layer 4: Qwen2DecoderLayer"></i><i class="c0" title="layer 5: Qwen2DecoderLayer"></i><i class="c0" title="layer 6: Qwen2DecoderLayer"></i><i class="c0" title="layer 7: Qwen2DecoderLayer"></i><i class="c0" title="layer 8: Qwen2DecoderLayer"></i><i class="c0" title="layer 9: Qwen2DecoderLayer"></i><i class="c0" title="layer 10: Qwen2DecoderLayer"></i><i class="c0" title="layer 11: Qwen2DecoderLayer"></i><i class="c0" title="layer 12: Qwen2DecoderLayer"></i><i class="c0" title="layer 13: Qwen2DecoderLayer"></i><i class="c0" title="layer 14: Qwen2DecoderLayer"></i><i class="c0" title="layer 15: Qwen2DecoderLayer"></i><i class="c0" title="layer 16: Qwen2DecoderLayer"></i><i class="c0" title="layer 17: Qwen2DecoderLayer"></i><i class="c0" title="layer 18: Qwen2DecoderLayer"></i><i class="c0" title="layer 19: Qwen2DecoderLayer"></i><i class="c0" title="layer 20: Qwen2DecoderLayer"></i><i class="c0" title="layer 21: Qwen2DecoderLayer"></i><i class="c0" title="layer 22: Qwen2DecoderLayer"></i><i class="c0" title="layer 23: Qwen2DecoderLayer"></i><i class="c0" title="layer 24: Qwen2DecoderLayer"></i><i class="c0" title="layer 25: Qwen2DecoderLayer"></i><i class="c0" title="layer 26: Qwen2DecoderLayer"></i><i class="c0" title="layer 27: Qwen2DecoderLayer"></i><i class="c0" title="layer 28: Qwen2DecoderLayer"></i><i class="c0" title="layer 29: Qwen2DecoderLayer"></i><i class="c0" title="layer 30: Qwen2DecoderLayer"></i><i class="c0" title="layer 31: Qwen2DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Qwen2DecoderLayer"></i><i class="c0" title="layer 1: Qwen2DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Qwen2DecoderLayer</td><td>32</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.771298" title="model.language_model.layers: 6.5B (77.1%)"></i><i class="c1" style="flex:0.076111" title="model.language_model: 639.1M (7.6%)"></i><i class="c2" style="flex:0.076110" title="lm_head: 639.1M (7.6%)"></i><i class="c3" style="flex:0.074983" title="model.audio_tower.layers: 629.6M (7.5%)"></i><i class="c4" style="flex:0.000873" title="model.audio_tower: 7.3M (0.1%)"></i><i class="c5" style="flex:0.000625" title="model.multi_modal_projector: 5.2M (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.004075" title="model.language_model.layers: 82.6K (0.4%)"></i><i class="c1" style="flex:0.492927" title="model.language_model: 10.0M (49.3%)"></i><i class="c2" style="flex:0.492924" title="lm_head: 10.0M (49.3%)"></i><i class="c3" style="flex:0.003298" title="model.audio_tower.layers: 66.8K (0.3%)"></i><i class="c4" style="flex:0.006571" title="model.audio_tower: 133.1K (0.7%)"></i><i class="c5" style="flex:0.000205" title="model.multi_modal_projector: 4.2K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.language_model.layers</td><td>6.5B</td><td>77.1%</td><td>82.6K</td><td>0.4%</td></tr><tr><td><i class="c1"></i>model.language_model</td><td>639.1M</td><td>7.6%</td><td>10.0M</td><td>49.3%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>639.1M</td><td>7.6%</td><td>10.0M</td><td>49.3%</td></tr><tr><td><i class="c3"></i>model.audio_tower.layers</td><td>629.6M</td><td>7.5%</td><td>66.8K</td><td>0.3%</td></tr><tr><td><i class="c4"></i>model.audio_tower</td><td>7.3M</td><td>0.1%</td><td>133.1K</td><td>0.7%</td></tr><tr><td><i class="c5"></i>model.multi_modal_projector</td><td>5.2M</td><td>0.1%</td><td>4.2K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>qwen2_audio</td><td>qwen2_audio</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>156032</td><td>156032</td><td>β kept</td></tr><tr class=""><td>audio_config</td><td>model_type</td><td>qwen2_audio_encoder</td><td>qwen2_audio_encoder</td><td>β kept</td></tr><tr class=""><td>audio_config</td><td>num_mel_bins</td><td>128</td><td>128</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>qwen2</td><td>qwen2</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>156032</td><td>156032</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Qwen2Audio omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>418</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>65</td><td>5</td></tr><tr class=""><td>Qwen2RMSNorm</td><td>65</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2Attention</td><td>32</td><td>2</td></tr><tr class=""><td>Attention</td><td>32</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2DecoderLayer</td><td>32</td><td>2</td></tr><tr class=""><td>Qwen2MLP</td><td>32</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>32</td><td>2</td></tr><tr class=""><td>Conv1d</td><td>2</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>AvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiModalProjector</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2Model</td><td>1</td><td>1</td></tr><tr class=""><td>Qwen2RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 55 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>audio_config.d_model</td><td>1280</td><td>64</td></tr><tr><td>audio_config.encoder_attention_heads</td><td>20</td><td>4</td></tr><tr><td>audio_config.encoder_ffn_dim</td><td>5120</td><td>128</td></tr><tr><td>audio_config.encoder_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.hidden_size</td><td>4096</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>11008</td><td>128</td></tr><tr><td>text_config.layer_types</td><td>32 entries: 32Γ full_attention</td><td>["full_attention", "full_attention"]</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>32</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/recurrentgemma-2b-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/recurrentgemma-2b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/recurrentgemma-2b-mic/tree/732c0b9399dcbbdb7c806f7043968c4aafea55c8" title="huggingface-mic/recurrentgemma-2b-mic@732c0b9399dcbbdb7c806f7043968c4aafea55c8">huggingface-mic/recurrentgemma-2b-mic@732c0b9399dc</a> β <a href="https://huggingface.co/google/recurrentgemma-2b/tree/3620f4ca9c5d16ee56c00180474a3201ec7f734a" title="google/recurrentgemma-2b@3620f4ca9c5d16ee56c00180474a3201ec7f734a">google/recurrentgemma-2b@3620f4ca9c5d</a> Β· RecurrentGemmaForCausalLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">24.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 26</span><div class="cells"><i class="c0" title="layer 0: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 1: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 2: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 3: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 4: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 5: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 6: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 7: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 8: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 9: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 10: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 11: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 12: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 13: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 14: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 15: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 16: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 17: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 18: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 19: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 20: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 21: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 22: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 23: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 24: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 25: Conv1d, RecurrentBlock, Rglru"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 1: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 2: is_causal=True, Attention, RotaryEmbedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv1d, RecurrentBlock, Rglru</td><td>18</td><td>2</td><td>β kept</td></tr><tr><td><i class="c1"></i>is_causal=True, Attention, RotaryEmbedding</td><td>8</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.755723" title="model.layers: 2.0B (75.6%)"></i><i class="c1" style="flex:0.244276" title="model.embed_tokens: 655.4M (24.4%)"></i><i class="c2" style="flex:0.000001" title="model.final_norm: 2.6K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.317904" title="model.layers: 7.6M (31.8%)"></i><i class="c1" style="flex:0.682094" title="model.embed_tokens: 16.4M (68.2%)"></i><i class="c2" style="flex:0.000003" title="model.final_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.0B</td><td>75.6%</td><td>7.6M</td><td>31.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>655.4M</td><td>24.4%</td><td>16.4M</td><td>68.2%</td></tr><tr><td><i class="c2"></i>model.final_norm</td><td>2.6K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MQA (10 heads, 1 KV)</td><td>MQA (4 heads, 1 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>recurrent_gemma</td><td>recurrent_gemma</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256000</td><td>256000</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix RecurrentGemma omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>165</td><td>20</td></tr><tr class=""><td>RMSNorm</td><td>53</td><td>7</td></tr><tr class=""><td>GELUTanh</td><td>44</td><td>5</td></tr><tr class=""><td>DecoderLayer</td><td>26</td><td>3</td></tr><tr class=""><td>Mlp</td><td>26</td><td>3</td></tr><tr class=""><td>Conv1d</td><td>18</td><td>2</td></tr><tr class=""><td>RecurrentBlock</td><td>18</td><td>2</td></tr><tr class=""><td>Rglru</td><td>18</td><td>2</td></tr><tr class=""><td>Attention</td><td>8</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>8</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 30 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>256</td><td>16</td></tr><tr><td>hidden_size</td><td>2560</td><td>64</td></tr><tr><td>intermediate_size</td><td>15360</td><td>128</td></tr><tr><td>num_attention_heads</td><td>10</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>26</td><td>3</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/recurrentgemma-2b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/recurrentgemma-2b-mic/tree/732c0b9399dcbbdb7c806f7043968c4aafea55c8" title="huggingface-mic/recurrentgemma-2b-mic@732c0b9399dcbbdb7c806f7043968c4aafea55c8">huggingface-mic/recurrentgemma-2b-mic@732c0b9399dc</a> β <a href="https://huggingface.co/google/recurrentgemma-2b/tree/3620f4ca9c5d16ee56c00180474a3201ec7f734a" title="google/recurrentgemma-2b@3620f4ca9c5d16ee56c00180474a3201ec7f734a">google/recurrentgemma-2b@3620f4ca9c5d</a> Β· RecurrentGemmaForCausalLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=recurrentgemma-2b-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">24.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 26</span><div class="cells"><i class="c0" title="layer 0: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 1: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 2: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 3: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 4: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 5: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 6: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 7: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 8: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 9: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 10: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 11: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 12: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 13: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 14: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 15: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 16: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 17: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 18: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 19: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 20: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 21: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 22: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 23: is_causal=True, Attention, RotaryEmbedding"></i><i class="c0" title="layer 24: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 25: Conv1d, RecurrentBlock, Rglru"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv1d, RecurrentBlock, Rglru"></i><i class="c0" title="layer 1: Conv1d, RecurrentBlock, Rglru"></i><i class="c1" title="layer 2: is_causal=True, Attention, RotaryEmbedding"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv1d, RecurrentBlock, Rglru</td><td>18</td><td>2</td><td>β kept</td></tr><tr><td><i class="c1"></i>is_causal=True, Attention, RotaryEmbedding</td><td>8</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.755723" title="model.layers: 2.0B (75.6%)"></i><i class="c1" style="flex:0.244276" title="model.embed_tokens: 655.4M (24.4%)"></i><i class="c2" style="flex:0.000001" title="model.final_norm: 2.6K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.317904" title="model.layers: 7.6M (31.8%)"></i><i class="c1" style="flex:0.682094" title="model.embed_tokens: 16.4M (68.2%)"></i><i class="c2" style="flex:0.000003" title="model.final_norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.0B</td><td>75.6%</td><td>7.6M</td><td>31.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>655.4M</td><td>24.4%</td><td>16.4M</td><td>68.2%</td></tr><tr><td><i class="c2"></i>model.final_norm</td><td>2.6K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MQA (10 heads, 1 KV)</td><td>MQA (4 heads, 1 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>recurrent_gemma</td><td>recurrent_gemma</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256000</td><td>256000</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix RecurrentGemma omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>165</td><td>20</td></tr><tr class=""><td>RMSNorm</td><td>53</td><td>7</td></tr><tr class=""><td>GELUTanh</td><td>44</td><td>5</td></tr><tr class=""><td>DecoderLayer</td><td>26</td><td>3</td></tr><tr class=""><td>Mlp</td><td>26</td><td>3</td></tr><tr class=""><td>Conv1d</td><td>18</td><td>2</td></tr><tr class=""><td>RecurrentBlock</td><td>18</td><td>2</td></tr><tr class=""><td>Rglru</td><td>18</td><td>2</td></tr><tr class=""><td>Attention</td><td>8</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>8</td><td>1</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 30 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>256</td><td>16</td></tr><tr><td>hidden_size</td><td>2560</td><td>64</td></tr><tr><td>intermediate_size</td><td>15360</td><td>128</td></tr><tr><td>num_attention_heads</td><td>10</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>26</td><td>3</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/resnet-18-mic.html
CHANGED
|
@@ -31,6 +31,7 @@ tr.bad td{color:var(--warn)}
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
-
<h1>microsoft/resnet-18</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/resnet-18-mic/tree/
|
| 35 |
-
<
|
|
|
|
| 36 |
</main></body></html>
|
|
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
+
<h1>microsoft/resnet-18</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/resnet-18-mic/tree/79f17282337052447d8e69c74df99f801e596790" title="huggingface-mic/resnet-18-mic@79f17282337052447d8e69c74df99f801e596790">huggingface-mic/resnet-18-mic@79f172823370</a> β <a href="https://huggingface.co/microsoft/resnet-18/tree/65a5785d9156231087c481e0c7dd33a5ff6f7e3e" title="microsoft/resnet-18@65a5785d9156231087c481e0c7dd33a5ff6f7e3e">microsoft/resnet-18@65a5785d9156</a> Β· ResNetForImageClassification Β· real weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=resnet-18-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
+
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">11.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">11.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>resnet.encoder.stages</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ShortCut"></i><i class="c1" title="layer 1: ShortCut"></i><i class="c1" title="layer 2: ShortCut"></i><i class="c1" title="layer 3: ShortCut"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: no ShortCut"></i><i class="c1" title="layer 1: ShortCut"></i><i class="c1" title="layer 2: ShortCut"></i><i class="c1" title="layer 3: ShortCut"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no ShortCut</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>ShortCut</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.955299" title="resnet.encoder.stages: 11.2M (95.5%)"></i><i class="c1" style="flex:0.043885" title="classifier.1: 513.0K (4.4%)"></i><i class="c2" style="flex:0.000816" title="resnet.embedder: 9.5K (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.955299" title="resnet.encoder.stages: 11.2M (95.5%)"></i><i class="c1" style="flex:0.043885" title="classifier.1: 513.0K (4.4%)"></i><i class="c2" style="flex:0.000816" title="resnet.embedder: 9.5K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>resnet.encoder.stages</td><td>11.2M</td><td>95.5%</td><td>11.2M</td><td>95.5%</td></tr><tr><td><i class="c1"></i>classifier.1</td><td>513.0K</td><td>4.4%</td><td>513.0K</td><td>4.4%</td></tr><tr><td><i class="c2"></i>resnet.embedder</td><td>9.5K</td><td>0.1%</td><td>9.5K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>resnet</td><td>resnet</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>relu</td><td>relu</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix ResNet omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>BatchNorm2d</td><td>20</td><td>20</td></tr><tr class=""><td>Conv2d</td><td>20</td><td>20</td></tr><tr class=""><td>ReLU</td><td>17</td><td>17</td></tr><tr class=""><td>ConvLayer</td><td>17</td><td>17</td></tr><tr class=""><td>Identity</td><td>13</td><td>13</td></tr><tr class=""><td>Sequential</td><td>13</td><td>13</td></tr><tr class=""><td>BasicLayer</td><td>8</td><td>8</td></tr><tr class=""><td>Stage</td><td>4</td><td>4</td></tr><tr class=""><td>ShortCut</td><td>3</td><td>3</td></tr><tr class=""><td>AdaptiveAvgPool2d</td><td>1</td><td>1</td></tr><tr class=""><td>Flatten</td><td>1</td><td>1</td></tr><tr class=""><td>Linear</td><td>1</td><td>1</td></tr><tr class=""><td>MaxPool2d</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">0 fields changed, 20 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr></table>
|
| 37 |
</main></body></html>
|
shrink/resnet-18-mic.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
{"fixture": "huggingface-mic/resnet-18-mic@
|
|
|
|
| 1 |
+
{"fixture": "huggingface-mic/resnet-18-mic@79f17282337052447d8e69c74df99f801e596790", "findings": []}
|
shrink/roberta-base-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>FacebookAI/roberta-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/roberta-base-mic/tree/ff3c8d1292ae8885553023bad9a04744ced2effe" title="huggingface-mic/roberta-base-mic@ff3c8d1292ae8885553023bad9a04744ced2effe">huggingface-mic/roberta-base-mic@ff3c8d1292ae</a> β <a href="https://huggingface.co/FacebookAI/roberta-base/tree/e2da8e2f811d1448a5b465c236feacd80ffbac7b" title="FacebookAI/roberta-base@e2da8e2f811d1448a5b465c236feacd80ffbac7b">FacebookAI/roberta-base@e2da8e2f811d</a> Β· RobertaForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">124.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>roberta.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.682087" title="roberta.encoder.layer: 85.1M (68.2%)"></i><i class="c1" style="flex:0.312762" title="roberta.embeddings: 39.0M (31.3%)"></i><i class="c2" style="flex:0.004736" title="lm_head.dense: 590.6K (0.5%)"></i><i class="c3" style="flex:0.000403" title="lm_head: 50.3K (0.0%)"></i><i class="c4" style="flex:0.000012" title="lm_head.layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.019856" title="roberta.encoder.layer: 66.9K (2.0%)"></i><i class="c1" style="flex:0.963964" title="roberta.embeddings: 3.3M (96.4%)"></i><i class="c2" style="flex:0.001234" title="lm_head.dense: 4.2K (0.1%)"></i><i class="c3" style="flex:0.014909" title="lm_head: 50.3K (1.5%)"></i><i class="c4" style="flex:0.000038" title="lm_head.layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>roberta.encoder.layer</td><td>85.1M</td><td>68.2%</td><td>66.9K</td><td>2.0%</td></tr><tr><td><i class="c1"></i>roberta.embeddings</td><td>39.0M</td><td>31.3%</td><td>3.3M</td><td>96.4%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.5%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>50.3K</td><td>0.0%</td><td>50.3K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>roberta</td><td>roberta</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50265</td><td>50265</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Roberta omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>37</td><td>7</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>FacebookAI/roberta-base</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/roberta-base-mic/tree/ff3c8d1292ae8885553023bad9a04744ced2effe" title="huggingface-mic/roberta-base-mic@ff3c8d1292ae8885553023bad9a04744ced2effe">huggingface-mic/roberta-base-mic@ff3c8d1292ae</a> β <a href="https://huggingface.co/FacebookAI/roberta-base/tree/e2da8e2f811d1448a5b465c236feacd80ffbac7b" title="FacebookAI/roberta-base@e2da8e2f811d1448a5b465c236feacd80ffbac7b">FacebookAI/roberta-base@e2da8e2f811d</a> Β· RobertaForMaskedLM Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=roberta-base-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">124.7M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>roberta.encoder.layer</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i><i class="c0" title="layer 2: Layer"></i><i class="c0" title="layer 3: Layer"></i><i class="c0" title="layer 4: Layer"></i><i class="c0" title="layer 5: Layer"></i><i class="c0" title="layer 6: Layer"></i><i class="c0" title="layer 7: Layer"></i><i class="c0" title="layer 8: Layer"></i><i class="c0" title="layer 9: Layer"></i><i class="c0" title="layer 10: Layer"></i><i class="c0" title="layer 11: Layer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Layer"></i><i class="c0" title="layer 1: Layer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Layer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.682087" title="roberta.encoder.layer: 85.1M (68.2%)"></i><i class="c1" style="flex:0.312762" title="roberta.embeddings: 39.0M (31.3%)"></i><i class="c2" style="flex:0.004736" title="lm_head.dense: 590.6K (0.5%)"></i><i class="c3" style="flex:0.000403" title="lm_head: 50.3K (0.0%)"></i><i class="c4" style="flex:0.000012" title="lm_head.layer_norm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.019856" title="roberta.encoder.layer: 66.9K (2.0%)"></i><i class="c1" style="flex:0.963964" title="roberta.embeddings: 3.3M (96.4%)"></i><i class="c2" style="flex:0.001234" title="lm_head.dense: 4.2K (0.1%)"></i><i class="c3" style="flex:0.014909" title="lm_head: 50.3K (1.5%)"></i><i class="c4" style="flex:0.000038" title="lm_head.layer_norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>roberta.encoder.layer</td><td>85.1M</td><td>68.2%</td><td>66.9K</td><td>2.0%</td></tr><tr><td><i class="c1"></i>roberta.embeddings</td><td>39.0M</td><td>31.3%</td><td>3.3M</td><td>96.4%</td></tr><tr><td><i class="c2"></i>lm_head.dense</td><td>590.6K</td><td>0.5%</td><td>4.2K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>50.3K</td><td>0.0%</td><td>50.3K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>lm_head.layer_norm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>roberta</td><td>roberta</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>50265</td><td>50265</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>514</td><td>514</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Roberta omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>74</td><td>14</td></tr><tr class=""><td>Dropout</td><td>37</td><td>7</td></tr><tr class=""><td>LayerNorm</td><td>26</td><td>6</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>2</td></tr><tr class=""><td>Attention</td><td>12</td><td>2</td></tr><tr class=""><td>Intermediate</td><td>12</td><td>2</td></tr><tr class=""><td>Layer</td><td>12</td><td>2</td></tr><tr class=""><td>Output</td><td>12</td><td>2</td></tr><tr class=""><td>SelfAttention</td><td>12</td><td>2</td></tr><tr class=""><td>SelfOutput</td><td>12</td><td>2</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForMaskedLM</td><td>1</td><td>1</td></tr><tr class=""><td>LMHead</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/rtdetr_r50vd-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>PekingU/rtdetr_r50vd</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/rtdetr_r50vd-mic/tree/5655fbe502f7ba0ce74c6037780272095348ca82" title="huggingface-mic/rtdetr_r50vd-mic@5655fbe502f7ba0ce74c6037780272095348ca82">huggingface-mic/rtdetr_r50vd-mic@5655fbe502f7</a> β <a href="https://huggingface.co/PekingU/rtdetr_r50vd/tree/df939e661d8c52e80608d1ec566561aabd25a4e7" title="PekingU/rtdetr_r50vd@df939e661d8c52e80608d1ec566561aabd25a4e7">PekingU/rtdetr_r50vd@df939e661d8c</a> Β· RTDetrForObjectDetection Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">42.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">10.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">10</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.backbone.model.encoder.stages</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: no AvgPool2d"></i><i class="c1" title="layer 1: AvgPool2d"></i><i class="c1" title="layer 2: AvgPool2d"></i><i class="c1" title="layer 3: AvgPool2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: no AvgPool2d"></i><i class="c1" title="layer 1: AvgPool2d"></i><i class="c1" title="layer 2: AvgPool2d"></i><i class="c1" title="layer 3: AvgPool2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no AvgPool2d</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>AvgPool2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder_input_proj</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.aifi</h3><div class="row"><span class="lab">Original Β· 1</span><div class="cells"><i class="c0" title="layer 0: AIFILayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: AIFILayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AIFILayer</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.lateral_convs</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConvNormLayer</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.fpn_blocks</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>RepVggBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.downsample_convs</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConvNormLayer</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.pan_blocks</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>RepVggBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder_input_proj</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.546625" title="model.backbone.model.encoder.stages: 23.4M (54.7%)"></i><i class="c1" style="flex:0.139311" title="model.decoder.layers: 6.0M (13.9%)"></i><i class="c2" style="flex:0.104092" title="model.encoder.pan_blocks: 4.5M (10.4%)"></i><i class="c3" style="flex:0.104092" title="model.encoder.fpn_blocks: 4.5M (10.4%)"></i><i class="c4" style="flex:0.027527" title="model.encoder.downsample_convs: 1.2M (2.8%)"></i><i class="c5" style="flex:0.021427" title="model.encoder_input_proj: 919.0K (2.1%)"></i><i class="c6" style="flex:0.018551" title="model.decoder.bbox_embed: 795.7K (1.9%)"></i><i class="c7" style="flex:0.038375" title="9 others: 1.6M (3.8%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.002971" title="model.backbone.model.encoder.stages: 32.4K (0.3%)"></i><i class="c1" style="flex:0.009406" title="model.decoder.layers: 102.6K (0.9%)"></i><i class="c2" style="flex:0.409459" title="model.encoder.pan_blocks: 4.5M (40.9%)"></i><i class="c3" style="flex:0.409459" title="model.encoder.fpn_blocks: 4.5M (40.9%)"></i><i class="c4" style="flex:0.108281" title="model.encoder.downsample_convs: 1.2M (10.8%)"></i><i class="c5" style="flex:0.005400" title="model.encoder_input_proj: 58.9K (0.5%)"></i><i class="c6" style="flex:0.001574" title="model.decoder.bbox_embed: 17.2K (0.2%)"></i><i class="c7" style="flex:0.053450" title="9 others: 582.8K (5.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.backbone.model.encoder.stages</td><td>23.4M</td><td>54.7%</td><td>32.4K</td><td>0.3%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>6.0M</td><td>13.9%</td><td>102.6K</td><td>0.9%</td></tr><tr><td><i class="c2"></i>model.encoder.pan_blocks</td><td>4.5M</td><td>10.4%</td><td>4.5M</td><td>40.9%</td></tr><tr><td><i class="c3"></i>model.encoder.fpn_blocks</td><td>4.5M</td><td>10.4%</td><td>4.5M</td><td>40.9%</td></tr><tr><td><i class="c4"></i>model.encoder.downsample_convs</td><td>1.2M</td><td>2.8%</td><td>1.2M</td><td>10.8%</td></tr><tr><td><i class="c5"></i>model.encoder_input_proj</td><td>919.0K</td><td>2.1%</td><td>58.9K</td><td>0.5%</td></tr><tr><td><i class="c6"></i>model.decoder.bbox_embed</td><td>795.7K</td><td>1.9%</td><td>17.2K</td><td>0.2%</td></tr><tr><td><i class="c7"></i>9 others</td><td>1.6M</td><td>3.8%</td><td>582.8K</td><td>5.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>rt_detr</td><td>rt_detr</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>rt_detr_resnet</td><td>rt_detr_resnet</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>relu</td><td>relu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix RTDetr omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Conv2d</td><td>97</td><td>61</td></tr><tr class=""><td>Linear</td><td>97</td><td>41</td></tr><tr class=""><td>ReLU</td><td>57</td><td>17</td></tr><tr class=""><td>Identity</td><td>56</td><td>32</td></tr><tr class=""><td>FrozenBatchNorm2d</td><td>55</td><td>19</td></tr><tr class=""><td>ResNetConvLayer</td><td>51</td><td>15</td></tr><tr class=""><td>BatchNorm2d</td><td>42</td><td>42</td></tr><tr class=""><td>ConvNormLayer</td><td>36</td><td>36</td></tr><tr class=""><td>Sequential</td><td>35</td><td>23</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>24</td></tr><tr class=""><td>LayerNorm</td><td>21</td><td>9</td></tr><tr class=""><td>ModuleList</td><td>20</td><td>16</td></tr><tr class=""><td>ResNetBottleNeckLayer</td><td>16</td><td>4</td></tr><tr class=""><td>RepVggBlock</td><td>12</td><td>12</td></tr><tr class=""><td>MLPPredictionHead</td><td>8</td><td>4</td></tr><tr class=""><td>MLP</td><td>7</td><td>3</td></tr><tr class=""><td>SelfAttention</td><td>7</td><td>3</td></tr><tr class=""><td>MultiScaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>MultiscaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>CSPRepLayer</td><td>4</td><td>4</td></tr><tr class=""><td>ResNetShortCut</td><td>4</td><td>4</td></tr><tr class=""><td>ResNetStage</td><td>4</td><td>4</td></tr><tr class=""><td>AvgPool2d</td><td>3</td><td>3</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>MaxPool2d</td><td>1</td><td>1</td></tr><tr class=""><td>AIFILayer</td><td>1</td><td>1</td></tr><tr class=""><td>ConvEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>EncoderLayer</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>HybridEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 71 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[3, 4, 6, 3]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.hidden_sizes</td><td>[256, 512, 1024, 2048]</td><td>[16, 32, 64, 128]</td></tr><tr><td>d_model</td><td>256</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>1024</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>1024</td><td>128</td></tr><tr><td>encoder_in_channels</td><td>[512, 1024, 2048]</td><td>[32, 64, 128]</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>PekingU/rtdetr_r50vd</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/rtdetr_r50vd-mic/tree/5655fbe502f7ba0ce74c6037780272095348ca82" title="huggingface-mic/rtdetr_r50vd-mic@5655fbe502f7ba0ce74c6037780272095348ca82">huggingface-mic/rtdetr_r50vd-mic@5655fbe502f7</a> β <a href="https://huggingface.co/PekingU/rtdetr_r50vd/tree/df939e661d8c52e80608d1ec566561aabd25a4e7" title="PekingU/rtdetr_r50vd@df939e661d8c52e80608d1ec566561aabd25a4e7">PekingU/rtdetr_r50vd@df939e661d8c</a> Β· RTDetrForObjectDetection Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=rtdetr_r50vd-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">42.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">10.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">10</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.backbone.model.encoder.stages</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: no AvgPool2d"></i><i class="c1" title="layer 1: AvgPool2d"></i><i class="c1" title="layer 2: AvgPool2d"></i><i class="c1" title="layer 3: AvgPool2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: no AvgPool2d"></i><i class="c1" title="layer 1: AvgPool2d"></i><i class="c1" title="layer 2: AvgPool2d"></i><i class="c1" title="layer 3: AvgPool2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>no AvgPool2d</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>AvgPool2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder_input_proj</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.aifi</h3><div class="row"><span class="lab">Original Β· 1</span><div class="cells"><i class="c0" title="layer 0: AIFILayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: AIFILayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>AIFILayer</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.lateral_convs</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConvNormLayer</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.fpn_blocks</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>RepVggBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.downsample_convs</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: ConvNormLayer"></i><i class="c0" title="layer 1: ConvNormLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConvNormLayer</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.encoder.pan_blocks</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: RepVggBlock"></i><i class="c0" title="layer 1: RepVggBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>RepVggBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder_input_proj</h3><div class="row"><span class="lab">Original Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><div class="row"><span class="lab">Fixture Β· 3</span><div class="cells"><i class="c0" title="layer 0: Conv2d"></i><i class="c0" title="layer 1: Conv2d"></i><i class="c0" title="layer 2: Conv2d"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Conv2d</td><td>3</td><td>3</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.decoder.bbox_embed</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i><i class="c0" title="layer 4: Linear"></i><i class="c0" title="layer 5: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.546625" title="model.backbone.model.encoder.stages: 23.4M (54.7%)"></i><i class="c1" style="flex:0.139311" title="model.decoder.layers: 6.0M (13.9%)"></i><i class="c2" style="flex:0.104092" title="model.encoder.pan_blocks: 4.5M (10.4%)"></i><i class="c3" style="flex:0.104092" title="model.encoder.fpn_blocks: 4.5M (10.4%)"></i><i class="c4" style="flex:0.027527" title="model.encoder.downsample_convs: 1.2M (2.8%)"></i><i class="c5" style="flex:0.021427" title="model.encoder_input_proj: 919.0K (2.1%)"></i><i class="c6" style="flex:0.018551" title="model.decoder.bbox_embed: 795.7K (1.9%)"></i><i class="c7" style="flex:0.038375" title="9 others: 1.6M (3.8%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.002971" title="model.backbone.model.encoder.stages: 32.4K (0.3%)"></i><i class="c1" style="flex:0.009406" title="model.decoder.layers: 102.6K (0.9%)"></i><i class="c2" style="flex:0.409459" title="model.encoder.pan_blocks: 4.5M (40.9%)"></i><i class="c3" style="flex:0.409459" title="model.encoder.fpn_blocks: 4.5M (40.9%)"></i><i class="c4" style="flex:0.108281" title="model.encoder.downsample_convs: 1.2M (10.8%)"></i><i class="c5" style="flex:0.005400" title="model.encoder_input_proj: 58.9K (0.5%)"></i><i class="c6" style="flex:0.001574" title="model.decoder.bbox_embed: 17.2K (0.2%)"></i><i class="c7" style="flex:0.053450" title="9 others: 582.8K (5.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.backbone.model.encoder.stages</td><td>23.4M</td><td>54.7%</td><td>32.4K</td><td>0.3%</td></tr><tr><td><i class="c1"></i>model.decoder.layers</td><td>6.0M</td><td>13.9%</td><td>102.6K</td><td>0.9%</td></tr><tr><td><i class="c2"></i>model.encoder.pan_blocks</td><td>4.5M</td><td>10.4%</td><td>4.5M</td><td>40.9%</td></tr><tr><td><i class="c3"></i>model.encoder.fpn_blocks</td><td>4.5M</td><td>10.4%</td><td>4.5M</td><td>40.9%</td></tr><tr><td><i class="c4"></i>model.encoder.downsample_convs</td><td>1.2M</td><td>2.8%</td><td>1.2M</td><td>10.8%</td></tr><tr><td><i class="c5"></i>model.encoder_input_proj</td><td>919.0K</td><td>2.1%</td><td>58.9K</td><td>0.5%</td></tr><tr><td><i class="c6"></i>model.decoder.bbox_embed</td><td>795.7K</td><td>1.9%</td><td>17.2K</td><td>0.2%</td></tr><tr><td><i class="c7"></i>9 others</td><td>1.6M</td><td>3.8%</td><td>582.8K</td><td>5.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>rt_detr</td><td>rt_detr</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>model_type</td><td>rt_detr_resnet</td><td>rt_detr_resnet</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>hidden_act</td><td>relu</td><td>relu</td><td>β kept</td></tr><tr class=""><td>backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix RTDetr omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Conv2d</td><td>97</td><td>61</td></tr><tr class=""><td>Linear</td><td>97</td><td>41</td></tr><tr class=""><td>ReLU</td><td>57</td><td>17</td></tr><tr class=""><td>Identity</td><td>56</td><td>32</td></tr><tr class=""><td>FrozenBatchNorm2d</td><td>55</td><td>19</td></tr><tr class=""><td>ResNetConvLayer</td><td>51</td><td>15</td></tr><tr class=""><td>BatchNorm2d</td><td>42</td><td>42</td></tr><tr class=""><td>ConvNormLayer</td><td>36</td><td>36</td></tr><tr class=""><td>Sequential</td><td>35</td><td>23</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>24</td></tr><tr class=""><td>LayerNorm</td><td>21</td><td>9</td></tr><tr class=""><td>ModuleList</td><td>20</td><td>16</td></tr><tr class=""><td>ResNetBottleNeckLayer</td><td>16</td><td>4</td></tr><tr class=""><td>RepVggBlock</td><td>12</td><td>12</td></tr><tr class=""><td>MLPPredictionHead</td><td>8</td><td>4</td></tr><tr class=""><td>MLP</td><td>7</td><td>3</td></tr><tr class=""><td>SelfAttention</td><td>7</td><td>3</td></tr><tr class=""><td>MultiScaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>MultiscaleDeformableAttention</td><td>6</td><td>2</td></tr><tr class=""><td>CSPRepLayer</td><td>4</td><td>4</td></tr><tr class=""><td>ResNetShortCut</td><td>4</td><td>4</td></tr><tr class=""><td>ResNetStage</td><td>4</td><td>4</td></tr><tr class=""><td>AvgPool2d</td><td>3</td><td>3</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>GELUActivation</td><td>1</td><td>1</td></tr><tr class=""><td>MaxPool2d</td><td>1</td><td>1</td></tr><tr class=""><td>AIFILayer</td><td>1</td><td>1</td></tr><tr class=""><td>ConvEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>EncoderLayer</td><td>1</td><td>1</td></tr><tr class=""><td>ForObjectDetection</td><td>1</td><td>1</td></tr><tr class=""><td>HybridEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetBackbone</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>ResNetEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 71 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>backbone_config.depths</td><td>[3, 4, 6, 3]</td><td>[1, 1, 1, 1]</td></tr><tr><td>backbone_config.hidden_sizes</td><td>[256, 512, 1024, 2048]</td><td>[16, 32, 64, 128]</td></tr><tr><td>d_model</td><td>256</td><td>64</td></tr><tr><td>decoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>1024</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>8</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>1024</td><td>128</td></tr><tr><td>encoder_in_channels</td><td>[512, 1024, 2048]</td><td>[32, 64, 128]</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/sam2.1-hiera-tiny-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/sam2.1-hiera-tiny</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/sam2.1-hiera-tiny-mic/tree/e740a31c3d860e3d217f291ba1dc2a154a98090c" title="huggingface-mic/sam2.1-hiera-tiny-mic@e740a31c3d860e3d217f291ba1dc2a154a98090c">huggingface-mic/sam2.1-hiera-tiny-mic@e740a31c3d86</a> β <a href="https://huggingface.co/facebook/sam2.1-hiera-tiny/tree/de431c4043854a71d8101e17995dfe596bf101a5" title="facebook/sam2.1-hiera-tiny@de431c4043854a71d8101e17995dfe596bf101a5">facebook/sam2.1-hiera-tiny@de431c404385</a> Β· Sam2Model Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">31.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">1.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">3</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">4</span><span class="k">to review</span></div></div><section class="findings warn"><h2>β To review</h2><ul><li>model: RoPE type axial became None</li><li>model: model_type sam2_video became sam2</li><li>prompt_encoder_config: model_type sam2_video_prompt_encoder became sam2_prompt_encoder</li><li>mask_decoder_config: model_type sam2_video_mask_decoder became sam2_mask_decoder</li></ul></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>vision_encoder.backbone.blocks</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: MultiScaleBlock"></i><i class="c0" title="layer 1: MultiScaleBlock"></i><i class="c0" title="layer 2: MultiScaleBlock"></i><i class="c0" title="layer 3: MultiScaleBlock"></i><i class="c0" title="layer 4: MultiScaleBlock"></i><i class="c0" title="layer 5: MultiScaleBlock"></i><i class="c0" title="layer 6: MultiScaleBlock"></i><i class="c0" title="layer 7: MultiScaleBlock"></i><i class="c0" title="layer 8: MultiScaleBlock"></i><i class="c0" title="layer 9: MultiScaleBlock"></i><i class="c0" title="layer 10: MultiScaleBlock"></i><i class="c0" title="layer 11: MultiScaleBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 5</span><div class="cells"><i class="c0" title="layer 0: MultiScaleBlock"></i><i class="c0" title="layer 1: MultiScaleBlock"></i><i class="c0" title="layer 2: MultiScaleBlock"></i><i class="c0" title="layer 3: MultiScaleBlock"></i><i class="c0" title="layer 4: MultiScaleBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MultiScaleBlock</td><td>12</td><td>5</td><td>β kept</td></tr></table></div><div class="stack"><h3>mask_decoder.transformer.layers</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: TwoWayAttentionBlock"></i><i class="c0" title="layer 1: TwoWayAttentionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: TwoWayAttentionBlock"></i><i class="c0" title="layer 1: TwoWayAttentionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>TwoWayAttentionBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>mask_decoder.output_hypernetworks_mlps</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: ReLU"></i><i class="c0" title="layer 1: ReLU"></i><i class="c0" title="layer 2: ReLU"></i><i class="c0" title="layer 3: ReLU"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: ReLU"></i><i class="c0" title="layer 1: ReLU"></i><i class="c0" title="layer 2: ReLU"></i><i class="c0" title="layer 3: ReLU"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ReLU</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.853174" title="vision_encoder.backbone.blocks: 26.8M (85.3%)"></i><i class="c1" style="flex:0.100476" title="mask_decoder.transformer.layers: 3.2M (10.0%)"></i><i class="c2" style="flex:0.017787" title="mask_decoder.output_hypernetworks_mlps: 559.2K (1.8%)"></i><i class="c3" style="flex:0.011757" title="vision_encoder.neck: 369.7K (1.2%)"></i><i class="c4" style="flex:0.004218" title="mask_decoder.iou_prediction_head: 132.6K (0.4%)"></i><i class="c5" style="flex:0.004206" title="mask_decoder.transformer: 132.2K (0.4%)"></i><i class="c6" style="flex:0.004193" title="mask_decoder.pred_obj_score_head: 131.8K (0.4%)"></i><i class="c7" style="flex:0.004189" title="14 others: 131.7K (0.4%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.611799" title="vision_encoder.backbone.blocks: 1.2M (61.2%)"></i><i class="c1" style="flex:0.209550" title="mask_decoder.transformer.layers: 398.3K (21.0%)"></i><i class="c2" style="flex:0.073834" title="mask_decoder.output_hypernetworks_mlps: 140.4K (7.4%)"></i><i class="c3" style="flex:0.032591" title="vision_encoder.neck: 62.0K (3.3%)"></i><i class="c4" style="flex:0.017644" title="mask_decoder.iou_prediction_head: 33.5K (1.8%)"></i><i class="c5" style="flex:0.017541" title="mask_decoder.transformer: 33.3K (1.8%)"></i><i class="c6" style="flex:0.017441" title="mask_decoder.pred_obj_score_head: 33.2K (1.7%)"></i><i class="c7" style="flex:0.019601" title="14 others: 37.3K (2.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>vision_encoder.backbone.blocks</td><td>26.8M</td><td>85.3%</td><td>1.2M</td><td>61.2%</td></tr><tr><td><i class="c1"></i>mask_decoder.transformer.layers</td><td>3.2M</td><td>10.0%</td><td>398.3K</td><td>21.0%</td></tr><tr><td><i class="c2"></i>mask_decoder.output_hypernetworks_mlps</td><td>559.2K</td><td>1.8%</td><td>140.4K</td><td>7.4%</td></tr><tr><td><i class="c3"></i>vision_encoder.neck</td><td>369.7K</td><td>1.2%</td><td>62.0K</td><td>3.3%</td></tr><tr><td><i class="c4"></i>mask_decoder.iou_prediction_head</td><td>132.6K</td><td>0.4%</td><td>33.5K</td><td>1.8%</td></tr><tr><td><i class="c5"></i>mask_decoder.transformer</td><td>132.2K</td><td>0.4%</td><td>33.3K</td><td>1.8%</td></tr><tr><td><i class="c6"></i>mask_decoder.pred_obj_score_head</td><td>131.8K</td><td>0.4%</td><td>33.2K</td><td>1.7%</td></tr><tr><td><i class="c7"></i>14 others</td><td>131.7K</td><td>0.4%</td><td>37.3K</td><td>2.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class="bad"><td>model</td><td>RoPE type</td><td>axial</td><td>None</td><td>β changed</td></tr><tr class="bad"><td>model</td><td>model_type</td><td>sam2_video</td><td>sam2</td><td>β changed</td></tr><tr class=""><td>model</td><td>image_size</td><td>1024</td><td>1024</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>sam2_vision_model</td><td>sam2_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>model_type</td><td>sam2_hiera_det_model</td><td>sam2_hiera_det_model</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>image_size</td><td>[1024, 1024]</td><td>[1024, 1024]</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class="bad"><td>prompt_encoder_config</td><td>model_type</td><td>sam2_video_prompt_encoder</td><td>sam2_prompt_encoder</td><td>β changed</td></tr><tr class=""><td>prompt_encoder_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>prompt_encoder_config</td><td>image_size</td><td>1024</td><td>1024</td><td>β kept</td></tr><tr class=""><td>prompt_encoder_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class="bad"><td>mask_decoder_config</td><td>model_type</td><td>sam2_video_mask_decoder</td><td>sam2_mask_decoder</td><td>β changed</td></tr><tr class=""><td>mask_decoder_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Sam2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>101</td><td>73</td></tr><tr class=""><td>LayerNorm</td><td>33</td><td>19</td></tr><tr class=""><td>ModuleList</td><td>24</td><td>17</td></tr><tr class=""><td>FeedForward</td><td>20</td><td>13</td></tr><tr class=""><td>GELUActivation</td><td>13</td><td>6</td></tr><tr class=""><td>MultiScaleAttention</td><td>12</td><td>5</td></tr><tr class=""><td>MultiScaleBlock</td><td>12</td><td>5</td></tr><tr class=""><td>Conv2d</td><td>10</td><td>10</td></tr><tr class=""><td>ReLU</td><td>8</td><td>8</td></tr><tr class=""><td>Attention</td><td>7</td><td>7</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>LayerNorm</td><td>3</td><td>3</td></tr><tr class=""><td>ConvTranspose2d</td><td>2</td><td>2</td></tr><tr class=""><td>PositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>TwoWayAttentionBlock</td><td>2</td><td>2</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>HieraDetModel</td><td>1</td><td>1</td></tr><tr class=""><td>MaskDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>MaskEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>PromptEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>TwoWayTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionNeck</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">18 fields changed, 134 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>mask_decoder_config.hidden_size</td><td>256</td><td>128</td></tr><tr><td>mask_decoder_config.iou_head_hidden_dim</td><td>256</td><td>128</td></tr><tr><td>mask_decoder_config.mlp_dim</td><td>2048</td><td>256</td></tr><tr><td>mask_decoder_config.model_type</td><td>sam2_video_mask_decoder</td><td>sam2_mask_decoder</td></tr><tr><td>mask_decoder_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>max_cond_frame_num</td><td>-1</td><td>β</td></tr><tr><td>memory_attention_rope_theta</td><td>β</td><td>10000</td></tr><tr><td>model_type</td><td>sam2_video</td><td>sam2</td></tr><tr><td>prompt_encoder_config.hidden_size</td><td>256</td><td>128</td></tr><tr><td>prompt_encoder_config.model_type</td><td>sam2_video_prompt_encoder</td><td>sam2_prompt_encoder</td></tr><tr><td>rope_parameters</td><td>{"rope_theta": 10000, "rope_type": "axial"}</td><td>β</td></tr><tr><td>vision_config.backbone_channel_list</td><td>[768, 384, 192, 96]</td><td>[256, 128, 64, 32]</td></tr><tr><td>vision_config.backbone_config.blocks_per_stage</td><td>[1, 2, 7, 2]</td><td>[1, 1, 2, 1]</td></tr><tr><td>vision_config.backbone_config.embed_dim_per_stage</td><td>[96, 192, 384, 768]</td><td>[32, 64, 128, 256]</td></tr><tr><td>vision_config.backbone_config.global_attention_blocks</td><td>[5, 7, 9]</td><td>[3]</td></tr><tr><td>vision_config.backbone_config.hidden_size</td><td>96</td><td>32</td></tr><tr><td>vision_config.backbone_config.num_attention_heads</td><td>1</td><td>4</td></tr><tr><td>vision_config.fpn_hidden_size</td><td>256</td><td>128</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/sam2.1-hiera-tiny</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/sam2.1-hiera-tiny-mic/tree/e740a31c3d860e3d217f291ba1dc2a154a98090c" title="huggingface-mic/sam2.1-hiera-tiny-mic@e740a31c3d860e3d217f291ba1dc2a154a98090c">huggingface-mic/sam2.1-hiera-tiny-mic@e740a31c3d86</a> β <a href="https://huggingface.co/facebook/sam2.1-hiera-tiny/tree/de431c4043854a71d8101e17995dfe596bf101a5" title="facebook/sam2.1-hiera-tiny@de431c4043854a71d8101e17995dfe596bf101a5">facebook/sam2.1-hiera-tiny@de431c404385</a> Β· Sam2Model Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=sam2.1-hiera-tiny-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">31.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">1.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">3</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">4</span><span class="k">to review</span></div></div><section class="findings warn"><h2>β To review</h2><ul><li>model: RoPE type axial became None</li><li>model: model_type sam2_video became sam2</li><li>prompt_encoder_config: model_type sam2_video_prompt_encoder became sam2_prompt_encoder</li><li>mask_decoder_config: model_type sam2_video_mask_decoder became sam2_mask_decoder</li></ul></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>vision_encoder.backbone.blocks</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: MultiScaleBlock"></i><i class="c0" title="layer 1: MultiScaleBlock"></i><i class="c0" title="layer 2: MultiScaleBlock"></i><i class="c0" title="layer 3: MultiScaleBlock"></i><i class="c0" title="layer 4: MultiScaleBlock"></i><i class="c0" title="layer 5: MultiScaleBlock"></i><i class="c0" title="layer 6: MultiScaleBlock"></i><i class="c0" title="layer 7: MultiScaleBlock"></i><i class="c0" title="layer 8: MultiScaleBlock"></i><i class="c0" title="layer 9: MultiScaleBlock"></i><i class="c0" title="layer 10: MultiScaleBlock"></i><i class="c0" title="layer 11: MultiScaleBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 5</span><div class="cells"><i class="c0" title="layer 0: MultiScaleBlock"></i><i class="c0" title="layer 1: MultiScaleBlock"></i><i class="c0" title="layer 2: MultiScaleBlock"></i><i class="c0" title="layer 3: MultiScaleBlock"></i><i class="c0" title="layer 4: MultiScaleBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>MultiScaleBlock</td><td>12</td><td>5</td><td>β kept</td></tr></table></div><div class="stack"><h3>mask_decoder.transformer.layers</h3><div class="row"><span class="lab">Original Β· 2</span><div class="cells"><i class="c0" title="layer 0: TwoWayAttentionBlock"></i><i class="c0" title="layer 1: TwoWayAttentionBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: TwoWayAttentionBlock"></i><i class="c0" title="layer 1: TwoWayAttentionBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>TwoWayAttentionBlock</td><td>2</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>mask_decoder.output_hypernetworks_mlps</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: ReLU"></i><i class="c0" title="layer 1: ReLU"></i><i class="c0" title="layer 2: ReLU"></i><i class="c0" title="layer 3: ReLU"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: ReLU"></i><i class="c0" title="layer 1: ReLU"></i><i class="c0" title="layer 2: ReLU"></i><i class="c0" title="layer 3: ReLU"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ReLU</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.853174" title="vision_encoder.backbone.blocks: 26.8M (85.3%)"></i><i class="c1" style="flex:0.100476" title="mask_decoder.transformer.layers: 3.2M (10.0%)"></i><i class="c2" style="flex:0.017787" title="mask_decoder.output_hypernetworks_mlps: 559.2K (1.8%)"></i><i class="c3" style="flex:0.011757" title="vision_encoder.neck: 369.7K (1.2%)"></i><i class="c4" style="flex:0.004218" title="mask_decoder.iou_prediction_head: 132.6K (0.4%)"></i><i class="c5" style="flex:0.004206" title="mask_decoder.transformer: 132.2K (0.4%)"></i><i class="c6" style="flex:0.004193" title="mask_decoder.pred_obj_score_head: 131.8K (0.4%)"></i><i class="c7" style="flex:0.004189" title="14 others: 131.7K (0.4%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.611799" title="vision_encoder.backbone.blocks: 1.2M (61.2%)"></i><i class="c1" style="flex:0.209550" title="mask_decoder.transformer.layers: 398.3K (21.0%)"></i><i class="c2" style="flex:0.073834" title="mask_decoder.output_hypernetworks_mlps: 140.4K (7.4%)"></i><i class="c3" style="flex:0.032591" title="vision_encoder.neck: 62.0K (3.3%)"></i><i class="c4" style="flex:0.017644" title="mask_decoder.iou_prediction_head: 33.5K (1.8%)"></i><i class="c5" style="flex:0.017541" title="mask_decoder.transformer: 33.3K (1.8%)"></i><i class="c6" style="flex:0.017441" title="mask_decoder.pred_obj_score_head: 33.2K (1.7%)"></i><i class="c7" style="flex:0.019601" title="14 others: 37.3K (2.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>vision_encoder.backbone.blocks</td><td>26.8M</td><td>85.3%</td><td>1.2M</td><td>61.2%</td></tr><tr><td><i class="c1"></i>mask_decoder.transformer.layers</td><td>3.2M</td><td>10.0%</td><td>398.3K</td><td>21.0%</td></tr><tr><td><i class="c2"></i>mask_decoder.output_hypernetworks_mlps</td><td>559.2K</td><td>1.8%</td><td>140.4K</td><td>7.4%</td></tr><tr><td><i class="c3"></i>vision_encoder.neck</td><td>369.7K</td><td>1.2%</td><td>62.0K</td><td>3.3%</td></tr><tr><td><i class="c4"></i>mask_decoder.iou_prediction_head</td><td>132.6K</td><td>0.4%</td><td>33.5K</td><td>1.8%</td></tr><tr><td><i class="c5"></i>mask_decoder.transformer</td><td>132.2K</td><td>0.4%</td><td>33.3K</td><td>1.8%</td></tr><tr><td><i class="c6"></i>mask_decoder.pred_obj_score_head</td><td>131.8K</td><td>0.4%</td><td>33.2K</td><td>1.7%</td></tr><tr><td><i class="c7"></i>14 others</td><td>131.7K</td><td>0.4%</td><td>37.3K</td><td>2.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class="bad"><td>model</td><td>RoPE type</td><td>axial</td><td>None</td><td>β changed</td></tr><tr class="bad"><td>model</td><td>model_type</td><td>sam2_video</td><td>sam2</td><td>β changed</td></tr><tr class=""><td>model</td><td>image_size</td><td>1024</td><td>1024</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>sam2_vision_model</td><td>sam2_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>model_type</td><td>sam2_hiera_det_model</td><td>sam2_hiera_det_model</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>image_size</td><td>[1024, 1024]</td><td>[1024, 1024]</td><td>β kept</td></tr><tr class=""><td>vision_config.backbone_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class="bad"><td>prompt_encoder_config</td><td>model_type</td><td>sam2_video_prompt_encoder</td><td>sam2_prompt_encoder</td><td>β changed</td></tr><tr class=""><td>prompt_encoder_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>prompt_encoder_config</td><td>image_size</td><td>1024</td><td>1024</td><td>β kept</td></tr><tr class=""><td>prompt_encoder_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class="bad"><td>mask_decoder_config</td><td>model_type</td><td>sam2_video_mask_decoder</td><td>sam2_mask_decoder</td><td>β changed</td></tr><tr class=""><td>mask_decoder_config</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Sam2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>101</td><td>73</td></tr><tr class=""><td>LayerNorm</td><td>33</td><td>19</td></tr><tr class=""><td>ModuleList</td><td>24</td><td>17</td></tr><tr class=""><td>FeedForward</td><td>20</td><td>13</td></tr><tr class=""><td>GELUActivation</td><td>13</td><td>6</td></tr><tr class=""><td>MultiScaleAttention</td><td>12</td><td>5</td></tr><tr class=""><td>MultiScaleBlock</td><td>12</td><td>5</td></tr><tr class=""><td>Conv2d</td><td>10</td><td>10</td></tr><tr class=""><td>ReLU</td><td>8</td><td>8</td></tr><tr class=""><td>Attention</td><td>7</td><td>7</td></tr><tr class=""><td>Embedding</td><td>6</td><td>6</td></tr><tr class=""><td>LayerNorm</td><td>3</td><td>3</td></tr><tr class=""><td>ConvTranspose2d</td><td>2</td><td>2</td></tr><tr class=""><td>PositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>TwoWayAttentionBlock</td><td>2</td><td>2</td></tr><tr class=""><td>GELU</td><td>1</td><td>1</td></tr><tr class=""><td>HieraDetModel</td><td>1</td><td>1</td></tr><tr class=""><td>MaskDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>MaskEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>PromptEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>SinePositionEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>TwoWayTransformer</td><td>1</td><td>1</td></tr><tr class=""><td>VisionModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionNeck</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">18 fields changed, 134 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>mask_decoder_config.hidden_size</td><td>256</td><td>128</td></tr><tr><td>mask_decoder_config.iou_head_hidden_dim</td><td>256</td><td>128</td></tr><tr><td>mask_decoder_config.mlp_dim</td><td>2048</td><td>256</td></tr><tr><td>mask_decoder_config.model_type</td><td>sam2_video_mask_decoder</td><td>sam2_mask_decoder</td></tr><tr><td>mask_decoder_config.num_attention_heads</td><td>8</td><td>4</td></tr><tr><td>max_cond_frame_num</td><td>-1</td><td>β</td></tr><tr><td>memory_attention_rope_theta</td><td>β</td><td>10000</td></tr><tr><td>model_type</td><td>sam2_video</td><td>sam2</td></tr><tr><td>prompt_encoder_config.hidden_size</td><td>256</td><td>128</td></tr><tr><td>prompt_encoder_config.model_type</td><td>sam2_video_prompt_encoder</td><td>sam2_prompt_encoder</td></tr><tr><td>rope_parameters</td><td>{"rope_theta": 10000, "rope_type": "axial"}</td><td>β</td></tr><tr><td>vision_config.backbone_channel_list</td><td>[768, 384, 192, 96]</td><td>[256, 128, 64, 32]</td></tr><tr><td>vision_config.backbone_config.blocks_per_stage</td><td>[1, 2, 7, 2]</td><td>[1, 1, 2, 1]</td></tr><tr><td>vision_config.backbone_config.embed_dim_per_stage</td><td>[96, 192, 384, 768]</td><td>[32, 64, 128, 256]</td></tr><tr><td>vision_config.backbone_config.global_attention_blocks</td><td>[5, 7, 9]</td><td>[3]</td></tr><tr><td>vision_config.backbone_config.hidden_size</td><td>96</td><td>32</td></tr><tr><td>vision_config.backbone_config.num_attention_heads</td><td>1</td><td>4</td></tr><tr><td>vision_config.fpn_hidden_size</td><td>256</td><td>128</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/seamless-m4t-v2-large-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/seamless-m4t-v2-large</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/seamless-m4t-v2-large-mic/tree/43e20937aec2b5592d27e162442c2db3532e010b" title="huggingface-mic/seamless-m4t-v2-large-mic@43e20937aec2b5592d27e162442c2db3532e010b">huggingface-mic/seamless-m4t-v2-large-mic@43e20937aec2</a> β <a href="https://huggingface.co/facebook/seamless-m4t-v2-large/tree/5f8cc790b19fc3f67a61c105133b20b34e3dcb76" title="facebook/seamless-m4t-v2-large@5f8cc790b19fc3f67a61c105133b20b34e3dcb76">facebook/seamless-m4t-v2-large@5f8cc790b19f</a> Β· SeamlessM4Tv2Model Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.3B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">95.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">7</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>speech_encoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: ConformerEncoderLayer"></i><i class="c0" title="layer 1: ConformerEncoderLayer"></i><i class="c0" title="layer 2: ConformerEncoderLayer"></i><i class="c0" title="layer 3: ConformerEncoderLayer"></i><i class="c0" title="layer 4: ConformerEncoderLayer"></i><i class="c0" title="layer 5: ConformerEncoderLayer"></i><i class="c0" title="layer 6: ConformerEncoderLayer"></i><i class="c0" title="layer 7: ConformerEncoderLayer"></i><i class="c0" title="layer 8: ConformerEncoderLayer"></i><i class="c0" title="layer 9: ConformerEncoderLayer"></i><i class="c0" title="layer 10: ConformerEncoderLayer"></i><i class="c0" title="layer 11: ConformerEncoderLayer"></i><i class="c0" title="layer 12: ConformerEncoderLayer"></i><i class="c0" title="layer 13: ConformerEncoderLayer"></i><i class="c0" title="layer 14: ConformerEncoderLayer"></i><i class="c0" title="layer 15: ConformerEncoderLayer"></i><i class="c0" title="layer 16: ConformerEncoderLayer"></i><i class="c0" title="layer 17: ConformerEncoderLayer"></i><i class="c0" title="layer 18: ConformerEncoderLayer"></i><i class="c0" title="layer 19: ConformerEncoderLayer"></i><i class="c0" title="layer 20: ConformerEncoderLayer"></i><i class="c0" title="layer 21: ConformerEncoderLayer"></i><i class="c0" title="layer 22: ConformerEncoderLayer"></i><i class="c0" title="layer 23: ConformerEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 24</span><div class="cells"><i class="c0" title="layer 0: ConformerEncoderLayer"></i><i class="c0" title="layer 1: ConformerEncoderLayer"></i><i class="c0" title="layer 2: ConformerEncoderLayer"></i><i class="c0" title="layer 3: ConformerEncoderLayer"></i><i class="c0" title="layer 4: ConformerEncoderLayer"></i><i class="c0" title="layer 5: ConformerEncoderLayer"></i><i class="c0" title="layer 6: ConformerEncoderLayer"></i><i class="c0" title="layer 7: ConformerEncoderLayer"></i><i class="c0" title="layer 8: ConformerEncoderLayer"></i><i class="c0" title="layer 9: ConformerEncoderLayer"></i><i class="c0" title="layer 10: ConformerEncoderLayer"></i><i class="c0" title="layer 11: ConformerEncoderLayer"></i><i class="c0" title="layer 12: ConformerEncoderLayer"></i><i class="c0" title="layer 13: ConformerEncoderLayer"></i><i class="c0" title="layer 14: ConformerEncoderLayer"></i><i class="c0" title="layer 15: ConformerEncoderLayer"></i><i class="c0" title="layer 16: ConformerEncoderLayer"></i><i class="c0" title="layer 17: ConformerEncoderLayer"></i><i class="c0" title="layer 18: ConformerEncoderLayer"></i><i class="c0" title="layer 19: ConformerEncoderLayer"></i><i class="c0" title="layer 20: ConformerEncoderLayer"></i><i class="c0" title="layer 21: ConformerEncoderLayer"></i><i class="c0" title="layer 22: ConformerEncoderLayer"></i><i class="c0" title="layer 23: ConformerEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConformerEncoderLayer</td><td>24</td><td>24</td><td>β kept</td></tr></table></div><div class="stack"><h3>speech_encoder.adapter.layers</h3><div class="row"><span class="lab">Original Β· 1</span><div class="cells"><i class="c0" title="layer 0: ConformerAdapterLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: ConformerAdapterLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConformerAdapterLayer</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>text_decoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>t2u_model.model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>t2u_model.model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: TextToUnitDecoderLayer"></i><i class="c0" title="layer 1: TextToUnitDecoderLayer"></i><i class="c0" title="layer 2: TextToUnitDecoderLayer"></i><i class="c0" title="layer 3: TextToUnitDecoderLayer"></i><i class="c0" title="layer 4: TextToUnitDecoderLayer"></i><i class="c0" title="layer 5: TextToUnitDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: TextToUnitDecoderLayer"></i><i class="c0" title="layer 1: TextToUnitDecoderLayer"></i><i class="c0" title="layer 2: TextToUnitDecoderLayer"></i><i class="c0" title="layer 3: TextToUnitDecoderLayer"></i><i class="c0" title="layer 4: TextToUnitDecoderLayer"></i><i class="c0" title="layer 5: TextToUnitDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>TextToUnitDecoderLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>vocoder.hifi_gan.resblocks</h3><div class="row"><span class="lab">Original Β· 15</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i><i class="c0" title="layer 12: HifiGanResidualBlock"></i><i class="c0" title="layer 13: HifiGanResidualBlock"></i><i class="c0" title="layer 14: HifiGanResidualBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 15</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i><i class="c0" title="layer 12: HifiGanResidualBlock"></i><i class="c0" title="layer 13: HifiGanResidualBlock"></i><i class="c0" title="layer 14: HifiGanResidualBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>HifiGanResidualBlock</td><td>15</td><td>15</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.261793" title="text_decoder.layers: 604.5M (26.2%)"></i><i class="c1" style="flex:0.251305" title="speech_encoder.encoder.layers: 580.3M (25.1%)"></i><i class="c2" style="flex:0.218138" title="text_encoder.layers: 503.7M (21.8%)"></i><i class="c3" style="flex:0.113564" title="shared: 262.2M (11.4%)"></i><i class="c4" style="flex:0.054534" title="t2u_model.model.encoder.layers: 125.9M (5.5%)"></i><i class="c5" style="flex:0.049067" title="t2u_model.model.decoder.layers: 113.3M (4.9%)"></i><i class="c6" style="flex:0.019988" title="speech_encoder.adapter.layers: 46.2M (2.0%)"></i><i class="c7" style="flex:0.031611" title="13 others: 73.0M (3.2%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.001057" title="text_decoder.layers: 100.5K (0.1%)"></i><i class="c1" style="flex:0.274973" title="speech_encoder.encoder.layers: 26.1M (27.5%)"></i><i class="c2" style="flex:0.000704" title="text_encoder.layers: 66.9K (0.1%)"></i><i class="c3" style="flex:0.172463" title="shared: 16.4M (17.2%)"></i><i class="c4" style="flex:0.067787" title="t2u_model.model.encoder.layers: 6.4M (6.8%)"></i><i class="c5" style="flex:0.004695" title="t2u_model.model.decoder.layers: 446.2K (0.5%)"></i><i class="c6" style="flex:0.007121" title="speech_encoder.adapter.layers: 676.8K (0.7%)"></i><i class="c7" style="flex:0.471198" title="13 others: 44.8M (47.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>text_decoder.layers</td><td>604.5M</td><td>26.2%</td><td>100.5K</td><td>0.1%</td></tr><tr><td><i class="c1"></i>speech_encoder.encoder.layers</td><td>580.3M</td><td>25.1%</td><td>26.1M</td><td>27.5%</td></tr><tr><td><i class="c2"></i>text_encoder.layers</td><td>503.7M</td><td>21.8%</td><td>66.9K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>shared</td><td>262.2M</td><td>11.4%</td><td>16.4M</td><td>17.2%</td></tr><tr><td><i class="c4"></i>t2u_model.model.encoder.layers</td><td>125.9M</td><td>5.5%</td><td>6.4M</td><td>6.8%</td></tr><tr><td><i class="c5"></i>t2u_model.model.decoder.layers</td><td>113.3M</td><td>4.9%</td><td>446.2K</td><td>0.5%</td></tr><tr><td><i class="c6"></i>speech_encoder.adapter.layers</td><td>46.2M</td><td>2.0%</td><td>676.8K</td><td>0.7%</td></tr><tr><td><i class="c7"></i>13 others</td><td>73.0M</td><td>3.2%</td><td>44.8M</td><td>47.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>seamless_m4t_v2</td><td>seamless_m4t_v2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256102</td><td>256102</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4096</td><td>4096</td><td>β kept</td></tr><tr class=""><td>model</td><td>sampling_rate</td><td>16000</td><td>16000</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SeamlessM4Tv2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>649</td><td>297</td></tr><tr class=""><td>Dropout</td><td>346</td><td>214</td></tr><tr class=""><td>LayerNorm</td><td>302</td><td>192</td></tr><tr class=""><td>Conv1d</td><td>182</td><td>182</td></tr><tr class=""><td>ReLU</td><td>88</td><td>22</td></tr><tr class=""><td>Attention</td><td>84</td><td>18</td></tr><tr class=""><td>SiLU</td><td>72</td><td>72</td></tr><tr class=""><td>FeedForwardNetwork</td><td>54</td><td>10</td></tr><tr class=""><td>ConformerFeedForward</td><td>50</td><td>50</td></tr><tr class=""><td>ModuleList</td><td>38</td><td>38</td></tr><tr class=""><td>Embedding</td><td>30</td><td>30</td></tr><tr class=""><td>EncoderLayer</td><td>30</td><td>8</td></tr><tr class=""><td>GLU</td><td>25</td><td>25</td></tr><tr class=""><td>ConformerSelfAttention</td><td>25</td><td>25</td></tr><tr class=""><td>ConformerConvolutionModule</td><td>24</td><td>24</td></tr><tr class=""><td>ConformerEncoderLayer</td><td>24</td><td>24</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>HifiGanResidualBlock</td><td>15</td><td>15</td></tr><tr class=""><td>TextToUnitDecoderLayer</td><td>6</td><td>6</td></tr><tr class=""><td>ConvTranspose1d</td><td>5</td><td>5</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>4</td><td>4</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>VariancePredictor</td><td>2</td><td>2</td></tr><tr class=""><td>CodeHifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerAdapter</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerAdapterLayer</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerFeatureProjection</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>HifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SpeechEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 77 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>8192</td><td>128</td></tr><tr><td>decoder_layers</td><td>24</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>8192</td><td>128</td></tr><tr><td>encoder_layers</td><td>24</td><td>2</td></tr><tr><td>hidden_size</td><td>1024</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>facebook/seamless-m4t-v2-large</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/seamless-m4t-v2-large-mic/tree/43e20937aec2b5592d27e162442c2db3532e010b" title="huggingface-mic/seamless-m4t-v2-large-mic@43e20937aec2b5592d27e162442c2db3532e010b">huggingface-mic/seamless-m4t-v2-large-mic@43e20937aec2</a> β <a href="https://huggingface.co/facebook/seamless-m4t-v2-large/tree/5f8cc790b19fc3f67a61c105133b20b34e3dcb76" title="facebook/seamless-m4t-v2-large@5f8cc790b19fc3f67a61c105133b20b34e3dcb76">facebook/seamless-m4t-v2-large@5f8cc790b19f</a> Β· SeamlessM4Tv2Model Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=seamless-m4t-v2-large-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.3B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">95.0M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">7</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>speech_encoder.encoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: ConformerEncoderLayer"></i><i class="c0" title="layer 1: ConformerEncoderLayer"></i><i class="c0" title="layer 2: ConformerEncoderLayer"></i><i class="c0" title="layer 3: ConformerEncoderLayer"></i><i class="c0" title="layer 4: ConformerEncoderLayer"></i><i class="c0" title="layer 5: ConformerEncoderLayer"></i><i class="c0" title="layer 6: ConformerEncoderLayer"></i><i class="c0" title="layer 7: ConformerEncoderLayer"></i><i class="c0" title="layer 8: ConformerEncoderLayer"></i><i class="c0" title="layer 9: ConformerEncoderLayer"></i><i class="c0" title="layer 10: ConformerEncoderLayer"></i><i class="c0" title="layer 11: ConformerEncoderLayer"></i><i class="c0" title="layer 12: ConformerEncoderLayer"></i><i class="c0" title="layer 13: ConformerEncoderLayer"></i><i class="c0" title="layer 14: ConformerEncoderLayer"></i><i class="c0" title="layer 15: ConformerEncoderLayer"></i><i class="c0" title="layer 16: ConformerEncoderLayer"></i><i class="c0" title="layer 17: ConformerEncoderLayer"></i><i class="c0" title="layer 18: ConformerEncoderLayer"></i><i class="c0" title="layer 19: ConformerEncoderLayer"></i><i class="c0" title="layer 20: ConformerEncoderLayer"></i><i class="c0" title="layer 21: ConformerEncoderLayer"></i><i class="c0" title="layer 22: ConformerEncoderLayer"></i><i class="c0" title="layer 23: ConformerEncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 24</span><div class="cells"><i class="c0" title="layer 0: ConformerEncoderLayer"></i><i class="c0" title="layer 1: ConformerEncoderLayer"></i><i class="c0" title="layer 2: ConformerEncoderLayer"></i><i class="c0" title="layer 3: ConformerEncoderLayer"></i><i class="c0" title="layer 4: ConformerEncoderLayer"></i><i class="c0" title="layer 5: ConformerEncoderLayer"></i><i class="c0" title="layer 6: ConformerEncoderLayer"></i><i class="c0" title="layer 7: ConformerEncoderLayer"></i><i class="c0" title="layer 8: ConformerEncoderLayer"></i><i class="c0" title="layer 9: ConformerEncoderLayer"></i><i class="c0" title="layer 10: ConformerEncoderLayer"></i><i class="c0" title="layer 11: ConformerEncoderLayer"></i><i class="c0" title="layer 12: ConformerEncoderLayer"></i><i class="c0" title="layer 13: ConformerEncoderLayer"></i><i class="c0" title="layer 14: ConformerEncoderLayer"></i><i class="c0" title="layer 15: ConformerEncoderLayer"></i><i class="c0" title="layer 16: ConformerEncoderLayer"></i><i class="c0" title="layer 17: ConformerEncoderLayer"></i><i class="c0" title="layer 18: ConformerEncoderLayer"></i><i class="c0" title="layer 19: ConformerEncoderLayer"></i><i class="c0" title="layer 20: ConformerEncoderLayer"></i><i class="c0" title="layer 21: ConformerEncoderLayer"></i><i class="c0" title="layer 22: ConformerEncoderLayer"></i><i class="c0" title="layer 23: ConformerEncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConformerEncoderLayer</td><td>24</td><td>24</td><td>β kept</td></tr></table></div><div class="stack"><h3>speech_encoder.adapter.layers</h3><div class="row"><span class="lab">Original Β· 1</span><div class="cells"><i class="c0" title="layer 0: ConformerAdapterLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 1</span><div class="cells"><i class="c0" title="layer 0: ConformerAdapterLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>ConformerAdapterLayer</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>text_decoder.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>t2u_model.model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>t2u_model.model.decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: TextToUnitDecoderLayer"></i><i class="c0" title="layer 1: TextToUnitDecoderLayer"></i><i class="c0" title="layer 2: TextToUnitDecoderLayer"></i><i class="c0" title="layer 3: TextToUnitDecoderLayer"></i><i class="c0" title="layer 4: TextToUnitDecoderLayer"></i><i class="c0" title="layer 5: TextToUnitDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 6</span><div class="cells"><i class="c0" title="layer 0: TextToUnitDecoderLayer"></i><i class="c0" title="layer 1: TextToUnitDecoderLayer"></i><i class="c0" title="layer 2: TextToUnitDecoderLayer"></i><i class="c0" title="layer 3: TextToUnitDecoderLayer"></i><i class="c0" title="layer 4: TextToUnitDecoderLayer"></i><i class="c0" title="layer 5: TextToUnitDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>TextToUnitDecoderLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>vocoder.hifi_gan.resblocks</h3><div class="row"><span class="lab">Original Β· 15</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i><i class="c0" title="layer 12: HifiGanResidualBlock"></i><i class="c0" title="layer 13: HifiGanResidualBlock"></i><i class="c0" title="layer 14: HifiGanResidualBlock"></i></div></div><div class="row"><span class="lab">Fixture Β· 15</span><div class="cells"><i class="c0" title="layer 0: HifiGanResidualBlock"></i><i class="c0" title="layer 1: HifiGanResidualBlock"></i><i class="c0" title="layer 2: HifiGanResidualBlock"></i><i class="c0" title="layer 3: HifiGanResidualBlock"></i><i class="c0" title="layer 4: HifiGanResidualBlock"></i><i class="c0" title="layer 5: HifiGanResidualBlock"></i><i class="c0" title="layer 6: HifiGanResidualBlock"></i><i class="c0" title="layer 7: HifiGanResidualBlock"></i><i class="c0" title="layer 8: HifiGanResidualBlock"></i><i class="c0" title="layer 9: HifiGanResidualBlock"></i><i class="c0" title="layer 10: HifiGanResidualBlock"></i><i class="c0" title="layer 11: HifiGanResidualBlock"></i><i class="c0" title="layer 12: HifiGanResidualBlock"></i><i class="c0" title="layer 13: HifiGanResidualBlock"></i><i class="c0" title="layer 14: HifiGanResidualBlock"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>HifiGanResidualBlock</td><td>15</td><td>15</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.261793" title="text_decoder.layers: 604.5M (26.2%)"></i><i class="c1" style="flex:0.251305" title="speech_encoder.encoder.layers: 580.3M (25.1%)"></i><i class="c2" style="flex:0.218138" title="text_encoder.layers: 503.7M (21.8%)"></i><i class="c3" style="flex:0.113564" title="shared: 262.2M (11.4%)"></i><i class="c4" style="flex:0.054534" title="t2u_model.model.encoder.layers: 125.9M (5.5%)"></i><i class="c5" style="flex:0.049067" title="t2u_model.model.decoder.layers: 113.3M (4.9%)"></i><i class="c6" style="flex:0.019988" title="speech_encoder.adapter.layers: 46.2M (2.0%)"></i><i class="c7" style="flex:0.031611" title="13 others: 73.0M (3.2%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.001057" title="text_decoder.layers: 100.5K (0.1%)"></i><i class="c1" style="flex:0.274973" title="speech_encoder.encoder.layers: 26.1M (27.5%)"></i><i class="c2" style="flex:0.000704" title="text_encoder.layers: 66.9K (0.1%)"></i><i class="c3" style="flex:0.172463" title="shared: 16.4M (17.2%)"></i><i class="c4" style="flex:0.067787" title="t2u_model.model.encoder.layers: 6.4M (6.8%)"></i><i class="c5" style="flex:0.004695" title="t2u_model.model.decoder.layers: 446.2K (0.5%)"></i><i class="c6" style="flex:0.007121" title="speech_encoder.adapter.layers: 676.8K (0.7%)"></i><i class="c7" style="flex:0.471198" title="13 others: 44.8M (47.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>text_decoder.layers</td><td>604.5M</td><td>26.2%</td><td>100.5K</td><td>0.1%</td></tr><tr><td><i class="c1"></i>speech_encoder.encoder.layers</td><td>580.3M</td><td>25.1%</td><td>26.1M</td><td>27.5%</td></tr><tr><td><i class="c2"></i>text_encoder.layers</td><td>503.7M</td><td>21.8%</td><td>66.9K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>shared</td><td>262.2M</td><td>11.4%</td><td>16.4M</td><td>17.2%</td></tr><tr><td><i class="c4"></i>t2u_model.model.encoder.layers</td><td>125.9M</td><td>5.5%</td><td>6.4M</td><td>6.8%</td></tr><tr><td><i class="c5"></i>t2u_model.model.decoder.layers</td><td>113.3M</td><td>4.9%</td><td>446.2K</td><td>0.5%</td></tr><tr><td><i class="c6"></i>speech_encoder.adapter.layers</td><td>46.2M</td><td>2.0%</td><td>676.8K</td><td>0.7%</td></tr><tr><td><i class="c7"></i>13 others</td><td>73.0M</td><td>3.2%</td><td>44.8M</td><td>47.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>seamless_m4t_v2</td><td>seamless_m4t_v2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>256102</td><td>256102</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4096</td><td>4096</td><td>β kept</td></tr><tr class=""><td>model</td><td>sampling_rate</td><td>16000</td><td>16000</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SeamlessM4Tv2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>649</td><td>297</td></tr><tr class=""><td>Dropout</td><td>346</td><td>214</td></tr><tr class=""><td>LayerNorm</td><td>302</td><td>192</td></tr><tr class=""><td>Conv1d</td><td>182</td><td>182</td></tr><tr class=""><td>ReLU</td><td>88</td><td>22</td></tr><tr class=""><td>Attention</td><td>84</td><td>18</td></tr><tr class=""><td>SiLU</td><td>72</td><td>72</td></tr><tr class=""><td>FeedForwardNetwork</td><td>54</td><td>10</td></tr><tr class=""><td>ConformerFeedForward</td><td>50</td><td>50</td></tr><tr class=""><td>ModuleList</td><td>38</td><td>38</td></tr><tr class=""><td>Embedding</td><td>30</td><td>30</td></tr><tr class=""><td>EncoderLayer</td><td>30</td><td>8</td></tr><tr class=""><td>GLU</td><td>25</td><td>25</td></tr><tr class=""><td>ConformerSelfAttention</td><td>25</td><td>25</td></tr><tr class=""><td>ConformerConvolutionModule</td><td>24</td><td>24</td></tr><tr class=""><td>ConformerEncoderLayer</td><td>24</td><td>24</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>HifiGanResidualBlock</td><td>15</td><td>15</td></tr><tr class=""><td>TextToUnitDecoderLayer</td><td>6</td><td>6</td></tr><tr class=""><td>ConvTranspose1d</td><td>5</td><td>5</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>4</td><td>4</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>ScaledWordEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>VariancePredictor</td><td>2</td><td>2</td></tr><tr class=""><td>CodeHifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerAdapter</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerAdapterLayer</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>ConformerFeatureProjection</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>HifiGan</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SpeechEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitDecoder</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>TextToUnitModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 77 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>decoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>8192</td><td>128</td></tr><tr><td>decoder_layers</td><td>24</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>16</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>8192</td><td>128</td></tr><tr><td>encoder_layers</td><td>24</td><td>2</td></tr><tr><td>hidden_size</td><td>1024</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/segformer-b0-finetuned-ade-512-512-mic.html
CHANGED
|
@@ -31,6 +31,7 @@ tr.bad td{color:var(--warn)}
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
-
<h1>nvidia/segformer-b0-finetuned-ade-512-512</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/segformer-b0-finetuned-ade-512-512-mic/tree/
|
| 35 |
-
<
|
|
|
|
| 36 |
</main></body></html>
|
|
|
|
| 31 |
.c0{background:var(--s0)}.c1{background:var(--s1)}.c2{background:var(--s2)}.c3{background:var(--s3)}.c4{background:var(--s4)}.c5{background:var(--s5)}.c6{background:var(--s6)}.c7{background:var(--s7)}.c8{background:var(--s8)}
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
+
<h1>nvidia/segformer-b0-finetuned-ade-512-512</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/segformer-b0-finetuned-ade-512-512-mic/tree/6677cf3e61099c794f921bcd38ac95eb8a7e9ecf" title="huggingface-mic/segformer-b0-finetuned-ade-512-512-mic@6677cf3e61099c794f921bcd38ac95eb8a7e9ecf">huggingface-mic/segformer-b0-finetuned-ade-512-512-mic@6677cf3e6109</a> β <a href="https://huggingface.co/nvidia/segformer-b0-finetuned-ade-512-512/tree/489d5cd81a0b59fab9b7ea758d3548ebe99677da" title="nvidia/segformer-b0-finetuned-ade-512-512@489d5cd81a0b59fab9b7ea758d3548ebe99677da">nvidia/segformer-b0-finetuned-ade-512-512@489d5cd81a0b</a> Β· SegformerForSemanticSegmentation Β· real weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=segformer-b0-finetuned-ade-512-512-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
+
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.8M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>segformer.stages</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: SequenceReduction"></i><i class="c0" title="layer 1: SequenceReduction"></i><i class="c0" title="layer 2: SequenceReduction"></i><i class="c1" title="layer 3: no SequenceReduction"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: SequenceReduction"></i><i class="c0" title="layer 1: SequenceReduction"></i><i class="c0" title="layer 2: SequenceReduction"></i><i class="c1" title="layer 3: no SequenceReduction"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>SequenceReduction</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no SequenceReduction</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><div class="stack"><h3>decode_head.linear_projections</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: Linear"></i><i class="c0" title="layer 1: Linear"></i><i class="c0" title="layer 2: Linear"></i><i class="c0" title="layer 3: Linear"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>Linear</td><td>4</td><td>4</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.884536" title="segformer.stages: 3.3M (88.5%)"></i><i class="c1" style="flex:0.069855" title="decode_head.linear_fuse: 262.1K (7.0%)"></i><i class="c2" style="flex:0.035200" title="decode_head.linear_projections: 132.1K (3.5%)"></i><i class="c3" style="flex:0.010273" title="decode_head.classifier: 38.5K (1.0%)"></i><i class="c4" style="flex:0.000136" title="decode_head.batch_norm: 512 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.884536" title="segformer.stages: 3.3M (88.5%)"></i><i class="c1" style="flex:0.069855" title="decode_head.linear_fuse: 262.1K (7.0%)"></i><i class="c2" style="flex:0.035200" title="decode_head.linear_projections: 132.1K (3.5%)"></i><i class="c3" style="flex:0.010273" title="decode_head.classifier: 38.5K (1.0%)"></i><i class="c4" style="flex:0.000136" title="decode_head.batch_norm: 512 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>segformer.stages</td><td>3.3M</td><td>88.5%</td><td>3.3M</td><td>88.5%</td></tr><tr><td><i class="c1"></i>decode_head.linear_fuse</td><td>262.1K</td><td>7.0%</td><td>262.1K</td><td>7.0%</td></tr><tr><td><i class="c2"></i>decode_head.linear_projections</td><td>132.1K</td><td>3.5%</td><td>132.1K</td><td>3.5%</td></tr><tr><td><i class="c3"></i>decode_head.classifier</td><td>38.5K</td><td>1.0%</td><td>38.5K</td><td>1.0%</td></tr><tr><td><i class="c4"></i>decode_head.batch_norm</td><td>512</td><td>0.0%</td><td>512</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>segformer</td><td>segformer</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Segformer omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>52</td><td>52</td></tr><tr class=""><td>LayerNorm</td><td>30</td><td>30</td></tr><tr class=""><td>Conv2d</td><td>20</td><td>20</td></tr><tr class=""><td>Dropout</td><td>17</td><td>17</td></tr><tr class=""><td>GELUActivation</td><td>8</td><td>8</td></tr><tr class=""><td>Attention</td><td>8</td><td>8</td></tr><tr class=""><td>DepthWiseConv</td><td>8</td><td>8</td></tr><tr class=""><td>Layer</td><td>8</td><td>8</td></tr><tr class=""><td>MixMLP</td><td>8</td><td>8</td></tr><tr class=""><td>DropPath</td><td>7</td><td>7</td></tr><tr class=""><td>ModuleList</td><td>6</td><td>6</td></tr><tr class=""><td>SequenceReduction</td><td>6</td><td>6</td></tr><tr class=""><td>MLP</td><td>4</td><td>4</td></tr><tr class=""><td>OverlapPatchEmbeddings</td><td>4</td><td>4</td></tr><tr class=""><td>Stage</td><td>4</td><td>4</td></tr><tr class=""><td>BatchNorm2d</td><td>1</td><td>1</td></tr><tr class=""><td>Identity</td><td>1</td><td>1</td></tr><tr class=""><td>ReLU</td><td>1</td><td>1</td></tr><tr class=""><td>DecodeHead</td><td>1</td><td>1</td></tr><tr class=""><td>ForSemanticSegmentation</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">0 fields changed, 30 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr></table>
|
| 37 |
</main></body></html>
|
shrink/segformer-b0-finetuned-ade-512-512-mic.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
{"fixture": "huggingface-mic/segformer-b0-finetuned-ade-512-512-mic@
|
|
|
|
| 1 |
+
{"fixture": "huggingface-mic/segformer-b0-finetuned-ade-512-512-mic@6677cf3e61099c794f921bcd38ac95eb8a7e9ecf", "findings": []}
|
shrink/siglip-base-patch16-224-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/siglip-base-patch16-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/siglip-base-patch16-224-mic/tree/13fcabe2240b7fb71b1d3f2889e4f71778df91eb" title="huggingface-mic/siglip-base-patch16-224-mic@13fcabe2240b7fb71b1d3f2889e4f71778df91eb">huggingface-mic/siglip-base-patch16-224-mic@13fcabe2240b</a> β <a href="https://huggingface.co/google/siglip-base-patch16-224/tree/7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed" title="google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed">google/siglip-base-patch16-224@7fd15f0689c7</a> Β· SiglipModel Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">203.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.418666" title="vision_model.encoder.layers: 85.1M (41.9%)"></i><i class="c1" style="flex:0.418666" title="text_model.encoder.layers: 85.1M (41.9%)"></i><i class="c2" style="flex:0.121213" title="text_model.embeddings: 24.6M (12.1%)"></i><i class="c3" style="flex:0.034885" title="vision_model.head: 7.1M (3.5%)"></i><i class="c4" style="flex:0.003648" title="vision_model.embeddings: 741.1K (0.4%)"></i><i class="c5" style="flex:0.002907" title="text_model.head: 590.6K (0.3%)"></i><i class="c6" style="flex:0.000008" title="vision_model.post_layernorm: 1.5K (0.0%)"></i><i class="c7" style="flex:0.000008" title="3 others: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.029290" title="vision_model.encoder.layers: 66.9K (2.9%)"></i><i class="c1" style="flex:0.029290" title="text_model.encoder.layers: 66.9K (2.9%)"></i><i class="c2" style="flex:0.897849" title="text_model.embeddings: 2.1M (89.8%)"></i><i class="c3" style="flex:0.014617" title="vision_model.head: 33.4K (1.5%)"></i><i class="c4" style="flex:0.027022" title="vision_model.embeddings: 61.8K (2.7%)"></i><i class="c5" style="flex:0.001820" title="text_model.head: 4.2K (0.2%)"></i><i class="c6" style="flex:0.000056" title="vision_model.post_layernorm: 128 (0.0%)"></i><i class="c7" style="flex:0.000057" title="3 others: 130 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>vision_model.encoder.layers</td><td>85.1M</td><td>41.9%</td><td>66.9K</td><td>2.9%</td></tr><tr><td><i class="c1"></i>text_model.encoder.layers</td><td>85.1M</td><td>41.9%</td><td>66.9K</td><td>2.9%</td></tr><tr><td><i class="c2"></i>text_model.embeddings</td><td>24.6M</td><td>12.1%</td><td>2.1M</td><td>89.8%</td></tr><tr><td><i class="c3"></i>vision_model.head</td><td>7.1M</td><td>3.5%</td><td>33.4K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>vision_model.embeddings</td><td>741.1K</td><td>0.4%</td><td>61.8K</td><td>2.7%</td></tr><tr><td><i class="c5"></i>text_model.head</td><td>590.6K</td><td>0.3%</td><td>4.2K</td><td>0.2%</td></tr><tr><td><i class="c6"></i>vision_model.post_layernorm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c7"></i>3 others</td><td>1.5K</td><td>0.0%</td><td>130</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>siglip</td><td>siglip</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>siglip_text_model</td><td>siglip_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>32000</td><td>32000</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>64</td><td>64</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>siglip_vision_model</td><td>siglip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Siglip omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>147</td><td>27</td></tr><tr class=""><td>LayerNorm</td><td>51</td><td>11</td></tr><tr class=""><td>GELUTanh</td><td>25</td><td>5</td></tr><tr class=""><td>MLP</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>MultiheadAttention</td><td>1</td><td>1</td></tr><tr class=""><td>NonDynamicallyQuantizableLinear</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiheadAttentionPoolingHead</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 42 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>text_config.projection_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>google/siglip-base-patch16-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/siglip-base-patch16-224-mic/tree/13fcabe2240b7fb71b1d3f2889e4f71778df91eb" title="huggingface-mic/siglip-base-patch16-224-mic@13fcabe2240b7fb71b1d3f2889e4f71778df91eb">huggingface-mic/siglip-base-patch16-224-mic@13fcabe2240b</a> β <a href="https://huggingface.co/google/siglip-base-patch16-224/tree/7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed" title="google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed">google/siglip-base-patch16-224@7fd15f0689c7</a> Β· SiglipModel Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=siglip-base-patch16-224-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">203.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">2.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>text_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.418666" title="vision_model.encoder.layers: 85.1M (41.9%)"></i><i class="c1" style="flex:0.418666" title="text_model.encoder.layers: 85.1M (41.9%)"></i><i class="c2" style="flex:0.121213" title="text_model.embeddings: 24.6M (12.1%)"></i><i class="c3" style="flex:0.034885" title="vision_model.head: 7.1M (3.5%)"></i><i class="c4" style="flex:0.003648" title="vision_model.embeddings: 741.1K (0.4%)"></i><i class="c5" style="flex:0.002907" title="text_model.head: 590.6K (0.3%)"></i><i class="c6" style="flex:0.000008" title="vision_model.post_layernorm: 1.5K (0.0%)"></i><i class="c7" style="flex:0.000008" title="3 others: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.029290" title="vision_model.encoder.layers: 66.9K (2.9%)"></i><i class="c1" style="flex:0.029290" title="text_model.encoder.layers: 66.9K (2.9%)"></i><i class="c2" style="flex:0.897849" title="text_model.embeddings: 2.1M (89.8%)"></i><i class="c3" style="flex:0.014617" title="vision_model.head: 33.4K (1.5%)"></i><i class="c4" style="flex:0.027022" title="vision_model.embeddings: 61.8K (2.7%)"></i><i class="c5" style="flex:0.001820" title="text_model.head: 4.2K (0.2%)"></i><i class="c6" style="flex:0.000056" title="vision_model.post_layernorm: 128 (0.0%)"></i><i class="c7" style="flex:0.000057" title="3 others: 130 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>vision_model.encoder.layers</td><td>85.1M</td><td>41.9%</td><td>66.9K</td><td>2.9%</td></tr><tr><td><i class="c1"></i>text_model.encoder.layers</td><td>85.1M</td><td>41.9%</td><td>66.9K</td><td>2.9%</td></tr><tr><td><i class="c2"></i>text_model.embeddings</td><td>24.6M</td><td>12.1%</td><td>2.1M</td><td>89.8%</td></tr><tr><td><i class="c3"></i>vision_model.head</td><td>7.1M</td><td>3.5%</td><td>33.4K</td><td>1.5%</td></tr><tr><td><i class="c4"></i>vision_model.embeddings</td><td>741.1K</td><td>0.4%</td><td>61.8K</td><td>2.7%</td></tr><tr><td><i class="c5"></i>text_model.head</td><td>590.6K</td><td>0.3%</td><td>4.2K</td><td>0.2%</td></tr><tr><td><i class="c6"></i>vision_model.post_layernorm</td><td>1.5K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr><tr><td><i class="c7"></i>3 others</td><td>1.5K</td><td>0.0%</td><td>130</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>siglip</td><td>siglip</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>siglip_text_model</td><td>siglip_text_model</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>32000</td><td>32000</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>64</td><td>64</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>siglip_vision_model</td><td>siglip_vision_model</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>16</td><td>16</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Siglip omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>147</td><td>27</td></tr><tr class=""><td>LayerNorm</td><td>51</td><td>11</td></tr><tr class=""><td>GELUTanh</td><td>25</td><td>5</td></tr><tr class=""><td>MLP</td><td>25</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>24</td><td>4</td></tr><tr class=""><td>Embedding</td><td>3</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Encoder</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>MultiheadAttention</td><td>1</td><td>1</td></tr><tr class=""><td>NonDynamicallyQuantizableLinear</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>MultiheadAttentionPoolingHead</td><td>1</td><td>1</td></tr><tr class=""><td>TextEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>TextModel</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionModel</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">9 fields changed, 42 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>12</td><td>2</td></tr><tr><td>text_config.projection_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.hidden_size</td><td>768</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>3072</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>12</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>12</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/smollm-1.7b-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolLM-1.7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smollm-1.7b-mic/tree/fc0e707f0977de7e4095792e1b56203e3af7b5c6" title="huggingface-mic/smollm-1.7b-mic@fc0e707f0977de7e4095792e1b56203e3af7b5c6">huggingface-mic/smollm-1.7b-mic@fc0e707f0977</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolLM-1.7B/tree/d7449ff7241c863f3e8accc475155f0f97afa011" title="HuggingFaceTB/SmolLM-1.7B@d7449ff7241c863f3e8accc475155f0f97afa011">HuggingFaceTB/SmolLM-1.7B@d7449ff7241c</a> Β· LlamaForCausalLM Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.941179" title="model.layers: 1.6B (94.1%)"></i><i class="c1" style="flex:0.058820" title="model.embed_tokens: 100.7M (5.9%)"></i><i class="c2" style="flex:0.000001" title="model.norm: 2.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.025458" title="model.layers: 82.2K (2.5%)"></i><i class="c1" style="flex:0.974523" title="model.embed_tokens: 3.1M (97.5%)"></i><i class="c2" style="flex:0.000020" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.6B</td><td>94.1%</td><td>82.2K</td><td>2.5%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>100.7M</td><td>5.9%</td><td>3.1M</td><td>97.5%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>2.0K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Llama omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 23 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>intermediate_size</td><td>8192</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr><tr><td>pad_token_id</td><td>null</td><td>0</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolLM-1.7B</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smollm-1.7b-mic/tree/fc0e707f0977de7e4095792e1b56203e3af7b5c6" title="huggingface-mic/smollm-1.7b-mic@fc0e707f0977de7e4095792e1b56203e3af7b5c6">huggingface-mic/smollm-1.7b-mic@fc0e707f0977</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolLM-1.7B/tree/d7449ff7241c863f3e8accc475155f0f97afa011" title="HuggingFaceTB/SmolLM-1.7B@d7449ff7241c863f3e8accc475155f0f97afa011">HuggingFaceTB/SmolLM-1.7B@d7449ff7241c</a> Β· LlamaForCausalLM Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=smollm-1.7b-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.7B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.941179" title="model.layers: 1.6B (94.1%)"></i><i class="c1" style="flex:0.058820" title="model.embed_tokens: 100.7M (5.9%)"></i><i class="c2" style="flex:0.000001" title="model.norm: 2.0K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.025458" title="model.layers: 82.2K (2.5%)"></i><i class="c1" style="flex:0.974523" title="model.embed_tokens: 3.1M (97.5%)"></i><i class="c2" style="flex:0.000020" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.6B</td><td>94.1%</td><td>82.2K</td><td>2.5%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>100.7M</td><td>5.9%</td><td>3.1M</td><td>97.5%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>2.0K</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Llama omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 23 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>intermediate_size</td><td>8192</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr><tr><td>pad_token_id</td><td>null</td><td>0</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/smollm-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolLM-135M</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smollm-mic/tree/d0da7b30e6c71b294e950e4abdbc7b858854bf28" title="huggingface-mic/smollm-mic@d0da7b30e6c71b294e950e4abdbc7b858854bf28">huggingface-mic/smollm-mic@d0da7b30e6c7</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolLM-135M/tree/1d461723eec654e65efdc40cf49301c89c0c92f4" title="HuggingFaceTB/SmolLM-135M@1d461723eec654e65efdc40cf49301c89c0c92f4">HuggingFaceTB/SmolLM-135M@1d461723eec6</a> Β· LlamaForCausalLM Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">134.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 30</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>30</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.789524" title="model.layers: 106.2M (79.0%)"></i><i class="c1" style="flex:0.210471" title="model.embed_tokens: 28.3M (21.0%)"></i><i class="c2" style="flex:0.000004" title="model.norm: 576 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.022978" title="model.layers: 74.0K (2.3%)"></i><i class="c1" style="flex:0.977002" title="model.embed_tokens: 3.1M (97.7%)"></i><i class="c2" style="flex:0.000020" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>106.2M</td><td>79.0%</td><td>74.0K</td><td>2.3%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>28.3M</td><td>21.0%</td><td>3.1M</td><td>97.7%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>576</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (9 heads, 3 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Llama omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>211</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>61</td><td>5</td></tr><tr class=""><td>Attention</td><td>30</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>30</td><td>2</td></tr><tr class=""><td>MLP</td><td>30</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>30</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>576</td><td>64</td></tr><tr><td>intermediate_size</td><td>1536</td><td>128</td></tr><tr><td>num_attention_heads</td><td>9</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>30</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>3</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolLM-135M</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smollm-mic/tree/d0da7b30e6c71b294e950e4abdbc7b858854bf28" title="huggingface-mic/smollm-mic@d0da7b30e6c71b294e950e4abdbc7b858854bf28">huggingface-mic/smollm-mic@d0da7b30e6c7</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolLM-135M/tree/1d461723eec654e65efdc40cf49301c89c0c92f4" title="HuggingFaceTB/SmolLM-135M@1d461723eec654e65efdc40cf49301c89c0c92f4">HuggingFaceTB/SmolLM-135M@1d461723eec6</a> Β· LlamaForCausalLM Β· overfit weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=smollm-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">134.5M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 30</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>30</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.789524" title="model.layers: 106.2M (79.0%)"></i><i class="c1" style="flex:0.210471" title="model.embed_tokens: 28.3M (21.0%)"></i><i class="c2" style="flex:0.000004" title="model.norm: 576 (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.022978" title="model.layers: 74.0K (2.3%)"></i><i class="c1" style="flex:0.977002" title="model.embed_tokens: 3.1M (97.7%)"></i><i class="c2" style="flex:0.000020" title="model.norm: 64 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>106.2M</td><td>79.0%</td><td>74.0K</td><td>2.3%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>28.3M</td><td>21.0%</td><td>3.1M</td><td>97.7%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>576</td><td>0.0%</td><td>64</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (9 heads, 3 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>2048</td><td>2048</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Llama omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>211</td><td>15</td></tr><tr class=""><td>RMSNorm</td><td>61</td><td>5</td></tr><tr class=""><td>Attention</td><td>30</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>30</td><td>2</td></tr><tr class=""><td>MLP</td><td>30</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>30</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">6 fields changed, 24 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>head_dim</td><td>64</td><td>16</td></tr><tr><td>hidden_size</td><td>576</td><td>64</td></tr><tr><td>intermediate_size</td><td>1536</td><td>128</td></tr><tr><td>num_attention_heads</td><td>9</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>30</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>3</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/smolvlm-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolVLM2-2.2B-Instruct</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smolvlm-mic/tree/8cefc6b12559831ef4a11f203b84848058513ee4" title="huggingface-mic/smolvlm-mic@8cefc6b12559831ef4a11f203b84848058513ee4">huggingface-mic/smolvlm-mic@8cefc6b12559</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct/tree/482adb537c021c86670beed01cd58990d01e72e4" title="HuggingFaceTB/SmolVLM2-2.2B-Instruct@482adb537c021c86670beed01cd58990d01e72e4">HuggingFaceTB/SmolVLM2-2.2B-Instruct@482adb537c02</a> Β· SmolVLMForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.2B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 27</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i><i class="c0" title="layer 24: EncoderLayer"></i><i class="c0" title="layer 25: EncoderLayer"></i><i class="c0" title="layer 26: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>27</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.text_model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: LlamaDecoderLayer"></i><i class="c0" title="layer 1: LlamaDecoderLayer"></i><i class="c0" title="layer 2: LlamaDecoderLayer"></i><i class="c0" title="layer 3: LlamaDecoderLayer"></i><i class="c0" title="layer 4: LlamaDecoderLayer"></i><i class="c0" title="layer 5: LlamaDecoderLayer"></i><i class="c0" title="layer 6: LlamaDecoderLayer"></i><i class="c0" title="layer 7: LlamaDecoderLayer"></i><i class="c0" title="layer 8: LlamaDecoderLayer"></i><i class="c0" title="layer 9: LlamaDecoderLayer"></i><i class="c0" title="layer 10: LlamaDecoderLayer"></i><i class="c0" title="layer 11: LlamaDecoderLayer"></i><i class="c0" title="layer 12: LlamaDecoderLayer"></i><i class="c0" title="layer 13: LlamaDecoderLayer"></i><i class="c0" title="layer 14: LlamaDecoderLayer"></i><i class="c0" title="layer 15: LlamaDecoderLayer"></i><i class="c0" title="layer 16: LlamaDecoderLayer"></i><i class="c0" title="layer 17: LlamaDecoderLayer"></i><i class="c0" title="layer 18: LlamaDecoderLayer"></i><i class="c0" title="layer 19: LlamaDecoderLayer"></i><i class="c0" title="layer 20: LlamaDecoderLayer"></i><i class="c0" title="layer 21: LlamaDecoderLayer"></i><i class="c0" title="layer 22: LlamaDecoderLayer"></i><i class="c0" title="layer 23: LlamaDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: LlamaDecoderLayer"></i><i class="c0" title="layer 1: LlamaDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>LlamaDecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.716896" title="model.text_model.layers: 1.6B (71.7%)"></i><i class="c1" style="flex:0.183136" title="model.vision_model.encoder.layers: 411.5M (18.3%)"></i><i class="c2" style="flex:0.044921" title="model.text_model: 100.9M (4.5%)"></i><i class="c3" style="flex:0.044920" title="lm_head: 100.9M (4.5%)"></i><i class="c4" style="flex:0.009451" title="model.connector: 21.2M (0.9%)"></i><i class="c5" style="flex:0.000677" title="model.vision_model: 1.5M (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.012492" title="model.text_model.layers: 82.2K (1.2%)"></i><i class="c1" style="flex:0.010176" title="model.vision_model.encoder.layers: 66.9K (1.0%)"></i><i class="c2" style="flex:0.479448" title="model.text_model: 3.2M (47.9%)"></i><i class="c3" style="flex:0.479438" title="lm_head: 3.2M (47.9%)"></i><i class="c4" style="flex:0.005604" title="model.connector: 36.9K (0.6%)"></i><i class="c5" style="flex:0.012842" title="model.vision_model: 84.5K (1.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.text_model.layers</td><td>1.6B</td><td>71.7%</td><td>82.2K</td><td>1.2%</td></tr><tr><td><i class="c1"></i>model.vision_model.encoder.layers</td><td>411.5M</td><td>18.3%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c2"></i>model.text_model</td><td>100.9M</td><td>4.5%</td><td>3.2M</td><td>47.9%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>100.9M</td><td>4.5%</td><td>3.2M</td><td>47.9%</td></tr><tr><td><i class="c4"></i>model.connector</td><td>21.2M</td><td>0.9%</td><td>36.9K</td><td>0.6%</td></tr><tr><td><i class="c5"></i>model.vision_model</td><td>1.5M</td><td>0.1%</td><td>84.5K</td><td>1.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>smolvlm</td><td>smolvlm</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49280</td><td>49280</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>smolvlm_vision</td><td>smolvlm_vision</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>384</td><td>384</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49280</td><td>49280</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>model_type</td><td>vllama3</td><td>vllama3</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SmolVLM omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>332</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>55</td><td>5</td></tr><tr class=""><td>LlamaRMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>27</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>27</td><td>2</td></tr><tr class=""><td>VisionAttention</td><td>27</td><td>2</td></tr><tr class=""><td>VisionMLP</td><td>27</td><td>2</td></tr><tr class=""><td>LlamaAttention</td><td>24</td><td>2</td></tr><tr class=""><td>LlamaDecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>LlamaMLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>LlamaModel</td><td>1</td><td>1</td></tr><tr class=""><td>LlamaRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Connector</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SimpleMLP</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 127 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>8192</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>32</td><td>4</td></tr><tr><td>vision_config.hidden_size</td><td>1152</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4304</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>27</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>HuggingFaceTB/SmolVLM2-2.2B-Instruct</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/smolvlm-mic/tree/8cefc6b12559831ef4a11f203b84848058513ee4" title="huggingface-mic/smolvlm-mic@8cefc6b12559831ef4a11f203b84848058513ee4">huggingface-mic/smolvlm-mic@8cefc6b12559</a> β <a href="https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct/tree/482adb537c021c86670beed01cd58990d01e72e4" title="HuggingFaceTB/SmolVLM2-2.2B-Instruct@482adb537c021c86670beed01cd58990d01e72e4">HuggingFaceTB/SmolVLM2-2.2B-Instruct@482adb537c02</a> Β· SmolVLMForConditionalGeneration Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=smolvlm-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 15 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">2.2B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">6.6M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">2</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.vision_model.encoder.layers</h3><div class="row"><span class="lab">Original Β· 27</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i><i class="c0" title="layer 12: EncoderLayer"></i><i class="c0" title="layer 13: EncoderLayer"></i><i class="c0" title="layer 14: EncoderLayer"></i><i class="c0" title="layer 15: EncoderLayer"></i><i class="c0" title="layer 16: EncoderLayer"></i><i class="c0" title="layer 17: EncoderLayer"></i><i class="c0" title="layer 18: EncoderLayer"></i><i class="c0" title="layer 19: EncoderLayer"></i><i class="c0" title="layer 20: EncoderLayer"></i><i class="c0" title="layer 21: EncoderLayer"></i><i class="c0" title="layer 22: EncoderLayer"></i><i class="c0" title="layer 23: EncoderLayer"></i><i class="c0" title="layer 24: EncoderLayer"></i><i class="c0" title="layer 25: EncoderLayer"></i><i class="c0" title="layer 26: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>27</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>model.text_model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: LlamaDecoderLayer"></i><i class="c0" title="layer 1: LlamaDecoderLayer"></i><i class="c0" title="layer 2: LlamaDecoderLayer"></i><i class="c0" title="layer 3: LlamaDecoderLayer"></i><i class="c0" title="layer 4: LlamaDecoderLayer"></i><i class="c0" title="layer 5: LlamaDecoderLayer"></i><i class="c0" title="layer 6: LlamaDecoderLayer"></i><i class="c0" title="layer 7: LlamaDecoderLayer"></i><i class="c0" title="layer 8: LlamaDecoderLayer"></i><i class="c0" title="layer 9: LlamaDecoderLayer"></i><i class="c0" title="layer 10: LlamaDecoderLayer"></i><i class="c0" title="layer 11: LlamaDecoderLayer"></i><i class="c0" title="layer 12: LlamaDecoderLayer"></i><i class="c0" title="layer 13: LlamaDecoderLayer"></i><i class="c0" title="layer 14: LlamaDecoderLayer"></i><i class="c0" title="layer 15: LlamaDecoderLayer"></i><i class="c0" title="layer 16: LlamaDecoderLayer"></i><i class="c0" title="layer 17: LlamaDecoderLayer"></i><i class="c0" title="layer 18: LlamaDecoderLayer"></i><i class="c0" title="layer 19: LlamaDecoderLayer"></i><i class="c0" title="layer 20: LlamaDecoderLayer"></i><i class="c0" title="layer 21: LlamaDecoderLayer"></i><i class="c0" title="layer 22: LlamaDecoderLayer"></i><i class="c0" title="layer 23: LlamaDecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: LlamaDecoderLayer"></i><i class="c0" title="layer 1: LlamaDecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>LlamaDecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.716896" title="model.text_model.layers: 1.6B (71.7%)"></i><i class="c1" style="flex:0.183136" title="model.vision_model.encoder.layers: 411.5M (18.3%)"></i><i class="c2" style="flex:0.044921" title="model.text_model: 100.9M (4.5%)"></i><i class="c3" style="flex:0.044920" title="lm_head: 100.9M (4.5%)"></i><i class="c4" style="flex:0.009451" title="model.connector: 21.2M (0.9%)"></i><i class="c5" style="flex:0.000677" title="model.vision_model: 1.5M (0.1%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.012492" title="model.text_model.layers: 82.2K (1.2%)"></i><i class="c1" style="flex:0.010176" title="model.vision_model.encoder.layers: 66.9K (1.0%)"></i><i class="c2" style="flex:0.479448" title="model.text_model: 3.2M (47.9%)"></i><i class="c3" style="flex:0.479438" title="lm_head: 3.2M (47.9%)"></i><i class="c4" style="flex:0.005604" title="model.connector: 36.9K (0.6%)"></i><i class="c5" style="flex:0.012842" title="model.vision_model: 84.5K (1.3%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.text_model.layers</td><td>1.6B</td><td>71.7%</td><td>82.2K</td><td>1.2%</td></tr><tr><td><i class="c1"></i>model.vision_model.encoder.layers</td><td>411.5M</td><td>18.3%</td><td>66.9K</td><td>1.0%</td></tr><tr><td><i class="c2"></i>model.text_model</td><td>100.9M</td><td>4.5%</td><td>3.2M</td><td>47.9%</td></tr><tr><td><i class="c3"></i>lm_head</td><td>100.9M</td><td>4.5%</td><td>3.2M</td><td>47.9%</td></tr><tr><td><i class="c4"></i>model.connector</td><td>21.2M</td><td>0.9%</td><td>36.9K</td><td>0.6%</td></tr><tr><td><i class="c5"></i>model.vision_model</td><td>1.5M</td><td>0.1%</td><td>84.5K</td><td>1.3%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>smolvlm</td><td>smolvlm</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49280</td><td>49280</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>model_type</td><td>smolvlm_vision</td><td>smolvlm_vision</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>image_size</td><td>384</td><td>384</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>patch_size</td><td>14</td><td>14</td><td>β kept</td></tr><tr class=""><td>vision_config</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>model_type</td><td>llama</td><td>llama</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>vocab_size</td><td>49280</td><td>49280</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>text_config</td><td>max_position_embeddings</td><td>8192</td><td>8192</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>model_type</td><td>vllama3</td><td>vllama3</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>text_config.perceiver_config</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SmolVLM omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>332</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>55</td><td>5</td></tr><tr class=""><td>LlamaRMSNorm</td><td>49</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>27</td><td>2</td></tr><tr class=""><td>EncoderLayer</td><td>27</td><td>2</td></tr><tr class=""><td>VisionAttention</td><td>27</td><td>2</td></tr><tr class=""><td>VisionMLP</td><td>27</td><td>2</td></tr><tr class=""><td>LlamaAttention</td><td>24</td><td>2</td></tr><tr class=""><td>LlamaDecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>LlamaMLP</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>ModuleList</td><td>2</td><td>2</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>LlamaModel</td><td>1</td><td>1</td></tr><tr class=""><td>LlamaRotaryEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>Connector</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForConditionalGeneration</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>SimpleMLP</td><td>1</td><td>1</td></tr><tr class=""><td>VisionEmbeddings</td><td>1</td><td>1</td></tr><tr class=""><td>VisionTransformer</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">10 fields changed, 127 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>text_config.head_dim</td><td>64</td><td>16</td></tr><tr><td>text_config.hidden_size</td><td>2048</td><td>64</td></tr><tr><td>text_config.intermediate_size</td><td>8192</td><td>128</td></tr><tr><td>text_config.num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>text_config.num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>text_config.num_key_value_heads</td><td>32</td><td>4</td></tr><tr><td>vision_config.hidden_size</td><td>1152</td><td>64</td></tr><tr><td>vision_config.intermediate_size</td><td>4304</td><td>128</td></tr><tr><td>vision_config.num_attention_heads</td><td>16</td><td>4</td></tr><tr><td>vision_config.num_hidden_layers</td><td>27</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/speech_t5-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/speecht5_asr</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/speech_t5-mic/tree/889b4e918f22534926a9817ffa5aa22df47af8df" title="huggingface-mic/speech_t5-mic@889b4e918f22534926a9817ffa5aa22df47af8df">huggingface-mic/speech_t5-mic@889b4e918f22</a> β <a href="https://huggingface.co/microsoft/speecht5_asr/tree/53615c10408485422e09a12cda191a747f4bbe34" title="microsoft/speecht5_asr@53615c10408485422e09a12cda191a747f4bbe34">microsoft/speecht5_asr@53615c104084</a> Β· SpeechT5ForSpeechToText Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">151.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">3</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>speecht5.encoder.prenet.feature_encoder.conv_layers</h3><div class="row"><span class="lab">Original Β· 7</span><div class="cells"><i class="c0" title="layer 0: GroupNorm, GroupNormConvLayer"></i><i class="c1" title="layer 1: NoLayerNormConvLayer"></i><i class="c1" title="layer 2: NoLayerNormConvLayer"></i><i class="c1" title="layer 3: NoLayerNormConvLayer"></i><i class="c1" title="layer 4: NoLayerNormConvLayer"></i><i class="c1" title="layer 5: NoLayerNormConvLayer"></i><i class="c1" title="layer 6: NoLayerNormConvLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 7</span><div class="cells"><i class="c0" title="layer 0: GroupNorm, GroupNormConvLayer"></i><i class="c1" title="layer 1: NoLayerNormConvLayer"></i><i class="c1" title="layer 2: NoLayerNormConvLayer"></i><i class="c1" title="layer 3: NoLayerNormConvLayer"></i><i class="c1" title="layer 4: NoLayerNormConvLayer"></i><i class="c1" title="layer 5: NoLayerNormConvLayer"></i><i class="c1" title="layer 6: NoLayerNormConvLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>GroupNorm, GroupNormConvLayer</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>NoLayerNormConvLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>speecht5.encoder.wrapped_encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>speecht5.decoder.wrapped_decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.562660" title="speecht5.encoder.wrapped_encoder.layers: 85.1M (56.3%)"></i><i class="c1" style="flex:0.375157" title="speecht5.decoder.wrapped_decoder.layers: 56.7M (37.5%)"></i><i class="c2" style="flex:0.033985" title="speecht5.encoder: 5.1M (3.4%)"></i><i class="c3" style="flex:0.027787" title="speecht5.encoder.prenet.feature_encoder.conv_layers: 4.2M (2.8%)"></i><i class="c4" style="flex:0.000412" title="speecht5.decoder: 62.2K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.015060" title="speecht5.encoder.wrapped_encoder.layers: 66.9K (1.5%)"></i><i class="c1" style="flex:0.022604" title="speecht5.decoder.wrapped_decoder.layers: 100.5K (2.3%)"></i><i class="c2" style="flex:0.016226" title="speecht5.encoder: 72.1K (1.6%)"></i><i class="c3" style="flex:0.944944" title="speecht5.encoder.prenet.feature_encoder.conv_layers: 4.2M (94.5%)"></i><i class="c4" style="flex:0.001166" title="speecht5.decoder: 5.2K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>speecht5.encoder.wrapped_encoder.layers</td><td>85.1M</td><td>56.3%</td><td>66.9K</td><td>1.5%</td></tr><tr><td><i class="c1"></i>speecht5.decoder.wrapped_decoder.layers</td><td>56.7M</td><td>37.5%</td><td>100.5K</td><td>2.3%</td></tr><tr><td><i class="c2"></i>speecht5.encoder</td><td>5.1M</td><td>3.4%</td><td>72.1K</td><td>1.6%</td></tr><tr><td><i class="c3"></i>speecht5.encoder.prenet.feature_encoder.conv_layers</td><td>4.2M</td><td>2.8%</td><td>4.2M</td><td>94.5%</td></tr><tr><td><i class="c4"></i>speecht5.decoder</td><td>62.2K</td><td>0.0%</td><td>5.2K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>speecht5</td><td>speecht5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>81</td><td>81</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_mel_bins</td><td>80</td><td>80</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SpeechT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>134</td><td>34</td></tr><tr class=""><td>Dropout</td><td>57</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>44</td><td>12</td></tr><tr class=""><td>GELUActivation</td><td>26</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>6</td></tr><tr class=""><td>FeedForward</td><td>18</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>Conv1d</td><td>7</td><td>7</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>NoLayerNormConvLayer</td><td>6</td><td>6</td></tr><tr class=""><td>ModuleList</td><td>3</td><td>3</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>GroupNorm</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>ParametrizationList</td><td>1</td><td>1</td></tr><tr class=""><td>ParametrizedConv1d</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>DecoderWithTextPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>EncoderWithSpeechPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>FeatureEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>FeatureProjection</td><td>1</td><td>1</td></tr><tr class=""><td>ForSpeechToText</td><td>1</td><td>1</td></tr><tr class=""><td>GroupNormConvLayer</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PositionalConvEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>RelativePositionalEncoding</td><td>1</td><td>1</td></tr><tr class=""><td>SamePadLayer</td><td>1</td><td>1</td></tr><tr class=""><td>SpeechEncoderPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>TextDecoderPostnet</td><td>1</td><td>1</td></tr><tr class=""><td>TextDecoderPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>_WeightNorm</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>decoder_attention_heads</td><td>12</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>12</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/speecht5_asr</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/speech_t5-mic/tree/889b4e918f22534926a9817ffa5aa22df47af8df" title="huggingface-mic/speech_t5-mic@889b4e918f22534926a9817ffa5aa22df47af8df">huggingface-mic/speech_t5-mic@889b4e918f22</a> β <a href="https://huggingface.co/microsoft/speecht5_asr/tree/53615c10408485422e09a12cda191a747f4bbe34" title="microsoft/speecht5_asr@53615c10408485422e09a12cda191a747f4bbe34">microsoft/speecht5_asr@53615c104084</a> Β· SpeechT5ForSpeechToText Β· random weights Β· built before the shrink rule (no recorded recipe)</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=speech_t5-mic&days=30&sticky_only=false" target="_blank" rel="noopener">passing now; 3 failed checks in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">151.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">4.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">3</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>speecht5.encoder.prenet.feature_encoder.conv_layers</h3><div class="row"><span class="lab">Original Β· 7</span><div class="cells"><i class="c0" title="layer 0: GroupNorm, GroupNormConvLayer"></i><i class="c1" title="layer 1: NoLayerNormConvLayer"></i><i class="c1" title="layer 2: NoLayerNormConvLayer"></i><i class="c1" title="layer 3: NoLayerNormConvLayer"></i><i class="c1" title="layer 4: NoLayerNormConvLayer"></i><i class="c1" title="layer 5: NoLayerNormConvLayer"></i><i class="c1" title="layer 6: NoLayerNormConvLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 7</span><div class="cells"><i class="c0" title="layer 0: GroupNorm, GroupNormConvLayer"></i><i class="c1" title="layer 1: NoLayerNormConvLayer"></i><i class="c1" title="layer 2: NoLayerNormConvLayer"></i><i class="c1" title="layer 3: NoLayerNormConvLayer"></i><i class="c1" title="layer 4: NoLayerNormConvLayer"></i><i class="c1" title="layer 5: NoLayerNormConvLayer"></i><i class="c1" title="layer 6: NoLayerNormConvLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>GroupNorm, GroupNormConvLayer</td><td>1</td><td>1</td><td>β kept</td></tr><tr><td><i class="c1"></i>NoLayerNormConvLayer</td><td>6</td><td>6</td><td>β kept</td></tr></table></div><div class="stack"><h3>speecht5.encoder.wrapped_encoder.layers</h3><div class="row"><span class="lab">Original Β· 12</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i><i class="c0" title="layer 2: EncoderLayer"></i><i class="c0" title="layer 3: EncoderLayer"></i><i class="c0" title="layer 4: EncoderLayer"></i><i class="c0" title="layer 5: EncoderLayer"></i><i class="c0" title="layer 6: EncoderLayer"></i><i class="c0" title="layer 7: EncoderLayer"></i><i class="c0" title="layer 8: EncoderLayer"></i><i class="c0" title="layer 9: EncoderLayer"></i><i class="c0" title="layer 10: EncoderLayer"></i><i class="c0" title="layer 11: EncoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: EncoderLayer"></i><i class="c0" title="layer 1: EncoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>EncoderLayer</td><td>12</td><td>2</td><td>β kept</td></tr></table></div><div class="stack"><h3>speecht5.decoder.wrapped_decoder.layers</h3><div class="row"><span class="lab">Original Β· 6</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>6</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.562660" title="speecht5.encoder.wrapped_encoder.layers: 85.1M (56.3%)"></i><i class="c1" style="flex:0.375157" title="speecht5.decoder.wrapped_decoder.layers: 56.7M (37.5%)"></i><i class="c2" style="flex:0.033985" title="speecht5.encoder: 5.1M (3.4%)"></i><i class="c3" style="flex:0.027787" title="speecht5.encoder.prenet.feature_encoder.conv_layers: 4.2M (2.8%)"></i><i class="c4" style="flex:0.000412" title="speecht5.decoder: 62.2K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.015060" title="speecht5.encoder.wrapped_encoder.layers: 66.9K (1.5%)"></i><i class="c1" style="flex:0.022604" title="speecht5.decoder.wrapped_decoder.layers: 100.5K (2.3%)"></i><i class="c2" style="flex:0.016226" title="speecht5.encoder: 72.1K (1.6%)"></i><i class="c3" style="flex:0.944944" title="speecht5.encoder.prenet.feature_encoder.conv_layers: 4.2M (94.5%)"></i><i class="c4" style="flex:0.001166" title="speecht5.decoder: 5.2K (0.1%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>speecht5.encoder.wrapped_encoder.layers</td><td>85.1M</td><td>56.3%</td><td>66.9K</td><td>1.5%</td></tr><tr><td><i class="c1"></i>speecht5.decoder.wrapped_decoder.layers</td><td>56.7M</td><td>37.5%</td><td>100.5K</td><td>2.3%</td></tr><tr><td><i class="c2"></i>speecht5.encoder</td><td>5.1M</td><td>3.4%</td><td>72.1K</td><td>1.6%</td></tr><tr><td><i class="c3"></i>speecht5.encoder.prenet.feature_encoder.conv_layers</td><td>4.2M</td><td>2.8%</td><td>4.2M</td><td>94.5%</td></tr><tr><td><i class="c4"></i>speecht5.decoder</td><td>62.2K</td><td>0.0%</td><td>5.2K</td><td>0.1%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>speecht5</td><td>speecht5</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>81</td><td>81</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_mel_bins</td><td>80</td><td>80</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix SpeechT5 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>134</td><td>34</td></tr><tr class=""><td>Dropout</td><td>57</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>44</td><td>12</td></tr><tr class=""><td>GELUActivation</td><td>26</td><td>12</td></tr><tr class=""><td>Attention</td><td>24</td><td>6</td></tr><tr class=""><td>FeedForward</td><td>18</td><td>4</td></tr><tr class=""><td>EncoderLayer</td><td>12</td><td>2</td></tr><tr class=""><td>Conv1d</td><td>7</td><td>7</td></tr><tr class=""><td>DecoderLayer</td><td>6</td><td>2</td></tr><tr class=""><td>NoLayerNormConvLayer</td><td>6</td><td>6</td></tr><tr class=""><td>ModuleList</td><td>3</td><td>3</td></tr><tr class=""><td>Embedding</td><td>2</td><td>2</td></tr><tr class=""><td>SinusoidalPositionalEmbedding</td><td>2</td><td>2</td></tr><tr class=""><td>GroupNorm</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleDict</td><td>1</td><td>1</td></tr><tr class=""><td>ParametrizationList</td><td>1</td><td>1</td></tr><tr class=""><td>ParametrizedConv1d</td><td>1</td><td>1</td></tr><tr class=""><td>Decoder</td><td>1</td><td>1</td></tr><tr class=""><td>DecoderWithTextPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>EncoderWithSpeechPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>FeatureEncoder</td><td>1</td><td>1</td></tr><tr class=""><td>FeatureProjection</td><td>1</td><td>1</td></tr><tr class=""><td>ForSpeechToText</td><td>1</td><td>1</td></tr><tr class=""><td>GroupNormConvLayer</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PositionalConvEmbedding</td><td>1</td><td>1</td></tr><tr class=""><td>RelativePositionalEncoding</td><td>1</td><td>1</td></tr><tr class=""><td>SamePadLayer</td><td>1</td><td>1</td></tr><tr class=""><td>SpeechEncoderPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>TextDecoderPostnet</td><td>1</td><td>1</td></tr><tr class=""><td>TextDecoderPrenet</td><td>1</td><td>1</td></tr><tr class=""><td>_WeightNorm</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">7 fields changed, 60 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>decoder_attention_heads</td><td>12</td><td>4</td></tr><tr><td>decoder_ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>decoder_layers</td><td>6</td><td>2</td></tr><tr><td>encoder_attention_heads</td><td>12</td><td>4</td></tr><tr><td>encoder_ffn_dim</td><td>3072</td><td>128</td></tr><tr><td>encoder_layers</td><td>12</td><td>2</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/stablelm-2-1_6b-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>stabilityai/stablelm-2-1_6b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/stablelm-2-1_6b-mic/tree/eec0b53366b61c85645d4c7ec7618033265b6dd3" title="huggingface-mic/stablelm-2-1_6b-mic@eec0b53366b61c85645d4c7ec7618033265b6dd3">huggingface-mic/stablelm-2-1_6b-mic@eec0b53366b6</a> β <a href="https://huggingface.co/stabilityai/stablelm-2-1_6b/tree/f499ead74c53749bd93cebc6ce8bc0d7bdf1eaef" title="stabilityai/stablelm-2-1_6b@f499ead74c53749bd93cebc6ce8bc0d7bdf1eaef">stabilityai/stablelm-2-1_6b@f499ead74c53</a> Β· StableLmForCausalLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.6B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">12.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.750050" title="model.layers: 1.2B (75.0%)"></i><i class="c1" style="flex:0.124974" title="model.embed_tokens: 205.5M (12.5%)"></i><i class="c2" style="flex:0.124974" title="lm_head: 205.5M (12.5%)"></i><i class="c3" style="flex:0.000002" title="model.norm: 4.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.006406" title="model.layers: 82.8K (0.6%)"></i><i class="c1" style="flex:0.496792" title="model.embed_tokens: 6.4M (49.7%)"></i><i class="c2" style="flex:0.496792" title="lm_head: 6.4M (49.7%)"></i><i class="c3" style="flex:0.000010" title="model.norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.2B</td><td>75.0%</td><td>82.8K</td><td>0.6%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>205.5M</td><td>12.5%</td><td>6.4M</td><td>49.7%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>205.5M</td><td>12.5%</td><td>6.4M</td><td>49.7%</td></tr><tr><td><i class="c3"></i>model.norm</td><td>4.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>stablelm</td><td>stablelm</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>100352</td><td>100352</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4096</td><td>4096</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix StableLm omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Dropout</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>intermediate_size</td><td>5632</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>stabilityai/stablelm-2-1_6b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/stablelm-2-1_6b-mic/tree/eec0b53366b61c85645d4c7ec7618033265b6dd3" title="huggingface-mic/stablelm-2-1_6b-mic@eec0b53366b61c85645d4c7ec7618033265b6dd3">huggingface-mic/stablelm-2-1_6b-mic@eec0b53366b6</a> β <a href="https://huggingface.co/stabilityai/stablelm-2-1_6b/tree/f499ead74c53749bd93cebc6ce8bc0d7bdf1eaef" title="stabilityai/stablelm-2-1_6b@f499ead74c53749bd93cebc6ce8bc0d7bdf1eaef">stabilityai/stablelm-2-1_6b@f499ead74c53</a> Β· StableLmForCausalLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=stablelm-2-1_6b-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">1.6B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">12.9M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 24</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>24</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.750050" title="model.layers: 1.2B (75.0%)"></i><i class="c1" style="flex:0.124974" title="model.embed_tokens: 205.5M (12.5%)"></i><i class="c2" style="flex:0.124974" title="lm_head: 205.5M (12.5%)"></i><i class="c3" style="flex:0.000002" title="model.norm: 4.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.006406" title="model.layers: 82.8K (0.6%)"></i><i class="c1" style="flex:0.496792" title="model.embed_tokens: 6.4M (49.7%)"></i><i class="c2" style="flex:0.496792" title="lm_head: 6.4M (49.7%)"></i><i class="c3" style="flex:0.000010" title="model.norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>1.2B</td><td>75.0%</td><td>82.8K</td><td>0.6%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>205.5M</td><td>12.5%</td><td>6.4M</td><td>49.7%</td></tr><tr><td><i class="c2"></i>lm_head</td><td>205.5M</td><td>12.5%</td><td>6.4M</td><td>49.7%</td></tr><tr><td><i class="c3"></i>model.norm</td><td>4.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>MHA (32 heads, 32 KV)</td><td>MHA (4 heads, 4 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>stablelm</td><td>stablelm</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>100352</td><td>100352</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>False</td><td>False</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>silu</td><td>silu</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>4096</td><td>4096</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix StableLm omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>169</td><td>15</td></tr><tr class=""><td>LayerNorm</td><td>49</td><td>5</td></tr><tr class=""><td>Dropout</td><td>24</td><td>2</td></tr><tr class=""><td>SiLUActivation</td><td>24</td><td>2</td></tr><tr class=""><td>Attention</td><td>24</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>24</td><td>2</td></tr><tr class=""><td>MLP</td><td>24</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 25 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>2048</td><td>64</td></tr><tr><td>intermediate_size</td><td>5632</td><td>128</td></tr><tr><td>num_attention_heads</td><td>32</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>24</td><td>2</td></tr><tr><td>num_key_value_heads</td><td>32</td><td>4</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/starcoder2-3b-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>bigcode/starcoder2-3b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/starcoder2-3b-mic/tree/f158c5574331920c1b49c8447d5bf4a4864df079" title="huggingface-mic/starcoder2-3b-mic@f158c5574331920c1b49c8447d5bf4a4864df079">huggingface-mic/starcoder2-3b-mic@f158c5574331</a> β <a href="https://huggingface.co/bigcode/starcoder2-3b/tree/733247c55e3f73af49ce8e9c7949bf14af205928" title="bigcode/starcoder2-3b@733247c55e3f73af49ce8e9c7949bf14af205928">bigcode/starcoder2-3b@733247c55e3f</a> Β· Starcoder2ForCausalLM Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.0B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 30</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>30</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.950171" title="model.layers: 2.9B (95.0%)"></i><i class="c1" style="flex:0.049827" title="model.embed_tokens: 151.0M (5.0%)"></i><i class="c2" style="flex:0.000002" title="model.norm: 6.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018294" title="model.layers: 58.6K (1.8%)"></i><i class="c1" style="flex:0.981666" title="model.embed_tokens: 3.1M (98.2%)"></i><i class="c2" style="flex:0.000040" title="model.norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.9B</td><td>95.0%</td><td>58.6K</td><td>1.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>151.0M</td><td>5.0%</td><td>3.1M</td><td>98.2%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>6.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (24 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>starcoder2</td><td>starcoder2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>16384</td><td>16384</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Starcoder2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>181</td><td>13</td></tr><tr class=""><td>LayerNorm</td><td>61</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>30</td><td>2</td></tr><tr class=""><td>Attention</td><td>30</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>30</td><td>2</td></tr><tr class=""><td>MLP</td><td>30</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>3072</td><td>64</td></tr><tr><td>intermediate_size</td><td>12288</td><td>128</td></tr><tr><td>num_attention_heads</td><td>24</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>30</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>bigcode/starcoder2-3b</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/starcoder2-3b-mic/tree/f158c5574331920c1b49c8447d5bf4a4864df079" title="huggingface-mic/starcoder2-3b-mic@f158c5574331920c1b49c8447d5bf4a4864df079">huggingface-mic/starcoder2-3b-mic@f158c5574331</a> β <a href="https://huggingface.co/bigcode/starcoder2-3b/tree/733247c55e3f73af49ce8e9c7949bf14af205928" title="bigcode/starcoder2-3b@733247c55e3f73af49ce8e9c7949bf14af205928">bigcode/starcoder2-3b@733247c55e3f</a> Β· Starcoder2ForCausalLM Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=starcoder2-3b-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">3.0B</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">3.2M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>model.layers</h3><div class="row"><span class="lab">Original Β· 30</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i><i class="c0" title="layer 2: DecoderLayer"></i><i class="c0" title="layer 3: DecoderLayer"></i><i class="c0" title="layer 4: DecoderLayer"></i><i class="c0" title="layer 5: DecoderLayer"></i><i class="c0" title="layer 6: DecoderLayer"></i><i class="c0" title="layer 7: DecoderLayer"></i><i class="c0" title="layer 8: DecoderLayer"></i><i class="c0" title="layer 9: DecoderLayer"></i><i class="c0" title="layer 10: DecoderLayer"></i><i class="c0" title="layer 11: DecoderLayer"></i><i class="c0" title="layer 12: DecoderLayer"></i><i class="c0" title="layer 13: DecoderLayer"></i><i class="c0" title="layer 14: DecoderLayer"></i><i class="c0" title="layer 15: DecoderLayer"></i><i class="c0" title="layer 16: DecoderLayer"></i><i class="c0" title="layer 17: DecoderLayer"></i><i class="c0" title="layer 18: DecoderLayer"></i><i class="c0" title="layer 19: DecoderLayer"></i><i class="c0" title="layer 20: DecoderLayer"></i><i class="c0" title="layer 21: DecoderLayer"></i><i class="c0" title="layer 22: DecoderLayer"></i><i class="c0" title="layer 23: DecoderLayer"></i><i class="c0" title="layer 24: DecoderLayer"></i><i class="c0" title="layer 25: DecoderLayer"></i><i class="c0" title="layer 26: DecoderLayer"></i><i class="c0" title="layer 27: DecoderLayer"></i><i class="c0" title="layer 28: DecoderLayer"></i><i class="c0" title="layer 29: DecoderLayer"></i></div></div><div class="row"><span class="lab">Fixture Β· 2</span><div class="cells"><i class="c0" title="layer 0: DecoderLayer"></i><i class="c0" title="layer 1: DecoderLayer"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>DecoderLayer</td><td>30</td><td>2</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.950171" title="model.layers: 2.9B (95.0%)"></i><i class="c1" style="flex:0.049827" title="model.embed_tokens: 151.0M (5.0%)"></i><i class="c2" style="flex:0.000002" title="model.norm: 6.1K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.018294" title="model.layers: 58.6K (1.8%)"></i><i class="c1" style="flex:0.981666" title="model.embed_tokens: 3.1M (98.2%)"></i><i class="c2" style="flex:0.000040" title="model.norm: 128 (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>model.layers</td><td>2.9B</td><td>95.0%</td><td>58.6K</td><td>1.8%</td></tr><tr><td><i class="c1"></i>model.embed_tokens</td><td>151.0M</td><td>5.0%</td><td>3.1M</td><td>98.2%</td></tr><tr><td><i class="c2"></i>model.norm</td><td>6.1K</td><td>0.0%</td><td>128</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>attention style</td><td>GQA (24 heads, 2 KV)</td><td>GQA (4 heads, 2 KV)</td><td>β kept</td></tr><tr class=""><td>model</td><td>RoPE type</td><td>default</td><td>default</td><td>β kept</td></tr><tr class=""><td>model</td><td>model_type</td><td>starcoder2</td><td>starcoder2</td><td>β kept</td></tr><tr class=""><td>model</td><td>vocab_size</td><td>49152</td><td>49152</td><td>β kept</td></tr><tr class=""><td>model</td><td>tie_word_embeddings</td><td>True</td><td>True</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu_pytorch_tanh</td><td>gelu_pytorch_tanh</td><td>β kept</td></tr><tr class=""><td>model</td><td>max_position_embeddings</td><td>16384</td><td>16384</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Starcoder2 omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>181</td><td>13</td></tr><tr class=""><td>LayerNorm</td><td>61</td><td>5</td></tr><tr class=""><td>GELUTanh</td><td>30</td><td>2</td></tr><tr class=""><td>Attention</td><td>30</td><td>2</td></tr><tr class=""><td>DecoderLayer</td><td>30</td><td>2</td></tr><tr class=""><td>MLP</td><td>30</td><td>2</td></tr><tr class=""><td>Embedding</td><td>1</td><td>1</td></tr><tr class=""><td>ModuleList</td><td>1</td><td>1</td></tr><tr class=""><td>ForCausalLM</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>RotaryEmbedding</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">4 fields changed, 28 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>hidden_size</td><td>3072</td><td>64</td></tr><tr><td>intermediate_size</td><td>12288</td><td>128</td></tr><tr><td>num_attention_heads</td><td>24</td><td>4</td></tr><tr><td>num_hidden_layers</td><td>30</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|
shrink/swin-tiny-patch4-window7-224-mic.html
CHANGED
|
@@ -32,5 +32,6 @@ tr.bad td{color:var(--warn)}
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/swin-tiny-patch4-window7-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/swin-tiny-patch4-window7-224-mic/tree/89ae45d56ca206c1ba47aee79182a5e98d039eb6" title="huggingface-mic/swin-tiny-patch4-window7-224-mic@89ae45d56ca206c1ba47aee79182a5e98d039eb6">huggingface-mic/swin-tiny-patch4-window7-224-mic@89ae45d56ca2</a> β <a href="https://huggingface.co/microsoft/swin-tiny-patch4-window7-224/tree/d00d478bfaf1f417d34a1186673ad4b4be264c51" title="microsoft/swin-tiny-patch4-window7-224@d00d478bfaf1f417d34a1186673ad4b4be264c51">microsoft/swin-tiny-patch4-window7-224@d00d478bfaf1</a> Β· SwinForImageClassification Β· random weights</p>
|
|
|
|
| 35 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">28.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">5.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: PatchMerging"></i><i class="c0" title="layer 1: PatchMerging"></i><i class="c0" title="layer 2: PatchMerging"></i><i class="c1" title="layer 3: no PatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: PatchMerging"></i><i class="c0" title="layer 1: PatchMerging"></i><i class="c0" title="layer 2: PatchMerging"></i><i class="c1" title="layer 3: no PatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>PatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no PatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.972588" title="swin.encoder.layers: 27.5M (97.3%)"></i><i class="c1" style="flex:0.027184" title="classifier: 769.0K (2.7%)"></i><i class="c2" style="flex:0.000173" title="swin.embeddings: 4.9K (0.0%)"></i><i class="c3" style="flex:0.000054" title="swin.layernorm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.904285" title="swin.encoder.layers: 4.9M (90.4%)"></i><i class="c1" style="flex:0.094921" title="classifier: 513.0K (9.5%)"></i><i class="c2" style="flex:0.000604" title="swin.embeddings: 3.3K (0.1%)"></i><i class="c3" style="flex:0.000189" title="swin.layernorm: 1.0K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>swin.encoder.layers</td><td>27.5M</td><td>97.3%</td><td>4.9M</td><td>90.4%</td></tr><tr><td><i class="c1"></i>classifier</td><td>769.0K</td><td>2.7%</td><td>513.0K</td><td>9.5%</td></tr><tr><td><i class="c2"></i>swin.embeddings</td><td>4.9K</td><td>0.0%</td><td>3.3K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>swin.layernorm</td><td>1.5K</td><td>0.0%</td><td>1.0K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Swin omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>76</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>29</td><td>13</td></tr><tr class=""><td>Dropout</td><td>13</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>4</td></tr><tr class=""><td>Attention</td><td>12</td><td>4</td></tr><tr class=""><td>Layer</td><td>12</td><td>4</td></tr><tr class=""><td>MLP</td><td>12</td><td>4</td></tr><tr class=""><td>RelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>DropPath</td><td>11</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>5</td><td>5</td></tr><tr class=""><td>Stage</td><td>4</td><td>4</td></tr><tr class=""><td>PatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>AdaptiveAvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Identity</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 27 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>embed_dim</td><td>96</td><td>64</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>num_layers</td><td>4</td><td>2</td></tr></table>
|
| 36 |
</main></body></html>
|
|
|
|
| 32 |
</style></head><body><main>
|
| 33 |
<p class="meta"><a href="../index.html">β MIC coverage of Transformers model types</a></p>
|
| 34 |
<h1>microsoft/swin-tiny-patch4-window7-224</h1><p class="meta"><a href="https://huggingface.co/huggingface-mic/swin-tiny-patch4-window7-224-mic/tree/89ae45d56ca206c1ba47aee79182a5e98d039eb6" title="huggingface-mic/swin-tiny-patch4-window7-224-mic@89ae45d56ca206c1ba47aee79182a5e98d039eb6">huggingface-mic/swin-tiny-patch4-window7-224-mic@89ae45d56ca2</a> β <a href="https://huggingface.co/microsoft/swin-tiny-patch4-window7-224/tree/d00d478bfaf1f417d34a1186673ad4b4be264c51" title="microsoft/swin-tiny-patch4-window7-224@d00d478bfaf1f417d34a1186673ad4b4be264c51">microsoft/swin-tiny-patch4-window7-224@d00d478bfaf1</a> Β· SwinForImageClassification Β· random weights</p>
|
| 35 |
+
<p class="meta"><!--ci-->CI: <a href="https://transformers-ci.huggingface.cool/mic?q=swin-tiny-patch4-window7-224-mic&days=30&sticky_only=false" target="_blank" rel="noopener">no failure on main in 30 days</a> Β· <a href="https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser" target="_blank" rel="noopener">MIC failures dashboard</a></p>
|
| 36 |
<div class="stats"><div class="stat"><span class="k">Original</span><span class="v">28.3M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Fixture</span><span class="v">5.4M</span><span class="k">parameters</span></div><div class="stat"><span class="k">Layer stacks</span><span class="v">1</span><span class="k">compared</span></div><div class="stat"><span class="k">Findings</span><span class="v">0</span><span class="k">to review</span></div></div><section class="findings ok"><h2>β Same architecture, smaller</h2><p>Every module class, layer kind and checked property of the original is in the fixture.</p></section><h2>Layer stacks</h2><p class="note">One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.</p><div class="stack"><h3>swin.encoder.layers</h3><div class="row"><span class="lab">Original Β· 4</span><div class="cells"><i class="c0" title="layer 0: PatchMerging"></i><i class="c0" title="layer 1: PatchMerging"></i><i class="c0" title="layer 2: PatchMerging"></i><i class="c1" title="layer 3: no PatchMerging"></i></div></div><div class="row"><span class="lab">Fixture Β· 4</span><div class="cells"><i class="c0" title="layer 0: PatchMerging"></i><i class="c0" title="layer 1: PatchMerging"></i><i class="c0" title="layer 2: PatchMerging"></i><i class="c1" title="layer 3: no PatchMerging"></i></div></div><table class="legend"><tr><th>Kind</th><th>Original</th><th>Fixture</th><th></th></tr><tr><td><i class="c0"></i>PatchMerging</td><td>3</td><td>3</td><td>β kept</td></tr><tr><td><i class="c1"></i>no PatchMerging</td><td>1</td><td>1</td><td>β kept</td></tr></table></div><h2>Where the parameters are</h2><p class="note">Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.</p><div class="row"><span class="lab">Original</span><div class="bar"><i class="c0" style="flex:0.972588" title="swin.encoder.layers: 27.5M (97.3%)"></i><i class="c1" style="flex:0.027184" title="classifier: 769.0K (2.7%)"></i><i class="c2" style="flex:0.000173" title="swin.embeddings: 4.9K (0.0%)"></i><i class="c3" style="flex:0.000054" title="swin.layernorm: 1.5K (0.0%)"></i></div></div><div class="row"><span class="lab">Fixture</span><div class="bar"><i class="c0" style="flex:0.904285" title="swin.encoder.layers: 4.9M (90.4%)"></i><i class="c1" style="flex:0.094921" title="classifier: 513.0K (9.5%)"></i><i class="c2" style="flex:0.000604" title="swin.embeddings: 3.3K (0.1%)"></i><i class="c3" style="flex:0.000189" title="swin.layernorm: 1.0K (0.0%)"></i></div></div><table class="legend"><tr><th>Component</th><th>Original</th><th>Share</th><th>Fixture</th><th>Share</th></tr><tr><td><i class="c0"></i>swin.encoder.layers</td><td>27.5M</td><td>97.3%</td><td>4.9M</td><td>90.4%</td></tr><tr><td><i class="c1"></i>classifier</td><td>769.0K</td><td>2.7%</td><td>513.0K</td><td>9.5%</td></tr><tr><td><i class="c2"></i>swin.embeddings</td><td>4.9K</td><td>0.0%</td><td>3.3K</td><td>0.1%</td></tr><tr><td><i class="c3"></i>swin.layernorm</td><td>1.5K</td><td>0.0%</td><td>1.0K</td><td>0.0%</td></tr></table><h2>Preserved properties</h2><table><tr><th>Config</th><th>Property</th><th>Original</th><th>Fixture</th><th></th></tr><tr class=""><td>model</td><td>model_type</td><td>swin</td><td>swin</td><td>β kept</td></tr><tr class=""><td>model</td><td>hidden_act</td><td>gelu</td><td>gelu</td><td>β kept</td></tr><tr class=""><td>model</td><td>image_size</td><td>224</td><td>224</td><td>β kept</td></tr><tr class=""><td>model</td><td>patch_size</td><td>4</td><td>4</td><td>β kept</td></tr><tr class=""><td>model</td><td>num_channels</td><td>3</td><td>3</td><td>β kept</td></tr></table><h2>Module classes</h2><p class="note">Instances of each class in the built model (prefix Swin omitted).</p><table><tr><th>Class</th><th>Original</th><th>Fixture</th></tr><tr class=""><td>Linear</td><td>76</td><td>28</td></tr><tr class=""><td>LayerNorm</td><td>29</td><td>13</td></tr><tr class=""><td>Dropout</td><td>13</td><td>5</td></tr><tr class=""><td>GELUActivation</td><td>12</td><td>4</td></tr><tr class=""><td>Attention</td><td>12</td><td>4</td></tr><tr class=""><td>Layer</td><td>12</td><td>4</td></tr><tr class=""><td>MLP</td><td>12</td><td>4</td></tr><tr class=""><td>RelativePositionBias</td><td>12</td><td>4</td></tr><tr class=""><td>DropPath</td><td>11</td><td>3</td></tr><tr class=""><td>ModuleList</td><td>5</td><td>5</td></tr><tr class=""><td>Stage</td><td>4</td><td>4</td></tr><tr class=""><td>PatchMerging</td><td>3</td><td>3</td></tr><tr class=""><td>AdaptiveAvgPool1d</td><td>1</td><td>1</td></tr><tr class=""><td>Conv2d</td><td>1</td><td>1</td></tr><tr class=""><td>Identity</td><td>1</td><td>1</td></tr><tr class=""><td>Embeddings</td><td>1</td><td>1</td></tr><tr class=""><td>Encoder</td><td>1</td><td>1</td></tr><tr class=""><td>ForImageClassification</td><td>1</td><td>1</td></tr><tr class=""><td>Model</td><td>1</td><td>1</td></tr><tr class=""><td>PatchEmbeddings</td><td>1</td><td>1</td></tr></table><h2>Config changes</h2><p class="note">5 fields changed, 27 unchanged.</p><table><tr><th>Field</th><th>Original</th><th>Fixture</th></tr><tr><td>depths</td><td>[2, 2, 6, 2]</td><td>[1, 1, 1, 1]</td></tr><tr><td>embed_dim</td><td>96</td><td>64</td></tr><tr><td>hidden_size</td><td>768</td><td>64</td></tr><tr><td>num_heads</td><td>[3, 6, 12, 24]</td><td>[2, 4, 8, 16]</td></tr><tr><td>num_layers</td><td>4</td><td>2</td></tr></table>
|
| 37 |
</main></body></html>
|