Spaces:
Running
Running
Landing page: the badges jump to the Checkpoints and Data sections
Browse files- index.html +4 -4
index.html
CHANGED
|
@@ -61,8 +61,8 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
|
|
| 61 |
<a href="https://github.com/jincan333/ShellElastic/blob/main/paper/ShellElastic.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
|
| 62 |
<a href="https://jincan333.github.io/ShellElastic_page"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
|
| 63 |
<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
|
| 64 |
-
<a href="
|
| 65 |
-
<a href="
|
| 66 |
</div>
|
| 67 |
</header>
|
| 68 |
</div>
|
|
@@ -74,7 +74,7 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
|
|
| 74 |
<figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
|
| 75 |
<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
|
| 76 |
|
| 77 |
-
<h2>Checkpoints</h2>
|
| 78 |
<p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
|
| 79 |
119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
|
| 80 |
(Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
|
|
@@ -90,7 +90,7 @@ holds the five models in the folders <code>low/</code> to <code>max/</code>. The
|
|
| 90 |
<tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B">675M–1.69B</a></td></tr>
|
| 91 |
</table></div>
|
| 92 |
|
| 93 |
-
<h2>Data</h2>
|
| 94 |
<p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
|
| 95 |
read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
|
| 96 |
(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
|
|
|
|
| 61 |
<a href="https://github.com/jincan333/ShellElastic/blob/main/paper/ShellElastic.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
|
| 62 |
<a href="https://jincan333.github.io/ShellElastic_page"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
|
| 63 |
<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
|
| 64 |
+
<a href="#checkpoints"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
| 65 |
+
<a href="#data"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
| 66 |
</div>
|
| 67 |
</header>
|
| 68 |
</div>
|
|
|
|
| 74 |
<figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
|
| 75 |
<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
|
| 76 |
|
| 77 |
+
<h2 id="checkpoints">Checkpoints</h2>
|
| 78 |
<p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
|
| 79 |
119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
|
| 80 |
(Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
|
|
|
|
| 90 |
<tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B">675M–1.69B</a></td></tr>
|
| 91 |
</table></div>
|
| 92 |
|
| 93 |
+
<h2 id="data">Data</h2>
|
| 94 |
<p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
|
| 95 |
read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
|
| 96 |
(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
|