Can111 commited on
Commit
135ecfb
·
verified ·
1 Parent(s): f441af2

Landing page: the badges jump to the Checkpoints and Data sections

Browse files
Files changed (1) hide show
  1. index.html +4 -4
index.html CHANGED
@@ -61,8 +61,8 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
61
  <a href="https://github.com/jincan333/ShellElastic/blob/main/paper/ShellElastic.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
62
  <a href="https://jincan333.github.io/ShellElastic_page"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
63
  <a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&amp;logo=github&amp;logoColor=white"></a>
64
- <a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
65
- <a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
66
  </div>
67
  </header>
68
  </div>
@@ -74,7 +74,7 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
74
  <figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
75
  <p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
76
 
77
- <h2>Checkpoints</h2>
78
  <p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
79
  119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
80
  (Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
@@ -90,7 +90,7 @@ holds the five models in the folders <code>low/</code> to <code>max/</code>. The
90
  <tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B">675M–1.69B</a></td></tr>
91
  </table></div>
92
 
93
- <h2>Data</h2>
94
  <p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
95
  read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
96
  (141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
 
61
  <a href="https://github.com/jincan333/ShellElastic/blob/main/paper/ShellElastic.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
62
  <a href="https://jincan333.github.io/ShellElastic_page"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
63
  <a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&amp;logo=github&amp;logoColor=white"></a>
64
+ <a href="#checkpoints"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
65
+ <a href="#data"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
66
  </div>
67
  </header>
68
  </div>
 
74
  <figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
75
  <p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
76
 
77
+ <h2 id="checkpoints">Checkpoints</h2>
78
  <p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
79
  119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
80
  (Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
 
90
  <tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B">675M–1.69B</a></td></tr>
91
  </table></div>
92
 
93
+ <h2 id="data">Data</h2>
94
  <p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
95
  read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
96
  (141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4