Spaces:
Running
Running
Landing page: Figure 2, and both figures from the current paper
Browse files- .gitattributes +1 -0
- figure1.png +2 -2
- figure2.png +3 -0
- index.html +1 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
figure1.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
figure1.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
figure2.png filter=lfs diff=lfs merge=lfs -text
|
figure1.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
figure2.png
ADDED
|
Git LFS Details
|
index.html
CHANGED
|
@@ -69,6 +69,7 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
|
|
| 69 |
<h2>About ShellElastic</h2>
|
| 70 |
<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
|
| 71 |
<p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
|
|
|
|
| 72 |
<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
|
| 73 |
|
| 74 |
<h2>Checkpoints</h2>
|
|
|
|
| 69 |
<h2>About ShellElastic</h2>
|
| 70 |
<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
|
| 71 |
<p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
|
| 72 |
+
<figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
|
| 73 |
<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
|
| 74 |
|
| 75 |
<h2>Checkpoints</h2>
|