Can111 commited on
Commit
319347f
·
verified ·
1 Parent(s): 654228b

Landing page: Figure 2, and both figures from the current paper

Browse files
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. figure1.png +2 -2
  3. figure2.png +3 -0
  4. index.html +1 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  figure1.png filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  figure1.png filter=lfs diff=lfs merge=lfs -text
37
+ figure2.png filter=lfs diff=lfs merge=lfs -text
figure1.png CHANGED

Git LFS Details

  • SHA256: 85e803ffce530adfc1ce0ce3493760a9c436a09221d4d81e814033b66aaa0b87
  • Pointer size: 131 Bytes
  • Size of remote file: 140 kB

Git LFS Details

  • SHA256: 749b46db887367550e6bd25f7a4d3b04f2646f218b5bf4efddd0466db61e645e
  • Pointer size: 131 Bytes
  • Size of remote file: 140 kB
figure2.png ADDED

Git LFS Details

  • SHA256: 5a3008f563e121c2d68ebbc91d23db150db60ab4487cd249b95959b59bec46b5
  • Pointer size: 131 Bytes
  • Size of remote file: 307 kB
index.html CHANGED
@@ -69,6 +69,7 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
69
  <h2>About ShellElastic</h2>
70
  <figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
71
  <p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
 
72
  <p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
73
 
74
  <h2>Checkpoints</h2>
 
69
  <h2>About ShellElastic</h2>
70
  <figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
71
  <p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
72
+ <figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
73
  <p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
74
 
75
  <h2>Checkpoints</h2>