index.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer</title>
<style>
:root { --fg: rgb(31,35,40); --muted: rgb(89,99,110); --line: rgb(208,215,222); --bg: white; --soft: rgb(246,248,250);
        --link: rgb(9,105,218); }
@media (prefers-color-scheme: dark) {
  :root { --fg: rgb(230,237,243); --muted: rgb(145,152,161); --line: rgb(61,68,77); --bg: rgb(13,17,23);
          --soft: rgb(22,27,34); --link: rgb(68,147,248); }
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg); color: var(--fg);
       font: 16px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif; }
main { max-width: 980px; margin: 0 auto; padding: 8px 16px 56px; }
.top { max-width: 1140px; margin: 0 auto; padding: 32px 16px 0; }
a { color: var(--link); text-decoration: none; }
a:hover { text-decoration: underline; }
header { text-align: center; }
h1 { font-size: 2.6em; margin: 0.2em 0 0.3em; padding-bottom: 0.25em; border-bottom: 1px solid var(--line); line-height: 1; }
h1 img.logo { height: 42px; width: auto; vertical-align: middle; }
.subtitle { font-size: 1.55em; font-weight: 600; line-height: 1.3; margin: 0.6em 0; }
.authors { font-size: min(16px, calc((100vw - 32px) * 0.0147)); white-space: nowrap; margin: 0.9em 0 0.4em; }
@media (max-width: 900px) { .authors { font-size: 15px; white-space: normal; } }
.authors b { font-weight: 600; }
.author { white-space: nowrap; }
.affil, .corr { margin: 0.4em 0; }
.badges { margin: 1.1em 0 0.2em; display: flex; flex-wrap: wrap; justify-content: center; gap: 6px; }
.badges img { height: 28px; display: block; }
h2 { font-size: 1.5em; margin: 1.6em 0 0.6em; padding-bottom: 0.3em; border-bottom: 1px solid var(--line); }
figure { margin: 1.4em 0; }
figure img { width: 100%; height: auto; background: white; border-radius: 6px; }
p { margin: 0.7em 0; }
.table-wrap { overflow-x: auto; }
table { width: 100%; border-collapse: collapse; font-size: 0.95em; }
th, td { border: 1px solid var(--line); padding: 8px 10px; vertical-align: top; text-align: left; }
th { background: var(--soft); font-weight: 600; }
td .note { color: var(--muted); font-size: 0.9em; }
td.links { white-space: nowrap; }
@media (max-width: 640px) { td.links { white-space: normal; } }
.scale-link { display: inline-block; margin: 2px 4px 2px 0; padding: 3px 10px; border: 1px solid var(--line);
        border-radius: 6px; background: var(--soft); font-size: 0.92em; }
code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace; font-size: 0.86em; }
.repo { overflow-wrap: anywhere; }
.muted { color: var(--muted); }
</style>
</head>
<body>
<div class="top">
<header>
  <h1><picture><source media="(prefers-color-scheme: dark)" srcset="shellelastic-logo-dark.svg">
    <img class="logo" alt="ShellElastic" src="shellelastic-logo-light.svg" height="42"></picture></h1>
  <div class="subtitle">Scalable Elastic Pretraining<br>with Simultaneous Maximal Updates and Hyperparameter Transfer</div>
  <p class="authors"><span class="author"><b>Can Jin</b><sup>1*</sup></span> · <span class="author"><b>Yan Kang</b><sup>2</sup></span> · <span class="author"><b>Yifan Gong</b><sup>2</sup></span> · <span class="author"><b>Chongjian Ge</b><sup>2</sup></span> · <span class="author"><b>Mingze Xu</b><sup>2</sup></span> · <span class="author"><b>Yuanjun Xiong</b><sup>2</sup></span> · <span class="author"><b>Dimitris N. Metaxas</b><sup>1</sup></span> · <span class="author"><b>Hongwu Peng</b><sup>2</sup></span></p>
  <p class="affil"><sup>1</sup>Rutgers University &emsp; <sup>2</sup>Adobe Research</p>
  <p class="corr"><sup>*</sup>Corresponding author: <a href="mailto:can.jin@rutgers.edu">can.jin@rutgers.edu</a>. Work done during an
    internship at Adobe Research.</p>
  <div class="badges">
    <a href="https://arxiv.org/pdf/2610.15124" target="_blank" rel="noopener"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
    <a href="https://jincan333.github.io/ShellElastic_page" target="_blank" rel="noopener"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
    <a href="https://github.com/jincan333/ShellElastic" target="_blank" rel="noopener"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&amp;logo=github&amp;logoColor=white"></a>
    <a href="#checkpoints"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
    <a href="#data"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
  </div>
</header>
</div>
<main>

<h2>About ShellElastic</h2>
<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
<p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
<figure><img src="figure2.png" alt="Figure 2 of the paper: overview of µ-Elastic"></figure>
<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>

<h2 id="checkpoints">Checkpoints</h2>
<p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
(Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
holds the five models in the folders <code>low/</code> to <code>max/</code>. The repositories are named
<code class="repo">Can111/ShellElastic-&lt;scale&gt;</code> and
<code class="repo">Can111/ShellElastic-baseline-&lt;method&gt;-&lt;scale&gt;</code>.</p>
<div class="table-wrap"><table>
<tr><th>Method</th><th>Checkpoints (sub-network sizes)</th></tr>
<tr><td><b>ShellElastic</b><br><span class="note">Supernet with its five sub-networks (Low, Medium, High, Extra, Max)</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-119M-285M" href="https://huggingface.co/Can111/ShellElastic-119M-285M" target="_blank" rel="noopener">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-375M-906M" href="https://huggingface.co/Can111/ShellElastic-375M-906M" target="_blank" rel="noopener">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-675M-1.69B" target="_blank" rel="noopener">675M–1.69B</a></td></tr>
<tr><td><b>MatFormer</b><br><span class="note">Baseline trained for the paper: a supernet that nests only the FFN hidden width</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-119M-285M" target="_blank" rel="noopener">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-375M-906M" target="_blank" rel="noopener">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-675M-1.69B" target="_blank" rel="noopener">675M–1.69B</a></td></tr>
<tr><td><b>Fully elastic</b><br><span class="note">Baseline trained for the paper: a supernet that slices shared dense matrices</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-119M-285M" target="_blank" rel="noopener">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-375M-906M" target="_blank" rel="noopener">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-675M-1.69B" target="_blank" rel="noopener">675M–1.69B</a></td></tr>
<tr><td><b>Standalone</b><br><span class="note">Baseline: five separately trained models, each on the full budget</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-119M-285M" target="_blank" rel="noopener">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-375M-906M" target="_blank" rel="noopener">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-675M-1.69B" target="_blank" rel="noopener">675M–1.69B</a></td></tr>
<tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M" target="_blank" rel="noopener">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M" target="_blank" rel="noopener">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" target="_blank" rel="noopener">675M–1.69B</a></td></tr>
</table></div>

<h2 id="data">Data</h2>
<p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data" target="_blank" rel="noopener">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
text (1.12 GB, 84,999 sequences).</p>

</main>
</body>
</html>