Landing page

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figure1.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,10 +1,12 @@
 ---
 title: ShellElastic
-emoji: 👀
-colorFrom: gray
+emoji: 🐚
+colorFrom: blue
 colorTo: yellow
 sdk: static
-pinned: false
+pinned: true
+license: apache-2.0
+short_description: "Scalable elastic pretraining: checkpoints and data"
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+The landing page of ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer. It links the checkpoints and the data of the paper.

index.html

@@ -1,19 +1,98 @@
 <!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer</title>
+<style>
+:root { --fg: rgb(31,35,40); --muted: rgb(89,99,110); --line: rgb(208,215,222); --bg: white; --soft: rgb(246,248,250);
+        --link: rgb(9,105,218); }
+@media (prefers-color-scheme: dark) {
+  :root { --fg: rgb(230,237,243); --muted: rgb(145,152,161); --line: rgb(61,68,77); --bg: rgb(13,17,23);
+          --soft: rgb(22,27,34); --link: rgb(68,147,248); }
+}
+* { box-sizing: border-box; }
+body { margin: 0; background: var(--bg); color: var(--fg);
+       font: 16px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif; }
+main { max-width: 980px; margin: 0 auto; padding: 32px 16px 56px; }
+a { color: var(--link); text-decoration: none; }
+a:hover { text-decoration: underline; }
+header { text-align: center; }
+h1 { font-size: 2.6em; margin: 0.2em 0 0.3em; padding-bottom: 0.25em; border-bottom: 1px solid var(--line); }
+.subtitle { font-size: 1.55em; font-weight: 600; line-height: 1.3; margin: 0.6em 0; }
+.authors { font-size: 1.15em; margin: 0.9em 0 0.4em; }
+.authors b { font-weight: 600; }
+.author { white-space: nowrap; }
+.affil, .corr { margin: 0.4em 0; }
+.badges { margin: 1.1em 0 0.2em; display: flex; flex-wrap: wrap; justify-content: center; gap: 6px; }
+.badges img { height: 28px; display: block; }
+h2 { font-size: 1.5em; margin: 1.6em 0 0.6em; padding-bottom: 0.3em; border-bottom: 1px solid var(--line); }
+figure { margin: 1.4em 0; }
+figure img { width: 100%; height: auto; background: white; border-radius: 6px; }
+p { margin: 0.7em 0; }
+.table-wrap { overflow-x: auto; }
+table { width: 100%; border-collapse: collapse; font-size: 0.95em; }
+th, td { border: 1px solid var(--line); padding: 8px 10px; vertical-align: top; text-align: left; }
+th { background: var(--soft); font-weight: 600; }
+td .note { color: var(--muted); font-size: 0.9em; }
+td.links { white-space: nowrap; }
+@media (max-width: 640px) { td.links { white-space: normal; } }
+.scale-link { display: inline-block; margin: 2px 4px 2px 0; padding: 3px 10px; border: 1px solid var(--line);
+        border-radius: 6px; background: var(--soft); font-size: 0.92em; }
+code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace; font-size: 0.86em; }
+.repo { overflow-wrap: anywhere; }
+.muted { color: var(--muted); }
+</style>
+</head>
+<body>
+<main>
+<header>
+  <h1>ShellElastic</h1>
+  <div class="subtitle">Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer</div>
+  <p class="authors"><span class="author"><b>Can Jin</b><sup>1*</sup></span> · <span class="author"><b>Yan Kang</b><sup>2</sup></span> · <span class="author"><b>Yifan Gong</b><sup>2</sup></span> · <span class="author"><b>Chongjian Ge</b><sup>2</sup></span> · <span class="author"><b>Mingze Xu</b><sup>2</sup></span> · <span class="author"><b>Yuanjun Xiong</b><sup>2</sup></span> · <span class="author"><b>Dimitris N. Metaxas</b><sup>1</sup></span> · <span class="author"><b>Hongwu Peng</b><sup>2</sup></span></p>
+  <p class="affil"><sup>1</sup>Rutgers University &emsp; <sup>2</sup>Adobe Research</p>
+  <p class="corr"><sup>*</sup>Corresponding author: <a href="mailto:can.jin@rutgers.edu">can.jin@rutgers.edu</a>. Work done during an
+    internship at Adobe Research.</p>
+  <div class="badges">
+    <a href="https://github.com/jincan333/ShellElastic/tree/main/paper"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
+    <a href="http://localhost:8765/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
+    <a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&amp;logo=github&amp;logoColor=white"></a>
+    <a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
+    <a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&amp;logo=huggingface&amp;logoColor=white"></a>
+    <a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-3da639?style=for-the-badge&amp;logo=apache&amp;logoColor=white"></a>
+  </div>
+</header>
+
+<h2>About ShellElastic</h2>
+<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
+<p>Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.</p>
+<p>µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.</p>
+
+<h2>Checkpoints</h2>
+<p>Final checkpoints of ShellElastic and four baselines at the three scales of the paper, with sub-networks of about
+119M–285M, 375M–906M and 675M–1.69B parameters (Section 4.1). Each supernet repository holds all five sub-networks
+(Low, Medium, High, Extra, Max) and its reference implementation; each Standalone and Sampled standalone repository
+holds the five models in the folders <code>low/</code> to <code>max/</code>. The repositories are named
+<code class="repo">Can111/ShellElastic-&lt;scale&gt;</code> and
+<code class="repo">Can111/ShellElastic-baseline-&lt;method&gt;-&lt;scale&gt;</code>.</p>
+<div class="table-wrap"><table>
+<tr><th>Method</th><th>Checkpoints (sub-network sizes)</th></tr>
+<tr><td><b>ShellElastic</b><br><span class="note">Supernet with its five sub-networks (Low, Medium, High, Extra, Max)</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-119M-285M" href="https://huggingface.co/Can111/ShellElastic-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-375M-906M" href="https://huggingface.co/Can111/ShellElastic-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-675M-1.69B">675M–1.69B</a></td></tr>
+<tr><td><b>MatFormer</b><br><span class="note">Baseline trained for the paper: a supernet that nests only the FFN hidden width</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-MatFormer-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-MatFormer-675M-1.69B">675M–1.69B</a></td></tr>
+<tr><td><b>Fully elastic</b><br><span class="note">Baseline trained for the paper: a supernet that slices shared dense matrices</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-FullyElastic-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-FullyElastic-675M-1.69B">675M–1.69B</a></td></tr>
+<tr><td><b>Standalone</b><br><span class="note">Baseline: five separately trained models, each on the full budget</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-Standalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-Standalone-675M-1.69B">675M–1.69B</a></td></tr>
+<tr><td><b>Sampled standalone</b><br><span class="note">Baseline: the five Standalone models trained within the steps of one elastic run</span></td><td class="links"><a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-119M-285M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-119M-285M">119M–285M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-375M-906M" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-375M-906M">375M–906M</a> <a class="scale-link" title="Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B" href="https://huggingface.co/Can111/ShellElastic-baseline-SampledStandalone-675M-1.69B">675M–1.69B</a></td></tr>
+</table></div>
+
+<h2>Data</h2>
+<p><a class="repo" href="https://huggingface.co/datasets/Can111/ShellElastic-data">Can111/ShellElastic-data</a>: the tokenized training and validation data that these runs
+read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
+(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
+text (1.12 GB, 84,999 sequences).</p>
+
+<h2>License</h2>
+<p>The code is released under the <a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE">Apache License 2.0</a>. The checkpoints and the data are
+released under CC BY 4.0; the data also carry the terms of their sources (see the dataset card).</p>
+</main>
+</body>
 </html>