Card: header and introduction of ShellElastic

README.md

@@ -15,9 +15,36 @@ tags:
 - supernet
 ---
 
-# Fully elastic baseline, 119M–285M scale (ShellElastic paper, Table 2)
+<div align="center">
 
-Final weights of the Fully elastic baseline in Table 2 of *ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer* (Jin et al.; Rutgers University and Adobe Research), at the 119M–285M scale. These weights are a baseline trained for this paper, not an official release of slimmable networks (Yu et al., 2018), of which Fully elastic is a transformer adaptation. Fully elastic nests the residual, attention and FFN widths by slicing shared dense matrices (Section 4.1), and the paper trains it with µ-Elastic (Section 3.2 and Appendix D). `model.safetensors` holds the supernet with its five sub-networks (Low, Medium, High, Extra, Max); `modeling_elastic.py` runs any of them and extracts it as a standalone dense model.
+# ShellElastic
+
+### Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
+
+**Can Jin**<sup>1*</sup> · **Yan Kang**<sup>2</sup> · **Yifan Gong**<sup>2</sup> · **Chongjian Ge**<sup>2</sup> · **Mingze Xu**<sup>2</sup> · **Yuanjun Xiong**<sup>2</sup> · **Dimitris N. Metaxas**<sup>1</sup> · **Hongwu Peng**<sup>2</sup>
+
+<sup>1</sup>Rutgers University &emsp; <sup>2</sup>Adobe Research
+
+<sup>*</sup>Corresponding author: [can.jin@rutgers.edu](mailto:can.jin@rutgers.edu). Work done during an internship at Adobe Research.
+
+<a href="https://github.com/jincan333/ShellElastic/tree/main/paper"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
+<a href="http://localhost:8765/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
+<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
+<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
+<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
+<a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-3da639?style=for-the-badge&logo=apache&logoColor=white"></a>
+
+</div>
+
+## About ShellElastic
+
+Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.
+
+µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.
+
+## Fully elastic baseline, 119M–285M scale
+
+Final weights of the Fully elastic baseline at the 119M–285M scale of the paper (Section 4.1). These weights are a baseline trained for this paper, not an official release of slimmable networks (Yu et al., 2018), of which Fully elastic is a transformer adaptation. Fully elastic nests the residual, attention and FFN widths by slicing shared dense matrices (Section 4.1), and the paper trains it with µ-Elastic (Section 3.2 and Appendix D). `model.safetensors` holds the supernet with its five sub-networks (Low, Medium, High, Extra, Max); `modeling_elastic.py` runs any of them and extracts it as a standalone dense model.
 
 ## Sub-networks
 
@@ -33,7 +60,7 @@ Parameter counts include the five learned readout scalars α_s; rounded to 0.1M,
 
 ## Results
 
-| Sub-network | Val. loss, this checkpoint | Val. loss, Table 2 | 5-shot acc. (%), this checkpoint | 5-shot acc. (%), Table 2 |
+| Sub-network | Val. loss, this checkpoint | Val. loss, paper | 5-shot acc. (%), this checkpoint | 5-shot acc. (%), paper |
 |---|---|---|---|---|
 | Low | 3.4298 | 3.429 | 34.6 | 34.9 |
 | Medium | 3.3549 | 3.354 | 36.0 | 35.5 |
@@ -42,7 +69,7 @@ Parameter counts include the five learned readout scalars α_s; rounded to 0.1M,
 | Max | 3.2914 | 3.290 | 36.4 | 36.0 |
 | Avg. | 3.3379 | 3.336 | 35.9 | 35.6 |
 
-*This checkpoint*: the final validation loss of each sub-network of these weights (per-token cross-entropy in nats on held-out C4 text, at step 25,000) and its mean 5-shot accuracy over the 13 downstream tasks of Appendix F. *Table 2*: the paper's entries in the Fully elastic row at the 119M–285M scale. *Avg.*: the mean over the five sub-networks. Appendix G of the paper gives the per-task accuracies.
+*This checkpoint*: the final validation loss of each sub-network of these weights (per-token cross-entropy in nats on held-out C4 text, at step 25,000) and its mean 5-shot accuracy over the 13 downstream tasks of Appendix F. *Paper*: the main results of the paper for Fully elastic at the 119M–285M scale (Table 2). *Avg.*: the mean over the five sub-networks. Appendix G of the paper gives the per-task accuracies.
 
 ## Architecture