Card: header and introduction of ShellElastic

README.md

@@ -13,9 +13,36 @@ tags:
 - supernet
 ---
 
-# ShellElastic, 675M–1.69B scale (Table 2)
+<div align="center">
 
-Final weights of the ShellElastic supernet in Table 2 of *ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer* (Jin et al.; Rutgers University and Adobe Research), at the 675M–1.69B scale. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular, and one full-width forward pass contains the exact hidden states of every sub-network (Section 3.1 and Proposition 1). `model.safetensors` holds the supernet with its five sub-networks (Low, Medium, High, Extra, Max); `modeling_shellelastic.py` runs any of them and extracts it as a standalone model (Section 3.3).
+# ShellElastic
+
+### Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
+
+**Can Jin**<sup>1*</sup> · **Yan Kang**<sup>2</sup> · **Yifan Gong**<sup>2</sup> · **Chongjian Ge**<sup>2</sup> · **Mingze Xu**<sup>2</sup> · **Yuanjun Xiong**<sup>2</sup> · **Dimitris N. Metaxas**<sup>1</sup> · **Hongwu Peng**<sup>2</sup>
+
+<sup>1</sup>Rutgers University &emsp; <sup>2</sup>Adobe Research
+
+<sup>*</sup>Corresponding author: [can.jin@rutgers.edu](mailto:can.jin@rutgers.edu). Work done during an internship at Adobe Research.
+
+<a href="https://github.com/jincan333/ShellElastic/tree/main/paper"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
+<a href="http://localhost:8765/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
+<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
+<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
+<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
+<a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-3da639?style=for-the-badge&logo=apache&logoColor=white"></a>
+
+</div>
+
+## About ShellElastic
+
+Foundation models need to support diverse inference budgets. Elastic models train several nested sub-networks inside one weight-shared supernet. ShellElastic is an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains the exact hidden states of every sub-network, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone.
+
+µ-Elastic extends maximal update parameterization (µP) to ShellElastic and other elastic architectures. Its scaling rules keep all sub-networks of a supernet maximally updated simultaneously, and hyperparameters tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets. A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.
+
+## ShellElastic supernet, 675M–1.69B scale
+
+Final weights of the ShellElastic supernet at the 675M–1.69B scale of the paper (Section 4.1). `model.safetensors` holds the supernet with its five sub-networks (Low, Medium, High, Extra, Max); `modeling_shellelastic.py` runs any of them and extracts it as a standalone model (Section 3.3).
 
 ## Sub-networks
 
@@ -31,7 +58,7 @@ Parameter counts include the five learned readout scalars α_s; rounded to 0.1M,
 
 ## Results
 
-| Sub-network | Val. loss, this checkpoint | Val. loss, Table 2 | 5-shot acc. (%), this checkpoint | 5-shot acc. (%), Table 2 |
+| Sub-network | Val. loss, this checkpoint | Val. loss, paper | 5-shot acc. (%), this checkpoint | 5-shot acc. (%), paper |
 |---|---|---|---|---|
 | Low | 2.9135 | 2.914 | 43.7 | 43.7 |
 | Medium | 2.8657 | 2.866 | 43.7 | 43.7 |
@@ -40,7 +67,7 @@ Parameter counts include the five learned readout scalars α_s; rounded to 0.1M,
 | Max | 2.7959 | 2.796 | 44.5 | 44.5 |
 | Avg. | 2.8447 | 2.845 | 44.0 | 44.0 |
 
-*This checkpoint*: the final validation loss of each sub-network of these weights (per-token cross-entropy in nats on held-out C4 text, at step 100,000) and its mean 5-shot accuracy over the 13 downstream tasks of Appendix F. *Table 2*: the paper's entries in the ShellElastic row at the 675M–1.69B scale. *Avg.*: the mean over the five sub-networks. Appendix G of the paper gives the per-task accuracies.
+*This checkpoint*: the final validation loss of each sub-network of these weights (per-token cross-entropy in nats on held-out C4 text, at step 100,000) and its mean 5-shot accuracy over the 13 downstream tasks of Appendix F. *Paper*: the main results of the paper for ShellElastic at the 675M–1.69B scale (Table 2). *Avg.*: the mean over the five sub-networks. Appendix G of the paper gives the per-task accuracies.
 
 ## Architecture