Spaces:
Running
Running
Landing page: paper PDF and project page links, no license, new Figure 1
Browse files- README.md +0 -1
- figure1.png +2 -2
- index.html +10 -10
README.md
CHANGED
|
@@ -5,7 +5,6 @@ colorFrom: blue
|
|
| 5 |
colorTo: yellow
|
| 6 |
sdk: static
|
| 7 |
pinned: true
|
| 8 |
-
license: apache-2.0
|
| 9 |
short_description: "Scalable elastic pretraining: checkpoints and data"
|
| 10 |
---
|
| 11 |
|
|
|
|
| 5 |
colorTo: yellow
|
| 6 |
sdk: static
|
| 7 |
pinned: true
|
|
|
|
| 8 |
short_description: "Scalable elastic pretraining: checkpoints and data"
|
| 9 |
---
|
| 10 |
|
figure1.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
index.html
CHANGED
|
@@ -14,13 +14,15 @@
|
|
| 14 |
* { box-sizing: border-box; }
|
| 15 |
body { margin: 0; background: var(--bg); color: var(--fg);
|
| 16 |
font: 16px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif; }
|
| 17 |
-
main { max-width: 980px; margin: 0 auto; padding:
|
|
|
|
| 18 |
a { color: var(--link); text-decoration: none; }
|
| 19 |
a:hover { text-decoration: underline; }
|
| 20 |
header { text-align: center; }
|
| 21 |
h1 { font-size: 2.6em; margin: 0.2em 0 0.3em; padding-bottom: 0.25em; border-bottom: 1px solid var(--line); }
|
| 22 |
.subtitle { font-size: 1.55em; font-weight: 600; line-height: 1.3; margin: 0.6em 0; }
|
| 23 |
-
.authors { font-size:
|
|
|
|
| 24 |
.authors b { font-weight: 600; }
|
| 25 |
.author { white-space: nowrap; }
|
| 26 |
.affil, .corr { margin: 0.4em 0; }
|
|
@@ -45,23 +47,24 @@ code, .repo { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Conso
|
|
| 45 |
</style>
|
| 46 |
</head>
|
| 47 |
<body>
|
| 48 |
-
<
|
| 49 |
<header>
|
| 50 |
<h1>ShellElastic</h1>
|
| 51 |
-
<div class="subtitle">Scalable Elastic Pretraining
|
| 52 |
<p class="authors"><span class="author"><b>Can Jin</b><sup>1*</sup></span> 路 <span class="author"><b>Yan Kang</b><sup>2</sup></span> 路 <span class="author"><b>Yifan Gong</b><sup>2</sup></span> 路 <span class="author"><b>Chongjian Ge</b><sup>2</sup></span> 路 <span class="author"><b>Mingze Xu</b><sup>2</sup></span> 路 <span class="author"><b>Yuanjun Xiong</b><sup>2</sup></span> 路 <span class="author"><b>Dimitris N. Metaxas</b><sup>1</sup></span> 路 <span class="author"><b>Hongwu Peng</b><sup>2</sup></span></p>
|
| 53 |
<p class="affil"><sup>1</sup>Rutgers University   <sup>2</sup>Adobe Research</p>
|
| 54 |
<p class="corr"><sup>*</sup>Corresponding author: <a href="mailto:can.jin@rutgers.edu">can.jin@rutgers.edu</a>. Work done during an
|
| 55 |
internship at Adobe Research.</p>
|
| 56 |
<div class="badges">
|
| 57 |
-
<a href="https://github.com/jincan333/ShellElastic/
|
| 58 |
-
<a href="
|
| 59 |
<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
|
| 60 |
<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
| 61 |
<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
| 62 |
-
<a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-3da639?style=for-the-badge&logo=apache&logoColor=white"></a>
|
| 63 |
</div>
|
| 64 |
</header>
|
|
|
|
|
|
|
| 65 |
|
| 66 |
<h2>About ShellElastic</h2>
|
| 67 |
<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
|
|
@@ -90,9 +93,6 @@ read, in the layout that the code reads. Training: 3,136 shards of DCLM-Baseline
|
|
| 90 |
(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
|
| 91 |
text (1.12 GB, 84,999 sequences).</p>
|
| 92 |
|
| 93 |
-
<h2>License</h2>
|
| 94 |
-
<p>The code is released under the <a href="https://github.com/jincan333/ShellElastic/blob/main/LICENSE">Apache License 2.0</a>. The checkpoints and the data are
|
| 95 |
-
released under CC BY 4.0; the data also carry the terms of their sources (see the dataset card).</p>
|
| 96 |
</main>
|
| 97 |
</body>
|
| 98 |
</html>
|
|
|
|
| 14 |
* { box-sizing: border-box; }
|
| 15 |
body { margin: 0; background: var(--bg); color: var(--fg);
|
| 16 |
font: 16px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif; }
|
| 17 |
+
main { max-width: 980px; margin: 0 auto; padding: 8px 16px 56px; }
|
| 18 |
+
.top { max-width: 1140px; margin: 0 auto; padding: 32px 16px 0; }
|
| 19 |
a { color: var(--link); text-decoration: none; }
|
| 20 |
a:hover { text-decoration: underline; }
|
| 21 |
header { text-align: center; }
|
| 22 |
h1 { font-size: 2.6em; margin: 0.2em 0 0.3em; padding-bottom: 0.25em; border-bottom: 1px solid var(--line); }
|
| 23 |
.subtitle { font-size: 1.55em; font-weight: 600; line-height: 1.3; margin: 0.6em 0; }
|
| 24 |
+
.authors { font-size: min(16px, calc((100vw - 32px) * 0.0147)); white-space: nowrap; margin: 0.9em 0 0.4em; }
|
| 25 |
+
@media (max-width: 900px) { .authors { font-size: 15px; white-space: normal; } }
|
| 26 |
.authors b { font-weight: 600; }
|
| 27 |
.author { white-space: nowrap; }
|
| 28 |
.affil, .corr { margin: 0.4em 0; }
|
|
|
|
| 47 |
</style>
|
| 48 |
</head>
|
| 49 |
<body>
|
| 50 |
+
<div class="top">
|
| 51 |
<header>
|
| 52 |
<h1>ShellElastic</h1>
|
| 53 |
+
<div class="subtitle">Scalable Elastic Pretraining<br>with Simultaneous Maximal Updates and Hyperparameter Transfer</div>
|
| 54 |
<p class="authors"><span class="author"><b>Can Jin</b><sup>1*</sup></span> 路 <span class="author"><b>Yan Kang</b><sup>2</sup></span> 路 <span class="author"><b>Yifan Gong</b><sup>2</sup></span> 路 <span class="author"><b>Chongjian Ge</b><sup>2</sup></span> 路 <span class="author"><b>Mingze Xu</b><sup>2</sup></span> 路 <span class="author"><b>Yuanjun Xiong</b><sup>2</sup></span> 路 <span class="author"><b>Dimitris N. Metaxas</b><sup>1</sup></span> 路 <span class="author"><b>Hongwu Peng</b><sup>2</sup></span></p>
|
| 55 |
<p class="affil"><sup>1</sup>Rutgers University   <sup>2</sup>Adobe Research</p>
|
| 56 |
<p class="corr"><sup>*</sup>Corresponding author: <a href="mailto:can.jin@rutgers.edu">can.jin@rutgers.edu</a>. Work done during an
|
| 57 |
internship at Adobe Research.</p>
|
| 58 |
<div class="badges">
|
| 59 |
+
<a href="https://github.com/jincan333/ShellElastic/blob/main/paper/ShellElastic.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-PDF-b31b1b?style=for-the-badge"></a>
|
| 60 |
+
<a href="https://jincan333.github.io/ShellElastic_page"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2f6db5?style=for-the-badge"></a>
|
| 61 |
<a href="https://github.com/jincan333/ShellElastic"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white"></a>
|
| 62 |
<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Checkpoints" src="https://img.shields.io/badge/Checkpoints-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
| 63 |
<a href="https://huggingface.co/collections/Can111/shellelastic-checkpoints-and-data-6ac41e3a763d5a3fd0e79111"><img alt="Datasets" src="https://img.shields.io/badge/Datasets-Hugging%20Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=white"></a>
|
|
|
|
| 64 |
</div>
|
| 65 |
</header>
|
| 66 |
+
</div>
|
| 67 |
+
<main>
|
| 68 |
|
| 69 |
<h2>About ShellElastic</h2>
|
| 70 |
<figure><img src="figure1.png" alt="Figure 1 of the paper: the ShellElastic architecture"></figure>
|
|
|
|
| 93 |
(141.32 GB, 25,690,112 sequences of 2,049 tokens); validation: 11 shards of held-out C4
|
| 94 |
text (1.12 GB, 84,999 sequences).</p>
|
| 95 |
|
|
|
|
|
|
|
|
|
|
| 96 |
</main>
|
| 97 |
</body>
|
| 98 |
</html>
|