Rename the dataset repository to Can111/ShellElastic-data
Browse files
README.md
CHANGED
|
@@ -4,7 +4,7 @@ language:
|
|
| 4 |
- en
|
| 5 |
datasets:
|
| 6 |
- mlfoundations/dclm-baseline-1.0
|
| 7 |
-
- Can111/ShellElastic-
|
| 8 |
library_name: pytorch
|
| 9 |
pipeline_tag: text-generation
|
| 10 |
tags:
|
|
@@ -71,7 +71,7 @@ Fully elastic models, a transformer adaptation of slimmable networks (Yu et al.,
|
|
| 71 |
|
| 72 |
## Training
|
| 73 |
|
| 74 |
-
**Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-
|
| 75 |
|
| 76 |
**Recipe.** Trained as described in the paper (Sections 3.2 and 4.1, Appendices D, E.1 and E.2, and Table 9).
|
| 77 |
|
|
@@ -148,7 +148,7 @@ Apache License 2.0 (`LICENSE`).
|
|
| 148 |
Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
|
| 149 |
the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
|
| 150 |
included in this repository; the tokenized data of these runs are in the dataset repository
|
| 151 |
-
[Can111/ShellElastic-
|
| 152 |
|
| 153 |
## Citation
|
| 154 |
|
|
|
|
| 4 |
- en
|
| 5 |
datasets:
|
| 6 |
- mlfoundations/dclm-baseline-1.0
|
| 7 |
+
- Can111/ShellElastic-data
|
| 8 |
library_name: pytorch
|
| 9 |
pipeline_tag: text-generation
|
| 10 |
tags:
|
|
|
|
| 71 |
|
| 72 |
## Training
|
| 73 |
|
| 74 |
+
**Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data).
|
| 75 |
|
| 76 |
**Recipe.** Trained as described in the paper (Sections 3.2 and 4.1, Appendices D, E.1 and E.2, and Table 9).
|
| 77 |
|
|
|
|
| 148 |
Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
|
| 149 |
the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
|
| 150 |
included in this repository; the tokenized data of these runs are in the dataset repository
|
| 151 |
+
[Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
|
| 152 |
|
| 153 |
## Citation
|
| 154 |
|