Can111 commited on
Commit
a05a7aa
·
verified ·
1 Parent(s): 54d3e70

Rename the dataset repository to Can111/ShellElastic-data

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -4,7 +4,7 @@ language:
4
  - en
5
  datasets:
6
  - mlfoundations/dclm-baseline-1.0
7
- - Can111/ShellElastic-pretraining-data
8
  library_name: pytorch
9
  pipeline_tag: text-generation
10
  tags:
@@ -71,7 +71,7 @@ Fully elastic models, a transformer adaptation of slimmable networks (Yu et al.,
71
 
72
  ## Training
73
 
74
- **Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-pretraining-data](https://huggingface.co/datasets/Can111/ShellElastic-pretraining-data).
75
 
76
  **Recipe.** Trained as described in the paper (Sections 3.2 and 4.1, Appendices D, E.1 and E.2, and Table 9).
77
 
@@ -148,7 +148,7 @@ Apache License 2.0 (`LICENSE`).
148
  Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
149
  the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
150
  included in this repository; the tokenized data of these runs are in the dataset repository
151
- [Can111/ShellElastic-pretraining-data](https://huggingface.co/datasets/Can111/ShellElastic-pretraining-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
152
 
153
  ## Citation
154
 
 
4
  - en
5
  datasets:
6
  - mlfoundations/dclm-baseline-1.0
7
+ - Can111/ShellElastic-data
8
  library_name: pytorch
9
  pipeline_tag: text-generation
10
  tags:
 
71
 
72
  ## Training
73
 
74
+ **Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data).
75
 
76
  **Recipe.** Trained as described in the paper (Sections 3.2 and 4.1, Appendices D, E.1 and E.2, and Table 9).
77
 
 
148
  Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
149
  the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
150
  included in this repository; the tokenized data of these runs are in the dataset repository
151
+ [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
152
 
153
  ## Citation
154