hyunwoongko commited on
Commit
2aebddf
·
1 Parent(s): c750458

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -3
README.md CHANGED
@@ -31,7 +31,7 @@ dimensions of each head. The model is trained with a tokenization vocabulary of
31
 
32
  ## Training data
33
 
34
- Polyglot-Ko was trained on 863 GB of Korean language data (1.2TB before processing), a large-scale dataset curated by [TUNiB](https://tunib.ai/). This dataset, while primarily Korean, does include other languages and scripts such as English and Hanja.
35
 
36
  | Source |Size (GB) | Link |
37
  |-------------------------------------|---------|------------------------------------------|
@@ -51,8 +51,6 @@ Polyglot-Ko was trained on 863 GB of Korean language data (1.2TB before processi
51
  | AIHub various tasks datasets | < 1.0 |aihub.or.kr/ |
52
  | Standard Korean language dictionary | < 1.0 | stdict.korean.go.kr/main/main.do |
53
 
54
- This dataset was collected for the purpose of training Polyglot-Ko models. The dataset will not be released for public use. Our data collection process has abided by South Korean laws.
55
-
56
  Furthermore, in order to avoid the model memorizing and generating personally identifiable information (PII) in the training data, we masked out the following sensitive information in the pre-processing stage:
57
 
58
  * `<|acc|>` : bank account number
 
31
 
32
  ## Training data
33
 
34
+ Polyglot-Ko was trained on 863 GB of Korean language data (1.2TB before processing), a large-scale dataset curated by [TUNiB](https://tunib.ai/), and the data collection process has abided by South Korean laws. This dataset was collected for the purpose of training Polyglot-Ko models, so it will not be released for public use.
35
 
36
  | Source |Size (GB) | Link |
37
  |-------------------------------------|---------|------------------------------------------|
 
51
  | AIHub various tasks datasets | < 1.0 |aihub.or.kr/ |
52
  | Standard Korean language dictionary | < 1.0 | stdict.korean.go.kr/main/main.do |
53
 
 
 
54
  Furthermore, in order to avoid the model memorizing and generating personally identifiable information (PII) in the training data, we masked out the following sensitive information in the pre-processing stage:
55
 
56
  * `<|acc|>` : bank account number