Card: component shares of the training split
Browse files
README.md
CHANGED
|
@@ -62,7 +62,8 @@ Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a9
|
|
| 62 |
1 880 520 445 tokens available, of which 1 059 × 1 048 576 were consumed.
|
| 63 |
|
| 64 |
Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens
|
| 65 |
-
the pack-01 stream is web 52.
|
|
|
|
| 66 |
Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
|
| 67 |
`[DOWOD]`); masking does not guarantee removal of every personal datum.
|
| 68 |
|
|
|
|
| 62 |
1 880 520 445 tokens available, of which 1 059 × 1 048 576 were consumed.
|
| 63 |
|
| 64 |
Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens
|
| 65 |
+
the training split (`train.bin`) of the pack-01 stream is web 52.16 %, core 22.14 %, legal 8.20 %, English 17.49 %
|
| 66 |
+
(pack-00 stream: 51.79 / 22.19 / 8.46 / 17.56).
|
| 67 |
Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
|
| 68 |
`[DOWOD]`); masking does not guarantee removal of every personal datum.
|
| 69 |
|