Describe short text without reference to an application
Browse files- ARCHITECTURE.md +1 -1
- millid.py +1 -1
ARCHITECTURE.md
CHANGED
|
@@ -204,7 +204,7 @@ flowchart LR
|
|
| 204 |
is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
|
| 205 |
100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
|
| 206 |
- **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
|
| 207 |
-
where the script has them: short text is the hard case, and what a document
|
| 208 |
headers, captions and table cells.
|
| 209 |
- **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
|
| 210 |
should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.
|
|
|
|
| 204 |
is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
|
| 205 |
100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
|
| 206 |
- **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
|
| 207 |
+
where the script has them: short text is the hard case, and what a document holds in its
|
| 208 |
headers, captions and table cells.
|
| 209 |
- **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
|
| 210 |
should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.
|
millid.py
CHANGED
|
@@ -657,7 +657,7 @@ class Sampler:
|
|
| 657 |
A language is drawn with probability proportional to n^alpha, n being its passage count:
|
| 658 |
alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
|
| 659 |
passage of it is drawn, and sometimes cut short, because a short snippet is where
|
| 660 |
-
language identification is hardest and what a
|
| 661 |
"""
|
| 662 |
|
| 663 |
def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):
|
|
|
|
| 657 |
A language is drawn with probability proportional to n^alpha, n being its passage count:
|
| 658 |
alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
|
| 659 |
passage of it is drawn, and sometimes cut short, because a short snippet is where
|
| 660 |
+
language identification is hardest and what a page holds in headers and table cells.
|
| 661 |
"""
|
| 662 |
|
| 663 |
def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):
|