byamasupatrick commited on
Commit
beaf38d
·
verified ·
1 Parent(s): d8cc4d3

Describe short text without reference to an application

Browse files
Files changed (2) hide show
  1. ARCHITECTURE.md +1 -1
  2. millid.py +1 -1
ARCHITECTURE.md CHANGED
@@ -204,7 +204,7 @@ flowchart LR
204
  is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
205
  100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
206
  - **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
207
- where the script has them: short text is the hard case, and what a document pipeline meets in
208
  headers, captions and table cells.
209
  - **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
210
  should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.
 
204
  is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
205
  100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
206
  - **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
207
+ where the script has them: short text is the hard case, and what a document holds in its
208
  headers, captions and table cells.
209
  - **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
210
  should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.
millid.py CHANGED
@@ -657,7 +657,7 @@ class Sampler:
657
  A language is drawn with probability proportional to n^alpha, n being its passage count:
658
  alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
659
  passage of it is drawn, and sometimes cut short, because a short snippet is where
660
- language identification is hardest and what a document pipeline sees in table cells.
661
  """
662
 
663
  def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):
 
657
  A language is drawn with probability proportional to n^alpha, n being its passage count:
658
  alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
659
  passage of it is drawn, and sometimes cut short, because a short snippet is where
660
+ language identification is hardest and what a page holds in headers and table cells.
661
  """
662
 
663
  def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):