README.md

---
license: apache-2.0
language:
  - id
  - sw
  - tr
  - vi
library_name: fasttext
tags:
  - fasttext
  - text-classification
  - data-quality
  - pretraining-data
  - multilingual
---

# SynRank quality classifiers

fastText classifiers that score a document by how much it looks like clean,
knowledge-dense text in its language. From the ACL 2026 paper
[How Can Synthetic Data Improve Multilingual Language Model Pretraining? A Data Quality Perspective](https://aclanthology.org/2026.acl-long.1002/).

Point one at a raw web corpus, keep the top-scoring documents, and you get a
pretraining corpus that, in the paper's controlled comparison, matches what
human experts produce with hand-written rules on knowledge benchmarks
(XARC-E/XARC-C). You do not need an expert who speaks the language.

| File | Language |
| --- | --- |
| `synrank_id.ftz` | Indonesian |
| `synrank_sw.ftz` | Swahili |
| `synrank_tr.ftz` | Turkish |
| `synrank_vi.ftz` | Vietnamese |

Each is the **round-1** model — the second iteration, which has seen real
high-scoring web text as well as synthetic positives. Round 0 has only seen
machine-translated positives and over-selects for translationese.
On 2026-09-28, each packaged file was checked against the original training
outputs: its complete serialized arguments, dictionary and unquantized output
matrix match `iterative_model_1_epoch.bin`, and differ from rounds 0 and 2.
See the [provenance record](classifier-provenance.md) and
[machine-readable evidence](classifier-provenance.json) for the checks and limits.
Use [SHA256SUMS](SHA256SUMS) to verify downloaded weight files.

Quantized with fastText product quantization for distribution: 7.71 GB → 969 MB
each. Most of the unquantized size is fastText's 2,000,000 hashed n-gram buckets
× 896 dims (≈7.17 GB). The 151,647 × 896 vocabulary embeddings add only
≈0.54 GB. Quantization can change both absolute scores and document rankings.

> **Recompute your threshold.** Use a percentile of the scores produced by the
> model you actually run, and evaluate selection overlap on your corpus before
> substituting a quantized classifier for the original. A cutoff from the
> unquantized model need not transfer.

## Usage

Install the scoring dependencies:

```bash
pip install fasttext transformers huggingface_hub
```

The example downloads the Indonesian classifier from
[`tyzhu/mulsyn_classifiers`](https://huggingface.co/tyzhu/mulsyn_classifiers).
For another language, choose its filename from the table above.

Documents must be encoded the same way they were during training: tokenized with
the Qwen2 tokenizer, with each token id written as the pseudo-word `w<id>`.
Feeding raw text will produce meaningless scores.

```python
import fasttext
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

model_path = hf_hub_download(
    repo_id="tyzhu/mulsyn_classifiers",
    filename="synrank_id.ftz",
)
model = fasttext.load_model(model_path)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B")

def quality_score(text: str) -> float:
    text = " ".join(text.replace("\\n", "\n").split())  # same flattening as mulsyn.io.extract_text
    ids = tokenizer(text, truncation=True, max_length=512)["input_ids"]
    words = " ".join(f"w{i}" for i in ids)
    labels, scores = model.predict([words], k=2)  # list input also works with NumPy 2
    return float(dict(zip(labels[0], scores[0]))["__label__positive"])

print(quality_score("Ilmu pengetahuan membantu kita memahami dunia."))
```

The returned score is the probability of `__label__positive`, even when the
negative label is the most likely prediction. Rank documents within the
classifier's language and choose a percentile cutoff for your corpus.

The paper measures ~980 documents/second on a standard CPU, so filtering a
web-scale corpus is a matter of CPU parallelism rather than GPUs.

## How they were trained

Positives: 10k documents from Cosmopedia machine-translated into the target
language. The paper adds the highest-scoring real documents to the positives
and filters with the round-1 model. With the MULSYN training code's defaults, training
runs three rounds, promoting the top and bottom 10k of a 200k-document
held-out pool into the positive and negative sets each round.

fastText: dim 896 (the Qwen 0.5B hidden size), lr 0.1, 5 epochs, word n-grams
up to 3, min count 3, initialized from Qwen 0.5B input embeddings. The code
defaults to `Qwen/Qwen2-0.5B`. A 2026-09-28 artifact audit found that sampled
rows of the original exported embeddings and trained input matrices match
Qwen2-0.5B; they do not match Qwen2.5-0.5B. Use Qwen2-0.5B for these files,
even though the paper text names Qwen2.5-0.5B. Both have hidden size 896.

Held-out P/R/F1 on 10k synthetic positives + 10k noisy negatives, measured
on the original unquantized models. The packaged files derive from round 1;
these measurements were not rerun on the quantized files.

| Language | Round 0 P / R / F1 | Round 1 (source) P / R / F1 |
| --- | --- | --- |
| Indonesian | 99.50 / 99.80 / 99.65 | 91.52 / 99.95 / 95.55 |
| Swahili | 98.75 / 99.82 / 99.28 | 92.46 / 99.97 / 96.07 |
| Turkish | 98.98 / 99.75 / 99.36 | 91.60 / 99.96 / 95.60 |
| Vietnamese | 99.52 / 99.74 / 99.63 | 92.09 / 99.91 / 95.84 |

The round-0 column matches paper Table 4. The round-1 values match the saved
`val_stats_epoch_1.json` files for the original round-1 models; they are not
in the paper.
Precision here measures agreement with the synthetic-versus-noisy labels.
Lower precision means more noisy-set documents receive a positive prediction;
it does not establish that those documents are fluent or useful for pretraining.

Neither column measures filter quality. The evidence for that is downstream,
from 1.1B models pretrained in six languages. SynRank-filtered MADLAD beats the
unfiltered corpus on XARC-C in all six languages and on XARC-E in five. It
matches human-expert-cleaned corpora on those knowledge benchmarks (XARC-E 35.31
vs 35.32, XARC-C 24.56 vs 24.54), but not on XCOPA or XHellaswag.

## Used in Sailor2

Several of the paper's authors are also authors of
[Sailor2](https://arxiv.org/abs/2502.12982). Sailor2 used a variant of this
recipe to select the high-quality SEA web data in its Stage-2 (annealing)
mixture (report §3.2). NLLB-3.3B
translated high-quality English into each Southeast Asian language. One fastText
classifier per language was trained on 10,000 translated positives (40%
Cosmopedia, 40% MADLAD, 20% UltraChat) and 10,000 random CommonCrawl negatives.
The **top 20%** of each language's CommonCrawl was kept for annealing (the
[blog](https://sea-sailor.github.io/blog/sailor2/#stage-2-synthetic-data-to-retrieve-high-quality-tokens)
says 10–20%). The Sailor2 organization card identifies
[`sailor2/sea-commoncrawl-high-quality`](https://huggingface.co/datasets/sailor2/sea-commoncrawl-high-quality)
as that selection; it has 13 language folders and no Malay, so it does not
match the Stage-2 table exactly.
The models trained on the resulting Stage-2 data are in the
[Sailor2 collection](https://huggingface.co/collections/sail/sailor2-language-models-674d7c9e6b4dbbd9a869906b)
(`sail/Sailor2-1B`, `-8B`, `-20B` and their chat versions).

**These are not Sailor2's classifiers.** Sailor2's classifiers are not public.
The four models here are the paper's, trained with MADLAD-400 negatives.
Indonesian and Vietnamese are Sailor2 languages; Swahili and Turkish are not.

## Limitations

* **Language-specific.** A classifier trained for Indonesian will not
  meaningfully score Turkish.
* **Selects for a style, not for truth.** High-scoring documents are fluent and
  knowledge-dense in form. Fluent falsehoods score well.
* **Trained on MADLAD-400 / CommonCrawl web text.** Scores on very different
  domains (code, chat logs, OCR output) are not calibrated.
* **Quantization can change scores and rankings.** Check its effect on your corpus.

## Citation

```bibtex
@inproceedings{zhu-etal-2026-synthetic,
    title = "How Can Synthetic Data Improve Multilingual Language Model Pretraining? A Data Quality Perspective",
    author = "Zhu, Tongyao and Liu, Qian and Ma, Chang and Zhang, Jinghan and Dou, Longxu and He, Junxian and Chen, Shiqi",
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    year = "2026",
    url = "https://aclanthology.org/2026.acl-long.1002/",
}
```

If you also use the Sailor2 corpora or models, please cite Sailor2:

```bibtex
@misc{dou2025sailor2sailingsoutheastasia,
      title={Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs},
      author={Longxu Dou and Qian Liu and Fan Zhou and Changyu Chen and Zili Wang and Ziqi Jin and Zichen Liu and Tongyao Zhu and Cunxiao Du and Penghui Yang and Haonan Wang and Jiaheng Liu and Yongchi Zhao and Xiachong Feng and Xin Mao and Man Tsung Yeung and Kunat Pipatanakul and Fajri Koto and Min Si Thu and Hynek Kydl{\'\i}{\v{c}}ek and Zeyi Liu and Qunshu Lin and Sittipong Sripaisarnmongkol and Kridtaphad Sae-Khow and Nirattisai Thongchim and Taechawat Konkaew and Narong Borijindargoon and Anh Dao and Matichon Maneegard and Phakphum Artkaew and Zheng-Xin Yong and Quan Nguyen and Wannaphong Phatthiyaphaibun and Hoang H. Tran and Mike Zhang and Shiqi Chen and Tianyu Pang and Chao Du and Xinyi Wan and Wei Lu and Min Lin},
      year={2025},
      eprint={2502.12982},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.12982},
}
```