Austinyu7's picture
Add trained language detector models
eb587ad verified
|
Raw History Blame Contribute Delete
1.8 kB
---
tags:
- language-identification
- edan20
- scikit-learn
- pytorch
license: apache-2.0
language:
- ar
- fr
- zh
- ja
- ko
- en
- sv
- da
---
# EDAN20 Assignment 4: CLD3-inspired language detector
This repository contains two small neural language classifiers trained on a
balanced subset of Tatoeba. The supported labels are Arabic (`ara`), French
(`fra`), Mandarin Chinese (`cmn`), Japanese (`jpn`), Korean (`kor`), English
(`eng`), Swedish (`swe`), and Danish (`dan`).
Both classifiers use relative frequencies of hashed, lowercased character
unigrams, bigrams, and trigrams. The 2,583-dimensional input is passed through
a 50-unit ReLU hidden layer and an eight-class output layer. This is inspired
by CLD3, but it uses the frequency vector directly instead of learning a dense
embedding for every hashed n-gram ID.
## Files
- `nld.joblib`: trained scikit-learn `MLPClassifier`
- `nld.pth`: PyTorch `state_dict` for a 2583--50--8 network
- `nld_vectorizer.joblib`: fitted `DictVectorizer`
- `nld_lang_codes.joblib`: mapping from class indices to language codes
- `predict.py`: preprocessing and example inference for both models
## Validation results
The deterministic split used 24,000 sentences (3,000 per language), with 80%
for training and 20% for validation.
| Implementation | Accuracy | Macro F1 |
|---|---:|---:|
| scikit-learn | 0.9923 | 0.9922 |
| PyTorch | 0.9915 | 0.9914 |
The main remaining errors are Swedish--Danish confusions. Results should not
be interpreted as performance on arbitrary web text: Tatoeba sentences are
short, the eight classes are balanced, and no unseen languages are represented.
## Example
```bash
python predict.py "Bonjour tout le monde"
```
The artifacts were produced with scikit-learn 1.0.2, PyTorch 2.5.0 (CPU), and
NumPy 1.26.4.