Upload README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- ar
|
| 4 |
+
- zh
|
| 5 |
+
- da
|
| 6 |
+
- en
|
| 7 |
+
- fr
|
| 8 |
+
- ja
|
| 9 |
+
- ko
|
| 10 |
+
- sv
|
| 11 |
+
library_name: pytorch
|
| 12 |
+
pipeline_tag: text-classification
|
| 13 |
+
tags:
|
| 14 |
+
- EDAN20
|
| 15 |
+
- language-identification
|
| 16 |
+
- scikit-learn
|
| 17 |
+
- pytorch
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
# Nano Language Detector (EDAN20, Lab 4)
|
| 21 |
+
|
| 22 |
+
A small language identifier inspired by Google's CLD3, built for the EDAN20 Language Technology course at Lund University. It recognises eight languages: Arabic (`ara`), Mandarin Chinese (`cmn`), Danish (`dan`), English (`eng`), French (`fra`), Japanese (`jpn`), Korean (`kor`), and Swedish (`swe`).
|
| 23 |
+
|
| 24 |
+
## Model
|
| 25 |
+
|
| 26 |
+
Each sentence is lowercased and split into character unigrams, bigrams, and trigrams. The n-grams are hashed with MD5 into 521 + 1031 + 1031 = 2583 buckets, and the input vector contains the relative frequency of each bucket. The network has one hidden layer:
|
| 27 |
+
|
| 28 |
+
```
|
| 29 |
+
input (2583) -> Linear + ReLU (50) -> Linear (8) -> softmax
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
The same architecture was trained twice: with scikit-learn (`MLPClassifier`, 5 epochs) and with PyTorch (Adam, cross-entropy, batches of 32, 7 epochs).
|
| 33 |
+
|
| 34 |
+
## Files
|
| 35 |
+
|
| 36 |
+
| File | Content |
|
| 37 |
+
|---|---|
|
| 38 |
+
| `nld.joblib` | scikit-learn `MLPClassifier` |
|
| 39 |
+
| `nld.pth` | PyTorch weights (`state_dict`) |
|
| 40 |
+
| `nld_vectorizer.joblib` | `DictVectorizer` mapping hash buckets to input columns (needed by both models) |
|
| 41 |
+
| `nld_lang_codes.joblib` | Dictionary from class index to language code |
|
| 42 |
+
| `predict.py` | Feature extraction and inference with both models |
|
| 43 |
+
|
| 44 |
+
## Usage
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
pip install torch scikit-learn joblib huggingface_hub
|
| 48 |
+
python predict.py "Hejsan grabbar!" "Salut les gars !"
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
The feature extraction in `predict.py` must be used as is: the models only work with the same hashing (MD5) and the same bucket sizes.
|
| 52 |
+
|
| 53 |
+
## Training data and results
|
| 54 |
+
|
| 55 |
+
Trained on Tatoeba sentences: 25,000 per language (11,674 for Korean), 80 % for training and 20 % (37,335 sentences) for validation.
|
| 56 |
+
|
| 57 |
+
| Model | Accuracy | Macro F1 |
|
| 58 |
+
|---|---|---|
|
| 59 |
+
| scikit-learn | 0.9912 | 0.9916 |
|
| 60 |
+
| PyTorch | 0.9924 | 0.9927 |
|
| 61 |
+
|
| 62 |
+
Most errors are between Swedish and Danish, and very short sentences can be misclassified.
|