File size: 2,040 Bytes
f344c50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
---
language:
- ar
- zh
- da
- en
- fr
- ja
- ko
- sv
library_name: pytorch
pipeline_tag: text-classification
tags:
- EDAN20
- language-identification
- scikit-learn
- pytorch
---

# Nano Language Detector (EDAN20, Lab 4)

A small language identifier inspired by Google's CLD3, built for the EDAN20 Language Technology course at Lund University. It recognises eight languages: Arabic (`ara`), Mandarin Chinese (`cmn`), Danish (`dan`), English (`eng`), French (`fra`), Japanese (`jpn`), Korean (`kor`), and Swedish (`swe`).

## Model

Each sentence is lowercased and split into character unigrams, bigrams, and trigrams. The n-grams are hashed with MD5 into 521 + 1031 + 1031 = 2583 buckets, and the input vector contains the relative frequency of each bucket. The network has one hidden layer:

```
input (2583) -> Linear + ReLU (50) -> Linear (8) -> softmax
```

The same architecture was trained twice: with scikit-learn (`MLPClassifier`, 5 epochs) and with PyTorch (Adam, cross-entropy, batches of 32, 7 epochs).

## Files

| File | Content |
|---|---|
| `nld.joblib` | scikit-learn `MLPClassifier` |
| `nld.pth` | PyTorch weights (`state_dict`) |
| `nld_vectorizer.joblib` | `DictVectorizer` mapping hash buckets to input columns (needed by both models) |
| `nld_lang_codes.joblib` | Dictionary from class index to language code |
| `predict.py` | Feature extraction and inference with both models |

## Usage

```bash
pip install torch scikit-learn joblib huggingface_hub
python predict.py "Hejsan grabbar!" "Salut les gars !"
```

The feature extraction in `predict.py` must be used as is: the models only work with the same hashing (MD5) and the same bucket sizes.

## Training data and results

Trained on Tatoeba sentences: 25,000 per language (11,674 for Korean), 80 % for training and 20 % (37,335 sentences) for validation.

| Model | Accuracy | Macro F1 |
|---|---|---|
| scikit-learn | 0.9912 | 0.9916 |
| PyTorch | 0.9924 | 0.9927 |

Most errors are between Swedish and Danish, and very short sentences can be misclassified.