Mauro1455 commited on
Commit
f344c50
·
verified ·
1 Parent(s): 8e4918e

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +62 -0
README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ar
4
+ - zh
5
+ - da
6
+ - en
7
+ - fr
8
+ - ja
9
+ - ko
10
+ - sv
11
+ library_name: pytorch
12
+ pipeline_tag: text-classification
13
+ tags:
14
+ - EDAN20
15
+ - language-identification
16
+ - scikit-learn
17
+ - pytorch
18
+ ---
19
+
20
+ # Nano Language Detector (EDAN20, Lab 4)
21
+
22
+ A small language identifier inspired by Google's CLD3, built for the EDAN20 Language Technology course at Lund University. It recognises eight languages: Arabic (`ara`), Mandarin Chinese (`cmn`), Danish (`dan`), English (`eng`), French (`fra`), Japanese (`jpn`), Korean (`kor`), and Swedish (`swe`).
23
+
24
+ ## Model
25
+
26
+ Each sentence is lowercased and split into character unigrams, bigrams, and trigrams. The n-grams are hashed with MD5 into 521 + 1031 + 1031 = 2583 buckets, and the input vector contains the relative frequency of each bucket. The network has one hidden layer:
27
+
28
+ ```
29
+ input (2583) -> Linear + ReLU (50) -> Linear (8) -> softmax
30
+ ```
31
+
32
+ The same architecture was trained twice: with scikit-learn (`MLPClassifier`, 5 epochs) and with PyTorch (Adam, cross-entropy, batches of 32, 7 epochs).
33
+
34
+ ## Files
35
+
36
+ | File | Content |
37
+ |---|---|
38
+ | `nld.joblib` | scikit-learn `MLPClassifier` |
39
+ | `nld.pth` | PyTorch weights (`state_dict`) |
40
+ | `nld_vectorizer.joblib` | `DictVectorizer` mapping hash buckets to input columns (needed by both models) |
41
+ | `nld_lang_codes.joblib` | Dictionary from class index to language code |
42
+ | `predict.py` | Feature extraction and inference with both models |
43
+
44
+ ## Usage
45
+
46
+ ```bash
47
+ pip install torch scikit-learn joblib huggingface_hub
48
+ python predict.py "Hejsan grabbar!" "Salut les gars !"
49
+ ```
50
+
51
+ The feature extraction in `predict.py` must be used as is: the models only work with the same hashing (MD5) and the same bucket sizes.
52
+
53
+ ## Training data and results
54
+
55
+ Trained on Tatoeba sentences: 25,000 per language (11,674 for Korean), 80 % for training and 20 % (37,335 sentences) for validation.
56
+
57
+ | Model | Accuracy | Macro F1 |
58
+ |---|---|---|
59
+ | scikit-learn | 0.9912 | 0.9916 |
60
+ | PyTorch | 0.9924 | 0.9927 |
61
+
62
+ Most errors are between Swedish and Danish, and very short sentences can be misclassified.