|
Download README.md from Austinyu7/edan20-assignment4-language-detector: direct link, hf CLI and curl.
- Browser
- Download file 1.8 kB
-
https://huggingface.co/Austinyu7/edan20-assignment4-language-detector/resolve/main/README.md
- Command line
-
hf download hf://Austinyu7/edan20-assignment4-language-detector/README.md
-
curl -L -o README.md https://huggingface.co/Austinyu7/edan20-assignment4-language-detector/resolve/main/README.md
1.8 kB
| tags: | |
| - language-identification | |
| - edan20 | |
| - scikit-learn | |
| - pytorch | |
| license: apache-2.0 | |
| language: | |
| - ar | |
| - fr | |
| - zh | |
| - ja | |
| - ko | |
| - en | |
| - sv | |
| - da | |
| # EDAN20 Assignment 4: CLD3-inspired language detector | |
| This repository contains two small neural language classifiers trained on a | |
| balanced subset of Tatoeba. The supported labels are Arabic (`ara`), French | |
| (`fra`), Mandarin Chinese (`cmn`), Japanese (`jpn`), Korean (`kor`), English | |
| (`eng`), Swedish (`swe`), and Danish (`dan`). | |
| Both classifiers use relative frequencies of hashed, lowercased character | |
| unigrams, bigrams, and trigrams. The 2,583-dimensional input is passed through | |
| a 50-unit ReLU hidden layer and an eight-class output layer. This is inspired | |
| by CLD3, but it uses the frequency vector directly instead of learning a dense | |
| embedding for every hashed n-gram ID. | |
| ## Files | |
| - `nld.joblib`: trained scikit-learn `MLPClassifier` | |
| - `nld.pth`: PyTorch `state_dict` for a 2583--50--8 network | |
| - `nld_vectorizer.joblib`: fitted `DictVectorizer` | |
| - `nld_lang_codes.joblib`: mapping from class indices to language codes | |
| - `predict.py`: preprocessing and example inference for both models | |
| ## Validation results | |
| The deterministic split used 24,000 sentences (3,000 per language), with 80% | |
| for training and 20% for validation. | |
| | Implementation | Accuracy | Macro F1 | | |
| |---|---:|---:| | |
| | scikit-learn | 0.9923 | 0.9922 | | |
| | PyTorch | 0.9915 | 0.9914 | | |
| The main remaining errors are Swedish--Danish confusions. Results should not | |
| be interpreted as performance on arbitrary web text: Tatoeba sentences are | |
| short, the eight classes are balanced, and no unseen languages are represented. | |
| ## Example | |
| ```bash | |
| python predict.py "Bonjour tout le monde" | |
| ``` | |
| The artifacts were produced with scikit-learn 1.0.2, PyTorch 2.5.0 (CPU), and | |
| NumPy 1.26.4. | |