werner1hugo commited on
Commit
b7ba0db
·
verified ·
1 Parent(s): fddf6af

Add model card

Browse files
Files changed (1) hide show
  1. README.md +168 -0
README.md ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ library_name: fasttext
4
+ pipeline_tag: text-classification
5
+ tags:
6
+ - language-identification
7
+ - short-text
8
+ - fasttext
9
+ - scikit-learn
10
+ - model-selection
11
+ language:
12
+ - ca
13
+ - da
14
+ - de
15
+ - el
16
+ - en
17
+ - es
18
+ - fi
19
+ - fr
20
+ - hr
21
+ - it
22
+ - ja
23
+ - ko
24
+ - lt
25
+ - mk
26
+ - nb
27
+ - nl
28
+ - pl
29
+ - pt
30
+ - ro
31
+ - ru
32
+ - sl
33
+ - sv
34
+ - uk
35
+ - zh
36
+ ---
37
+
38
+ # LID framework: trained candidate models
39
+
40
+ These are the 102 trained language identification (LID) models that populate the
41
+ Meta-Knowledge Base of the LID model selection framework
42
+ ([GitHub: Whugo226/lid_framework](https://github.com/Whugo226/lid_framework)),
43
+ developed for the MEng thesis *An Interactive Framework for Language
44
+ Identification Model Recommendation* (Werner Hugo, Stellenbosch University, 2026).
45
+
46
+ The framework profiles an unlabelled short-text corpus and recommends one of
47
+ these models (or one of three off-the-shelf detectors). Its dashboard downloads
48
+ only the model it is asked to run, so you do not need to download this whole
49
+ repository.
50
+
51
+ ## Contents
52
+
53
+ Six configurations were trained on each of 17 benchmark corpora
54
+ (6 × 17 = 102 models):
55
+
56
+ | Configuration | Family | Features | File |
57
+ |---|---|---|---|
58
+ | `bow_char_ngram_3_5` | Multinomial Naive Bayes | character 3–5-grams, raw counts | `best_pipeline.pkl` |
59
+ | `tfidf_char_ngram_3_5` | Multinomial Naive Bayes | character 3–5-grams, TF-IDF | `best_pipeline.pkl` |
60
+ | `bow_maxabs_lr_char_ngram_3_5` | Logistic regression | character 3–5-grams, max-abs scaled counts | `best_pipeline.pkl` |
61
+ | `tfidf_lr_char_ngram_3_5` | Logistic regression | character 3–5-grams, TF-IDF | `best_pipeline.pkl` |
62
+ | `fasttext_word` | FastText | word unigrams, no subwords | `model.bin` |
63
+ | `fasttext_subword` | FastText | word bigrams + character subwords (2–4) | `model.bin` |
64
+
65
+ Files are laid out as `{family}/{training_corpus}/{configuration}/`, where
66
+ `{family}` is `naive_bayes`, `logistic_regression` or `fasttext`. Each folder
67
+ also holds `run_metadata.json` (selected hyperparameters and training-split
68
+ metrics), `classification_report.txt`, and the hyperparameter search results.
69
+
70
+ Every model predicts ISO 639-1 codes for the subset of the 24 languages below
71
+ that its training corpus contains. The FastText models are large (up to about
72
+ 2.3 GB each); the Naive Bayes and logistic regression pipelines are 1–63 MB.
73
+
74
+ ## Usage
75
+
76
+ With the framework installed, the dashboard's *Run model* tab downloads and runs
77
+ a model on demand. Programmatically:
78
+
79
+ ```python
80
+ from lid_toolkit.explainer.model_runner import ModelRunner
81
+
82
+ # store_datasets: the training-corpus names the variant names may end in
83
+ runner = ModelRunner(store_datasets=["xnli"])
84
+ runner.predict("fasttext_subword_xnli", ["Dit is een korte zin."]) # downloads on first use
85
+ ```
86
+
87
+ Without the framework, the files load directly:
88
+
89
+ ```python
90
+ from huggingface_hub import hf_hub_download
91
+ import fasttext, pickle
92
+
93
+ ft = fasttext.load_model(hf_hub_download(
94
+ "werner1hugo/lid-framework-models", "fasttext/xnli/fasttext_subword/model.bin"))
95
+ ft.predict("Dit is een korte zin.")
96
+
97
+ with open(hf_hub_download(
98
+ "werner1hugo/lid-framework-models",
99
+ "logistic_regression/xnli/tfidf_lr_char_ngram_3_5/best_pipeline.pkl"), "rb") as fh:
100
+ lr = pickle.load(fh)
101
+ lr.predict(["Dit is een korte zin."])
102
+ ```
103
+
104
+ The scikit-learn pipelines were saved with scikit-learn 1.8.0. The framework pins
105
+ 1.7.2 (the version its Meta-Knowledge Base was built with); under 1.7.2 the
106
+ pipelines load and predict correctly but scikit-learn prints an
107
+ `InconsistentVersionWarning`. Only unpickle files from sources you trust.
108
+
109
+ ## Training
110
+
111
+ - Every source document was truncated at a word boundary to at most 120
112
+ characters (40 for Chinese, Japanese and Korean); documents shorter than 30
113
+ characters (10 for CJK) were discarded.
114
+ - Each corpus was split into four disjoint row-level parts under a fixed seed:
115
+ 50% knowledge-training, 20% knowledge-benchmarking, 15% evaluation and
116
+ 15% validation. Models were trained on the knowledge-training split only.
117
+ - Hyperparameters were selected by macro-F1 on a stratified 80/20 split of the
118
+ training data: randomised search with 5-fold (Naive Bayes) or 3-fold
119
+ (logistic regression) cross-validation, and a grid over epochs and learning
120
+ rate on a 90/10 split for FastText.
121
+
122
+ ## Performance and limitations
123
+
124
+ In-domain (evaluated on the same corpus they were trained on), mean weighted F1
125
+ ranges from 0.939 (FastText word) to 0.995 (FastText subword). Cross-domain
126
+ performance collapses to a mean of about 0.55–0.61, with worst cases below 0.06:
127
+ a model trained on one corpus can be unusable on another. That is why the
128
+ framework recommends a model per corpus rather than one model for everything.
129
+
130
+ - Only 24 high-resource languages are covered; any other language is misclassified.
131
+ - The models were trained and evaluated on short text (30–120 characters).
132
+ - They assign one language per document and do not handle code-switching.
133
+
134
+ ## Training corpora and licences
135
+
136
+ These models are derived from the corpora below. They are released under
137
+ **CC BY-NC-SA 4.0** (non-commercial, share-alike), the most restrictive terms
138
+ among the corpora with stated licences. Use of an individual model is also
139
+ subject to the terms of the corpus it was trained on. Licences are as stated on
140
+ each dataset card at the time of release; "not stated" means the card gives none.
141
+
142
+ | Training corpus (folder name) | Hugging Face dataset | Languages | Licence on dataset card |
143
+ |---|---|---|---|
144
+ | `OpenLID-v2` | laurievb/OpenLID-v2 | 23 | other (gated) |
145
+ | `amazon_reviews_multi` | neonwatty/amazon_reviews_multi | 6 | not stated |
146
+ | `europarl` | Helsinki-NLP/europarl | 13 | unknown |
147
+ | `exorde-social-media-december-2024-week1` | Exorde/exorde-social-media-december-2024-week1 | 24 | MIT |
148
+ | `flores_plus` | openlanguagedata/flores_plus | 23 | CC BY-SA 4.0 (gated) |
149
+ | `language-identification` | papluca/language-identification | 12 | not stated |
150
+ | `massive` | AmazonScience/massive | 18 | CC BY 4.0 |
151
+ | `mmarco` | unicamp-dl/mmarco | 10 | not stated |
152
+ | `multi_eurlex` | coastalcph/multi_eurlex | 16 | CC BY-SA 4.0 |
153
+ | `multilingual_cc_news` | hotchpotch/multilingual_cc_news | 21 | not stated |
154
+ | `multilingual_toxicity_dataset` | textdetox/multilingual_toxicity_dataset | 9 | OpenRAIL++ |
155
+ | `stsb_multi_mt` | PhilipMay/stsb_multi_mt | 10 | other |
156
+ | `tweet_sentiment_multilingual` | cardiffnlp/tweet_sentiment_multilingual | 6 | not stated |
157
+ | `tydiqa` | google-research-datasets/tydiqa | 5 | Apache 2.0 |
158
+ | `wikipedia` | wikimedia/wikipedia | 24 | CC BY-SA 3.0, GFDL |
159
+ | `xlsum` | csebuetnlp/xlsum | 9 | CC BY-NC-SA 4.0 |
160
+ | `xnli` | facebook/xnli | 7 | not stated |
161
+
162
+ *Languages* counts the in-scope languages each corpus contains after filtering
163
+ to the 24 supported languages.
164
+
165
+ ## Citation
166
+
167
+ If you use these models, please cite the thesis and the software archive (see
168
+ `CITATION.cff` in the GitHub repository).