| ---
|
| language:
|
| - ba
|
| license: apache-2.0
|
| pretty_name: Bashkir LID (Binary BA vs NON_BA)
|
| library_name: onnxruntime
|
| pipeline_tag: text-classification
|
| tags:
|
| - bashkir
|
| - language-identification
|
| - text-classification
|
| - onnx
|
| - onnxruntime
|
| - low-resource
|
| ---
|
|
|
| # Bashkir LID (Binary BA vs NON_BA)
|
|
|
| > Compact ONNX language gate that decides whether text is Bashkir, for corpus
|
| > filtering and local CPU inference.
|
|
|
| ## Overview
|
|
|
| A compact binary language gate for Bashkir (`ba` vs `non_ba`), designed for
|
| corpus filtering, OCR triage and local applications that need lightweight CPU
|
| inference. It was trained with hard negatives from sister Turkic languages,
|
| Russian and technical/noisy text, with special attention to Bashkir/Tatar
|
| separation.
|
|
|
| | At a glance | |
|
| | --- | --- |
|
| | Task | Binary language identification (`ba` vs `non_ba`) |
|
| | Default artifact | `model.onnx` + `lid.py` |
|
| | Source | Bashkir, Turkic, Russian and technical text (not redistributed) |
|
| | Version / license | v3 / Apache-2.0 |
|
|
|
| ## Contents
|
|
|
| ### Files and Configurations
|
|
|
| | File | Purpose |
|
| | --- | --- |
|
| | `model.onnx` | Exported classifier graph and weights (opset 17) |
|
| | `vectorizer.json` | Dual-channel character and word feature vocabularies |
|
| | `lid.py` | Portable ONNX Runtime adapter |
|
| | `config.json` | Runtime contract |
|
| | `META.json` | Model passport, validation results and artifact hashes |
|
| | `__init__.py` | Package marker |
|
| | `LICENSE` | Full license text |
|
| | `SHA256SUMS` | Release checksums |
|
|
|
| Training texts and the original `joblib` pipeline are not distributed.
|
|
|
| ### Model Architecture
|
|
|
| | Property | Description |
|
| | --- | --- |
|
| | Task | Binary language identification: `ba` or `non_ba` |
|
| | Features | Dual-channel TF-IDF: `char_wb` (2–5) + `word` (1–2), 240,000 features |
|
| | Classifier | Linear SGD classifier exported to ONNX with Sigmoid normalization |
|
| | Runtime | ONNX Runtime on CPU |
|
| | Preprocessing | Unicode lowercase and dual-channel sparse features, provided by `lid.py` |
|
| | Outputs | Class labels (`ba`, `non_ba`) and probabilities |
|
|
|
| The ONNX graph takes sparse feature tensors (`char_ids`, `char_counts`,
|
| `word_ids`, `word_counts`); the adapter handles preprocessing, batching and
|
| checksum validation. No custom ONNX operators or Transformers installation are
|
| required.
|
|
|
| ### Examples
|
|
|
| | Text | Prediction | `ba` probability |
|
| | --- | --- | ---: |
|
| | `Мин башҡорт телен яратам.` | `ba` | 0.998 |
|
| | `Бары бала бара.` | `ba` | 0.927 |
|
| | `Сегодня хорошая погода.` | `non_ba` | 0.044 |
|
| | `SELECT id FROM users WHERE active = TRUE;` | `non_ba` | 0.011 |
|
|
|
| Note the second row: valid Bashkir text does not always contain
|
| Bashkir-specific letters.
|
|
|
| ## Method
|
|
|
| The model is a dual-channel linear classifier: TF-IDF `char_wb` (2–5) and word
|
| (1–2) features are fused and passed to an SGD classifier exported to ONNX. It was
|
| trained with hard negatives from sister Turkic languages, Russian and
|
| technical/noisy text. Training texts are not distributed in this repository.
|
|
|
| ```text
|
| training data (Bashkir + hard negative Turkic/Russian/noise)
|
| ├── 1. extract character n-grams: char_wb (2–5)
|
| ├── 2. extract word n-grams: words (1–2)
|
| ├── 3. feature union & TF-IDF vectorization
|
| ├── 4. train SGDClassifier with log-loss
|
| └── 5. export calibrated ONNX graph (fast single-pass classification)
|
| ```
|
|
|
| ### Evaluation
|
|
|
| On a weak-label 4,000-sentence benchmark the model reached **93.25% binary
|
| accuracy**: at the default 0.50 threshold it retained 730/1,000 Bashkir examples
|
| and rejected 3,000/3,000 non-Bashkir examples. A hand-authored adversarial suite
|
| passed 26/26 (Tatar, Kazakh, Chuvash, Russian with Bashkir inserts, code and
|
| noise), and an independent 100-example stress set reached 97/100. Labels are weak
|
| or manually constructed, not an expert-reviewed Gold benchmark; exact numbers are
|
| recorded in [`META.json`](META.json).
|
|
|
| ### Operating thresholds
|
|
|
| `predict()` selects `ba` when its probability is at least 0.50. Measured
|
| operating points (from [`META.json`](META.json)):
|
|
|
| | BA probability threshold | BA recall | Non-BA false acceptance |
|
| | ---: | ---: | ---: |
|
| | 0.50 | 73.0% (730/1,000) | 0/3,000 |
|
| | 0.40 | 78.3% (783/1,000) | 2/3,000 |
|
| | 0.35 | 80.8% (808/1,000) | 4/3,000 |
|
|
|
| Use `0.50` for the strictest rejection of non-Bashkir text. Use `0.35–0.40` when
|
| preserving more Bashkir text matters, preferably with dictionary and quality
|
| checks. The price of the strict gate is false rejection of some valid Bashkir
|
| sentences, especially short or BA/Tatar-like sentences.
|
|
|
| ## Quality and Use
|
|
|
| This is a binary gate, not a general multilingual language detector. It does not
|
| identify which non-BA language was found. Tatar and other languages, short
|
| strings, names, OCR artifacts and mixed-language text can be ambiguous.
|
|
|
| ### Limitations
|
|
|
| - Binary decision only: `ba` vs `non_ba`, with no language label for rejects.
|
| - Weak-label benchmark; not an expert-reviewed Gold evaluation.
|
| - Confidence values are model scores, not calibrated human probabilities.
|
| - Empty strings, OCR fragments and technical noise should be handled with
|
| explicit pipeline rules.
|
| - For high-recall corpus construction, combine with the multiclass LID,
|
| dictionaries, quality rules and human review.
|
|
|
| ## Related Resources
|
|
|
| - [Bashkir Multiclass LID](https://huggingface.co/failed09/bashkir-lid-multiclass) —
|
| distinguishes `ba`, `tt`, `ru` and `other`; use it when you need the language
|
| label instead of a pass/reject gate.
|
|
|
| ## Usage
|
|
|
| ```bash
|
| pip install huggingface_hub onnxruntime numpy
|
| ```
|
|
|
| ```python
|
| import sys
|
| from huggingface_hub import snapshot_download
|
|
|
| model_dir = snapshot_download(
|
| "failed09/bashkir-lid",
|
| allow_patterns=[
|
| "lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
|
| ],
|
| revision="main", # pin a reviewed commit for reproducible deployments
|
| )
|
| sys.path.insert(0, model_dir)
|
| from lid import LanguageIdentifier
|
|
|
| lid = LanguageIdentifier(model_dir=model_dir)
|
| texts = ["Мин башҡорт телен яратам.", "Сегодня хорошая погода."]
|
| print(lid.predict(texts).tolist())
|
| print(lid.predict_proba(texts))
|
| print(lid.classes.tolist()) # probability-column order
|
| ```
|
|
|
| After downloading, inference runs locally without network access. For a local
|
| checkout with the model files alongside `lid.py`, `LanguageIdentifier()` also
|
| works without extra arguments.
|
|
|
| ## License
|
|
|
| The code and model export are released under the
|
| [Apache-2.0 license](https://huggingface.co/failed09/bashkir-lid/blob/main/LICENSE). Training texts
|
| are not included in this repository.
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @software{failed09_bashkir_lid_2026,
|
| title = {Bashkir LID},
|
| author = {failed09},
|
| year = {2026},
|
| publisher = {Hugging Face},
|
| url = {https://huggingface.co/failed09/bashkir-lid},
|
| note = {Compact binary Bashkir language gate}
|
| }
|
| ```
|
|
|
| ## Open Bashkir Data and Sources 🐝
|
|
|
| This release is part of an open-source effort to support the development,
|
| preservation and practical use of the Bashkir language. Other related models,
|
| datasets and tools are available on the author's Hugging Face profile.
|
|
|
| The author does not claim ownership or authorship of the source texts or other
|
| materials used to derive this release; rights and licensing remain with the
|
| original authors, publishers and dataset providers. Source texts are not
|
| redistributed in this repository, so users should follow the licenses and
|
| attribution requirements of the relevant upstream resources.
|
|
|
