Card: unified structure, real examples, license link

README.md

@@ -22,50 +22,118 @@ tags:
 
 # Bashkir Multiclass LID
 
-A compact ONNX language-identification model for Bashkir corpus filtering and
-local CPU applications. The model predicts four classes:
+> Compact ONNX language identification for `ba`, `tt`, `ru` and `other`, for
+> Bashkir corpus filtering and local CPU inference.
 
-- `ba` — Bashkir;
-- `tt` — Tatar;
-- `ru` — Russian;
-- `other` — other languages, code, URLs, technical noise and corrupted text.
+## Overview
 
-The model is intended to separate closely related Bashkir and Tatar text while
-also preventing foreign-language and technical material from being silently
+A compact ONNX language-identification model for Bashkir corpus filtering and
+local CPU applications. It predicts four classes: `ba` (Bashkir), `tt` (Tatar),
+`ru` (Russian) and `other` (other languages, code, URLs, technical noise and
+corrupted text). It is designed to separate closely related Bashkir and Tatar
+text while preventing foreign-language and technical material from being silently
 classified as Bashkir.
 
-## Model Architecture
-
-| Property | Description |
+| At a glance | |
 | --- | --- |
 | Task | Four-class language identification |
+| Default artifact | `model.onnx` + `lid.py` |
 | Classes | `ba`, `tt`, `ru`, `other` |
-| Character features | TF-IDF `char_wb` n-grams, lengths 2–5 |
-| Word features | TF-IDF word n-grams, lengths 1–2 |
-| Classifier | `SGDClassifier(loss="log_loss")` |
-| Export | ONNX, opset 17 |
-| Runtime | ONNX Runtime on CPU |
-| Input handling | Unicode-aware adapter in `lid.py` |
+| Source | Bashkir, Tatar, Russian and technical text (not redistributed) |
+| Version / license | v3 / Apache-2.0 |
 
-The ONNX graph consumes sparse feature tensors rather than raw strings. The
-provided adapter performs tokenization, feature lookup, batching and checksum
-validation. No custom ONNX operators or Transformers installation are needed.
+## Contents
 
-## Files
+### Files and Configurations
 
 | File | Purpose |
 | --- | --- |
-| `model.onnx` | Exported classifier graph and weights |
+| `model.onnx` | Exported classifier graph and weights (opset 17) |
 | `vectorizer.json` | Character and word feature vocabularies |
 | `lid.py` | Portable ONNX Runtime adapter |
 | `config.json` | Runtime contract and feature settings |
-| `META.json` | Model passport, validation and artifact hashes |
+| `META.json` | Model passport, validation results and artifact hashes |
 | `requirements.txt` | Minimal runtime dependencies |
 | `SHA256SUMS` | Release checksums |
 
 Training texts and the original `joblib` pipeline are not distributed.
 
-## Loading
+### Model Architecture
+
+| Property | Description |
+| --- | --- |
+| Task | Four-class language identification |
+| Classes | `ba`, `tt`, `ru`, `other` |
+| Character features | TF-IDF `char_wb` n-grams, lengths 2–5 |
+| Word features | TF-IDF word n-grams, lengths 1–2 |
+| Classifier | `SGDClassifier(loss="log_loss")` |
+| Export | ONNX, opset 17 |
+| Runtime | ONNX Runtime on CPU |
+| Input handling | Unicode-aware adapter in `lid.py` |
+
+The ONNX graph consumes sparse feature tensors rather than raw strings; the
+adapter performs tokenization, feature lookup, batching and checksum validation.
+No custom ONNX operators or Transformers installation are needed.
+
+### Examples
+
+| Text | Prediction | Top probability |
+| --- | --- | ---: |
+| `Бары бала бара.` | `ba` | 0.829 |
+| `Бүген Казанда яңа мәдәни үзәк ачылды.` | `tt` | 0.919 |
+| `В Уфе открыли новый корпус университета.` | `ru` | 0.930 |
+| `SELECT id FROM users WHERE active = TRUE;` | `other` | 0.982 |
+
+## Method
+
+The model is a dual-channel linear classifier: TF-IDF `char_wb` (2–5) and word
+(1–2) features are fused and passed to an `SGDClassifier` exported to ONNX. It is
+trained on Bashkir, Tatar, Russian and other-language material with technical
+noise. Training texts are not distributed in this repository.
+
+### Evaluation
+
+The clean multi-source benchmark contains 1,000 examples each from Bashkir,
+Tatar, Russian and other-language/noise sources; the adversarial set includes
+closely related Bashkir/Tatar pairs, code-switching, Kazakh, Chuvash, Russian
+regional text and technical noise.
+
+| Evaluation | Result |
+| --- | ---: |
+| Clean multi-source benchmark | 92.67% accuracy (3,707/4,000) |
+| Bashkir subset | 81.70% recall (817/1,000) |
+| Tatar subset | 100.00% recall (1,000/1,000) |
+| Russian subset | 91.40% recall (914/1,000) |
+| Other subset | 97.60% recall (976/1,000) |
+| Linguistic adversarial stress test | 39/39 (100%) |
+| ONNX / sklearn label parity | 0 disagreements |
+
+The benchmark is not expert-reviewed Gold and some examples come from the same
+source pools used during model development; source overlap was not fully
+excluded. Exact validation fields are recorded in [`META.json`](META.json).
+
+## Quality and Use
+
+Mixed-language text does not have a separate class: predictions should be read as
+the dominant language of the sentence. In a corpus pipeline, confident Bashkir can
+be retained, while uncertain BA/TT or mixed cases should be sent to a review queue
+instead of being deleted automatically. Short sentences with common Turkic
+vocabulary remain intrinsically ambiguous.
+
+### Limitations
+
+- A lightweight text classifier, not a universal language detector.
+- The reported benchmark is not an expert-reviewed Gold benchmark.
+- Confidence values are model scores, not calibrated human probabilities.
+- Empty strings, OCR fragments and technical noise should be handled with explicit pipeline rules.
+
+## Related Resources
+
+- [Bashkir LID (Binary BA vs NON_BA)](https://huggingface.co/failed09/bashkir-lid) —
+  a strict `ba` vs `non_ba` gate with configurable thresholds; use it when only a
+  pass/reject decision is needed.
+
+## Usage
 
 ```bash
 pip install huggingface_hub onnxruntime numpy
@@ -78,9 +146,9 @@ from huggingface_hub import snapshot_download
 model_dir = snapshot_download(
     "failed09/bashkir-lid-multiclass",
     allow_patterns=[
-        "lid.py", "config.json", "META.json", "model.onnx",
-        "vectorizer.json"
+        "lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
     ],
+    revision="main",  # pin a reviewed commit for reproducible deployments
 )
 sys.path.insert(0, model_dir)
 from lid import LanguageIdentifier
@@ -98,62 +166,14 @@ print(lid.predict_proba(texts))
 print(lid.classes.tolist())  # probability-column order
 ```
 
-For reproducible deployments, pin a reviewed Hub revision in
-`snapshot_download`. After downloading, inference runs locally without a
-network connection. The probability-column order is always the order in
-`META.json`.
-
-## Related Binary Gate
-
-Use this multilingual model when you need to distinguish between `ba`, `tt`,
-`ru` and `other`. If the task is instead a strict Bashkir corpus gate, use the
-companion [Bashkir Binary LID](https://huggingface.co/failed09/bashkir-lid),
-which predicts only `ba` versus `non_ba` and documents configurable thresholds.
-
-## Evaluation Summary
-
-| Evaluation | Result |
-| --- | ---: |
-| Clean multi-source benchmark | 92.67% accuracy (3,707/4,000) |
-| Bashkir subset | 81.70% recall (817/1,000) |
-| Tatar subset | 100.00% recall (1,000/1,000) |
-| Russian subset | 91.40% recall (914/1,000) |
-| Other subset | 97.60% recall (976/1,000) |
-| Linguistic adversarial stress test | 39/39 (100%) |
-| ONNX/sklearn label parity | 0 disagreements |
-
-The clean benchmark contains 1,000 examples each from Bashkir, Tatar,
-Russian and other-language/noise sources. The adversarial set includes
-closely related Bashkir/Tatar pairs, code-switching, Kazakh, Chuvash, Russian
-regional text and technical noise.
-
-## Interpretation and Mixed Text
-
-Mixed-language text does not have a separate class. Predictions should be
-interpreted as the dominant language of the sentence. In a corpus pipeline,
-confident Bashkir can be retained, while uncertain BA/TT or mixed cases should
-be sent to a review queue instead of being deleted automatically.
-
-Short sentences with common Turkic vocabulary remain intrinsically ambiguous.
-This is expected for Bashkir and Tatar, especially in border regions and
-informal online writing.
-
-## Limitations
-
-- This is a lightweight text classifier, not a universal language detector.
-- The reported benchmark is not an expert-reviewed Gold benchmark.
-- Some benchmark examples come from the same source pools used during model
-  development; source overlap was not fully excluded.
-- The adversarial test is small and hand-authored.
-- Confidence values are model scores, not calibrated human probabilities.
-- Empty strings, OCR fragments and technical noise should be handled with
-  explicit pipeline rules.
+After downloading, inference runs locally without network access. The
+probability-column order is always the order in [`META.json`](META.json).
 
 ## License
 
 Distributed under the [Apache-2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
-This release contains derived model and tokenizer assets, not the source
-corpus, scans or original document layouts.
+This release contains derived model and tokenizer assets, not the source corpus,
+scans or original document layouts.
 
 ## Citation
 
@@ -170,12 +190,12 @@ corpus, scans or original document layouts.
 
 ## Open Bashkir Data and Sources 🐝
 
-This model is part of an open-source effort to support the development,
+This release is part of an open-source effort to support the development,
 preservation and practical use of the Bashkir language. Other related models,
 datasets and tools are available on the author's Hugging Face profile.
 
 The author does not claim ownership or authorship of the source texts or other
-training materials used to derive this model. Rights and licensing remain with
-the original authors, publishers and dataset providers. Source texts are not
-redistributed in this repository; users should review and follow the licenses
-and attribution requirements of the relevant upstream resources.
+materials used to derive this release; rights and licensing remain with the
+original authors, publishers and dataset providers. Source texts are not
+redistributed in this repository, so users should follow the licenses and
+attribution requirements of the relevant upstream resources.

SHA256SUMS

@@ -1,7 +1,7 @@
+ee51fe10152186b053dcaf9ee9799b6bc09c6cce568926c323cd43bd4bd4436d  META.json
+8530a32e22a33c26d73e678d223f605e8967c3d78cab71f6197d76a00dcda3b9  README.md
 1c9f27d5d0e56968f472417b8258a4188d3df8c3bbde0fefc6f1c4d3510a58dd  config.json
 a64200ae65f0d6449500ec35d4c74a5ed0c43e10902886add7aa3ef406a5724c  lid.py
-ee51fe10152186b053dcaf9ee9799b6bc09c6cce568926c323cd43bd4bd4436d  META.json
 09d04d16bfd319da0bd931af0ce9311472fee8a9b75e7fd67c0a31a03e90fa07  model.onnx
-8783337bcc753eb92de9ec446ccc3fa83cbe4caf13f29b0917b2a2972b67302d  README.md
 0eb4ce313d9058b8cf7e5b358f96615d7df224c897e08904333956349cbd1785  requirements.txt
 9e2d3fd13186274867f06b2b2be02e6c769dc5b445621e676a76666f00265828  vectorizer.json