Improve model card and add citation

README.md

@@ -14,79 +14,82 @@ tags:
 
 # BashkirRoBERTa
 
-BashkirRoBERTa is a 50M-parameter masked language model for Bashkir.
-It predicts a missing token from both sides of its context. The release includes
-the original 16,384-piece SentencePiece BPE tokenizer and an exact Pre-LayerNorm
-Transformer implementation.
+A masked language model for Bashkir. Given a sentence with one `[MASK]` token,
+it predicts the most probable missing Bashkir token from its context. The model
+is useful for fill-mask experiments and as a starting point for further
+fine-tuning.
 
-It is intended for fill-mask experiments and further fine-tuning. It is not a
-chat or text-generation model.
+## Examples
 
-## Quick start
+| Input | Top prediction |
+| --- | --- |
+| `Башҡортостан — беҙҙең [MASK].` | `республика` |
+| `Мин башҡорт телен [MASK].` | `яратам` |
+| `Өфө — ҙур [MASK].` | `ҡала` |
+| `Бөгөн Өфөлә яңы [MASK] асылды.` | `мәктәп` |
 
-```python
-from transformers import AutoModelForMaskedLM, AutoTokenizer
-
-repo_id = "failed09/bashkir-roberta"
-tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
-model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)
-
-inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
-logits = model(**inputs).logits
-mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
-top_id = logits[0, mask_index].argmax().item()
-print(tokenizer.decode([top_id]))  # яратам
-```
-
-`trust_remote_code=True` is required because this checkpoint uses a custom
-Pre-LayerNorm architecture, not the post-LayerNorm implementation in the stock
-`RobertaForMaskedLM` class.
-
-## Architecture
+## Model
 
 | Property | Value |
-|---|---:|
-| Transformer blocks | 8, Pre-LayerNorm |
+| --- | --- |
+| Task | Masked language modelling / fill-mask |
+| Architecture | Pre-LayerNorm Transformer encoder |
+| Transformer blocks | 8 |
 | Hidden size / attention heads | 640 / 10 |
 | Feed-forward size | 2,560 |
 | Context window | 256 subword tokens |
-| Vocabulary | SentencePiece BPE, 16,384 tokens |
 | Parameters | 50.04M |
-| MLM head | tied to input embeddings |
+| Tokenizer | SentencePiece BPE, 16,384 tokens |
 
-Special IDs are fixed: `<pad>` 0, `<unk>` 1, `<s>` 2, `</s>` 3, `[CLS]` 4,
-`[SEP]` 5, and `[MASK]` 6.
+The output embedding matrix is tied to the input word embeddings. Token IDs are
+fixed: `<pad>` 0, `<unk>` 1, `<s>` 2, `</s>` 3, `[CLS]` 4, `[SEP]` 5, and
+`[MASK]` 6.
 
 ## Training and evaluation
 
 The model was pretrained with dynamic masked-language modelling on a Bashkir
 text collection assembled from encyclopedic, periodical, and literary sources.
-The source texts are not distributed in this repository. Public Bashkir
-Wikipedia is one component; the remaining source materials may carry separate
-terms. This release contains only model weights, code, and tokenizer assets.
+The source texts are not distributed in this repository. On a held-out Bashkir
+encyclopedic evaluation set, the project reports 24.7% top-1 and 54.0% top-5
+accuracy for masked subword prediction.
 
-On a held-out Bashkir encyclopedic evaluation set, the project reports 24.7%
-top-1 and 54.0% top-5 accuracy for masked subword prediction. These results are
-diagnostic MLM measurements, not a general-purpose language understanding score.
+These are diagnostic MLM results, not a general-purpose language-understanding
+score. A mask may represent a whole word or a SentencePiece subword fragment.
 
-## Limitations
+## Loading
 
-- A mask corresponds to one SentencePiece token, which may be a full word or a
-  subword fragment.
-- The model can reflect errors, source bias, and rare memorized sequences from
-  its training material. Do not use it to reconstruct non-public source text.
-- It has not been contrastively trained for semantic retrieval. Do not treat raw
-  MLM outputs as sentence embeddings.
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+repo_id = "failed09/bashkir-roberta"
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)
+
+inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
+logits = model(**inputs).logits
+mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
+prediction_id = logits[0, mask_index].argmax().item()
+print(tokenizer.decode([prediction_id]))  # яратам
+```
+
+`trust_remote_code=True` is required: the model preserves the original custom
+Pre-LayerNorm architecture rather than using the stock post-LayerNorm RoBERTa
+implementation.
 
 ## License and provenance
 
-The checkpoint is released under the repository's **custom terms** (`other` on
-the Hub) while the source-rights audit is completed. No training texts are
-redistributed. Please contact the maintainer through the Hub for provenance or
-removal requests before commercial redistribution.
+The checkpoint is released under custom terms (`other` on the Hub) while the
+source-rights audit is completed. No training texts are redistributed. For
+provenance or removal requests, please contact the maintainer through the Hub.
 
-## Reproducibility
+## Citation
 
-The original checkpoint was converted to `model.safetensors` without
-retraining. The included exporter verifies that the converted model produces
-exactly the same logits as the original checkpoint for a fixed inference input.
+```bibtex
+@model{failed09_bashkir_roberta_2026,
+  title     = {BashkirRoBERTa},
+  author    = {failed09},
+  year      = {2026},
+  publisher = {Hugging Face},
+  url       = {https://huggingface.co/failed09/bashkir-roberta}
+}
+```