Fill-Mask
Transformers
ONNX
Safetensors
Bashkir
bashkir-roberta-preln
bashkir
masked-language-modeling
roberta
sentencepiece
custom-code
onnxruntime
custom_code
Instructions to use failed09/bashkir-roberta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use failed09/bashkir-roberta with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="failed09/bashkir-roberta", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("failed09/bashkir-roberta", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs: format Method pipeline as tree
Browse files
README.md
CHANGED
|
@@ -17,20 +17,20 @@ tags:
|
|
| 17 |
|
| 18 |
# BashkirRoBERTa
|
| 19 |
|
| 20 |
-
> A Bashkir masked language model for fill-mask suggestions and further fine-tuning.
|
| 21 |
|
| 22 |
## Overview
|
| 23 |
|
| 24 |
-
BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide
|
| 25 |
-
fill-mask suggestions and serve as an encoder for further fine-tuning, including
|
| 26 |
-
the related person-name model. The release includes Transformers weights and FP32,
|
| 27 |
-
FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the
|
| 28 |
-
included model code when loaded with Transformers.
|
| 29 |
|
| 30 |
| At a glance | |
|
| 31 |
| --- | --- |
|
| 32 |
| Task | Masked language modelling / fill-mask |
|
| 33 |
-
| Default artifact | `model.safetensors` (Transformers); `onnx/model_int8.onnx` (CPU ONNX) |
|
| 34 |
| Source | A monolingual Bashkir-language dataset |
|
| 35 |
| Version / license | v1 / Apache-2.0 |
|
| 36 |
|
|
@@ -40,10 +40,10 @@ included model code when loaded with Transformers.
|
|
| 40 |
|
| 41 |
| File | Purpose | Size |
|
| 42 |
| --- | --- | ---: |
|
| 43 |
-
| **`model.safetensors`** | Transformers weights and fine-tuning starting point | 200.2 MB |
|
| 44 |
-
| `onnx/model_fp32.onnx` | Full-precision ONNX reference for CPU inference | 242.2 MB |
|
| 45 |
-
| `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
|
| 46 |
-
| **`onnx/model_int8.onnx`** | Compact INT8 ONNX graph for CPU use | 60.9 MB |
|
| 47 |
| `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
|
| 48 |
| `config.json` | Model configuration (`auto_map` for custom code) | — |
|
| 49 |
| `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
|
|
@@ -82,45 +82,48 @@ Outputs from the INT8 ONNX model on CPU:
|
|
| 82 |
|
| 83 |
## Method
|
| 84 |
|
| 85 |
-
The model was pretrained with dynamic masked-language modelling on a monolingual
|
| 86 |
-
Bashkir-language dataset assembled from encyclopedic, periodical and literary
|
| 87 |
-
sources. The source texts are not distributed in this repository.
|
| 88 |
-
|
| 89 |
-
```text
|
| 90 |
-
Bashkir text
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
| 95 |
|
| 96 |
On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
|
| 97 |
top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
|
| 98 |
-
diagnostic MLM results, not a general-purpose language-understanding score: a mask
|
| 99 |
-
may represent a whole word or a SentencePiece subword fragment.
|
| 100 |
-
The FP32 ONNX graph was checked against the source PyTorch model on multiple
|
| 101 |
-
sequence lengths and batch sizes, with matching top-token predictions.
|
| 102 |
|
| 103 |
## Quality and Use
|
| 104 |
|
| 105 |
-
Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the
|
| 106 |
-
INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision
|
| 107 |
-
ONNX reference is needed. Predictions are token suggestions that need review in
|
| 108 |
-
context, especially when the mask represents only part of a word.
|
| 109 |
|
| 110 |
### Limitations
|
| 111 |
|
| 112 |
- Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
|
| 113 |
- A mask may correspond to a partial subword, not always a full word.
|
| 114 |
- Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
|
| 115 |
-
- No training texts are redistributed; provenance or removal requests go through the maintainer.
|
| 116 |
-
|
| 117 |
-
## Related Resources
|
| 118 |
-
|
| 119 |
-
- [BashkirRoBERTa NER](https://huggingface.co/failed09/bashkir-roberta-ner)
|
| 120 |
-
fine-tunes this encoder to find person names and return their text spans. Use it
|
| 121 |
-
when the task is person-name extraction rather than masked-token prediction.
|
| 122 |
-
|
| 123 |
-
## Usage
|
| 124 |
|
| 125 |
```bash
|
| 126 |
pip install transformers torch huggingface_hub
|
|
@@ -143,7 +146,7 @@ prediction_id = logits[0, mask_index].argmax().item()
|
|
| 143 |
print(tokenizer.decode([prediction_id])) # яратам
|
| 144 |
```
|
| 145 |
|
| 146 |
-
ONNX Runtime for CPU and edge deployment:
|
| 147 |
|
| 148 |
```python
|
| 149 |
import numpy as np
|
|
@@ -161,16 +164,16 @@ tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(
|
|
| 161 |
mask_idx = tokens.index(6)
|
| 162 |
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
|
| 163 |
top_tokens = np.argsort(logits)[::-1][:5]
|
| 164 |
-
print([sp.decode([int(t)]) for t in top_tokens]) # ['яратам', 'беләм', 'өйрәнә', ...]
|
| 165 |
-
```
|
| 166 |
-
|
| 167 |
-
For full-precision ONNX inference, change the downloaded filename to
|
| 168 |
-
`onnx/model_fp32.onnx`; the input and output names are the same.
|
| 169 |
|
| 170 |
## License
|
| 171 |
|
| 172 |
The model weights, tokenizer and release code are distributed under the
|
| 173 |
-
[Apache-2.0 license](https://huggingface.co/failed09/bashkir-roberta/blob/main/LICENSE). Training texts are not redistributed; their
|
| 174 |
rights remain with their respective owners. For provenance or removal requests,
|
| 175 |
contact the maintainer through the Hub.
|
| 176 |
|
|
|
|
| 17 |
|
| 18 |
# BashkirRoBERTa
|
| 19 |
|
| 20 |
+
> A Bashkir masked language model for fill-mask suggestions and further fine-tuning.
|
| 21 |
|
| 22 |
## Overview
|
| 23 |
|
| 24 |
+
BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide
|
| 25 |
+
fill-mask suggestions and serve as an encoder for further fine-tuning, including
|
| 26 |
+
the related person-name model. The release includes Transformers weights and FP32,
|
| 27 |
+
FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the
|
| 28 |
+
included model code when loaded with Transformers.
|
| 29 |
|
| 30 |
| At a glance | |
|
| 31 |
| --- | --- |
|
| 32 |
| Task | Masked language modelling / fill-mask |
|
| 33 |
+
| Default artifact | `model.safetensors` (Transformers); `onnx/model_int8.onnx` (CPU ONNX) |
|
| 34 |
| Source | A monolingual Bashkir-language dataset |
|
| 35 |
| Version / license | v1 / Apache-2.0 |
|
| 36 |
|
|
|
|
| 40 |
|
| 41 |
| File | Purpose | Size |
|
| 42 |
| --- | --- | ---: |
|
| 43 |
+
| **`model.safetensors`** | Transformers weights and fine-tuning starting point | 200.2 MB |
|
| 44 |
+
| `onnx/model_fp32.onnx` | Full-precision ONNX reference for CPU inference | 242.2 MB |
|
| 45 |
+
| `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
|
| 46 |
+
| **`onnx/model_int8.onnx`** | Compact INT8 ONNX graph for CPU use | 60.9 MB |
|
| 47 |
| `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
|
| 48 |
| `config.json` | Model configuration (`auto_map` for custom code) | — |
|
| 49 |
| `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
|
|
|
|
| 82 |
|
| 83 |
## Method
|
| 84 |
|
| 85 |
+
The model was pretrained with dynamic masked-language modelling on a monolingual
|
| 86 |
+
Bashkir-language dataset assembled from encyclopedic, periodical and literary
|
| 87 |
+
sources. The source texts are not distributed in this repository.
|
| 88 |
+
|
| 89 |
+
```text
|
| 90 |
+
raw Bashkir text (monolingual corpus)
|
| 91 |
+
├── 1. SentencePiece unigram tokenization (vocab size: 32,000)
|
| 92 |
+
├── 2. dynamic masked-language pretraining (15% subword masking rate)
|
| 93 |
+
├── 3. export PyTorch Transformers checkpoint
|
| 94 |
+
└── 4. export optimized ONNX graphs: FP32, FP16 & dynamic INT8
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
### Benchmark
|
| 98 |
|
| 99 |
On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
|
| 100 |
top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
|
| 101 |
+
diagnostic MLM results, not a general-purpose language-understanding score: a mask
|
| 102 |
+
may represent a whole word or a SentencePiece subword fragment.
|
| 103 |
+
The FP32 ONNX graph was checked against the source PyTorch model on multiple
|
| 104 |
+
sequence lengths and batch sizes, with matching top-token predictions.
|
| 105 |
|
| 106 |
## Quality and Use
|
| 107 |
|
| 108 |
+
Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the
|
| 109 |
+
INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision
|
| 110 |
+
ONNX reference is needed. Predictions are token suggestions that need review in
|
| 111 |
+
context, especially when the mask represents only part of a word.
|
| 112 |
|
| 113 |
### Limitations
|
| 114 |
|
| 115 |
- Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
|
| 116 |
- A mask may correspond to a partial subword, not always a full word.
|
| 117 |
- Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
|
| 118 |
+
- No training texts are redistributed; provenance or removal requests go through the maintainer.
|
| 119 |
+
|
| 120 |
+
## Related Resources
|
| 121 |
+
|
| 122 |
+
- [BashkirRoBERTa NER](https://huggingface.co/failed09/bashkir-roberta-ner)
|
| 123 |
+
fine-tunes this encoder to find person names and return their text spans. Use it
|
| 124 |
+
when the task is person-name extraction rather than masked-token prediction.
|
| 125 |
+
|
| 126 |
+
## Usage
|
| 127 |
|
| 128 |
```bash
|
| 129 |
pip install transformers torch huggingface_hub
|
|
|
|
| 146 |
print(tokenizer.decode([prediction_id])) # яратам
|
| 147 |
```
|
| 148 |
|
| 149 |
+
ONNX Runtime for CPU and edge deployment:
|
| 150 |
|
| 151 |
```python
|
| 152 |
import numpy as np
|
|
|
|
| 164 |
mask_idx = tokens.index(6)
|
| 165 |
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
|
| 166 |
top_tokens = np.argsort(logits)[::-1][:5]
|
| 167 |
+
print([sp.decode([int(t)]) for t in top_tokens]) # ['яратам', 'беләм', 'өйрәнә', ...]
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
For full-precision ONNX inference, change the downloaded filename to
|
| 171 |
+
`onnx/model_fp32.onnx`; the input and output names are the same.
|
| 172 |
|
| 173 |
## License
|
| 174 |
|
| 175 |
The model weights, tokenizer and release code are distributed under the
|
| 176 |
+
[Apache-2.0 license](https://huggingface.co/failed09/bashkir-roberta/blob/main/LICENSE). Training texts are not redistributed; their
|
| 177 |
rights remain with their respective owners. For provenance or removal requests,
|
| 178 |
contact the maintainer through the Hub.
|
| 179 |
|