Fill-Mask
Transformers
ONNX
Safetensors
Bashkir
bashkir-roberta-preln
bashkir
masked-language-modeling
roberta
sentencepiece
custom-code
onnxruntime
custom_code
Instructions to use failed09/bashkir-roberta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use failed09/bashkir-roberta with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="failed09/bashkir-roberta", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("failed09/bashkir-roberta", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Align BashkirRoBERTa model card with release template
Browse files- README.md +31 -22
- SHA256SUMS +1 -1
README.md
CHANGED
|
@@ -17,21 +17,20 @@ tags:
|
|
| 17 |
|
| 18 |
# BashkirRoBERTa
|
| 19 |
|
| 20 |
-
> A masked language model for
|
| 21 |
-
> foundation fine-tuning.
|
| 22 |
|
| 23 |
## Overview
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
|
| 31 |
| At a glance | |
|
| 32 |
| --- | --- |
|
| 33 |
| Task | Masked language modelling / fill-mask |
|
| 34 |
-
| Default artifact | `model.safetensors` (Transformers)
|
| 35 |
| Source | A monolingual Bashkir-language dataset |
|
| 36 |
| Version / license | v1 / Apache-2.0 |
|
| 37 |
|
|
@@ -41,9 +40,9 @@ stock post-LayerNorm RoBERTa implementation.
|
|
| 41 |
|
| 42 |
| File | Purpose | Size |
|
| 43 |
| --- | --- | ---: |
|
| 44 |
-
| `model.safetensors` |
|
| 45 |
| `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
|
| 46 |
-
| `onnx/model_int8.onnx` | INT8 ONNX
|
| 47 |
| `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
|
| 48 |
| `config.json` | Model configuration (`auto_map` for custom code) | — |
|
| 49 |
| `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
|
|
@@ -82,11 +81,16 @@ Outputs from the INT8 ONNX model on CPU:
|
|
| 82 |
|
| 83 |
## Method
|
| 84 |
|
| 85 |
-
The model was pretrained with dynamic masked-language modelling on a monolingual
|
| 86 |
-
Bashkir-language dataset assembled from encyclopedic, periodical and literary
|
| 87 |
-
sources. The source texts are not distributed in this repository.
|
| 88 |
-
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
|
| 92 |
top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
|
|
@@ -95,19 +99,24 @@ may represent a whole word or a SentencePiece subword fragment.
|
|
| 95 |
|
| 96 |
## Quality and Use
|
| 97 |
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
under Apache-2.0; source-text provenance remains documented separately.
|
| 102 |
|
| 103 |
### Limitations
|
| 104 |
|
| 105 |
- Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
|
| 106 |
- A mask may correspond to a partial subword, not always a full word.
|
| 107 |
- Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
|
| 108 |
-
- No training texts are redistributed; provenance or removal requests go through the maintainer.
|
| 109 |
-
|
| 110 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
```bash
|
| 113 |
pip install transformers torch huggingface_hub
|
|
|
|
| 17 |
|
| 18 |
# BashkirRoBERTa
|
| 19 |
|
| 20 |
+
> A Bashkir masked language model for fill-mask suggestions and further fine-tuning.
|
|
|
|
| 21 |
|
| 22 |
## Overview
|
| 23 |
|
| 24 |
+
BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide
|
| 25 |
+
fill-mask suggestions and serve as an encoder for further fine-tuning, including
|
| 26 |
+
the related person-name model. The release includes Transformers weights and FP16
|
| 27 |
+
and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the included
|
| 28 |
+
model code when loaded with Transformers.
|
| 29 |
|
| 30 |
| At a glance | |
|
| 31 |
| --- | --- |
|
| 32 |
| Task | Masked language modelling / fill-mask |
|
| 33 |
+
| Default artifact | `model.safetensors` (Transformers); `onnx/model_int8.onnx` (CPU ONNX) |
|
| 34 |
| Source | A monolingual Bashkir-language dataset |
|
| 35 |
| Version / license | v1 / Apache-2.0 |
|
| 36 |
|
|
|
|
| 40 |
|
| 41 |
| File | Purpose | Size |
|
| 42 |
| --- | --- | ---: |
|
| 43 |
+
| **`model.safetensors`** | Transformers weights and fine-tuning starting point | 200.2 MB |
|
| 44 |
| `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
|
| 45 |
+
| **`onnx/model_int8.onnx`** | Compact INT8 ONNX graph for CPU use | 60.9 MB |
|
| 46 |
| `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
|
| 47 |
| `config.json` | Model configuration (`auto_map` for custom code) | — |
|
| 48 |
| `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
|
|
|
|
| 81 |
|
| 82 |
## Method
|
| 83 |
|
| 84 |
+
The model was pretrained with dynamic masked-language modelling on a monolingual
|
| 85 |
+
Bashkir-language dataset assembled from encyclopedic, periodical and literary
|
| 86 |
+
sources. The source texts are not distributed in this repository.
|
| 87 |
+
|
| 88 |
+
```text
|
| 89 |
+
Bashkir text → SentencePiece tokens → masked-language pretraining
|
| 90 |
+
→ Transformers checkpoint → FP16 and INT8 ONNX exports
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
### Benchmark
|
| 94 |
|
| 95 |
On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
|
| 96 |
top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
|
|
|
|
| 99 |
|
| 100 |
## Quality and Use
|
| 101 |
|
| 102 |
+
Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the
|
| 103 |
+
INT8 ONNX graph for compact CPU inference. Predictions are token suggestions that
|
| 104 |
+
need review in context, especially when the mask represents only part of a word.
|
|
|
|
| 105 |
|
| 106 |
### Limitations
|
| 107 |
|
| 108 |
- Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
|
| 109 |
- A mask may correspond to a partial subword, not always a full word.
|
| 110 |
- Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
|
| 111 |
+
- No training texts are redistributed; provenance or removal requests go through the maintainer.
|
| 112 |
+
|
| 113 |
+
## Related Resources
|
| 114 |
+
|
| 115 |
+
- [BashkirRoBERTa NER](https://huggingface.co/failed09/bashkir-roberta-ner)
|
| 116 |
+
fine-tunes this encoder to find person names and return their text spans. Use it
|
| 117 |
+
when the task is person-name extraction rather than masked-token prediction.
|
| 118 |
+
|
| 119 |
+
## Usage
|
| 120 |
|
| 121 |
```bash
|
| 122 |
pip install transformers torch huggingface_hub
|
SHA256SUMS
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 2 |
f6d59748711ddcc45af84d09cf7970a276ee93820a83dda4fb5fda8209f878ca META.json
|
| 3 |
-
|
| 4 |
59b773953af5b2a69adb7e3c5d345a90f34f848f6a1543309d5acc7fb78b28c4 config.json
|
| 5 |
e9f179aad7619153724920c86a90ecc3c161fd15ffc3332f200e9f2073a07f72 configuration_bashkir_roberta.py
|
| 6 |
48e4d4e7b8af675d222e58e14776abac24b7e83a937d7884bd9d735ea9dd2173 model.safetensors
|
|
|
|
| 1 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 2 |
f6d59748711ddcc45af84d09cf7970a276ee93820a83dda4fb5fda8209f878ca META.json
|
| 3 |
+
d11fdb4dffe9cc59684392c54e798f194493bb48862196291e96a067a8471efb README.md
|
| 4 |
59b773953af5b2a69adb7e3c5d345a90f34f848f6a1543309d5acc7fb78b28c4 config.json
|
| 5 |
e9f179aad7619153724920c86a90ecc3c161fd15ffc3332f200e9f2073a07f72 configuration_bashkir_roberta.py
|
| 6 |
48e4d4e7b8af675d222e58e14776abac24b7e83a937d7884bd9d735ea9dd2173 model.safetensors
|