failed09 commited on
Commit
7c7d9ed
·
verified ·
1 Parent(s): 96349dd

docs: format Method pipeline as tree

Browse files
Files changed (1) hide show
  1. README.md +48 -45
README.md CHANGED
@@ -17,20 +17,20 @@ tags:
17
 
18
  # BashkirRoBERTa
19
 
20
- > A Bashkir masked language model for fill-mask suggestions and further fine-tuning.
21
 
22
  ## Overview
23
 
24
- BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide
25
- fill-mask suggestions and serve as an encoder for further fine-tuning, including
26
- the related person-name model. The release includes Transformers weights and FP32,
27
- FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the
28
- included model code when loaded with Transformers.
29
 
30
  | At a glance | |
31
  | --- | --- |
32
  | Task | Masked language modelling / fill-mask |
33
- | Default artifact | `model.safetensors` (Transformers); `onnx/model_int8.onnx` (CPU ONNX) |
34
  | Source | A monolingual Bashkir-language dataset |
35
  | Version / license | v1 / Apache-2.0 |
36
 
@@ -40,10 +40,10 @@ included model code when loaded with Transformers.
40
 
41
  | File | Purpose | Size |
42
  | --- | --- | ---: |
43
- | **`model.safetensors`** | Transformers weights and fine-tuning starting point | 200.2 MB |
44
- | `onnx/model_fp32.onnx` | Full-precision ONNX reference for CPU inference | 242.2 MB |
45
- | `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
46
- | **`onnx/model_int8.onnx`** | Compact INT8 ONNX graph for CPU use | 60.9 MB |
47
  | `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
48
  | `config.json` | Model configuration (`auto_map` for custom code) | — |
49
  | `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
@@ -82,45 +82,48 @@ Outputs from the INT8 ONNX model on CPU:
82
 
83
  ## Method
84
 
85
- The model was pretrained with dynamic masked-language modelling on a monolingual
86
- Bashkir-language dataset assembled from encyclopedic, periodical and literary
87
- sources. The source texts are not distributed in this repository.
88
-
89
- ```text
90
- Bashkir text → SentencePiece tokens → masked-language pretraining
91
- → Transformers checkpoint → FP32 / FP16 / INT8 ONNX exports
92
- ```
93
-
94
- ### Benchmark
 
 
 
95
 
96
  On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
97
  top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
98
- diagnostic MLM results, not a general-purpose language-understanding score: a mask
99
- may represent a whole word or a SentencePiece subword fragment.
100
- The FP32 ONNX graph was checked against the source PyTorch model on multiple
101
- sequence lengths and batch sizes, with matching top-token predictions.
102
 
103
  ## Quality and Use
104
 
105
- Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the
106
- INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision
107
- ONNX reference is needed. Predictions are token suggestions that need review in
108
- context, especially when the mask represents only part of a word.
109
 
110
  ### Limitations
111
 
112
  - Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
113
  - A mask may correspond to a partial subword, not always a full word.
114
  - Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
115
- - No training texts are redistributed; provenance or removal requests go through the maintainer.
116
-
117
- ## Related Resources
118
-
119
- - [BashkirRoBERTa NER](https://huggingface.co/failed09/bashkir-roberta-ner)
120
- fine-tunes this encoder to find person names and return their text spans. Use it
121
- when the task is person-name extraction rather than masked-token prediction.
122
-
123
- ## Usage
124
 
125
  ```bash
126
  pip install transformers torch huggingface_hub
@@ -143,7 +146,7 @@ prediction_id = logits[0, mask_index].argmax().item()
143
  print(tokenizer.decode([prediction_id])) # яратам
144
  ```
145
 
146
- ONNX Runtime for CPU and edge deployment:
147
 
148
  ```python
149
  import numpy as np
@@ -161,16 +164,16 @@ tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(
161
  mask_idx = tokens.index(6)
162
  logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
163
  top_tokens = np.argsort(logits)[::-1][:5]
164
- print([sp.decode([int(t)]) for t in top_tokens]) # ['яратам', 'беләм', 'өйрәнә', ...]
165
- ```
166
-
167
- For full-precision ONNX inference, change the downloaded filename to
168
- `onnx/model_fp32.onnx`; the input and output names are the same.
169
 
170
  ## License
171
 
172
  The model weights, tokenizer and release code are distributed under the
173
- [Apache-2.0 license](https://huggingface.co/failed09/bashkir-roberta/blob/main/LICENSE). Training texts are not redistributed; their
174
  rights remain with their respective owners. For provenance or removal requests,
175
  contact the maintainer through the Hub.
176
 
 
17
 
18
  # BashkirRoBERTa
19
 
20
+ > A Bashkir masked language model for fill-mask suggestions and further fine-tuning.
21
 
22
  ## Overview
23
 
24
+ BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide
25
+ fill-mask suggestions and serve as an encoder for further fine-tuning, including
26
+ the related person-name model. The release includes Transformers weights and FP32,
27
+ FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the
28
+ included model code when loaded with Transformers.
29
 
30
  | At a glance | |
31
  | --- | --- |
32
  | Task | Masked language modelling / fill-mask |
33
+ | Default artifact | `model.safetensors` (Transformers); `onnx/model_int8.onnx` (CPU ONNX) |
34
  | Source | A monolingual Bashkir-language dataset |
35
  | Version / license | v1 / Apache-2.0 |
36
 
 
40
 
41
  | File | Purpose | Size |
42
  | --- | --- | ---: |
43
+ | **`model.safetensors`** | Transformers weights and fine-tuning starting point | 200.2 MB |
44
+ | `onnx/model_fp32.onnx` | Full-precision ONNX reference for CPU inference | 242.2 MB |
45
+ | `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
46
+ | **`onnx/model_int8.onnx`** | Compact INT8 ONNX graph for CPU use | 60.9 MB |
47
  | `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
48
  | `config.json` | Model configuration (`auto_map` for custom code) | — |
49
  | `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
 
82
 
83
  ## Method
84
 
85
+ The model was pretrained with dynamic masked-language modelling on a monolingual
86
+ Bashkir-language dataset assembled from encyclopedic, periodical and literary
87
+ sources. The source texts are not distributed in this repository.
88
+
89
+ ```text
90
+ raw Bashkir text (monolingual corpus)
91
+ ├── 1. SentencePiece unigram tokenization (vocab size: 32,000)
92
+ ├── 2. dynamic masked-language pretraining (15% subword masking rate)
93
+ ├── 3. export PyTorch Transformers checkpoint
94
+ └── 4. export optimized ONNX graphs: FP32, FP16 & dynamic INT8
95
+ ```
96
+
97
+ ### Benchmark
98
 
99
  On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
100
  top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
101
+ diagnostic MLM results, not a general-purpose language-understanding score: a mask
102
+ may represent a whole word or a SentencePiece subword fragment.
103
+ The FP32 ONNX graph was checked against the source PyTorch model on multiple
104
+ sequence lengths and batch sizes, with matching top-token predictions.
105
 
106
  ## Quality and Use
107
 
108
+ Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the
109
+ INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision
110
+ ONNX reference is needed. Predictions are token suggestions that need review in
111
+ context, especially when the mask represents only part of a word.
112
 
113
  ### Limitations
114
 
115
  - Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
116
  - A mask may correspond to a partial subword, not always a full word.
117
  - Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
118
+ - No training texts are redistributed; provenance or removal requests go through the maintainer.
119
+
120
+ ## Related Resources
121
+
122
+ - [BashkirRoBERTa NER](https://huggingface.co/failed09/bashkir-roberta-ner)
123
+ fine-tunes this encoder to find person names and return their text spans. Use it
124
+ when the task is person-name extraction rather than masked-token prediction.
125
+
126
+ ## Usage
127
 
128
  ```bash
129
  pip install transformers torch huggingface_hub
 
146
  print(tokenizer.decode([prediction_id])) # яратам
147
  ```
148
 
149
+ ONNX Runtime for CPU and edge deployment:
150
 
151
  ```python
152
  import numpy as np
 
164
  mask_idx = tokens.index(6)
165
  logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
166
  top_tokens = np.argsort(logits)[::-1][:5]
167
+ print([sp.decode([int(t)]) for t in top_tokens]) # ['яратам', 'беләм', 'өйрәнә', ...]
168
+ ```
169
+
170
+ For full-precision ONNX inference, change the downloaded filename to
171
+ `onnx/model_fp32.onnx`; the input and output names are the same.
172
 
173
  ## License
174
 
175
  The model weights, tokenizer and release code are distributed under the
176
+ [Apache-2.0 license](https://huggingface.co/failed09/bashkir-roberta/blob/main/LICENSE). Training texts are not redistributed; their
177
  rights remain with their respective owners. For provenance or removal requests,
178
  contact the maintainer through the Hub.
179