README.md

metadata

language:
  - ba
license: apache-2.0
pretty_name: BashkirRoBERTa
library_name: transformers
pipeline_tag: fill-mask
tags:
  - bashkir
  - masked-language-modeling
  - roberta
  - sentencepiece
  - custom-code
  - onnx
  - onnxruntime

BashkirRoBERTa

A Bashkir masked language model for fill-mask suggestions and further fine-tuning.

Overview

BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide fill-mask suggestions and serve as an encoder for further fine-tuning, including the related person-name model. The release includes Transformers weights and FP32, FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the included model code when loaded with Transformers.

At a glance
Task	Masked language modelling / fill-mask
Default artifact	model.safetensors (Transformers); onnx/model_int8.onnx (CPU ONNX)
Source	A monolingual Bashkir-language dataset
Version / license	v1 / Apache-2.0

Contents

Files and Configurations

File	Purpose	Size
model.safetensors	Transformers weights and fine-tuning starting point	200.2 MB
onnx/model_fp32.onnx	Full-precision ONNX reference for CPU inference	242.2 MB
onnx/model_fp16.onnx	FP16 ONNX model for GPU / DirectML	121.2 MB
onnx/model_int8.onnx	Compact INT8 ONNX graph for CPU use	60.9 MB
spm_bashkir_bert_16k.model	SentencePiece tokenizer	—
config.json	Model configuration (auto_map for custom code)	—
configuration_bashkir_roberta.py, modeling_bashkir_roberta.py, tokenization_bashkir_roberta.py	Custom Pre-LayerNorm implementation	—
tokenizer_config.json	Tokenizer configuration	—
META.json	Release passport and artifact hashes	—
LICENSE	Full license text	—
SHA256SUMS	Release checksums	—

Model Architecture

Property	Value
Task	Masked language modelling / fill-mask
Architecture	Pre-LayerNorm Transformer encoder
Transformer blocks	8
Hidden size / attention heads	640 / 10
Feed-forward size	2,560
Context window	256 subword tokens
Parameters	50.04M
Tokenizer	SentencePiece BPE, 16,384 tokens

The output embedding matrix is tied to the input word embeddings. Token IDs are fixed: <pad> 0, <unk> 1, <s> 2, </s> 3, [CLS] 4, [SEP] 5 and [MASK] 6.

Examples

Outputs from the INT8 ONNX model on CPU:

Input	Top prediction
Мин башҡорт телен [MASK].	яратам
Башҡортостан — беҙҙең [MASK].	республика
Өфө — ҙур [MASK].	ҡала
Бөгөн Өфөлә яңы [MASK] асылды.	мәсет

Method

The model was pretrained with dynamic masked-language modelling on a monolingual Bashkir-language dataset assembled from encyclopedic, periodical and literary sources. The source texts are not distributed in this repository.

Bashkir text → SentencePiece tokens → masked-language pretraining
    → Transformers checkpoint → FP32 / FP16 / INT8 ONNX exports

Benchmark

On a held-out Bashkir encyclopedic evaluation set the project reports 24.7% top-1 and 54.0% top-5 accuracy for masked subword prediction. These are diagnostic MLM results, not a general-purpose language-understanding score: a mask may represent a whole word or a SentencePiece subword fragment. The FP32 ONNX graph was checked against the source PyTorch model on multiple sequence lengths and batch sizes, with matching top-token predictions.

Quality and Use

Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision ONNX reference is needed. Predictions are token suggestions that need review in context, especially when the mask represents only part of a word.

Limitations

• Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
• A mask may correspond to a partial subword, not always a full word.
• Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
• No training texts are redistributed; provenance or removal requests go through the maintainer.

Related Resources

• BashkirRoBERTa NER fine-tunes this encoder to find person names and return their text spans. Use it when the task is person-name extraction rather than masked-token prediction.

Usage

pip install transformers torch huggingface_hub

PyTorch (Transformers), which requires trust_remote_code=True because of the custom Pre-LayerNorm architecture:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "failed09/bashkir-roberta"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
logits = model(**inputs).logits
mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
prediction_id = logits[0, mask_index].argmax().item()
print(tokenizer.decode([prediction_id]))  # яратам

ONNX Runtime for CPU and edge deployment:

import numpy as np
import onnxruntime as ort
import sentencepiece as spm
from huggingface_hub import hf_hub_download

model_path = hf_hub_download("failed09/bashkir-roberta", "onnx/model_int8.onnx")
sp_path = hf_hub_download("failed09/bashkir-roberta", "spm_bashkir_bert_16k.model")

session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sp = spm.SentencePieceProcessor(model_file=sp_path)

tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(".") + [3]
mask_idx = tokens.index(6)
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
top_tokens = np.argsort(logits)[::-1][:5]
print([sp.decode([int(t)]) for t in top_tokens])  # ['яратам', 'беләм', 'өйрәнә', ...]

For full-precision ONNX inference, change the downloaded filename to onnx/model_fp32.onnx; the input and output names are the same.

License

The model weights, tokenizer and release code are distributed under the Apache-2.0 license. Training texts are not redistributed; their rights remain with their respective owners. For provenance or removal requests, contact the maintainer through the Hub.

Citation

@software{failed09_bashkir_roberta_2026,
  title = {BashkirRoBERTa},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-roberta},
  note = {Masked language model for Bashkir}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.