metadata
license: mit
language:
- ba
tags:
- bashkir
- full-fine-tuning
- gpt2
- low-resource
Bashkir Full Fine-Tuning Models
Full fine-tuned GPT-2 family models on Bashkir Web Corpus (71,567 documents).
Part of the BashkirNLP project. See the paper for details.
Structure
Each top-level subfolder <model>_seed<N>/ contains a full fine-tuned language model.
The classification/ folder contains sequence-classification heads trained on top of the same base models.
distilgpt2_seed42/
distilgpt2_seed123/
distilgpt2_seed2024/
gpt2_seed42/
...
classification/
cls_binary_distilgpt2_seed42/
cls_multiclass_distilgpt2_seed42/
...
Training
- Base models:
distilgpt2(82M),gpt2(124M),gpt2-medium(355M) - Corpus: Bashkir Web Corpus (57,254 train / 7,157 val / 7,156 test)
- Epochs: 3
- Seeds: 42, 123, 2024
max_seq_len: 128
Evaluation
See step1_report.txt in the paper repository for metrics (PPL, BPB, catastrophic forgetting, target-language ratio).