Instructions to use Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct") model = PeftModel.from_pretrained(base_model, "Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA") - Notebooks
- Google Colab
- Kaggle
Qwen2.5-7B-Instruct-KazSTEM-LoRA
LoRA adapter for Qwen2.5-7B-Instruct, trained to answer Kazakh school STEM multiple-choice questions with one letter. Training data: 1,408 questions from Kazakh STEM MCQ 2000.
It improves accuracy on new questions from the same dataset by 18.1 points but does not improve KazMMLU (−0.8, not significant). I published it as part of that experiment, not as a better general Kazakh model. Details: github.com/sanim-science/kazakh-stem-llm.
Results
Accuracy in %. This repository holds training seed 1; I trained three seeds with the same settings and report their mean.
| Test | Base | Seed 1 | Mean of 3 seeds |
|---|---|---|---|
| Held-out questions from the dataset (413) | 43.3 | 61.7 | 61.4 (+18.1) |
| KazMMLU Kazakh STEM (4,174) | 47.0 | 45.4 | 46.2 (−0.8) |
| KazMMLU Kazakh non-STEM (2,045) | 36.9 | 37.6 | 37.5 (+0.6) |
95% bootstrap interval for the mean change: +13.2 to +22.8 on the held-out questions, −2.4 to +0.8 on KazMMLU STEM.
Usage
The adapter expects the prompt it was trained with.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA").eval()
def answer(question, options, subject="mathematics"):
letters = [L for L in "ABCDE" if options.get(L)]
prompt = "\n".join(
[f"The following is a multiple choice question about {subject} (high school level, in Kazakh).",
f"Reply with only the letter of the correct answer ({', '.join(letters)}).", "", question.strip(), ""]
+ [f"{L}. {options[L].strip()}" for L in letters] + ["", "Answer:"])
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
logits = model(**enc).logits[0, -1]
ids = {L: tok.encode(L, add_special_tokens=False)[0] for L in letters}
return max(letters, key=lambda L: logits[ids[L]].item())
print(answer("(2a³)² · a өрнегін ықшамдаңыз.",
{"A": "2a⁶", "B": "4a⁷", "C": "4a⁶", "D": "2a⁷", "E": "4a⁵"})) # B
Subjects for the prompt: mathematics, physics, chemistry, biology, informatics. Tested with transformers 5.20.2, peft 0.21.2, torch 2.11.
Training
LoRA r = 16, alpha = 16, dropout 0, on q, k, v, o, gate, up and down projections. AdamW, learning rate 2e-4 with 5% warm-up and linear decay, batch 8, 2 epochs; the epoch with the best accuracy on 169 validation questions is kept (seed 1: epoch 1). Loss on the answer letter only. Base model in float16, adapter in float32, 2 × T4 on Kaggle. The 413 test questions and 10 questions too similar to KazMMLU were not used.
Limitations
- Trained to output a letter for one prompt format; it does not add reasoning.
- The gain is on questions like the training set and did not carry over to KazMMLU.
- Training questions have five options; KazMMLU mostly has four. After training the model picks "D" too often on 4-option questions.
License
Apache 2.0, the same as the base model. Training data: CC BY 4.0.
- Downloads last month
- 53