Qwen2.5-7B-Instruct-KazSTEM-LoRA

LoRA adapter for Qwen2.5-7B-Instruct, trained to answer Kazakh school STEM multiple-choice questions with one letter. Training data: 1,408 questions from Kazakh STEM MCQ 2000.

It improves accuracy on new questions from the same dataset by 18.1 points but does not improve KazMMLU (−0.8, not significant). I published it as part of that experiment, not as a better general Kazakh model. Details: github.com/sanim-science/kazakh-stem-llm.

Results

Accuracy in %. This repository holds training seed 1; I trained three seeds with the same settings and report their mean.

Test Base Seed 1 Mean of 3 seeds
Held-out questions from the dataset (413) 43.3 61.7 61.4 (+18.1)
KazMMLU Kazakh STEM (4,174) 47.0 45.4 46.2 (−0.8)
KazMMLU Kazakh non-STEM (2,045) 36.9 37.6 37.5 (+0.6)

95% bootstrap interval for the mean change: +13.2 to +22.8 on the held-out questions, −2.4 to +0.8 on KazMMLU STEM.

Usage

The adapter expects the prompt it was trained with.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA").eval()

def answer(question, options, subject="mathematics"):
    letters = [L for L in "ABCDE" if options.get(L)]
    prompt = "\n".join(
        [f"The following is a multiple choice question about {subject} (high school level, in Kazakh).",
         f"Reply with only the letter of the correct answer ({', '.join(letters)}).", "", question.strip(), ""]
        + [f"{L}. {options[L].strip()}" for L in letters] + ["", "Answer:"])
    text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
    enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
    with torch.no_grad():
        logits = model(**enc).logits[0, -1]
    ids = {L: tok.encode(L, add_special_tokens=False)[0] for L in letters}
    return max(letters, key=lambda L: logits[ids[L]].item())

print(answer("(2a³)² · a өрнегін ықшамдаңыз.",
             {"A": "2a⁶", "B": "4a⁷", "C": "4a⁶", "D": "2a⁷", "E": "4a⁵"}))   # B

Subjects for the prompt: mathematics, physics, chemistry, biology, informatics. Tested with transformers 5.20.2, peft 0.21.2, torch 2.11.

Training

LoRA r = 16, alpha = 16, dropout 0, on q, k, v, o, gate, up and down projections. AdamW, learning rate 2e-4 with 5% warm-up and linear decay, batch 8, 2 epochs; the epoch with the best accuracy on 169 validation questions is kept (seed 1: epoch 1). Loss on the answer letter only. Base model in float16, adapter in float32, 2 × T4 on Kaggle. The 413 test questions and 10 questions too similar to KazMMLU were not used.

Limitations

  • Trained to output a letter for one prompt format; it does not add reasoning.
  • The gain is on questions like the training set and did not carry over to KazMMLU.
  • Training questions have five options; KazMMLU mostly has four. After training the model picks "D" too often on 4-option questions.

License

Apache 2.0, the same as the base model. Training data: CC BY 4.0.

Downloads last month
53
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA

Base model

Qwen/Qwen2.5-7B
Adapter
(2,931)
this model

Dataset used to train Sanim09/Qwen2.5-7B-Instruct-KazSTEM-LoRA