TR_QUIZ_GEN_SIMPLE_LLAMA7B

Llama-2-7b-chat fine-tuned to write short-answer quiz questions in Turkish

Given a Turkish educational text, this model writes short-answer questions about it. It is a LoRA adapter for Llama-2-7b-chat trained on Turkish-Quiz-Instruct, from the paper Automating Turkish Educational Quiz Generation Using Large Language Models (ISPR 2024).

Usage

The generator script in the GitHub repository handles prompting and parsing:

git clone https://github.com/KamyarZeinalipour/Turkish_Quiz_Generator && cd Turkish_Quiz_Generator
python quiz_gen_main.py --help

Results

Multiple-choice questions (ROUGE-1 / ROUGE-2 / ROUGE-L, base → fine-tuned):

Model	ROUGE-1	ROUGE-2	ROUGE-L
Llama-2-7b-chat	4.71 → 17.33	1.42 → 2.88	3.88 → 12.11
Llama-2-13b-chat	5.33 → 24.34	1.73 → 11.44	4.28 → 19.75
GPT-3.5-Turbo	27.53 → 36.64	13.18 → 20.36	21.38 → 28.82

Short-answer questions:

Model	ROUGE-1	ROUGE-2	ROUGE-L
Llama-2-7b-chat	3.31 → 54.37	1.04 → 47.33	2.86 → 49.01
Llama-2-13b-chat	3.81 → 52.99	1.33 → 46.88	3.22 → 49.54
GPT-3.5-Turbo	28.19 → 45.31	15.87 → 31.59	22.22 → 40.32

Models from this paper

Task	Model	Adapter
Multiple-choice	Llama-2-7b-chat	TR_QUIZ_GEN_MULTI_LLAMA7B
Multiple-choice	Llama-2-13b-chat	TR_QUIZ_GEN_MULTI_LLAMA13B
Short-answer	Llama-2-7b-chat	TR_QUIZ_GEN_SIMPLE_LLAMA7B
Short-answer	Llama-2-13b-chat	TR_QUIZ_GEN_SIMPLE_LLAMA13B

Citation

@inproceedings{zeinalipour2024turkishquiz,
  title     = {Automating Turkish Educational Quiz Generation Using Large Language Models},
  author    = {Kamyar Zeinalipour and Yusuf Gokberk Keptig and Marco Maggini and Marco Gori},
  booktitle = {Artificial Neural Networks in Pattern Recognition (ANNPR / ISPR 2024)},
  year      = {2024},
  url       = {https://arxiv.org/abs/2406.03397}
}