RefQA — Gemma-3-27B mixed adapter (v0.7)

A LoRA adapter that fine-tunes google/gemma-3-27b-it on a mix of medical instruction data and the RefQA CitationQA data. It learns the RefQA CitationQA task while retaining general medical question-answering ability.

Task

Two capabilities from one adapter:

1. Structured CitationQA generation on the RefQA schema (given the RefQA system prompt and a citation context), and
2. Retained general medical question answering.

Results

• Final eval_loss 0.215, token accuracy 92.6% (2 epochs / 50,224 steps)
• Out-of-domain evaluation: PubMedQA 74.10%, MedQA 24.59%

These numbers are reported as measured, not as a claim of state of the art.

Limitations

• Research artifact. No clinical validation; not for clinical decision-making.
• The CitationQA behavior requires the RefQA system prompt and chat format (below); without them the structured output is not reproduced.

Training

• Base: google/gemma-3-27b-it (snapshot 005ad340)
• Data: medical instruction mix + RefQA
• LoRA: r=64, alpha=128, dropout=0.05; language-model attention and MLP projections
• 2 epochs / 50,224 steps
• Stack: torch 2.11 (cu130), transformers 5.7, peft 0.19, trl 1.3

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

adapter = "taejoon89/refqa"
base = "google/gemma-3-27b-it"   # gated: accept the Gemma license first

tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

For CitationQA generation, the system prompt and input builder are in extract_qa_glm_v03.py at https://github.com/jin-0311/refqa. The training chat template is included here as chat_template.jinja.

Dataset and code

• Dataset: RefQA — https://doi.org/10.5281/zenodo.20805692
• Pipeline code: https://github.com/jin-0311/refqa

License

Use is governed by the base model's license (Gemma Terms of Use). You must accept the Gemma license to download and use the base model.