refqa / README.md

RefQA Gemma-3-27B LoRA adapter (v0.7) — clean release

cdac5f8 verified 7 days ago

2.34 kB

license: gemma
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: text-generation
tags:
  - lora
  - peft
  - sft
  - medical
  - clinical-nlp
  - citation
  - refqa

RefQA — Gemma-3-27B mixed adapter (v0.7)

A LoRA adapter that fine-tunes google/gemma-3-27b-it on a mix of medical instruction data and the RefQA CitationQA data. It learns the RefQA CitationQA task while retaining general medical question-answering ability.

Task

Two capabilities from one adapter:

Structured CitationQA generation on the RefQA schema (given the RefQA system prompt and a citation context), and
Retained general medical question answering.

Results

Final eval_loss 0.215, token accuracy 92.6% (2 epochs / 50,224 steps)
Out-of-domain evaluation: PubMedQA 74.10%, MedQA 24.59%

These numbers are reported as measured, not as a claim of state of the art.

Limitations

Research artifact. No clinical validation; not for clinical decision-making.
The CitationQA behavior requires the RefQA system prompt and chat format (below); without them the structured output is not reproduced.

Training

Base: google/gemma-3-27b-it (snapshot 005ad340)
Data: medical instruction mix + RefQA
LoRA: r=64, alpha=128, dropout=0.05; language-model attention and MLP projections
2 epochs / 50,224 steps
Stack: torch 2.11 (cu130), transformers 5.7, peft 0.19, trl 1.3

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

adapter = "taejoon89/refqa"
base = "google/gemma-3-27b-it"   # gated: accept the Gemma license first

tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

For CitationQA generation, the system prompt and input builder are in extract_qa_glm_v03.py at https://github.com/jin-0311/refqa. The training chat template is included here as chat_template.jinja.

Dataset and code

Dataset: RefQA — https://doi.org/10.5281/zenodo.20805692
Pipeline code: https://github.com/jin-0311/refqa

License

Use is governed by the base model's license (Gemma Terms of Use). You must accept the Gemma license to download and use the base model.