---
license: gemma
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- peft
- sft
- medical
- clinical-nlp
- citation
- refqa
---

# RefQA — Gemma-3-27B mixed adapter (v0.7)

A LoRA adapter that fine-tunes **google/gemma-3-27b-it** on a mix of medical instruction data and the [RefQA](https://doi.org/10.5281/zenodo.20805692) CitationQA data. It learns the RefQA CitationQA task while retaining general medical question-answering ability.

## Task

Two capabilities from one adapter:
1. Structured **CitationQA** generation on the RefQA schema (given the RefQA system prompt and a citation context), and
2. Retained general medical question answering.

## Results

- Final `eval_loss` 0.215, token accuracy 92.6% (2 epochs / 50,224 steps)
- Out-of-domain evaluation: **PubMedQA 74.10%**, **MedQA 24.59%**

These numbers are reported as measured, not as a claim of state of the art.

## Limitations

- Research artifact. No clinical validation; not for clinical decision-making.
- The CitationQA behavior requires the RefQA system prompt and chat format (below); without them the structured output is not reproduced.

## Training

- Base: `google/gemma-3-27b-it` (snapshot `005ad340`)
- Data: medical instruction mix + RefQA
- LoRA: r=64, alpha=128, dropout=0.05; language-model attention and MLP projections
- 2 epochs / 50,224 steps
- Stack: torch 2.11 (cu130), transformers 5.7, peft 0.19, trl 1.3

## How to use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

adapter = "taejoon89/refqa"
base = "google/gemma-3-27b-it"   # gated: accept the Gemma license first

tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
```

For CitationQA generation, the system prompt and input builder are in `extract_qa_glm_v03.py` at https://github.com/jin-0311/refqa. The training chat template is included here as `chat_template.jinja`.

## Dataset and code

- Dataset: RefQA — https://doi.org/10.5281/zenodo.20805692
- Pipeline code: https://github.com/jin-0311/refqa

## License

Use is governed by the base model's license (Gemma Terms of Use). You must accept the Gemma license to download and use the base model.