SweMedMCQ-LFM2.5-1.2B-Answer-EN

LFM2.5-1.2B fine-tuned for English and Swedish medical multiple-choice questions, trained on answer-level data in English. It is one of the 14 student models released with the paper "Benchmarking and Adapting LLMs for Englishโ€“Swedish Medical Question Answering" and the SweMedMCQ benchmark (code and data in the repository linked from the paper).

Training

  • Base model: LiquidAI/LFM2.5-1.2B-Instruct.
  • Data: 135,093 English training questions, from the training splits of the ten MedicalAgentsBench datasets. No test item was used for training or model selection.
  • Supervision: Answer-level supervision: the target is the gold option alone, for example {"answer_index": "B"}, after the answer-only prompt. No teacher model is involved.
  • Training language: English.
  • Settings: full fine-tuning (embeddings trained), two epochs, effective batch 128 (32 per GPU on 4 GPUs), learning rate 5e-5 with cosine schedule and 5% warm-up, weight decay 0.01, gradient clipping 1.0, bfloat16, padding-free packing, sequences up to the full model context, loss on the response tokens, seed 13.
  • Checkpoint: best validation loss, step 2,000 of 2,110, validation loss 0.0757 (evaluated every 200 steps).

Prompt format

{question} is the question text and {options} the options, one per paragraph, as A) โ€ฆ, B) โ€ฆ (letters in order, separated by a blank line). The prompt is sent as a single user message, with no system prompt. The model answers with {"answer_index": "<letter>"}.

English answer-only prompt
You are a highly knowledgeable and careful medical expert.

Your task is to answer the following multiple-choice medical question by selecting the single best option.

Follow these steps carefully:
1. Read the question and all available answer options.
2. Select the single best answer from the listed options.

---

Return your final output **only** in valid JSON format that matches this schema:

{
  "answer_index": "The letter corresponding to the correct answer option (e.g., 'A', 'B', 'C', etc.)."
}

---

## Input

**Question:**
{question}

**Options:**

{options}

---
Swedish answer-only prompt
Du รคr en mycket kunnig och noggrann medicinsk expert.

Din uppgift รคr att besvara fรถljande flervalsfrรฅga genom att vรคlja det enda bรคsta alternativet.

Fรถlj dessa steg noggrant:
1. Lรคs frรฅgan och alla tillgรคngliga svarsalternativ.
2. Vรคlj det enda bรคsta svaret bland de angivna alternativen.

---

Returnera ditt slutliga svar **endast** i giltigt JSON-format som fรถljer detta schema:

{
  "answer_index": "Bokstaven som motsvarar det korrekta svarsalternativet (t.ex. 'A', 'B', 'C' osv.)."
}

---

## Indata

**Frรฅga:**
{question}

**Alternativ:**

{options}

---

The chat template ends the prompt at the assistant turn; the JSON answer was generated with structured (JSON-schema) decoding from the first token.

Usage with vLLM

Greedy decoding, as in the evaluation. The two prompts above are in prompts.json in this repository (sv and en); fill them with str.format. The example is in English, the training language of this model. Swedish options are listed in label order; English options in their original order.

import json
from huggingface_hub import hf_hub_download
from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

MODEL = "Anonymiserad/SweMedMCQ-LFM2.5-1.2B-Answer-EN"
PROMPTS = json.load(open(hf_hub_download(MODEL, "prompts.json"), encoding="utf-8"))
llm = LLM(MODEL, max_model_len=12288)
tok = llm.get_tokenizer()

question = "What is the band of electromagnetic waves used in a thermographic camera?"
options = {"A": "X-ray", "B": "Microwave", "C": "Ultraviolet", "D": "Infrared"}
opts = "\n\n".join(f"{k}) {v}" for k, v in options.items() if v)
user = PROMPTS["en"].format(question=question, options=opts)
messages = [{"role": "user", "content": user}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

schema = {"type": "object", "properties": {"answer_index": {"type": "string"}}, "required": ["answer_index"], "additionalProperties": False}
params = SamplingParams(temperature=0.0, max_tokens=12288,
                        structured_outputs=StructuredOutputsParams(json=schema))
out = llm.generate([prompt], params)[0].outputs[0].text
print(json.loads(out)["answer_index"])

del llm  # shut the engine down, so the script exits

Serving with vLLM

Start an OpenAI-compatible server:

vllm serve Anonymiserad/SweMedMCQ-LFM2.5-1.2B-Answer-EN --max-model-len 12288

and send the same prompt (here messages as in the example above). The server rejects requests whose prompt + max_tokens exceed the context, so ask for at most 10,240 new tokens:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
schema = {"type": "object", "properties": {"answer_index": {"type": "string"}}, "required": ["answer_index"], "additionalProperties": False}
r = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0, max_tokens=10240,
                                   extra_body={"structured_outputs": {"json": schema}})
print(json.loads(r.choices[0].message.content)["answer_index"])

Results

Accuracy (%) on the 9,733 core items of the SweMedMCQ test set (English source and Swedish benchmark text, greedy decoding):

Decoding English Swedish Gap (EN โˆ’ SV)
answer only 35.2 24.8 10.4

The base LFM2.5-1.2B reaches 36.3% in English and 23.8% in Swedish with the same prompt.

Limitations

  • Research model. It is not a medical device and must not be used for diagnosis, treatment, or any clinical decision.
  • Trained and evaluated on exam-style multiple-choice questions only; accuracy is far below that of large models, and explanations can be wrong even when the answer is right.
  • Swedish training and test text is machine-translated from English and may contain translation errors.

Licence

LFM Open License v1.0, inherited from the base model (see LICENSE). This model is a fine-tuned derivative of LFM2.5-1.2B-Instruct; all weight files were changed by fine-tuning.

Citation

@misc{anonymous2026swemedmcq,
  title  = {Benchmarking and Adapting {LLMs} for {English}--{Swedish} Medical Question Answering},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review}
}
Downloads last month
66
Safetensors
Model size
1B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Anonymiserad/SweMedMCQ-LFM2.5-1.2B-Answer-EN

Finetuned
(142)
this model