segestic/MediLITE-QA-GGUF

MediLITE-QA — GGUF

GGUF files of MediLITE-QA, a 3.8B-parameter medical multiple-choice question-answering model built on Phi-3.5-mini-instruct, for CPU-only inference with llama.cpp and compatible runtimes. The model is described in:

Odewole, O.O.; Başaran, S. MediLITE-QA: Energy-Efficient Medical Question Answering with a Quantized Domain-Specific Language Model on a Commodity Laptop CPU. Sustainability 2026, 18(21), 11245. https://doi.org/10.3390/su182111245

For research use only. The model is not a medical device and must not be used to inform patient care.

Files

File	Size	Description
MediLITE-QA.Q4_K_M.gguf	2.39 GB	4-bit quantization; the file used for all CPU measurements in the paper
MediLITE-QA.Q5_K_M.gguf	2.82 GB	5-bit quantization
MediLITE-QA.Q8_0.gguf	4.06 GB	8-bit quantization
MediLITE-QA.F32.gguf	7.64 GB	Unquantized

The accuracy and efficiency results below refer to the Q4_K_M file.

Related repositories

Artifact	Repository
Merged model (Transformers)	segestic/MediliteQA-phi3.5-mini-all-med
LoRA adapter	segestic/phi3.5-mini-4k-qlora-medical-seg-vall_med

Code and evaluation logs: https://github.com/segestic/Medilite-QA

How to use

With llama-cpp-python:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="segestic/MediLITE-QA-GGUF",
    filename="MediLITE-QA.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=4,
    n_gpu_layers=0,
)

question = "..."
options = "A. ...\nB. ...\nC. ...\nD. ..."

reply = llm.create_chat_completion(
    messages=[{"role": "user", "content": f"{question}\n\n Input: {options}"}],
    max_tokens=4,
    temperature=0.0,
)
print(reply["choices"][0]["message"]["content"])

The model answers multiple-choice questions with a single letter; use the prompt layout above for best results.

Performance

Accuracy. Zero-shot accuracy of the Q4_K_M file, averaged over nine medical benchmarks (7,045 items), is 71.87% (bootstrap 95% CI 70.05 to 73.76), compared with 72.79% for the fp16 model and 68.73% for the base model. Per-benchmark results are in the paper.

Efficiency. Q4_K_M file, llama-cpp-python, 4 threads, no GPU; Intel Core i5-1135G7 with 6.96 GiB RAM.

Metric	Value
Steady-state time to first token	1.34 s
Time per output token	117 ms (8.19 tokens/s)
Peak resident memory (2,000-token context)	4,403 MiB (61.8% of RAM)
CPU-package energy	3.17 J per generated token

The 7–8B open models measured on the same hardware used 1.56–1.98× more CPU energy per token.

Limitations

• Not clinically validated. An exploratory blinded review by five clinicians found clinical correctness below that of a frontier reference model, and some responses were flagged for potential harm.
• Output format. The model is trained to answer with a single letter and follows free-form, explanation-style instructions less reliably.
• Hardware scope. Efficiency results come from one consumer-laptop CPU and should not be assumed to transfer to other devices.

Citation

@article{odewole2026medilite,
  author  = {Odewole, Olusegun Oladele and Başaran, Seren},
  title   = {MediLITE-QA: Energy-Efficient Medical Question Answering with a Quantized
             Domain-Specific Language Model on a Commodity Laptop CPU},
  journal = {Sustainability},
  volume  = {18},
  number  = {21},
  pages   = {11245},
  year    = {2026},
  doi     = {10.3390/su182111245}
}