LuJa111/GMM-sefai-model

GMM-Sefai Team Project for University

LoRA adapter that fine-tunes Qwen3-VL-8B-Instruct (4-bit) to caption images in Lithuanian. Built to show captioning changes before vs. after fine-tuning.

• Base model: unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit
• Dataset: LuJa111/GMM-Sefai-Dataset).
• Method: QLoRA via Unsloth

Usage

from unsloth import FastVisionModel
from PIL import Image

model, tok = FastVisionModel.from_pretrained("LuJa111/GMM-sefai-model", load_in_4bit=True)
FastVisionModel.for_inference(model)

INSTRUCTION = "Aprašyk šį vaizdą taisyklinga, sklandžia lietuvių kalba vienu ar dviem sakiniais."
img = Image.open("photo.jpg").convert("RGB")
msgs = [{"role": "user", "content": [{"type": "text", "text": INSTRUCTION}, {"type": "image"}]}]
text = tok.apply_chat_template(msgs, add_generation_prompt=True)
inputs = tok(img, text, add_special_tokens=False, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=False, repetition_penalty=1.3, no_repeat_ngram_size=3)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Careful for greedy decoding GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA GELTONA