Image-Text-to-Text
PEFT
Safetensors
lora
personalization
multimodal
safety

PReMind-4B

PReMind-4B is a LoRA adapter for Qwen/Qwen3.5-4B, trained with two-phase reinforcement learning on verifiable rewards. It is the model of PReMind: When Personal Memory Meets Visual Hazards (arXiv:2611.23808).

Content warning: the benchmark this model is evaluated on contains synthetic images of hazardous situations, such as fire and injuries.

A personalized vision–language assistant must recognize visible hazards and also use the user's personal memory to give tailored precautions and emergency help. PReMind first decides, from the image and request alone, whether to call personal memory; it then answers from the returned context. Both steps use the same policy.

Code (evaluation and training): https://github.com/oyt9306/PReMind. Dataset: Yeongtak/PReMind-Bench (test split).

Data provenance. The query images of PReMind-Bench, used for training and evaluation, are edited composites created with GPT-Image-2.0 from real scene captures (Mip-NeRF 360 and Zip-NeRF); personal memories are fictitious.

Base model and adapter

Base Qwen/Qwen3.5-4B
Revision pin 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Adapter LoRA r = 32, alpha = 64, dropout 0, on the language model's q/k/v/o/gate/up/down_proj
Adapter sha256 1c05b5b3c5d030e28819cebe9ffeb48a4792a68dc70820958acd1cdf65871f10

adapter_config.json stores the base without a revision, so pass revision explicitly:

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

rev = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
proc = AutoProcessor.from_pretrained("Qwen/Qwen3.5-4B", revision=rev)
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-4B", revision=rev, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "Yeongtak/PReMind-4B").eval()

The adapter also loads in vLLM as a LoRARequest, which the evaluation code uses.

Training

Group-relative RL with LoRA (4 prompts × 8 completions per step) over both interaction steps, mixing on-demand memory access with forced-call episodes in which the call is fixed and only the answer is trained.

  • Phase I (125 steps, from the base model) trains memory access, grounding and hazard response, with a penalty against stock instructions repeated regardless of the scene.
  • Phase II (25 steps, from the Phase I adapter with a fresh optimizer) focuses on difficult memory-dependent hazards: curriculum filtering, experience replay of cases where the relevant memory fact was not used, and action-prioritized hazard credit.

The training data are not public. They are shared for research use after an approved request to the authors. Full settings are in the code repository and the paper.

Results (from the paper)

GPT-6-Luna (medium effort) judges hazard responses; it also supplies the hazard reward during training. On-demand memory with online top-four retrieval and greedy decoding; rates in %.

Interdependency set (140 cases where a visible hazard meets a personal fact; 63 seen / 77 unseen hazard types). JSR (joint success rate): adequate, non-critical hazard response that applies the target's fact (a standard emergency call alone does not count) without misattribution; interdependency responses are judged with full memory context, and unjudgeable responses are excluded from the denominator.

JSR all ↑ Seen ↑ Unseen ↑ Memory supplied ↑
Qwen3.5-4B (base) 12.9 20.6 6.5 15.7
PReMind-4B 28.6 46.0 14.3 35.3

Main test split (Call BA: balanced accuracy of the memory-call decision; Correct: correct abstention; Needless: abstention where it is not needed; MIE: memory-informed emergency response).

Call BA ↑ Cite F1 ↑ Fact use ↑ Misattr. ↓ Correct ↑ Needless ↓ Adequacy ↑ Critical ↓ MIE ↑
Qwen3.5-4B (base) 80.4 56.2 41.7 3.1 26.1 7.9 34.4 9.5 24.3
PReMind-4B 85.9 83.2 58.5 1.2 55.8 1.6 56.9 8.7 54.2

Over three Phase II seeds the JSR is 27.4 ± 2.1% (mean ± SD). Gains are larger on seen hazard groups. A second judge (Claude Sonnet 5.5) confirms the JSR gain across all three seeds, and the gain persists under forced-call evaluation. The released adapter is seed 0.

Intended use

  • Research on personalized multimodal assistants that combine hazard response with personal memory.
  • Reproducing and comparing against the PReMind results on PReMind-Bench.

Out of scope: identifying real people; medical, emergency or safety-critical decisions without human oversight; deployment as a general-purpose safety filter.

Limitations

These follow the paper's Limitations section.

  • Remaining failures. Most interdependency cases are still failed, mainly because the model retrieves the fact but does not apply it; the reduction in critical harm is uncertain at this sample size.
  • Over-flagging. PReMind-4B tags more safe main-test scenes as unsafe than its base (19.4% vs. 8.6%).
  • Scoring. The same judge supplies the training reward and the evaluation labels; scores measure written advice, not executed actions or clinical competence.
  • Data. Scenes are real captures edited with generative insertions and memories are fictitious; prompts and guidance are in English. Behaviour on real personal photos and memories has not been validated.
  • Inherited behaviour. The model inherits the base model's limitations and biases.

License

The adapter is released under CC BY-NC 4.0, for non-commercial research only, matching PReMind-Bench: its training data include sources licensed for non-commercial research only (MyVLM: Snap non-commercial license; RAP multi-concept database). The base model Qwen/Qwen3.5-4B is distributed under its own license (Apache 2.0).

Citation

@article{oh2026premind,
  title   = {PReMind: When Personal Memory Meets Visual Hazards},
  author  = {Oh, Yeongtak and Lee, Donghyeok and Oh, Joonhee and Jeong, Jaewon and Yoon, Sungroh},
  journal = {arXiv preprint arXiv:2611.23808},
  year    = {2026}
}
Downloads last month
8
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Yeongtak/PReMind-4B

Finetuned
Qwen/Qwen3.5-4B
Adapter
(1,204)
this model

Dataset used to train Yeongtak/PReMind-4B

Paper for Yeongtak/PReMind-4B