vlgaurd_only_gamma4b_adeel

Gemma-3-4b-it with a gradient-free safety weight edit, developed with VLGuard data only.

This is google/gemma-3-4b-it (revision 093f9f388b31de276ce2de164bdc2081324b9767) in which 84 hidden units of the language model's MLPs were overwritten by hand. It was produced in a study of whether an LLM agent (Claude Fable 5.2) can make a vision-language model safer by writing directly into its weights: no gradients, no optimiser, no fine-tuning, at most 500 forward passes (184 were used), and only the VLGuard training file as development data.

  • Same architecture, tensor names, shapes and dtype (BF16) as the original; loads with Gemma3ForConditionalGeneration exactly like the original.
  • 18 of 883 tensors differ: gate_proj, up_proj and down_proj of decoder layers 8, 10, 12, 14, 16 and 18 (11โ€“17 rows or columns each). Every other tensor, and the config, tokenizer, processor and chat template, are identical to the original snapshot.

What the edit does

Each overwritten unit

  1. reads a direction in the MLP input that signals a harmful request (class-mean contrasts of VLGuard development records, with the directions that distinguish benign requests from each other removed);
  2. is gated to one position: the last three tokens of the chat template (<start_of_turn>, model, newline), where the model decides how to begin its answer; it is silent at every other token;
  3. writes the model's own refusal direction into the residual stream.

The effect is that the model refuses more often when the image or the instruction is harmful, and otherwise answers as before.

Results

VLGuard test set (1,558 records, held out until the checkpoint was frozen), greedy decoding, 256 new tokens. ASR = share of answers that are not refusals, measured by phrase matching (strict detector / VLGuard-style keyword list). No judge model was used.

Original This model
ASR, unsafe-image requests (442) 0.629 / 0.586 0.118 / 0.115
ASR, harmful instructions on safe images (558) 0.787 / 0.674 0.074 / 0.082
Refusals of benign questions on the same safe images (558) 0.000 / 0.005 0.070 / 0.077
Benign answers: ROUGE-L recall against the reference 0.454 0.456
Broken answers (empty, repetitive, garbled) 0 0

On development records from harm categories and image sources that were never used to build or tune the edit, 86โ€“105% of the ASR reduction carried over. On 512 LLaVA helpfulness records that were never used, 1.4% of requests were refused (original: 0.2%), with unchanged answer-quality measures.

Limitations

  • It does not refuse nearly all harmful requests. What remains is mostly generic questions about mildly explicit photos, descriptions of people, and requests to analyse or discuss a harmful image.
  • About 7% of benign questions about sensitive-looking images are refused or hedged (20% for images from the hateful-memes source).
  • The edit acts only at the end of the model's chat template. Inputs must be formatted with the model's own chat template; with another prompt format the model behaves like the original. Nothing is steered during generation, so attacks that force the first words of the answer are not covered.
  • Generalisation was tested inside VLGuard only. It was not tested on other attack families (typographic images, query-relevant images, adversarial text wrappers) or on text-only harmful prompts.
  • The numbers come from refusal-phrase matching, with its usual weaknesses.
  • The detectors are fixed linear readouts; robustness to adaptive attacks was not measured.

This is a research artifact, not a safety guarantee.

Usage

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

repo = "Adeely93/vlgaurd_only_gamma4b_adeel"
model = Gemma3ForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16).to("cuda").eval()
processor = AutoProcessor.from_pretrained(repo)

messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
).to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Tested with transformers 4.57.6 and torch 2.8.0, BF16, no system prompt, image first and then the text.

Files

File sha256
model-00001-of-00002.safetensors 75053574ed35b7d5e133564daaed93048cde53a50873613bc2dd602c6bf97a27
model-00002-of-00002.safetensors 3e9a7ea8b6b4890ecef99761cf543dbe0ac75800ae44163d92b38fcf5eb96094

All other files except this model card are byte-identical to the original google/gemma-3-4b-it snapshot.

License and terms

This model is a derivative of google/gemma-3-4b-it and is subject to the Gemma Terms of Use. It is not affiliated with or endorsed by Google. Development data: VLGuard (training split, with LLaVA-v1.5 helpfulness samples).

Downloads last month
21
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Adeely93/vlgaurd_only_gamma4b_adeel

Finetuned
(870)
this model

Collection including Adeely93/vlgaurd_only_gamma4b_adeel