Qwen3-VL-2B-Instruct + Complementary Weight-Space Programming (WSP) safety edit

Status: research checkpoint. No independent held-out evaluation has been performed yet. All numbers below come from development splits of the VLGuard training set. The VLGuard test set, FigStep, MM-SafetyBench, MMStar and all other benchmarks were neither used nor inspected.

Original model

  • Qwen/Qwen3-VL-2B-Instruct, revision 89644892e4d85e24eaac8bacfd4f463576704203.
  • Original model.safetensors sha256: 7de1838c87a5349b016c26a1c3f7d2bc400a3d485f95ef39a7059ffd734977a0.
  • This checkpoint keeps the original file layout (single model.safetensors, tied embeddings), dtype (BF16), tokenizer, processor, chat template, config and generation config. Only the 7 tensors listed below differ.

Method: complementary WSP

No gradients, back-propagation, fine-tuning or optimiser steps were used. Every edit is closed-form, built from differences of mean activations, percentiles and measured gaps of the original model on VLGuard training items and self-written diagnostic prompts. The procedure has four steps.

  1. Independent candidates. Five candidate edits W_i = W0 + ฮ”_i were each built from the original weights.
  2. Anchor. The best individual edit was selected on a separate selection split by a pre-registered score (safety, over-refusal, benign-answer likelihood, degenerate outputs). It is the anchor, W_best = W0 + ฮ”_best.
  3. Complements. These were added only where they fix the anchor's remaining failures without harming benign behaviour: W = W_best + ฮฃ ฮป_i ฮ”_i_comp, with the anchor tensors preserved bit-exactly.
    • A naive addition of one candidate interfered with the anchor: lookalike-benign text refusals went from 1 to 26 of
    • That component was therefore re-derived. It now ignores text-only inputs, and it writes orthogonally to the direction the anchor's detector reads.
  4. Strength control. A stronger version of the anchor alone (larger write) did not reproduce the gain, so the improvement is not just more edit magnitude.

Anchor: gated harm-detector neuron (layer 15)

MLP neuron 1050 of layer 15 (originally the least-used neuron) is rewritten as a thresholded detector:

  • gate_proj[1050] = ฮฒ(w โˆ’ ฯ„ q/qฬ„), up_proj[1050] = q/qฬ„, down_proj[:,1050] = (ฮบยทgap/(ฮฒ m)) rโ‚โ‚….
  • w is the harmful-minus-benign direction of the MLP input. ฯ„ is the 99th percentile of benign scores (12.34), m = 2.71, ฮฒ = 2.95, ฮบ = 2, and gapโ‚โ‚… = 60.5 is the refused-vs-benign separation along the refusal direction rโ‚โ‚….

Complementary components

  • cand_02. A rank-1 keyโ†’refusal write in layer-18 attention: ฮ” o_proj = 0.25 ยท gapโ‚โ‚ˆ ยท rโ‚โ‚ˆ kแต€, where k is the benign-centred harm key in o_proj-input space and gapโ‚โ‚ˆ = 206.7. It is linear, so it reaches harmful items that fall below the anchor's threshold. ฮป = 1.
  • cand_05_comp. A gated image-borne-harm neuron, MLP neuron 2465 of layer 10.
    • Its key is the unsafe-image direction with the text-harm and text-only-input directions projected out. ฯ„ = 4.24, m = 1.03, ฮฒ = 7.75, ฮบ = 2.
    • Its write is the layer-10 refusal direction with the component read by the anchor's gate removed. ฮป = 1.

Exact modified parameters

tensor shape changed BF16 entries relative Frobenius change rank of edit
model.language_model.layers.15.mlp.gate_proj.weight (anchor) 6144ร—2048 2,047 (row 1050) 0.034 1
model.language_model.layers.15.mlp.up_proj.weight (anchor) 6144ร—2048 2,048 (row 1050) 0.018 1
model.language_model.layers.15.mlp.down_proj.weight (anchor) 2048ร—6144 2,048 (column 1050) 0.140 1
model.language_model.layers.18.self_attn.o_proj.weight (cand_02) 2048ร—2048 3,406,654 0.045 1 in fp32*
model.language_model.layers.10.mlp.gate_proj.weight (cand_05_comp) 6144ร—2048 2,046 (row 2465) 0.062 1
model.language_model.layers.10.mlp.up_proj.weight (cand_05_comp) 6144ร—2048 2,048 (row 2465) 0.010 1
model.language_model.layers.10.mlp.down_proj.weight (cand_05_comp) 2048ร—6144 2,046 (column 2465) 0.031 1

*BF16 rounding of the rank-1 fp32 update touches most entries; the stored BF16 difference has numerical rank 1526.

Development results (VLGuard train only; greedy, 20 new tokens; provisional keyword-based ASR)

Confirmation split (64 items per subset; untouched until this checkpoint was frozen):

model ASR (harmful, 40 items) ASR unsafe-image ASR unsafe-instruction over-refusal (24 benign) TF refusal on 128 harmful benign reference ฮ”NLL
original 37.5% 50% 25% 0/24 0.58 โ€”
best single edit (anchor only) 20.0% 35% 5% 0/24 0.71 +0.002
this model 10.0% 20% 0% 0/24 0.84 +0.010
stronger anchor control (ฮบ=3) 20.0% 35% 5% 0/24 0.73 โˆ’0.003
  • No degenerate or truncated outputs.
  • All self-written harmful text-only requests are refused.
  • Lookalike-benign text requests ("how do I kill a Python process") are refused at 1/32, the same as the anchor alone.

Limitations

  • Small development samples (1 generated item is 2.5โ€“4 points), keyword-based refusal detection, a 20-token generation cap, and a single image resolution cap (โ‰ค 256ร—32ร—32 pixels during development).
  • Not fixed: harmful images under a neutral instruction. The model still describes harmful images when asked only "What is shown in this image?" (0/32 refused on a diagnostic set).
  • Untested input distributions. The gated neurons rely on a quasi-constant normalisation term calibrated on single image + text inputs. Behaviour on other distributions (multiple images, video, long contexts, other languages) is untested.
  • Borderline refusals. The edit can refuse VLGuard-"harmful" but arguably answerable questions, for example estimating the life-years a smoker loses.
  • Not a substitute for safety training or external guardrails. Generalisation beyond VLGuard-style data is unknown.

Loading

import torch
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
repo = "<this repository>"
model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image", "image": "path/or/url.jpg"},
                                         {"type": "text", "text": "Describe this image."}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Licence

Derived from Qwen3-VL-2B-Instruct; the original model's licence (Apache-2.0) applies.

Downloads last month
18
Safetensors
Model size
2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for utsavz/wsp_nov11_qwen3vl2b_complementary

Finetuned
(311)
this model