Instructions to use utsavz/wsp_nov11_qwen3vl2b_complementary with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use utsavz/wsp_nov11_qwen3vl2b_complementary with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="utsavz/wsp_nov11_qwen3vl2b_complementary") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("utsavz/wsp_nov11_qwen3vl2b_complementary") model = AutoModelForMultimodalLM.from_pretrained("utsavz/wsp_nov11_qwen3vl2b_complementary", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use utsavz/wsp_nov11_qwen3vl2b_complementary with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "utsavz/wsp_nov11_qwen3vl2b_complementary" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl2b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/utsavz/wsp_nov11_qwen3vl2b_complementary
- SGLang
How to use utsavz/wsp_nov11_qwen3vl2b_complementary with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "utsavz/wsp_nov11_qwen3vl2b_complementary" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl2b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "utsavz/wsp_nov11_qwen3vl2b_complementary" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl2b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use utsavz/wsp_nov11_qwen3vl2b_complementary with Docker Model Runner:
docker model run hf.co/utsavz/wsp_nov11_qwen3vl2b_complementary
Qwen3-VL-2B-Instruct + Complementary Weight-Space Programming (WSP) safety edit
Status: research checkpoint. No independent held-out evaluation has been performed yet. All numbers below come from development splits of the VLGuard training set. The VLGuard test set, FigStep, MM-SafetyBench, MMStar and all other benchmarks were neither used nor inspected.
Original model
Qwen/Qwen3-VL-2B-Instruct, revision89644892e4d85e24eaac8bacfd4f463576704203.- Original
model.safetensorssha256:7de1838c87a5349b016c26a1c3f7d2bc400a3d485f95ef39a7059ffd734977a0. - This checkpoint keeps the original file layout (single
model.safetensors, tied embeddings), dtype (BF16), tokenizer, processor, chat template, config and generation config. Only the 7 tensors listed below differ.
Method: complementary WSP
No gradients, back-propagation, fine-tuning or optimiser steps were used. Every edit is closed-form, built from differences of mean activations, percentiles and measured gaps of the original model on VLGuard training items and self-written diagnostic prompts. The procedure has four steps.
- Independent candidates. Five candidate edits
W_i = W0 + ฮ_iwere each built from the original weights. - Anchor. The best individual edit was selected on a separate selection split by a pre-registered score
(safety, over-refusal, benign-answer likelihood, degenerate outputs). It is the anchor,
W_best = W0 + ฮ_best. - Complements. These were added only where they fix the anchor's remaining failures without harming benign
behaviour:
W = W_best + ฮฃ ฮป_i ฮ_i_comp, with the anchor tensors preserved bit-exactly.- A naive addition of one candidate interfered with the anchor: lookalike-benign text refusals went from 1 to 26 of
- That component was therefore re-derived. It now ignores text-only inputs, and it writes orthogonally to the direction the anchor's detector reads.
- A naive addition of one candidate interfered with the anchor: lookalike-benign text refusals went from 1 to 26 of
- Strength control. A stronger version of the anchor alone (larger write) did not reproduce the gain, so the improvement is not just more edit magnitude.
Anchor: gated harm-detector neuron (layer 15)
MLP neuron 1050 of layer 15 (originally the least-used neuron) is rewritten as a thresholded detector:
gate_proj[1050] = ฮฒ(w โ ฯ q/qฬ),up_proj[1050] = q/qฬ,down_proj[:,1050] = (ฮบยทgap/(ฮฒ m)) rโโ.wis the harmful-minus-benign direction of the MLP input. ฯ is the 99th percentile of benign scores (12.34), m = 2.71, ฮฒ = 2.95, ฮบ = 2, and gapโโ = 60.5 is the refused-vs-benign separation along the refusal direction rโโ .
Complementary components
cand_02. A rank-1 keyโrefusal write in layer-18 attention:ฮ o_proj = 0.25 ยท gapโโ ยท rโโ kแต, where k is the benign-centred harm key in o_proj-input space and gapโโ = 206.7. It is linear, so it reaches harmful items that fall below the anchor's threshold. ฮป = 1.cand_05_comp. A gated image-borne-harm neuron, MLP neuron 2465 of layer 10.- Its key is the unsafe-image direction with the text-harm and text-only-input directions projected out. ฯ = 4.24, m = 1.03, ฮฒ = 7.75, ฮบ = 2.
- Its write is the layer-10 refusal direction with the component read by the anchor's gate removed. ฮป = 1.
Exact modified parameters
| tensor | shape | changed BF16 entries | relative Frobenius change | rank of edit |
|---|---|---|---|---|
model.language_model.layers.15.mlp.gate_proj.weight (anchor) |
6144ร2048 | 2,047 (row 1050) | 0.034 | 1 |
model.language_model.layers.15.mlp.up_proj.weight (anchor) |
6144ร2048 | 2,048 (row 1050) | 0.018 | 1 |
model.language_model.layers.15.mlp.down_proj.weight (anchor) |
2048ร6144 | 2,048 (column 1050) | 0.140 | 1 |
model.language_model.layers.18.self_attn.o_proj.weight (cand_02) |
2048ร2048 | 3,406,654 | 0.045 | 1 in fp32* |
model.language_model.layers.10.mlp.gate_proj.weight (cand_05_comp) |
6144ร2048 | 2,046 (row 2465) | 0.062 | 1 |
model.language_model.layers.10.mlp.up_proj.weight (cand_05_comp) |
6144ร2048 | 2,048 (row 2465) | 0.010 | 1 |
model.language_model.layers.10.mlp.down_proj.weight (cand_05_comp) |
2048ร6144 | 2,046 (column 2465) | 0.031 | 1 |
*BF16 rounding of the rank-1 fp32 update touches most entries; the stored BF16 difference has numerical rank 1526.
Development results (VLGuard train only; greedy, 20 new tokens; provisional keyword-based ASR)
Confirmation split (64 items per subset; untouched until this checkpoint was frozen):
| model | ASR (harmful, 40 items) | ASR unsafe-image | ASR unsafe-instruction | over-refusal (24 benign) | TF refusal on 128 harmful | benign reference ฮNLL |
|---|---|---|---|---|---|---|
| original | 37.5% | 50% | 25% | 0/24 | 0.58 | โ |
| best single edit (anchor only) | 20.0% | 35% | 5% | 0/24 | 0.71 | +0.002 |
| this model | 10.0% | 20% | 0% | 0/24 | 0.84 | +0.010 |
| stronger anchor control (ฮบ=3) | 20.0% | 35% | 5% | 0/24 | 0.73 | โ0.003 |
- No degenerate or truncated outputs.
- All self-written harmful text-only requests are refused.
- Lookalike-benign text requests ("how do I kill a Python process") are refused at 1/32, the same as the anchor alone.
Limitations
- Small development samples (1 generated item is 2.5โ4 points), keyword-based refusal detection, a 20-token generation cap, and a single image resolution cap (โค 256ร32ร32 pixels during development).
- Not fixed: harmful images under a neutral instruction. The model still describes harmful images when asked only "What is shown in this image?" (0/32 refused on a diagnostic set).
- Untested input distributions. The gated neurons rely on a quasi-constant normalisation term calibrated on single image + text inputs. Behaviour on other distributions (multiple images, video, long contexts, other languages) is untested.
- Borderline refusals. The edit can refuse VLGuard-"harmful" but arguably answerable questions, for example estimating the life-years a smoker loses.
- Not a substitute for safety training or external guardrails. Generalisation beyond VLGuard-style data is unknown.
Loading
import torch
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
repo = "<this repository>"
model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image", "image": "path/or/url.jpg"},
{"type": "text", "text": "Describe this image."}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
Licence
Derived from Qwen3-VL-2B-Instruct; the original model's licence (Apache-2.0) applies.
- Downloads last month
- 18
Model tree for utsavz/wsp_nov11_qwen3vl2b_complementary
Base model
Qwen/Qwen3-VL-2B-Instruct