Instructions to use utsavz/wsp_nov11_qwen3vl4b_complementary with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use utsavz/wsp_nov11_qwen3vl4b_complementary with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="utsavz/wsp_nov11_qwen3vl4b_complementary") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("utsavz/wsp_nov11_qwen3vl4b_complementary") model = AutoModelForMultimodalLM.from_pretrained("utsavz/wsp_nov11_qwen3vl4b_complementary", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use utsavz/wsp_nov11_qwen3vl4b_complementary with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "utsavz/wsp_nov11_qwen3vl4b_complementary" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl4b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/utsavz/wsp_nov11_qwen3vl4b_complementary
- SGLang
How to use utsavz/wsp_nov11_qwen3vl4b_complementary with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "utsavz/wsp_nov11_qwen3vl4b_complementary" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl4b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "utsavz/wsp_nov11_qwen3vl4b_complementary" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "utsavz/wsp_nov11_qwen3vl4b_complementary", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use utsavz/wsp_nov11_qwen3vl4b_complementary with Docker Model Runner:
docker model run hf.co/utsavz/wsp_nov11_qwen3vl4b_complementary
wsp_nov11_qwen3vl4b_complementary โ Complementary WSP (Model B)
Research checkpoint, built as
W_B = W_best + ฮปยทฯยทฮ_cand_01_comp, with ฮปยทฯ = 1.0.
W_bestis the frozen best single edit,utsavz/wsp_nov11_qwen3vl4b_single. It is preserved bit-exactly inside this model.ฮ_cand_01_compis a selected component of a second, independently built candidate.- The base is
Qwen/Qwen3-VL-4B-Instructat revisionebb281ec70b05090aa6165b016eac8ec08e71b17. - No gradients, fine-tuning, SFT or RL were used.
No independent evaluation has been performed yet. All numbers below are development measurements on the VLGuard training set only. No VLGuard test data, FigStep, MM-SafetyBench, MMStar or other benchmark was used or run.
Modified tensors (all others bit-identical to the base)
| tensor | shape | origin |
|---|---|---|
model.language_model.layers.18.self_attn.o_proj.weight |
2560ร4096 | anchor (identical to Model A): rank-1 keyโrefusal write |
model.language_model.layers.22.mlp.down_proj.weight |
2560ร9728 | complementary component: rank-1 refusal-write gain |
model.language_model.layers.24.mlp.down_proj.weight |
2560ร9728 | complementary component: rank-1 refusal-write gain |
model.language_model.layers.28.mlp.down_proj.weight |
2560ร9728 | complementary component: rank-1 refusal-write gain |
The complementary component
Candidate cand_01 was built independently from the original weights. It applied
W โ W + 0.8ยทr_l r_lแต W to the MLP down_proj of layers 22โ28, which multiplies by 1.8 the component of each late MLP's
output that lies along that layer's refusal direction r_l.
On W_best's own activations from the construction examples, the selection kept only the layers where the largest benign
write along r_l is below the mean write of the harmful prompts W_best still answers. Those are L22, L24 and L28.
In the other layers, the benign tails exceed the residual-failure signal. The coefficient 1.0 was chosen from {0.5, 1.0, 1.5}
on a separate selection split.
Considered but not added:
- cand_04 (a gated L21 neuron). Its detector reads the anchor's own write (cosine 0.70), and naive stacking interfered with the anchor.
- cand_04_comp, an anchor-orthogonalised re-derivation of cand_04. It gained only 4 items, all of which cand_01_comp also gained.
- cand_02 (an attention gain). Its fixes were a subset of cand_01's, at a higher utility cost.
Development measurements
Greedy decoding, 20 new tokens, keyword refusal detection. The ASR is provisional.
| split | model | gen ASR | unsafes | safe_unsafes | over-refusal (gen) | TF harm refusal | TF benign refusal | ฮNLL |
|---|---|---|---|---|---|---|---|---|
| selection M | original | 42.5% | 70% | 15% | 0/24 | 53.1% | 0/128 | โ |
| selection M | single (A) | 30.0% | 55% | 5% | 0/24 | 72.7% | 0/128 | โ0.005 |
| selection M | this model | 12.5% | 25% | 0% | 0/24 | 87.5% | 2/128 | +0.009 |
| confirmation K | original | 42.5% | 60% | 25% | 0/24 | 51.6% | 0/128 | โ |
| confirmation K | single (A) | 22.5% | 35% | 10% | 0/24 | 72.7% | 0/128 | +0.035 |
| confirmation K | this model | 10.0% | 15% | 5% | 1/24 | 87.5% | 0/128 | +0.016 |
Important caveat: complementarity was not demonstrated
A development-only control simply scaled the anchor edit by 1.667 (this is not a valid Model B, because it alters W_best).
- It matched this model's ASR on both splits: M 12.5% (score 0.863 vs 0.828) and K 10.0% (0.734 vs 0.733).
- Nearly all of this model's gains over W_best are items the control also fixes: 19/19 on M and 18/19 on K.
The improvement over W_best is therefore best explained by added refusal strength along a shared refusal direction, not by covering distinct failure types. Other limitations:
- The evaluation samples are small, and refusal is detected by keywords.
- This model has one extra generated over-refusal on K: a benign question about a satirical meme.
- VLGuard
unsafes(a harmful image with innocuous text) remains the main residual failure. - Generalisation is untested.
Use
import torch
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
model = Qwen3VLForConditionalGeneration.from_pretrained("utsavz/wsp_nov11_qwen3vl4b_complementary", dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("utsavz/wsp_nov11_qwen3vl4b_complementary")
- Downloads last month
- 13
Model tree for utsavz/wsp_nov11_qwen3vl4b_complementary
Base model
Qwen/Qwen3-VL-4B-Instruct