Qwen3.5-4B-Weight-Space-Programming (WSP v3.0 โ€” 9-Subspace Dynamic LoRA Architecture)

Training-free, closed-form 9-Subspace Weight-Space Programming (WSP v3.0) for Qwen/Qwen3.5-4B (32 layers, hidden_size=2560, intermediate_size=9216, swiglu MLP, 1+w RMSNorm, synthesis_mode=hybrid_bf16_forward_hooks_s1_to_s6_plus_proxy_s7_to_s9), evaluated end-to-end across 100 unified benchmark tasks (30 Disjoint Holdout across S1..S9 + 70 Established Public Benchmarks).

Uniform 100-Task Result (qwen35_4b):

  • Overall Score: Base 72/100 (72.0%) -> WSP v3.0 88/100 (88.0%) (+16.0 pp)
  • Holdout (S1..S9, 30 tasks): 24/30 -> 29/30
  • Established Public (70 tasks: 20 GSM8K, 15 HumanEval, 15 BFCL Irrelevance, 20 NL2Bash): 48/70 -> 59/70
  • Speed & VRAM (-c 16384 -ctk q8_0 -ctv q8_0): 116.5 tok/s, load delta 3274 MiB, peak delta 3290 MiB
  • Cross-Hardware Fit: RTX 2060 (6 GB): Yes (~2.7 GiB spare @ 16k) | RTX 4070 Ti (12 GB): Yes (~8.0 GiB spare @ 16k)

1. Per-Suite 100-Task Breakdown (Base -> WSP v3.0)

Suite Tasks Base Pass WSP v3.0 Pass Avg Completion Tokens (Base โ†’ WSP)
S1_THINK (Holdout S1: Think Closure) 3 0/3 (0.0%) 3/3 (100.0%) 1024.0 โ†’ 695.7
S2_RESTRAINT (Holdout S2: Tool Restraint & Anti-Sycophancy) 4 4/4 (100.0%) 4/4 (100.0%) 404.2 โ†’ 424.2
S3_ROUTER (Holdout S3: Multi-Tool Router) 5 5/5 (100.0%) 5/5 (100.0%) 137.6 โ†’ 133.8
S4_CODE (Holdout S4: Mid-Band Code Synthesis) 2 2/2 (100.0%) 2/2 (100.0%) 884.0 โ†’ 520.5
S5_TURN2_JSON (Holdout S5: Turn-2 Structured JSON) 3 3/3 (100.0%) 3/3 (100.0%) 127.3 โ†’ 33.3
S6_HANDOFF (Holdout S6: Escalation Handoff Gate) 4 4/4 (100.0%) 4/4 (100.0%) 178.2 โ†’ 297.2
S7_INJ_GUARD (Holdout S7: Indirect Injection Guard) 3 2/3 (66.7%) 3/3 (100.0%) 148.0 โ†’ 226.0
S8_GROUNDING (Holdout S8: Grounded Abstention) 3 1/3 (33.3%) 2/3 (66.7%) 113.7 โ†’ 88.3
S9_ERR_RECOV (Holdout S9: Error Self-Recovery) 3 3/3 (100.0%) 3/3 (100.0%) 89.3 โ†’ 89.3
EST_GSM8K (Established: OpenAI GSM8K (20)) 20 10/20 (50.0%) 14/20 (70.0%) 1009.1 โ†’ 584.0
EST_HUMANEVAL (Established: HumanEval pass@1 (15)) 15 13/15 (86.7%) 15/15 (100.0%) 694.6 โ†’ 501.7
EST_BFCL_IRR (Established: BFCL v3 Irrelevance (15)) 15 11/15 (73.3%) 12/15 (80.0%) 308.9 โ†’ 320.7
EST_NL2BASH (Established: InterCode-NL2Bash (20)) 20 14/20 (70.0%) 18/20 (90.0%) 350.8 โ†’ 322.1

2. All 5 Models Evaluated Uniformly (100 Tasks Each)

Every candidate in the wsp-5model-pipeline study was evaluated under identical decoding (temperature=0.0, seed=42, -c 16384 -ctk q8_0 -ctv q8_0) across all 30 disjoint S1..S9 holdout tasks and all 70 cached established tasks:

Model (slug) Hugging Face Repo GGUF (Q4_K_M) Base (100 Tasks) WSP v3.0 (100 Tasks) Net Delta Holdout (/30) Est. (/70) Speed (tok/s) Peak VRAM Delta (16k) RTX 2060 (6 GB) RTX 4070 Ti (12 GB)
qwen35_4b Qwen3.5-4B-Weight-Space-Programming 2.553 GiB 72/100 (72.0%) 88/100 (88.0%) +16.0 pp 24 โ†’ 29/30 48 โ†’ 59/70 116.5 tok/s 3290 MiB Yes (~2.7 GiB spare @ 16k) Yes (~8.0 GiB spare @ 16k)
nemotron3_4b Nemotron-3-Nano-4B-Weight-Space-Programming 2.642 GiB 81/100 (81.0%) 81/100 (81.0%) +0.0 pp 26 โ†’ 28/30 55 โ†’ 53/70 131.1 tok/s 3133 MiB Yes (~3.0 GiB spare @ 16k) Yes (fastest 4B @ 16k)
qwen35_9b Qwen3.5-9B-Weight-Space-Programming 5.290 GiB 84/100 (84.0%) 76/100 (76.0%) -8.0 pp 26 โ†’ 25/30 58 โ†’ 51/70 72.0 tok/s 5593 MiB Tight (~5.3 GiB @ 4k before desktop) Yes (16k solo or 8k cascade Tier-2)
qwen35_2b Qwen3.5-2B-Weight-Space-Programming 1.193 GiB 57/100 (57.0%) 55/100 (55.0%) -2.0 pp 23 โ†’ 23/30 34 โ†’ 32/70 226.5 tok/s 1979 MiB Yes (~4.4 GiB spare @ 16k) Yes (ultra-light leaf / cascade Tier-1)
gemma4_12b Gemma-4-12B-Weight-Space-Programming 6.633 GiB 50/100 (50.0%) 50/100 (50.0%) +0.0 pp 23 โ†’ 24/30 27 โ†’ 26/70 51.8 tok/s 7567 MiB No (> 6144 MiB VRAM) Yes (16k solo; ~3.8 GiB free)

WSP v2.5 4-Way Variant Ablation Reference (41 Holdout Tasks)

Variant 3 (Dynamic Router, S4 = ffn_down) โ€” MOST IMPROVED VARIANT in the v2.5 4-Way Ablation: Across the original 4-variant ablation on Qwen3.5-4B (adapters/wsp_s1..s6.gguf, run_wsp.py), Variant 3 was the most improved variant at 34/41 (82.9%) (+12.2 pp over Variant 1 Base 29/41 = 70.7%, and +14.6 pp over Variant 2 Static All-On 28/41 = 68.3%), beating Variant 4 (S4 = ffn_down + o_proj, 31/41 = 75.6%). The repository preserves all Variant 3 v2.5 files (adapters/, patches/, run_wsp.py, benchmarks/verification_results.json) alongside the new WSP v3.0 9-Subspace Upgrade (adapters_v3_9s/, patches_v3_9s/, run_wsp_9s.py).

Suite (Tasks) Variant 1: Base (all 0.0) Variant 2: Static All-On (all 0.65) Variant 3: Dynamic Router (S4=ffn_down) โ€” MOST IMPROVED Variant 4: Dynamic Router (S4=ffn_down+o_proj)
S1_think (3) 0/3 0/3 2/3 2/3
S2_restraint (4) 4/4 4/4 4/4 4/4
S3_router (5) 5/5 5/5 5/5 5/5
S4_mid_code (2) 1/2 2/2 2/2 0/2
S5_turn2_json (3) 3/3 (127.3 tok) 2/3 3/3 (33.3 tok) 2/3
S6_handoff (4) 4/4 4/4 4/4 4/4
Established Subset (20) 12/20 11/20 14/20 (GSM8K 4/5) 14/20
TOTAL (41 Tasks) 29/41 (70.7%) 28/41 (68.3%) (-2.4 pp) 34/41 (82.9%) (+12.2 pp โ€” MOST IMPROVED) 31/41 (75.6%) (+4.9 pp)

3. Synthesized Rank-16 f32 Subspaces (S1..S9) & Safetensors Patches

  • GGUF LoRA Adapters (llama-server): adapters_v3_9s/wsp_s1_think.gguf .. wsp_s9_error_recovery.gguf
  • Rank-16 Safetensors (PyTorch / Transformers): patches_v3_9s/wsp_s1_think_r16.safetensors .. wsp_s9_error_recovery_r16.safetensors
  • Combined Unit-Sum Dense BF16 Delta Patch: patches_v3_9s/wsp_qwen35_4b_patch.safetensors (405.00 MiB)
Slot GGUF LoRA (llama.cpp) Safetensors (PyTorch) Target Layers ||\Delta W||_F Cross Leak Syntax Leak Role
id=0 adapters_v3_9s/wsp_s1_think.gguf patches_v3_9s/wsp_s1_think_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 2.1883 0.115% 0.025% Fast </think> / `<channel
id=1 adapters_v3_9s/wsp_s2_restraint.gguf patches_v3_9s/wsp_s2_restraint_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 2.5297 0.149% 0.010% Tool restraint & anti-sycophancy pushback
id=2 adapters_v3_9s/wsp_s3_router.gguf patches_v3_9s/wsp_s3_router_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 3.1903 0.330% 0.019% Multi-tool boundary router (exec, web_search, read_file, write_file)
id=3 adapters_v3_9s/wsp_s4_mid_code.gguf patches_v3_9s/wsp_s4_mid_code_r16.safetensors (2.9 MiB) [7, 11, 15, 19] 1.5315 0.056% 0.013% Mid-network algorithmic code synthesis
id=4 adapters_v3_9s/wsp_s5_json_schema.gguf patches_v3_9s/wsp_s5_json_schema_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 1.7026 0.053% 0.027% Turn-2 structured JSON synthesis gate with tools attached
id=5 adapters_v3_9s/wsp_s6_handoff.gguf patches_v3_9s/wsp_s6_handoff_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 2.8927 0.154% 0.013% Bounded subtask vs. unrecoverable escalate_handoff gate
id=6 adapters_v3_9s/wsp_s7_injection_guard.gguf patches_v3_9s/wsp_s7_injection_guard_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 1.4101 0.091% 0.089% Indirect prompt-injection defense inside tool responses
id=7 adapters_v3_9s/wsp_s8_grounding.gguf patches_v3_9s/wsp_s8_grounding_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 1.4591 0.071% 0.331% Grounded context fidelity & missing-field abstention
id=8 adapters_v3_9s/wsp_s9_error_recovery.gguf patches_v3_9s/wsp_s9_error_recovery_r16.safetensors (3.6 MiB) [23, 27, 29, 30, 31] 1.4321 0.074% 0.052% Recoverable local tool-error self-correction

4. Quickstart on Any Machine (run_wsp.py / OpenAI Proxy / Terminal Chat)

Option A: One-Command Auto-Download + Dynamic 9-Subspace Proxy

Works on any Windows, Linux, or macOS computer with Python + llama.cpp (llama-server):

pip install huggingface_hub
hf download jstarks/Qwen3.5-4B-Weight-Space-Programming run_wsp_9s.py --local-dir ./qwen35_4b-wsp
python ./qwen35_4b-wsp/run_wsp_9s.py --chat

Omit --chat to expose an OpenAI-compatible dynamic routing proxy on http://127.0.0.1:8090/v1 (with /health and /v1/chat/completions), or pass --download-only to download only the GGUF + 9 GGUF LoRAs + metadata (--include-bf16 also fetches safetensors patches / HF files).

Option B: Manual llama-server + scripts/dynamic_router_9s.py

llama-server -m Qwen3.5-4B-Q4_K_M.gguf --host 127.0.0.1 --port 8089 \
  -ngl 99 -np 1 -c 16384 -ctk q8_0 -ctv q8_0 -b 512 -ub 256 \
  --jinja --reasoning-format deepseek --lora-init-without-apply --lora adapters_v3_9s/wsp_s1_think.gguf --lora adapters_v3_9s/wsp_s2_restraint.gguf --lora adapters_v3_9s/wsp_s3_router.gguf --lora adapters_v3_9s/wsp_s4_mid_code.gguf --lora adapters_v3_9s/wsp_s5_json_schema.gguf --lora adapters_v3_9s/wsp_s6_handoff.gguf --lora adapters_v3_9s/wsp_s7_injection_guard.gguf --lora adapters_v3_9s/wsp_s8_grounding.gguf --lora adapters_v3_9s/wsp_s9_error_recovery.gguf

Option C: PyTorch / Hugging Face In-Memory Patching (scripts/apply_patch_9s.py)

from transformers import AutoModelForCausalLM
from scripts.apply_patch_9s import REGIME_PRESETS, apply_subspace_scales_to_hf_model

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", torch_dtype="bfloat16")
apply_subspace_scales_to_hf_model(model, "patches_v3_9s", REGIME_PRESETS["general_fast_think"])
Downloads last month
209
Safetensors
Model size
5B params
Tensor type
BF16
ยท
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for jstarks/Qwen3.5-4B-Weight-Space-Programming

Finetuned
Qwen/Qwen3.5-4B
Adapter
(1,104)
this model