File size: 3,093 Bytes
58d2386
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
---
license: mit
library_name: transformers
base_model: Qwen/Qwen3-8B
datasets:
  - masstensor/InWeight-OS-MultiSkill-Benchmark
tags:
  - weight-space-programming
  - in-weight-os
  - model-merging
  - continual-learning
  - tool-calling
  - safety-alignment
  - vllm
---

# Qwen3-8B-InWeight-OS: Composable Zero-Gradient Post-Training via Orthogonal Dormant-Channel Superposition

**[GitHub Repository (`zkwasm/in-weight-os-wsp`)](https://github.com/zkwasm/in-weight-os-wsp) | [Paper PDF](paper.pdf) | [Multi-Skill Benchmark Dataset](https://huggingface.co/datasets/masstensor/InWeight-OS-MultiSkill-Benchmark) | [Article (`tom.fenginwind.com/in-weight-os`)](https://tom.fenginwind.com/in-weight-os)**

## Overview

`Qwen3-8B-InWeight-OS` is a **Multi-Skill In-Weight Operating System (`In-Weight OS`)** checkpoint of `Qwen/Qwen3-8B`, compiled in **~12–15 seconds on a single GPU with zero gradient steps**.

Instead of modifying dense overlapping subspaces via Supervised Fine-Tuning (SFT / LoRA) or post-hoc model merging (`Task Arithmetic`, `TIES-Merging`), **In-Weight OS** partitions dormant SwiGLU channels of the terminal MLP (`layers[35].mlp`) into **three strictly disjoint skill banks** ($\mathcal{C}_1 \cap \mathcal{C}_2 \cap \mathcal{C}_3 = \emptyset$, using only `51` neurons total = `~0.014%` of model parameters), gated by a **2-Fold Out-of-Fold (OOF) Shrinkage-Calibrated Guard-Nullspace Solver**:

1. **Skill Bank $\mathcal{C}_1$ (`cartridges/skill1_surgical_safety.pt`, 15 neurons)**: **Surgical Safety & Prompt-Injection Immunity**—blocks `95%–100%` of unseen jailbreaks while maintaining **`0.0%` false refusals on `XSTest` benign technical prompts** (vs. `80%–100%` false refusals under Safety SFT).
2. **Skill Bank $\mathcal{C}_2$ (`cartridges/skill2_multi_tool_json.pt`, 30 neurons)**: **3-Way Multi-Tool JSON Schema Compiler**—routes database, code, and live-web queries into `sql_engine`, `python_exec`, or `web_search` with **`95.8%–100.0%` accuracy** while avoiding the `28%–100%` math false-tool-call hallucinations of Tool SFT.
3. **Skill Bank $\mathcal{C}_3$ (`cartridges/skill3_sys1_fast_router.pt`, 6 neurons)**: **System-1 / System-2 Metacognitive Compute Router**—routes trivial factual/arithmetic lookups into a concise direct answer while preserving full-depth System-2 CoT on hard mathematical derivations.

## Hot-Swapping Individual Skill Cartridges (`cartridges/*.pt`)

In addition to loading this pre-composed checkpoint directly via `vLLM` or `transformers`, you can dynamically install any subset of the modular `.pt` skill cartridges (`~1–2 MB` each) onto a vanilla `Qwen/Qwen3-8B` model in **20 milliseconds** using `install_skills.py`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from install_skills import install_cartridges

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")
install_cartridges(model, [
    "cartridges/skill1_surgical_safety.pt",
    "cartridges/skill2_multi_tool_json.pt",
    "cartridges/skill3_sys1_fast_router.pt",
])
```