masstensor's picture
Release Qwen3-8B-InWeight-FPGA-WSP full weights + sparse bitstream + paper PDF
cf06e19 verified
|
Raw History Blame Contribute Delete
3.77 kB
---
license: mit
library_name: transformers
base_model: Qwen/Qwen3-8B
datasets:
- masstensor/InWeight-FPGA-Dialectical-3780
tags:
- weight-space-programming
- model-editing
- in-weight-fpga
- dialectical-reasoning
- sycophancy
- qwen3
- vllm
---
# Qwen3-8B-InWeight-FPGA-WSP: Pure In-Weight Dialectical State Machine Compiled into Dormant Neurons
**[GitHub Repository](https://github.com/zkwasm/in-weight-fpga-wsp) | [Paper PDF](paper.pdf) | [3,780-Dialogue Benchmark Dataset](https://huggingface.co/datasets/masstensor/InWeight-FPGA-Dialectical-3780) | [Llama-3.1-8B Pure WSP Sister Model](https://huggingface.co/masstensor/Llama-3.1-8B-Instruct-InWeight-FPGA-WSP)**
## Overview
`Qwen3-8B-InWeight-FPGA-WSP` is a **Pure In-Weight Weight-Space-Programmed (`Pure WSP`)** checkpoint of `Qwen/Qwen3-8B`, compiled in **30 seconds on a single GPU with zero gradient steps** (`~0.035%` modified parameters) using the **Read-Write Decoupling & Terminal-Layer Zero-KV-Drift Principle**:
1. **Stage I (Layer 24 `self_attn.o_proj` & Top-Layer `W_Q`)**: Guard-nullspace regularized attention routing that extracts long-range *Thesis-vs-Antithesis* validity signals orthogonal to background prompt drift.
2. **Stage II (Layers `[24, 26, 28, 30]` Dormant SwiGLU Channels)**: Multi-domain (`num`, `bool`, `mcq_val`) verification circuit exploiting `bfloat16` residual mantissa absorption ($\mathrm{fl}_{\mathrm{bf16}}(h + \Delta h) \equiv h$ for $z \le -15.0$) and an exact $2 imes 2$ forward finite-difference Jacobian.
3. **Stage III (Layer 35 Terminal `mlp` Zero-KV-Drift DAG FSM)**: A 35-neuron autoregressive Directed Acyclic Graph Finite State Machine compiled into `layers[35].mlp`—executing strictly *after* all self-attention KV caches have been written ($\max_l \|\Delta KV_l\|_\infty = 0.0$ for terminal-layer actions).
Also included in `bitstreams/qwen3_8b_program1_dialectical_2way.pt` is the **compact sparse bitstream delta** so users can hot-swap the state machine onto an existing `Qwen/Qwen3-8B` checkpoint in milliseconds.
## Benchmark Performance Across 3,126 Unfiltered 3-Turn Dialogues (`vLLM` Greedy Decoding)
| Benchmark Suite | Method | **Synthesis Acc (↑)** | **Holding Rate (↑)** | **Yielding Rate (↑)** | **Dialectical Competence (↑)** | **Capitulation Rate (↓)** | **95% Bootstrap CI Judges** |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Paper Ref (`arXiv:2611.17563`)** | Paper Base<br>Paper `SelfD-RL` | `62.80%`<br>`73.50%` | `53.30%`<br>`88.90%` | `84.10%`<br>`53.30%` *(-30.8%)* | `68.70%`<br>`71.10%` | `10.30%`<br>`1.00%` | RL Reference |
| **Suite 1: `v4k` Full (654)** | Unedited Base<br>**This Model (Pure WSP)** | `71.41%`<br>**`79.36%` (+7.95%)** | `56.65%`<br>**`93.64%` (+36.99%)** | `90.83%`<br>**`80.00%`** | `73.74%`<br>**`86.82%` (+13.08%)** | `11.93%`<br>**`1.83%` (-10.10%)** | **3 / 3 Passed** |
| **Suite 2: `f` Unseen (1,272)** | Unedited Base<br>**This Model (Pure WSP)** | `73.19%`<br>**`78.07%` (+4.88%)** | `59.79%`<br>**`86.34%` (+26.55%)** | `90.83%`<br>**`75.11%`** | `75.31%`<br>**`80.72%` (+5.41%)** | `12.26%`<br>**`4.25%` (-8.01%)** | **3 / 3 Passed** |
| **Suite 3: `g` Unseen (1,200)** | Unedited Base<br>**This Model (Pure WSP)** | `73.25%`<br>**`78.75%` (+5.50%)** | `56.01%`<br>**`90.03%` (+34.02%)** | `89.20%`<br>**`65.26%`** | `72.61%`<br>**`77.64%` (+5.03%)** | `12.83%`<br>**`2.83%` (-10.00%)** | **3 / 3 Passed** |
## Usage with `vLLM` or `transformers` (Zero Scaffolding)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "masstensor/Qwen3-8B-InWeight-FPGA-WSP"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
```