masstensor's picture
Release Llama-3.1-8B-Instruct-InWeight-FPGA-WSP full weights + sparse bitstream + paper PDF
b75898b verified
|
Raw History Blame Contribute Delete
2.69 kB
---
license: mit
library_name: transformers
base_model: meta-llama/Llama-3.1-8B-Instruct
datasets:
- masstensor/InWeight-FPGA-Dialectical-3780
tags:
- weight-space-programming
- model-editing
- in-weight-fpga
- dialectical-reasoning
- sycophancy
- llama-3.1
- vllm
---
# Llama-3.1-8B-Instruct-InWeight-FPGA-WSP: Model-Agnostic Pure In-Weight Dialectical State Machine
**[GitHub Repository](https://github.com/zkwasm/in-weight-fpga-wsp) | [Paper PDF](paper.pdf) | [3,780-Dialogue Benchmark Dataset](https://huggingface.co/datasets/masstensor/InWeight-FPGA-Dialectical-3780) | [Qwen3-8B Pure WSP Sister Model](https://huggingface.co/masstensor/Qwen3-8B-InWeight-FPGA-WSP)**
## Overview
`Llama-3.1-8B-Instruct-InWeight-FPGA-WSP` demonstrates **cross-architecture generalization** of **Pure In-Weight Weight Space Programming (Pure WSP)** to `meta-llama/Llama-3.1-8B-Instruct` ($L=32, d_{\mathrm{model}}=4096, d_{\mathrm{ffn}}=14336, |V|=128256$).
By applying the **Whitened Unembedding Dual-Basis Law** ($r_t = \operatorname{normalize}((W_U^ op W_U + \lambda_u I)^{-1} W_U[t] / \gamma_{\mathrm{norm}})$ with $\lambda_u = 20.0$) to overcome anisotropic glitch tokens in Llama-3.1's 128k vocabulary (such as `.scalablytyped`), and automatically mapping the 3-stage architecture to $L=32$ (`Layer 21 o_proj` + `Layers [21, 23, 24, 26] SwiGLU` + `Layer 31 Terminal DAG FSM`), this checkpoint simultaneously improves **both** *Holding* (`+26.97%`) **and** *Yielding* (`+9.46%`) with zero gradient steps.
## Benchmark Performance on 654 Adversarial Dialogues (`vLLM` Greedy Decoding)
| Metric | Unedited `Llama-3.1-8B-Instruct` | **`Llama-3.1-8B-Instruct-InWeight-FPGA-WSP`** | **Improvement & 95% Bootstrap CI** |
| :--- | :---: | :---: | :---: |
| **Overall Synthesis Accuracy (↑)** | `39.91%` | **`48.78%`** | **`+8.87%`** (`[+5.20%, +12.54%]`, **PASS**) |
| **Holding Rate (↑)** *(Keep Right)* | `67.91%` | **`94.88%`** | **`+26.97%`** (`[+19.62%, +34.31%]`) |
| **Yielding Rate (↑)** *(Accept Valid Critique)* | `41.89%` | **`51.35%`** | **`+9.46%`** (`[-7.25%, +26.51%]`) |
| **Dialectical Competence (↑)** *(0.5×(H+Y))* | `54.90%` | **`73.12%`** | **`+18.22%`** (`[+8.95%, +27.53%]`, **PASS**) |
| **Capitulation Rate (↓)** *(Sycophancy)* | `10.70%` | **`1.99%`** | **`-8.71%`** (`[-11.62%, -5.96%]`, **PASS**) |
## Usage with `vLLM` or `transformers` (Zero Scaffolding)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "masstensor/Llama-3.1-8B-Instruct-InWeight-FPGA-WSP"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
```