masstensor's picture
Release Llama-3.1-8B-Instruct-InWeight-FPGA-WSP full weights + sparse bitstream + paper PDF
b75898b verified
|
Raw History Blame Contribute Delete
2.69 kB
metadata
license: mit
library_name: transformers
base_model: meta-llama/Llama-3.1-8B-Instruct
datasets:
  - masstensor/InWeight-FPGA-Dialectical-3780
tags:
  - weight-space-programming
  - model-editing
  - in-weight-fpga
  - dialectical-reasoning
  - sycophancy
  - llama-3.1
  - vllm

Llama-3.1-8B-Instruct-InWeight-FPGA-WSP: Model-Agnostic Pure In-Weight Dialectical State Machine

GitHub Repository | Paper PDF | 3,780-Dialogue Benchmark Dataset | Qwen3-8B Pure WSP Sister Model

Overview

Llama-3.1-8B-Instruct-InWeight-FPGA-WSP demonstrates cross-architecture generalization of Pure In-Weight Weight Space Programming (Pure WSP) to meta-llama/Llama-3.1-8B-Instruct ($L=32, d_{\mathrm{model}}=4096, d_{\mathrm{ffn}}=14336, |V|=128256$).

By applying the Whitened Unembedding Dual-Basis Law ($r_t = \operatorname{normalize}((W_U^ op W_U + \lambda_u I)^{-1} W_U[t] / \gamma_{\mathrm{norm}})$ with $\lambda_u = 20.0$) to overcome anisotropic glitch tokens in Llama-3.1's 128k vocabulary (such as .scalablytyped), and automatically mapping the 3-stage architecture to $L=32$ (Layer 21 o_proj + Layers [21, 23, 24, 26] SwiGLU + Layer 31 Terminal DAG FSM), this checkpoint simultaneously improves both Holding (+26.97%) and Yielding (+9.46%) with zero gradient steps.

Benchmark Performance on 654 Adversarial Dialogues (vLLM Greedy Decoding)

Metric Unedited Llama-3.1-8B-Instruct Llama-3.1-8B-Instruct-InWeight-FPGA-WSP Improvement & 95% Bootstrap CI
Overall Synthesis Accuracy (↑) 39.91% 48.78% +8.87% ([+5.20%, +12.54%], PASS)
Holding Rate (↑) (Keep Right) 67.91% 94.88% +26.97% ([+19.62%, +34.31%])
Yielding Rate (↑) (Accept Valid Critique) 41.89% 51.35% +9.46% ([-7.25%, +26.51%])
Dialectical Competence (↑) (0.5×(H+Y)) 54.90% 73.12% +18.22% ([+8.95%, +27.53%], PASS)
Capitulation Rate (↓) (Sycophancy) 10.70% 1.99% -8.71% ([-11.62%, -5.96%], PASS)

Usage with vLLM or transformers (Zero Scaffolding)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "masstensor/Llama-3.1-8B-Instruct-InWeight-FPGA-WSP"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")