Update pb27_nex/README.md with O(1) decode and latest benchmarks
Browse files- pb27_nex/README.md +66 -0
pb27_nex/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: vllm
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
tags:
|
| 6 |
+
- pseudobrain
|
| 7 |
+
- qwen3.5-moe
|
| 8 |
+
- qwen3.8
|
| 9 |
+
- abliterated
|
| 10 |
+
- moe
|
| 11 |
+
- gated-deltanet
|
| 12 |
+
- long-context
|
| 13 |
+
- 1m-context
|
| 14 |
+
- constant-memory
|
| 15 |
+
- nvfp4
|
| 16 |
+
- vllm
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# PseudoBrain-Nex-35B-A3B (`pb27-nexc` / `PB27MoEForConditionalGeneration`)
|
| 20 |
+
|
| 21 |
+
**GitHub Repository & vLLM Plugin:** [https://github.com/defnotean/pseudobrain](https://github.com/defnotean/pseudobrain)
|
| 22 |
+
**Hugging Face Weights:** [https://huggingface.co/defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4](https://huggingface.co/defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4)
|
| 23 |
+
**Companion Models:** [`PseudoBrain-Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/defnotean1/PseudoBrain-Qwen3.8-Flash-Next-NVFP4) (`45–183 tok/s`) | [`PseudoBrain-27B-NVFP4`](https://huggingface.co/defnotean1/PseudoBrain-27B-NVFP4) (`27B` Dense)
|
| 24 |
+
**Architecture Paper:** [`docs/PSEUDOBRAIN_ARCHITECTURE_PAPER.md`](https://github.com/defnotean/pseudobrain/blob/main/docs/PSEUDOBRAIN_ARCHITECTURE_PAPER.md)
|
| 25 |
+
|
| 26 |
+
## Overview
|
| 27 |
+
|
| 28 |
+
**PseudoBrain-Nex-35B-A3B (`pb27-nexc`)** is a sparse Mixture-of-Experts (`35B` total parameters, `~3B` active parameters per token across `256` routed experts + `1` shared expert) constant-GPU-memory hybrid long-context conversion of `Huihui-Qwen3.5-35B-A3B-abliterated` / `Nex-N2.5` (`Qwen3_5MoeForConditionalGeneration`). All `10` full-attention layers (`[3, 7, 11, ..., 39]`) are converted into **Hybrid Windowed-Attention + Parallel Gated DeltaNet** blocks (`PB27MoEDecoderLayer`) alongside `30` native Gated DeltaNet layers and PseudoBrain's **Three-Tier NVMe/Page-Cache Exact Memory Archive** with **True $O(1)$ Constant-Speed Decode** (`pb27_vllm`).
|
| 29 |
+
|
| 30 |
+
- **True $O(1)$ Constant-Speed Decode up to `1,000,000` Tokens (`~25–33.6 tok/s` Flat):**
|
| 31 |
+
- **Decoupled Prefill vs. Decode Retrieval Budget:** Full scaled retrieval (`PB27_MEMORY_SCALE=2048:384:4096:512:8192:1536`) runs once on the final prefill question chunk (`PB27_MEMORY_PREFILL=pt`) and pins the winning blocks in `Archive.prompt_top` + GPU-resident `cached_pre_k` / `cached_pre_v`.
|
| 32 |
+
- **Zero-NVMe Stride-`64` Decode Cache:** During decode, `PB27_MEMORY_DECODE_STRIDE=64`, `PB27_MEMORY_DECODE_BLOCKS=64`, and `PB27_MEMORY_DECODE_RERANK=0` keep LSE-merge FlashAttention-2 running over the fixed GPU buffer (`[4 sinks + prompt_top + 16K window]`), eliminating the historical `500K–1M` NVMe decode slowdown (`~8–10 tok/s` → **`~25–33.6 tok/s` constant $O(1)$ decode speed**).
|
| 33 |
+
- **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`):** Stores block `kmax`/`kmin` in `bf16` with `4,096`-token super-block bounds (`256` super-blocks at `1M` tokens), reducing bound scoring to `<0.05 ms` ($O(1)$).
|
| 34 |
+
- **Priority-Aware RoPE Sink Repositioning (`_reposition_rope_sinks`) & Cross-KV-Head Unification:** Dynamically re-rotates retrieved archive blocks into contiguous local RoPE positions just before the sliding window (`[sinks .. win_start - 1]`) with priority ordering preserved (`prio_rank`).
|
| 35 |
+
- **$O(1)$ Constant GPU KV Cache (`~27–31 GiB` Total GPU Footprint up to `1,000,000` Tokens):** Uses only `8.1%` of a fixed `6 GiB` KV cache pool even at `1,000,000` tokens.
|
| 36 |
+
- **100% (`40/40 = 1.000`) Exact Single-Needle, 4-Needle (`multi4`), and 4-Hop Call-Chain Trace (`trace4`) Retrieval from `8K` to `1,000,000` Tokens:** Combines median-centered Quest block bounds, per-query Z-score normalization, post-merge `[b, b+1]` successor expansion (`k=1024, c=8192` at 1M), immutable `prompt_top` + rolling `decode_top` retention, and position-invariant 2-hop key-to-key bridge retrieval over the $75\%$ unrotated key subspace (`d // 4 :`).
|
| 37 |
+
- **W4A16 Marlin NVFP4 MoE Scale Alignment:** Includes automatic `w1`/`w3` `weight_global_scale` harmonization (`_install_nvfp4_scale_hooks`), reducing fused SwiGLU scale discrepancy from $38.1\%$ to $2.9\%$.
|
| 38 |
+
|
| 39 |
+
## Benchmark Summary
|
| 40 |
+
|
| 41 |
+
| Context Length ($L$) | `needle@0.1` | `needle@0.5` | `needle@0.9` | `multi4` (`4` constants) | `trace4` (`4`-hop chain) | Mean TTFT | KV Pool Used |
|
| 42 |
+
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 43 |
+
| **`8,192`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`2.0s`** (`33.59 tok/s`) | `8.1%` |
|
| 44 |
+
| **`32,768`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`17.0s`** (`8.8s` warm) | `8.1%` |
|
| 45 |
+
| **`65,536`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`21.2s`** | `8.1%` |
|
| 46 |
+
| **`120,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`36.7s`** | `8.1%` |
|
| 47 |
+
| **`200,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`81.2s`** | `8.1%` |
|
| 48 |
+
| **`250,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`90.3s`** | `8.1%` |
|
| 49 |
+
| **`500,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`238.3s`** | `8.1%` |
|
| 50 |
+
| **`1,000,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`615.4s`** | `8.1%` |
|
| 51 |
+
|
| 52 |
+
- **Custom Medium-Hard Coding Benchmark (`nexc_med`, 33 problems × 2 samples):** **`26 / 33` problems solved (`78.8%`)**, **`43 / 66` sample pass rate (`65.2%`)** (`reports/pb27/nexc_med.summary.json`).
|
| 53 |
+
|
| 54 |
+
## Quickstart with vLLM
|
| 55 |
+
|
| 56 |
+
1. Clone the PseudoBrain repository and install the `pb27_vllm` out-of-tree vLLM plugin:
|
| 57 |
+
```bash
|
| 58 |
+
git clone https://github.com/defnotean/pseudobrain.git
|
| 59 |
+
cd pseudobrain
|
| 60 |
+
pip install -e serving/pb27_vllm
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
2. Serve the NVFP4 + BF16 GDN checkpoint directly (`O(1)` constant-speed decode defaults are enabled automatically):
|
| 64 |
+
```bash
|
| 65 |
+
bash serving/pb27_vllm/serve.sh defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4 8000
|
| 66 |
+
```
|