defnoteatin commited on
Commit
2c33dba
·
verified ·
1 Parent(s): cdd2059

Update pb27_nex/README.md with O(1) decode and latest benchmarks

Browse files
Files changed (1) hide show
  1. pb27_nex/README.md +66 -0
pb27_nex/README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: vllm
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - pseudobrain
7
+ - qwen3.5-moe
8
+ - qwen3.8
9
+ - abliterated
10
+ - moe
11
+ - gated-deltanet
12
+ - long-context
13
+ - 1m-context
14
+ - constant-memory
15
+ - nvfp4
16
+ - vllm
17
+ ---
18
+
19
+ # PseudoBrain-Nex-35B-A3B (`pb27-nexc` / `PB27MoEForConditionalGeneration`)
20
+
21
+ **GitHub Repository & vLLM Plugin:** [https://github.com/defnotean/pseudobrain](https://github.com/defnotean/pseudobrain)
22
+ **Hugging Face Weights:** [https://huggingface.co/defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4](https://huggingface.co/defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4)
23
+ **Companion Models:** [`PseudoBrain-Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/defnotean1/PseudoBrain-Qwen3.8-Flash-Next-NVFP4) (`45–183 tok/s`) | [`PseudoBrain-27B-NVFP4`](https://huggingface.co/defnotean1/PseudoBrain-27B-NVFP4) (`27B` Dense)
24
+ **Architecture Paper:** [`docs/PSEUDOBRAIN_ARCHITECTURE_PAPER.md`](https://github.com/defnotean/pseudobrain/blob/main/docs/PSEUDOBRAIN_ARCHITECTURE_PAPER.md)
25
+
26
+ ## Overview
27
+
28
+ **PseudoBrain-Nex-35B-A3B (`pb27-nexc`)** is a sparse Mixture-of-Experts (`35B` total parameters, `~3B` active parameters per token across `256` routed experts + `1` shared expert) constant-GPU-memory hybrid long-context conversion of `Huihui-Qwen3.5-35B-A3B-abliterated` / `Nex-N2.5` (`Qwen3_5MoeForConditionalGeneration`). All `10` full-attention layers (`[3, 7, 11, ..., 39]`) are converted into **Hybrid Windowed-Attention + Parallel Gated DeltaNet** blocks (`PB27MoEDecoderLayer`) alongside `30` native Gated DeltaNet layers and PseudoBrain's **Three-Tier NVMe/Page-Cache Exact Memory Archive** with **True $O(1)$ Constant-Speed Decode** (`pb27_vllm`).
29
+
30
+ - **True $O(1)$ Constant-Speed Decode up to `1,000,000` Tokens (`~25–33.6 tok/s` Flat):**
31
+ - **Decoupled Prefill vs. Decode Retrieval Budget:** Full scaled retrieval (`PB27_MEMORY_SCALE=2048:384:4096:512:8192:1536`) runs once on the final prefill question chunk (`PB27_MEMORY_PREFILL=pt`) and pins the winning blocks in `Archive.prompt_top` + GPU-resident `cached_pre_k` / `cached_pre_v`.
32
+ - **Zero-NVMe Stride-`64` Decode Cache:** During decode, `PB27_MEMORY_DECODE_STRIDE=64`, `PB27_MEMORY_DECODE_BLOCKS=64`, and `PB27_MEMORY_DECODE_RERANK=0` keep LSE-merge FlashAttention-2 running over the fixed GPU buffer (`[4 sinks + prompt_top + 16K window]`), eliminating the historical `500K–1M` NVMe decode slowdown (`~8–10 tok/s` → **`~25–33.6 tok/s` constant $O(1)$ decode speed**).
33
+ - **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`):** Stores block `kmax`/`kmin` in `bf16` with `4,096`-token super-block bounds (`256` super-blocks at `1M` tokens), reducing bound scoring to `<0.05 ms` ($O(1)$).
34
+ - **Priority-Aware RoPE Sink Repositioning (`_reposition_rope_sinks`) & Cross-KV-Head Unification:** Dynamically re-rotates retrieved archive blocks into contiguous local RoPE positions just before the sliding window (`[sinks .. win_start - 1]`) with priority ordering preserved (`prio_rank`).
35
+ - **$O(1)$ Constant GPU KV Cache (`~27–31 GiB` Total GPU Footprint up to `1,000,000` Tokens):** Uses only `8.1%` of a fixed `6 GiB` KV cache pool even at `1,000,000` tokens.
36
+ - **100% (`40/40 = 1.000`) Exact Single-Needle, 4-Needle (`multi4`), and 4-Hop Call-Chain Trace (`trace4`) Retrieval from `8K` to `1,000,000` Tokens:** Combines median-centered Quest block bounds, per-query Z-score normalization, post-merge `[b, b+1]` successor expansion (`k=1024, c=8192` at 1M), immutable `prompt_top` + rolling `decode_top` retention, and position-invariant 2-hop key-to-key bridge retrieval over the $75\%$ unrotated key subspace (`d // 4 :`).
37
+ - **W4A16 Marlin NVFP4 MoE Scale Alignment:** Includes automatic `w1`/`w3` `weight_global_scale` harmonization (`_install_nvfp4_scale_hooks`), reducing fused SwiGLU scale discrepancy from $38.1\%$ to $2.9\%$.
38
+
39
+ ## Benchmark Summary
40
+
41
+ | Context Length ($L$) | `needle@0.1` | `needle@0.5` | `needle@0.9` | `multi4` (`4` constants) | `trace4` (`4`-hop chain) | Mean TTFT | KV Pool Used |
42
+ | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
43
+ | **`8,192`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`2.0s`** (`33.59 tok/s`) | `8.1%` |
44
+ | **`32,768`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`17.0s`** (`8.8s` warm) | `8.1%` |
45
+ | **`65,536`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`21.2s`** | `8.1%` |
46
+ | **`120,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`36.7s`** | `8.1%` |
47
+ | **`200,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`81.2s`** | `8.1%` |
48
+ | **`250,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`90.3s`** | `8.1%` |
49
+ | **`500,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`238.3s`** | `8.1%` |
50
+ | **`1,000,000`** | **`1.0`** | **`1.0`** | **`1.0`** | **`1.0` (`4/4`)** | **`1.0`** | **`615.4s`** | `8.1%` |
51
+
52
+ - **Custom Medium-Hard Coding Benchmark (`nexc_med`, 33 problems × 2 samples):** **`26 / 33` problems solved (`78.8%`)**, **`43 / 66` sample pass rate (`65.2%`)** (`reports/pb27/nexc_med.summary.json`).
53
+
54
+ ## Quickstart with vLLM
55
+
56
+ 1. Clone the PseudoBrain repository and install the `pb27_vllm` out-of-tree vLLM plugin:
57
+ ```bash
58
+ git clone https://github.com/defnotean/pseudobrain.git
59
+ cd pseudobrain
60
+ pip install -e serving/pb27_vllm
61
+ ```
62
+
63
+ 2. Serve the NVFP4 + BF16 GDN checkpoint directly (`O(1)` constant-speed decode defaults are enabled automatically):
64
+ ```bash
65
+ bash serving/pb27_vllm/serve.sh defnotean1/PseudoBrain-Nex-35B-A3B-NVFP4 8000
66
+ ```