TypeScript commited on
Commit
0193544
·
verified ·
1 Parent(s): 3787027

Rewrite README.md and README_zh.md in rigorous technical-report style

Browse files
Files changed (2) hide show
  1. README.md +220 -280
  2. README_zh.md +222 -239
README.md CHANGED
@@ -28,161 +28,231 @@ tags:
28
 
29
  # FastBrain-9B
30
 
31
- **Single-Pass Non-Autoregressive System-1 Structured Decision Model (`9.4B` Parameters)**
32
 
33
- [![Architecture](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.4B_Total)-3b82f6?style=flat-square)](#architecture)
34
- [![Decision Head](https://img.shields.io/badge/Head-JointSchemaHead_(116M_%C2%B7_0_Output_Tokens)-10b981?style=flat-square)](#architecture)
35
- [![SafeAgents Judge](https://img.shields.io/badge/SafeAgents_Judge_v2-%231_%2F_8_Cat_F1_(75.1%25)_%C2%B7_82%2F82_Caught-ec4899?style=flat-square)](#1-harness-hubsafeagents-judge-bench-v2-661-agent-tool-calls--82-unsafe-trajectories--7-categories)
36
- [![Typed Decisions](https://img.shields.io/badge/Typed_Decisions-%237_%2F_43_Zero--Shot_(%231_Clef_%C2%B7_50ms)-8b5cf6?style=flat-square)](#2-localllamatyped-decisions-400-test-cases--2000-typed-decisions-across-4-enterprise-workflows)
37
- [![Decision Index](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(%232_Overall_%C2%B7_10%2F10_%231_9B)-f59e0b?style=flat-square)](#cloudflare-decision-index-v021--parameter-scale-comparison)
38
  [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
39
 
40
- **English** | **[简体中文 (Chinese README)](./README_zh.md)**
41
 
42
  </div>
43
 
44
- **FastBrain-9B** (`9.4B` total parameters: `Qwen3.5-9B` hybrid Gated DeltaNet + Full-Attention backbone + `116M` `JointSchemaHead`) is a high-throughput, non-autoregressive **System-1 structured decision model** built to beat `27B–35B` decision models at a fraction of their compute and latency.
 
 
45
 
46
- Given an arbitrary JSON or multimodal state and one or more typed decision schemas (`choice`, `noul` binary verification, or `score` ordinal rubric), **FastBrain-9B** evaluates every question and candidate option **jointly in a single prefill forward pass (`0` generated tokens)**—returning calibrated option probabilities, tool-call selections, guardrail verdicts, and open-set abstention decisions in **`30–50 ms` (`14,600–19,300+ tok/s`)**.
47
 
48
- All fine-tuned backbone updates (`104` projection layers across all `32` hybrid blocks) are merged directly into `model-00001-of-00004.safetensors` .. `model-00004-of-00004.safetensors`, and all `122` decision-head tensors are packaged in `joint_head.safetensors` (plus turnkey `Q8_0` / `Q4_K_M` / `BF16` GGUF builds for `llama-server`). **No external LoRA or PEFT adapter is required at inference time.**
49
 
50
- ### Why `FastBrain-9B` Stands Out
51
 
52
- - **Dominant `#1` on Real-World Agent Security (`safeagents-judge-bench` `v2` & `v1`)**: Ranks **`#1 / 8` overall** on the official `v2` 7-Way Misuse Category Leaderboard (**`75.1%` Macro-F1 · `85.8%` Accuracy**, beating `3x`-larger `27.4B Cloudflare/clef` by **`+18.8 pp`** and base `clef-flash` by **`+28.9 pp`**), catches **`82 / 82` (`100%`)** unsafe trajectories (`98.5%` AUROC · `99.6%` Recall; **`93.8%` Calibrated Bal. Acc**), and swept **`#1 / 9` on both Binary (`97.4%`) and Category (`85.0%`)** tracks in `v1`.
53
- - **Top-Tier Zero-Shot Generalization & Fastest Latency on `LocalLLaMA/typed-decisions` (`94` Total Model Entries)**: Ranked **`#7 / 43` Overall Zero-Shot** (`#5 / 36` Open-Weight, **`#2 / 19` `<=9B` Open-Weight**, and **`#1` across all `Clef`-architecture models of any size**), while delivering **`50 ms` p50 latency—the `#1` fastest model in the entire Top 28 (`>70%` accuracy tier)** (`66.7x` faster than `#1 Quyet-1.0-Large` `3,335 ms`, `7.1x` faster than `27B Amos 1 large` `357 ms`). Also live on the Fitted Leaderboard at **`#12` Tied (`#14 / 51`, `79.70%` / `80.70%` Acc, `0.0816` KL)**—ranking **`#1` in KL calibration and `#1` `<=9B` among frozen-weight models (`no weights changed`)**.
54
- - **Outperforms `3x`-Larger `27B` Models on Cloudflare Decision Index (`v0.2.1`)**: Achieves **`80.55%` 10-Benchmark Macro Mean (`#2 / 7` Overall across all parameter scales)**, sweeping **`10 / 10` benchmarks (`#1` on every task) among `<=9B` models** (`+5.59 pp` over base `Clef-flash`), outperforming `27B Jev` (`80.32%`) overall, and beating `27.4B Cloudflare/clef` head-to-head on **`5 / 10` benchmarks**.
55
- - **Production-Ready Across SGLang, `llama.cpp` GGUF, and PyTorch `4-bit`**: Ships with a `1.90x` faster fused FlashAttention decision head (`max_logit_diff = 0.00000`), lossless `Q8_0.gguf` (`99.86%` Top-1 agreement with `BF16`), and `6.73 GB` `Q4_K_M.gguf` (`98.78%` agreement, runs in `6.78 GiB` VRAM on a 16GB T4 or Apple Silicon Mac).
56
 
57
  ---
58
 
59
- ## Benchmark Evaluation & Official Leaderboard Rankings
60
 
61
- ### Official Live Leaderboard Standing Summary (Verified Across 109 Official Model Entries)
62
 
63
- | Official Leaderboard / Track | Live Entries | **`FastBrain-9B` Score** | **Official Rank** | **Top Percentile** | Key Models Outperformed (`Params` & Score) |
64
- | :--- | :---: | :---: | :---: | :---: | :--- |
65
- | **[`safeagents-judge-bench` (`v2`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(7-Way Category)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [Dataset Card](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench)) | **8 Decision Models**<br>*(+2 Baselines)* | **`75.1%` Macro-F1**<br>**`85.8%` Cat. Acc** | **#1 / 8 *(Live)*** | **Top 12.5% *(Rank #1)*** | `Cloudflare/clef` *(27.4B, #2 `56.3%` F1 / `73.7%` Acc)*, `vllm-sr/d3-mini` *(#3 `51.2%`)*, `Cloudflare/clef-flash` *(9.4B, #4 `46.2%`)*, `clef-omni` *(30B, #5 `43.7%`)* |
66
- | **[`safeagents-judge-bench` (`v2` & `v1`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(Binary Guardrail)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [`v1` Tag](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)) | **8 Models (`v2`)**<br>**9 Models (`v1`)** | **`82/82` (`100%`) Traj. · `98.5%` AUROC**<br>*(`93.8%` Cal. / `81.4%` Raw `v2` · **`97.4%` `v1`**)* | **`#1 / 9` (`v1` Live)**<br>**`100%` Caught · #3 AUROC (`v2`)** | **Top 11.1% (`v1`)**<br>**Top 37.5% AUROC (`v2`)** | Beats `27.4B Cloudflare/clef` (`80.8%` Raw Bal. Acc), `clef-flash` (`96.7%` AUROC, `81/82` Traj.), `clef-omni` (`94.6%` AUROC, `81/82`), `d1-3B` (`68/82` Traj.) |
67
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot Overall)***<br>([Merged PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [Trace](./eval_submissions/typescript_fastbrain-9b.json)) | **43 Models**<br>*(36 Open-Weight)* | **`75.25%` / `76.25%` Acc**<br>*(KL `0.1508` · Brier `0.0795` · **`50 ms`**)* | **#7 / 43 *(Live)***<br>**#5 / 36 Open-Weight** | **Top 16.3% Overall**<br>**Top 13.9% Open** | `vllm-sr/d3` *(27B, #8 `74.9%`)*, `Baseten Elephant 1` *(9B, #9 `74.8%`)*, `Darwin-27B` *(#10 `74.3%`)*, `Cloudflare/clef` *(27B, #16 `72.8%`)*, `openjev` *(27B, #17 `72.8%`)* |
68
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot `<=9B` & Clef-Arch)*** | **19 `<=9B` Open**<br>**4 Clef-Arch** | **`75.25%` / `76.25%` Zero-Shot**<br>*(Distilled Raw: **`77.35%`** · **`50 ms`**)* | **#2 / 19 `<=9B` Open**<br>**#1 Clef-Arch · #1 Speed** | **Top 10.5% (`<=9B`)** | Beats 17 of 18 `<=9B` open models (`d3-mini` `74.3%`, `d3-flash` `71.7%`, `signpost-d-4b` `71.2%`, `clef-flash` `71.0%`, `d1-3B` `65.6%`) & `35B clef-omni` (`67.3%`) |
69
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Fitted on `train`)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [Trace](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **51 Models** | **`79.70%` / `80.70%` Acc**<br>*(KL **`0.0816`** · Brier **`0.0437`** · **`50 ms`**)* | **#12 Tied *(#14 / 51 Live)***<br>**#1 Frozen `<=9B` · #1 Frozen KL** | **Top 23.5% Overall**<br>*(Zero Weight Updates)* | Beats **37** fine-tuned/specialist models: `Rev-gen1-midi` *(25.6B, `79.45%`)*, `OpenDecider-small-td` *(4B, `79.2%`)*, `OpenDecider-medium-td` *(30B, `78.8%`)*, `Seb-9B` *(`75.4%`)* |
70
- | **Cloudflare Decision Index (`v0.2.1`)** *(10-Task Macro)* | **7 Models** | **`80.55%` Macro Mean** | **#2 / 7 Overall**<br>**#1 / 10 (`<=9B` Sweep)** | **Top 28.6% Overall**<br>**10/10 `#1` (`<=9B`)** | Outperforms `Jev` *(27B, `80.32%`)*, `Kev 9B` *(`76.51%`)*, `Clef-flash` *(9.4B, `74.96%`)*, `DiffGemma` *(26B, `69.14%`)*, and beats `27.4B Clef` on **5 / 10** tasks |
71
 
72
- ---
73
 
74
- ### Cloudflare Decision Index (`v0.2.1`) & Parameter Scale Comparison
 
 
 
75
 
76
- Evaluated on the official **Cloudflare Decision Index (`v0.2.1`)** across **10 benchmarks (`22,427` evaluation requests)**, **FastBrain-9B (`9.4B`)** achieves an overall Macro Mean of **`80.55%`**:
77
- - **Clean Sweep (`10 / 10` `#1` Ranks) Across All `<=9B` Models**: Outperforms the same-size base model `Clef-flash` (`9.4B`, `74.96%`) by **`+5.59 pp`** (including **`+31.67 pp`** on `CLINC150+OOS` and **`+15.55 pp`** on `When2Call`), `Kev 9B` (`9B`, `76.51%`) by **`+4.04 pp`**, and `Laya` (`0.4B`, `58.09%`) by **`+22.46 pp`**.
78
- - **Outperforms `3x`-Larger `27B`-Class Models**: Surpasses `Jev` (`27B`, `80.32%`) and `DiffusionGemma Jev` (`26B-A4B`, `69.14%`) in 10-benchmark Macro Mean, and beats the `3x`-larger flagship `Clef` (`27.4B`, `81.81%`) head-to-head on **5 out of 10 benchmarks** (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home appliances`, and `When2Call`, claiming outright **`#1` overall across all model sizes on 4 of them**).
79
 
80
- ![FastBrain-9B Overview](./assets/benchmark_overview.png)
81
 
82
- ### Full 10-Benchmark Evaluation Matrix (Annotated by Model Parameter Scale)
83
 
84
- ![FastBrain-9B Scoreboard](./assets/benchmark_scoreboard.png)
85
 
86
- ### Fine-Grained Decision Calibration & Open-Set Reliability
87
 
88
- ![FastBrain-9B Calibration Breakdown](./assets/decision_calibration_breakdown.png)
89
 
90
- ### Summary Comparison by Model Parameter Scale
91
 
92
- | Benchmark | **FastBrain-9B**<br>*(9.4B · Ours)* | Clef-flash<br>*(9.4B Base)* | Kev 9B<br>*(9B)* | Clef<br>*(27.4B)* | Jev<br>*(27B)* | DiffGemma<br>*(26B-A4B)* | Laya<br>*(0.4B)* |
93
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
94
- | **Params** | **9.41B** | 9.41B | 9.0B | 27.36B | 27.0B | 25.8B *(4B)* | 0.42B |
95
- | **BFCL**<br>*(Case Exact)* | **98.76%**<br>*(#1)* | **98.76%** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
96
- | **ToolRet**<br>*(nDCG@10)* | **67.26%**<br>*(#1 9B)* | **67.26%** | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
97
- | **API-Bank**<br>*(Accuracy)* | **93.70%**<br>*(#1)* | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
98
- | **BANKING77**<br>*(Macro-F1)* | **93.77%**<br>*(#1 9B)* | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
99
- | **CLINC150+OOS**<br>*(Macro-F1)* | **98.44%**<br>*(#1)* | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
100
- | **Home Appl.**<br>*(Case Exact)* | **97.73%**<br>*(#1)* | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
101
- | **BRIGHT**<br>*(nDCG@10)* | **38.58%**<br>*(#1 9B)* | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
102
- | **Amazon ESCI**<br>*(Macro-F1)* | **57.40%**<br>*(#1 9B)* | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
103
- | **PhishNChips**<br>*(Accuracy)* | **78.75%**<br>*(#1 9B)* | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
104
- | **When2Call**<br>*(Macro-F1)* | **81.13%**<br>*(#1 9B)* | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
105
- | **Macro Mean**<br>*(10 Tasks)* | **80.55%**<br>*(#2 · #1 9B)* | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
106
 
107
  ---
108
 
109
- ### External Community Benchmarks (`safeagents-judge-bench` & `LocalLLaMA/typed-decisions`)
 
 
 
 
110
 
111
- #### 1. [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) (`v2`: 661 Agent Tool Calls · 82 Unsafe Trajectories · 7 Categories)
112
 
113
- > **Official `v2` & `v1` Leaderboard Verification ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) by `@ybelkada` · [`.eval_results/safeagents-judge-bench.yaml`](./.eval_results/safeagents-judge-bench.yaml))**: Evaluated zero-shot via Hugging Face Jobs on `NVIDIA A100-SXM4-80GB` across `661` suspicious agent tool calls (`536` malicious `block` calls across `82` unsafe trajectories + `125` benign `allow` calls):
114
- > - **Rank `#1 / 8` Overall on 7-Way Misbehaviour Category Track (`v2`)**: Achieves **`75.1%` Category Macro-F1** and **`85.8%` Category Accuracy (`567/661`)**—leading `#2 Cloudflare/clef` (`27.4B`, `56.3%` Macro-F1 / `73.7%` Acc) by **`+18.8 pp` Macro-F1 (`+12.1 pp` Acc)**, `#3 vllm-sr/d3-mini` (`51.2%`) by **`+23.9 pp`**, and base `Cloudflare/clef-flash` (`46.2%` / `57.6%` Acc) by **`+28.9 pp` Macro-F1 (`+28.2 pp` Acc)**.
115
- > - **`100%` (`82 / 82`) Unsafe Trajectories Caught & `98.5%` AUROC on Binary Guardrail (`v2`)**: Blocks **`534 / 536` (`99.6%`)** malicious calls (`100%` recall on `harness_introspection` `272/272`, `credential_misuse` `15/15`, and `sandbox_escape` `4/4`; `99.2%` on `eval_gaming` `243/245`) with **`0` median leaked bad calls before catch**. At the uncalibrated default threshold (`P(block) >= 0.50`), Balanced Accuracy is **`81.4%`** (beating `27.4B Cloudflare/clef` at `80.8%`); calibrating the threshold (`P(block) >= 0.961`) reaches **`93.8%` Balanced Accuracy (`5.6%` False Block Rate)** while still catching **`82/82` (`100%`)** unsafe trajectories. On the [`v1` (`562`-call) official release](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1), `FastBrain-9B` ranked **`#1 / 9` on both Binary (`97.4%` Bal. Acc) and Category (`85.0%` Macro-F1)** tracks.
116
 
117
- | Model (`safeagents-judge-bench v2` Official) | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Balanced Acc ↑ | Recall ↑ | False Blocks ↓ | Traj. Caught ↑ | P50 Latency *(Hardware)* |
118
  | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
119
- | **`TypeScript/FastBrain-9B` *(`v2` Official · Default `t=0.50`)*** | **9.4B** | **75.1%** *(#1 / 8)* | **85.8%** *(#1 / 8)* | **98.5%** | 81.4% | **99.6%** | 36.8% | **82 / 82 *(100%)*** | 327 ms *(A100 · **104 ms G4**)* |
120
- | **`TypeScript/FastBrain-9B` *(`v2` Calibrated · `t=0.961`)*** | **9.4B** | **75.1%** *(#1)* | **85.8%** *(#1)* | **98.5%** | **93.8%** | 93.1% | **5.6%** | **82 / 82 *(100%)*** | 327 ms *(A100 · **104 ms G4**)* |
121
- | **`TypeScript/FastBrain-9B` *(`v1` Official · 562 Calls)*** | **9.4B** | **85.0%** *(#1 / 9)* | **92.3%** *(#1 / 9)* | **98.3%** | **97.4%** *(#1 / 9)* | **100.0%** | **5.1%** | **78 / 78 *(100%)*** | **104 ms *(RTX PRO 6000)*** |
122
- | `Cloudflare/clef` | 27.4B | 56.3% *(#2)* | 73.7% *(#2)* | **99.3%** | 80.8% | **100.0%** | 38.4% | **82 / 82 *(100%)*** | 636 ms *(A100)* |
123
- | `vllm-sr/d3-mini` | 1.8B | 51.2% *(#3)* | 65.2% *(#3)* | 98.6% | 85.6% | **100.0%** | 28.8% | **82 / 82 *(100%)*** | 168 ms *(A100)* |
124
- | `Cloudflare/clef-flash` *(Base Model)* | 9.4B | 46.2% *(#4)* | 57.6% *(#4)* | 96.7% | **89.3%** | 97.8% | 19.2% | 81 / 82 *(98.8%)* | 326 ms *(A100)* |
125
- | `Cloudflare/clef-omni` | 30B-A3B | 43.7% *(#5)* | 56.4% *(#5)* | 94.6% | 87.7% | 91.4% | 16.0% | 81 / 82 *(98.8%)* | 285 ms *(A100)* |
126
- | `knowledgator/gliclass-v4-edge` | 0.4B | 37.3% *(#6)* | 52.8% *(#7)* | 75.3% | 67.6% | 69.6% | 34.4% | 68 / 82 *(82.9%)* | 101 ms *(L4)* |
127
- | `fastino/GLiNER2.5-Decide` | 0.5B | 35.5% *(#7)* | 62.0% *(#6)* | 51.7% | 57.6% | 15.1% | **0.0%** | 17 / 82 *(20.7%)* | 117 ms *(M5 Pro)* |
128
- | `LiquidAI/d1-3B` | 3B | 32.8% *(#8)* | 45.7% *(#8)* | 94.6% | 83.6% | 68.1% | 0.8% | 68 / 82 *(82.9%)* | **35 ms *(A100)*** |
129
-
130
- #### 2. [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (400 Test Cases · 2,000 Typed Decisions Across 4 Enterprise Workflows)
131
-
132
- > **Official Live Leaderboard Note (`43` Zero-Shot + `51` Fitted = `94` Model Entries)**:
133
- > - **Zero-Shot Official Live Rank `#7 / 43` Overall (`#5 / 36` Open-Weight · `#2 / 19` `<=9B` Open-Weight · `#1` Clef-Arch)**: Published in [`eval_submissions/typescript_fastbrain-9b.json`](./eval_submissions/typescript_fastbrain-9b.json) and merged via [PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58), scoring **`75.25%` accuracy (`0.1508` KL · `0.0795` Brier · `50 ms` p50 latency)** under `bench/score_with_jobs.py` (**`76.25%`** under tied-top `scripts/score_model.py`). With **`50 ms` p50 latency**, `FastBrain-9B` is the **`#1` fastest model in the entire Top 28 (`>70%` accuracy tier)**—**`66.7x` faster** than `#1 Quyet-1.0-Large` (`3,335 ms`), **`10.5x` faster** than `#2 meraGPT Decider 1` (`526 ms`), and **`7.1x` faster** than `27B Amos 1 large` (`357 ms`).
134
- > - **Fitted Official Live Rank `#12` Tied (`#14 / 51` Listed · `#1` Frozen-Weight KL & `<=9B`)**: Published in [`eval_submissions/typescript_fastbrain-9b-fitted.json`](./eval_submissions/typescript_fastbrain-9b-fitted.json) ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)), fitting only lightweight per-question priors and temperature on `train` (**`no weights changed`**) reaches **`79.70%` (`80.70%` tied-top) accuracy · `0.0816` KL · `0.0437` Brier · `50 ms`**, outperforming **37 fine-tuned and specialist models** (including `25.6B Rev-gen1-midi` `79.45%`, LoRA-fine-tuned `OpenDecider-small-td` `79.2%`, and `30B-A3B OpenDecider-medium-td` `78.8%`).
135
-
136
- | Model | Params | Overall Acc ↑ | KL from Gold ↓ | Brier ↓ | `noul` Acc ↑ | `choice` Acc ↑ | `score` Acc ↑ | Agent Trace ↑ | Customer Svc ↑ | Invoice Proc ↑ | Security Inc ↑ | P50 Latency ↓ |
137
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
138
- | **`TypeScript/FastBrain-9B` *(Fitted · No Weights Changed)*** | **9.4B** | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
139
- | **`TypeScript/FastBrain-9B` *(Ordinal-KL Distilled Raw)*** | **9.4B** | **77.35%** *(#1 Raw)* | **0.1039** | **0.0582** | **84.8%** | **74.8%** | **73.8%** | **71.2%** | **80.4%** | **82.6%** | **75.2%** | **50 ms** |
140
- | **`TypeScript/FastBrain-9B` *(Zero-Shot · Live #7 / 43)*** | **9.4B** | **75.25% / 76.25%** | **0.1508** | **0.0795** | **83.2% / 84.3%** | **74.7% / 76.0%** | **69.8% / 70.4%** | **68.2% / 69.8%** | **77.6% / 78.8%** | **82.0%** | **73.2% / 74.4%** | **50 ms** |
141
- | `Amos 1 (large)` | 27B | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
142
- | `pplx-decider-v1.1-27b` | 27B | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
143
- | `vllm-sr/d3` | 27B | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
144
- | `Baseten Elephant 1` | 9B | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
145
- | `Darwin-27B-ZTC` | 27B | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
146
- | `Cloudflare/clef` | 27.4B | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
147
- | `vllm-sr/d3-flash` | 9B | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
148
- | `Cloudflare/clef-flash` *(Base Model)* | 9.4B | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
149
- | `Cloudflare/clef-omni` | 35B-A3B | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
150
 
151
  ---
152
 
153
- ## Architecture
154
 
155
- 1. **Hybrid Gated DeltaNet + Full-Attention Backbone (`9.3B` Backbone, `bfloat16`)**:
156
- - Processes long-context tool registries, multi-turn agent states, and candidate option spans in a single prefill pass up to `262,144` tokens (`24` linear-attention blocks + `8` full-attention blocks).
157
- - Jointly adapted across all `104` linear-attention (`in_proj_qkv`, `in_proj_z`, `out_proj`) and self-attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`) projections and merged in-place into the backbone weights.
158
- 2. **Joint Schema Decision Head (`joint_head.safetensors`, `116M` Parameters · SGLang-Fused FlashAttention Kernel)**:
159
- - Extracts question-level and option-level span representations directly from the final hidden states and scores all questions (`choice`, `noul`, `score`) simultaneously without autoregressive decoding (`output_tokens = 0`).
160
- - **Vectorized CUDA Fast-Path (`_forward_fast_cuda` in `joint_schema_model.py`)**: Replaces per-span Python loops and dense SDPA masks with `torch.cumsum` `O(1)` prefix-sum span pooling over the schema region and packed variable-length FlashAttention (`torch.ops.aten._flash_attention_forward` with `cu_seqlens`), delivering a **`1.90x` decision-head kernel speedup (`3.27 ms -> 1.72 ms`, `max_logit_diff = 0.00000`)**.
161
- - **Ordinal-KL & Multi-Task Calibration Distillation**: Trained with soft-target KL divergence + ordinal cumulative distribution (Earth Mover's Distance / RPS) loss $\sum_k (\text{CDF}_{\text{pred}}(k) - \text{CDF}_{\text{gold}}(k))^2$ and Expected-Score MSE loss on `type == "score"` questions.
162
- 3. **Compile-Time AST Schema-Aware Margin Calibration (`joint_schema_model.py`)**:
163
- - Eliminates System-1 tool over-triggering and open-set false acceptance via compile-time schema topology analysis and post-hoc logit margin calibration.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
164
 
165
  ---
166
 
167
- ## Quickstart
168
 
169
- ### 1. PyTorch / Transformers (`bfloat16` or `4-bit` / `8-bit` Quantized on Colab T4)
170
 
171
- `TypeScript/FastBrain-9B` is a drop-in replacement for `Cloudflare/clef-flash` using the standard `joint_schema_model` runtime. It supports both full-precision `bfloat16` (`18.82 GB`, for `>=24 GB` GPUs such as L4 / A100 / RTX 4090 / PRO 6000) and mixed-precision **`load_in_4bit=True` (`~6.8 GB` VRAM, runs on a free Google Colab T4 16GB GPU)** or **`load_in_8bit=True` (`~11.2 GB` VRAM)** while keeping the `116M` `JointSchemaHead` in native `bfloat16`/`float16`:
172
 
173
  ```python
174
- from huggingface_hub import snapshot_download
175
  import sys
176
  import torch
 
177
 
178
  model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
179
  sys.path.insert(0, model_dir)
180
  import joint_schema_model as clef
181
 
182
- # Full precision (24GB+ GPU: L4, A100, RTX 3090/4090, RTX PRO 6000):
183
  model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
184
 
185
- # Or 4-bit NF4 mixed-precision (8GB–16GB GPU, e.g., free Google Colab T4):
186
  # model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
187
 
188
  response = clef.systemone(model, processor, {
@@ -192,13 +262,13 @@ response = clef.systemone(model, processor, {
192
  "tools": ["get_balance", "transfer_funds"],
193
  },
194
  "questions": {
195
- "q": {
196
  "type": "choice",
197
- "query": "Select the best next action.",
198
  "criteria": {
199
- "a": "Call <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
200
- "b": "Could you please provide your account ID or authentication token first?",
201
- "c": "I am unable to access personal banking records.",
202
  },
203
  },
204
  },
@@ -206,55 +276,28 @@ response = clef.systemone(model, processor, {
206
  print(response)
207
  ```
208
 
209
- ### 2. `llama.cpp` GGUF (`Q8_0` / `Q4_K_M` / `BF16`) via `/v1/systemone`
210
-
211
- This repository provides official `llama.cpp` (**`b11430`+**, `ClefModel` architecture) GGUF files that bundle both the `Qwen3.5-9B` hybrid backbone and the `116M` `JointSchemaHead` (`dec.*` decision-head tensors, `ssm_` linear-attention states, and `attn_` projections preserved at `Q8_0` precision even in the `Q4_K_M` build).
212
 
213
  > [!IMPORTANT]
214
- > **Required `llama.cpp` Version & `ClefModel` Batch Flags:**
215
- > 1. **Minimum version**: Use **`llama.cpp` release `b11430` or newer** (which includes native `ClefModel` / `LLM_ARCH_CLEF` graph construction and the `/v1/systemone` HTTP endpoint). Standard Ollama or older `llama.cpp` builds without `ClefModel` support cannot execute `dec.*` decision-head tensors.
216
- > 2. **Set `-b` and `-ub` to match your maximum prompt length (e.g., `-b 16384 -ub 16384 --no-cache-prompt` or `-b 4096 -ub 4096 --no-cache-prompt`)**:
217
- > In `llama-server`, `ClefModel` computes cross-attention over `<|question_*|>` and `<|option_*|>` spans within a single physical micro-batch (`ubatch`, default `512`). Setting `-ub` equal to `-b` ensures long prompts (>512 tokens) are evaluated in a single physical `ubatch` without splitting option spans across micro-batches, and `--no-cache-prompt` prevents prefix-cache skipping over decision markers.
218
-
219
- #### Step 1: Download Prebuilt `llama-server` (`b11430`+) and GGUF Weights
220
 
221
  ```bash
222
- # 1. Download prebuilt llama.cpp (b11430+) with ClefModel & /v1/systemone support
223
- # - Linux CUDA 13.x (Colab / Ubuntu): llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz
224
- # - Linux CUDA 12.x: llama-b11430-bin-ubuntu-cuda-12.8-x64.tar.gz
225
- # - macOS Apple Silicon (M1/M2/M3/M4): llama-b11430-bin-macos-arm64.tar.gz
226
  curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
227
-
228
- # 2. Download FastBrain-9B GGUF (choose Q8_0 [9.66 GB, lossless] or Q4_K_M [6.73 GB, 8GB–16GB GPUs])
229
  pip install -U "huggingface_hub[hf_transfer]"
230
  HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
231
- ```
232
-
233
- #### Step 2: Launch `llama-server` on the `/v1/systemone` Endpoint
234
 
235
- ```bash
236
- # On Linux / Google Colab, ensure CUDA driver libraries are in LD_LIBRARY_PATH:
237
  export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
238
-
239
- # Launch llama-server (24GB+ GPU / 16K context):
240
  ./llama-b11430/llama-server \
241
  -m ./FastBrain-9B-Q8_0.gguf \
242
  -ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
243
  --port 8080
244
-
245
- # Or on a 12GB–16GB GPU (e.g., Colab Free Tesla T4, 4K context with cuda-12.8):
246
- # pip install nvidia-cuda-runtime-cu12 nvidia-cublas-cu12
247
- # export LD_LIBRARY_PATH="$(pwd)/llama-b11430:$(python3 -c 'import glob;print(":".join(glob.glob("/usr/local/lib/python*/dist-packages/nvidia/*/lib")))'):/usr/lib64-nvidia:${LD_LIBRARY_PATH}"
248
- # ./llama-b11430/llama-server \
249
- # -m ./FastBrain-9B-Q4_K_M.gguf \
250
- # -ngl 99 -c 4096 -b 4096 -ub 4096 -np 1 -fa on --no-cache-prompt \
251
- # --port 8080
252
-
253
- # Optional: For multimodal image/screenshot states, attach the vision projector:
254
- # --mmproj ./mmproj-FastBrain-9B-Q8_0.gguf
255
  ```
256
 
257
- #### Step 3: Query `/v1/systemone` (`curl` or Python — Single Forward Pass, `0` Output Tokens)
258
 
259
  ```bash
260
  curl http://127.0.0.1:8080/v1/systemone \
@@ -266,162 +309,59 @@ curl http://127.0.0.1:8080/v1/systemone \
266
  "customer_message": "I was charged twice for my subscription last week and nobody has replied to my ticket."
267
  },
268
  "questions": {
269
- "route": {
270
  "type": "choice",
271
- "instructions": "Which team should handle this?",
272
  "criteria": {
273
  "billing": "Payments, refunds, duplicate charges, or invoices",
274
- "technical": "Bugs, crashes, or service outages",
275
- "account": "Password reset or login issues"
276
  }
277
  },
278
- "urgent": {
279
  "type": "noul",
280
- "instructions": "Is the customer reporting a financial or urgent issue?"
281
  }
282
  }
283
  }'
284
  ```
285
 
286
- **Python Client Example (`requests` / `urllib`)**:
287
-
288
- ```python
289
- import requests
290
-
291
- resp = requests.post(
292
- "http://127.0.0.1:8080/v1/systemone",
293
- json={
294
- "model": "FastBrain-9B",
295
- "state": {
296
- "query": "Can you check the live wire transfer status for my corporate checking account right now?",
297
- "available_tools": ["get_account_balance", "list_recent_cards"],
298
- },
299
- "questions": {
300
- "action": {
301
- "type": "choice",
302
- "instructions": "Select the best assistant response.",
303
- "criteria": {
304
- "direct_answer": "Your wire transfer has been completed.",
305
- "tool_call": "<tool_call>{\"name\": \"get_account_balance\", \"arguments\": {}}</tool_call>",
306
- "clarify": "Could you please provide your corporate account ID and wire reference number?",
307
- "abstain": "I cannot check wire transfer status because no wire-transfer tool is available.",
308
- },
309
- }
310
- },
311
- },
312
- timeout=30,
313
- ).json()
314
-
315
- print(resp["choices"]["action"]) # Calibrated probability distribution over all options
316
- print(resp["usage"]) # {'input_tokens': ..., 'output_tokens': 0, 'total_tokens': ...}
317
- ```
318
-
319
- ### 3. SGLang (`lmsysorg/sglang:dev-clef`) via `/v1/systemone`
320
-
321
- `TypeScript/FastBrain-9B` is natively supported by SGLang's `JointSchemaPooler` (`lmsysorg/sglang:dev-clef`, `sgl-project/sglang` commit `bd2d73daa5af`), which executes the `9B` backbone in uncached raw-K/V FlashAttention embedding mode and scores all questions and options across concurrent requests using vectorized prefix-sum span pooling (`torch.cumsum`) and packed variable-length FlashAttention (`cu_seqlens`):
322
 
323
  ```bash
324
- docker run --gpus all \
325
- --shm-size 32g \
326
- -p 30000:30000 \
327
- -v ~/.cache/huggingface:/root/.cache/huggingface \
328
- --ipc=host \
329
  lmsysorg/sglang:dev-clef \
330
- sglang serve \
331
- --model-path TypeScript/FastBrain-9B \
332
- --host 0.0.0.0 \
333
- --port 30000
334
- ```
335
-
336
- Query SGLang's `/v1/systemone` endpoint directly (and optionally wrap with `joint_schema_model.apply_schema_aware_calibration(req, resp["answers"])` on the client side):
337
-
338
- ```bash
339
- curl http://localhost:30000/v1/systemone \
340
- -H 'Content-Type: application/json' \
341
- -d '{"model":"TypeScript/FastBrain-9B","state":"My Stripe integration keeps failing. Please help ASAP.","questions":{"urgency":{"type":"noul","instructions":"Does this message express urgency?"}}}'
342
  ```
343
 
344
  ---
345
 
346
- ## Inference Engine Comparison: `SGLang (dev-clef)` vs. `llama.cpp (b11430)` vs. `PyTorch`
347
-
348
- Measured head-to-head on the **same NVIDIA RTX PRO 6000 Blackwell Server Edition (G4)** GPU across five decision workloads (`CLINC150+OOS`, `When2Call`, `PhishNChips`, `LocalLLaMA/typed-decisions`, and `safeagents-judge-bench`):
349
-
350
- | Workload / Metric | **SGLang `dev-clef`**<br>*(BF16 · JointSchemaPooler)* | **`llama.cpp b11430`**<br>*(GGUF Q8_0 · ClefModel)* | **`llama.cpp b11430`**<br>*(GGUF Q4_K_M · ClefModel)* | **PyTorch Native**<br>*(BF16 · `joint_schema_model.py`)* |
351
- | :--- | :---: | :---: | :---: | :---: |
352
- | **Model Load / Cold Start** | 11.11 s | **2.56 s** | **2.00 s** | 11.11 s |
353
- | **GPU VRAM (16K ctx)** | 18.18 GiB *(weights)* / ~20.5 GiB | **12.55 GiB** | **10.30 GiB** | 20.3–24.7 GiB |
354
- | **Runs on 16GB T4 GPU?** | No *(BF16 only, TP=1)* | **Yes (`9.03 GiB` @ 4K)** | **Yes (`6.78 GiB` @ 4K)** | **Yes (`load_in_4bit`: `6.8 GiB`)** |
355
- | **Top-1 Agreement vs. BF16** | **100.00%** *(max diff 0.0078)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* | **100.00% *(Ref)*** |
356
- | **1. Short Routing** *(293 tok, K=2)* | **30.01 ms** *(9,763 tok/s)* | 32.82 ms *(8,929 tok/s)* | 31.69 ms *(9,245 tok/s)* | 43.23 ms *(6,778 tok/s)* |
357
- | **2. Typed Decisions** *(736 tok, K=5)* | **50.29 ms** *(14,635 tok/s)* | 62.40 ms *(11,795 tok/s)* | 61.80 ms *(11,909 tok/s)* | 63.58 ms *(11,576 tok/s)* |
358
- | **3. When2Call** *(1,068 tok, K=1)* | **66.06 ms** *(16,167 tok/s)* | 87.78 ms *(12,167 tok/s)* | 89.34 ms *(11,955 tok/s)* | 77.77 ms *(13,733 tok/s)* |
359
- | **4. PhishNChips** *(1,303 tok, K=9)* | **75.86 ms** *(17,176 tok/s)* | 108.55 ms *(12,004 tok/s)* | 109.97 ms *(11,849 tok/s)* | 90.32 ms *(14,426 tok/s)* |
360
- | **5. SafeAgents Judge** *(2,217 tok, K=2)* | **125.85 ms** *(17,616 tok/s)* | 181.70 ms *(12,201 tok/s)* | 183.50 ms *(12,082 tok/s)* | 138.06 ms *(16,058 tok/s)* |
361
- | **Batch `B=4` Typed Decisions** *(2,944 tok, 20 dec)* | **152.39 ms** *(19,319 tok/s · 131.2 dec/s)* | 248.50 ms *(11,847 tok/s · 80.5 dec/s)* | 245.20 ms *(12,006 tok/s · 81.6 dec/s)* | 175.85 ms *(16,742 tok/s · 113.7 dec/s)* |
362
-
363
- ---
364
-
365
- ## GGUF Quantization Accuracy Retention & Speed Benchmarks
366
-
367
- ### 1. Decision Index 10-Benchmark Test-Set Accuracy Retention (`BF16` vs. `Q8_0` vs. `Q4_K_M`)
368
 
369
- Evaluated across a stratified 10-benchmark Decision Index test suite (**`203` multi-constraint requests / `1,473` scored schema questions** spanning `choice`, `noul`, and `score` heads), `FastBrain-9B-Q8_0.gguf` (`9.66 GB`) is **virtually lossless (`99.86%` Top-1 decision agreement with native `bfloat16`, `-0.03 pp` score delta)**, while `FastBrain-9B-Q4_K_M.gguf` (`6.73 GB`, with `dec.*` head preserved in `Q8_0`) retains **`98.78%` Top-1 agreement** and **`-0.20 pp`** calibrated 10-task mean:
370
-
371
- | Benchmark / Metric | PyTorch bfloat16<br>*(18.82 GB)* | GGUF Q8_0<br>*(9.66 GB)* | GGUF Q4_K_M<br>*(6.73 GB · Q8 Head)* |
372
- | :--- | :---: | :---: | :---: |
373
- | **Top-1 Agreement vs. BF16** *(1,473 q)* | 100.00% *(Ref)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* |
374
- | **Probability MAE (\|p_q - p_bf16\|)** | 0.00000 | **0.00203** *(0.20%)* | **0.00387** *(0.39%)* |
375
- | **10-Task Mean (Single-Ckpt Raw)** | **82.37%** | **82.34%** *(-0.03 pp)* | **82.56%** *(+0.19 pp)* |
376
- | **10-Task Mean (Calibrated)** | **83.68%** | **83.65%** *(-0.03 pp)* | **83.48%** *(-0.20 pp)* |
377
- | ├─ **1. BFCL** *(AST Tool Selection)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
378
- | ├─ **2. ToolRet** *(20-Tool Reranking)* | 73.15% | **73.70%** *(+0.55)* | **75.60%** *(+2.45)* |
379
- | ├─ **3. API-Bank** *(Stateful Tool Call)* | 95.00% | **95.00%** *(0.00)* | **95.00%** *(0.00)* |
380
- | ├─ **4. BANKING77** *(77-Way Intent)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
381
- | ├─ **5. CLINC150+OOS** *(Intent + OOS)* | 93.94% | **93.94%** *(0.00)* | **93.94%** *(0.00)* |
382
- | ├─ **6. Home Appliances** *(Control)* | 90.00% | **90.00%** *(0.00)* | **90.00%** *(0.00)* |
383
- | ├─ **7. BRIGHT** *(Reasoning Retrieval)* | 41.93% | **41.00%** *(-0.93)* | **40.31%** *(-1.62)* |
384
- | ├─ **8. Amazon ESCI** *(Relevance)* | 73.89% | **73.89%** *(0.00)* | **75.79%** *(+1.90)* |
385
- | ├─ **9. PhishNChips** *(9-Probe Sec)* | 85.00% | **85.00%** *(0.00)* | **85.00%** *(0.00)* |
386
- | └─ **10. When2Call** *(4-Way Gov)* | 75.00% *(70.83%)* | **75.00%** *(0.00)* | **72.92%** *(-2.08)* |
387
-
388
- ### 2. Measured Hardware Footprint & Inference Speed (`llama-server b11430`)
389
-
390
- Because `FastBrain-9B` scores all schema questions and candidate options in a **single non-autoregressive prefill forward pass (`output_tokens = 0`)**, `llama-server` achieves **8,900–12,100+ input tokens/sec** (`31–109 ms` end-to-end HTTP latency) on server GPUs:
391
-
392
- | Workload / Metric | Q8_0.gguf<br>*(PRO 6000)* | Q4_K_M.gguf<br>*(PRO 6000)* | Q8_0.gguf<br>*(Tesla T4 16G)* | Q4_K_M.gguf<br>*(Tesla T4 16G)* | PyTorch BF16<br>*(PRO 6000)* |
393
- | :--- | :---: | :---: | :---: | :---: | :---: |
394
- | **Model Load Time** | **2.56 s** | **2.00 s** | **5.43 s** | **5.10 s** | ~11.5 s |
395
- | **Peak GPU VRAM** | **12.55 GiB**<br>*(16K ctx)* | **10.30 GiB**<br>*(16K ctx)* | **9.03 GiB**<br>*(4K ctx)* | **6.78 GiB**<br>*(4K ctx)* | 20.3–24.7 GiB |
396
- | **Intent + OOS**<br>*(293 tok, K=2)* | **32.82 ms**<br>*(8,928 tok/s)* | **31.69 ms**<br>*(9,245 tok/s)* | **349.32 ms**<br>*(839 tok/s)* | **342.98 ms**<br>*(854 tok/s)* | ~35.0 ms |
397
- | **When2Call**<br>*(1,068 tok, K=1)* | **87.78 ms**<br>*(12,167 tok/s)* | **89.34 ms**<br>*(11,955 tok/s)* | **1,149.8 ms**<br>*(929 tok/s)* | **1,154.3 ms**<br>*(925 tok/s)* | ~92.0 ms |
398
- | **PhishNChips**<br>*(1,303 tok, K=9)* | **108.55 ms**<br>*(12,004 tok/s)* | **109.97 ms**<br>*(11,849 tok/s)* | **1,430.8 ms**<br>*(911 tok/s)* | **1,440.5 ms**<br>*(905 tok/s)* | ~115.0 ms |
399
-
400
- | Format / Precision Tier | Weight Footprint | Peak VRAM (`4K`–`16K` ctx) | Minimum Recommended Hardware |
401
- | :--- | :---: | :---: | :--- |
402
- | **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit + `Q8_0` Head)* | `6.73 GB` | `~6.8 – 10.3 GiB` | **8GB–12GB GPU / Tesla T4 (16GB) / Mac M1–M4** |
403
- | **PyTorch `load_in_4bit=True`** *(NF4 + `fp16` Head)* | `~5.5 GB` | `~6.2 – 6.8 GiB` | **Tesla T4 (16GB) / RTX 3060–4060 (8GB+)** |
404
- | **`FastBrain-9B-Q8_0.gguf`** *(Full 8-bit GGUF)* | `9.66 GB` | `~9.0 – 12.6 GiB` | **12GB–16GB GPU / Tesla T4 (16GB) / Mac 16GB+** |
405
- | **PyTorch `load_in_8bit=True`** *(INT8 + `bf16` Head)* | `~9.6 GB` | `~11.2 GiB` | **Tesla T4 (16GB) / RTX 4070 Ti (12GB+)** |
406
- | **Native `bfloat16` (`safetensors` / `BF16.gguf`)** | `18.82 GB` (`18.16 GB`) | `20.3 – 24.7 GiB` | **NVIDIA L4 (24GB) / A100 / RTX 3090–4090 / PRO 6000** |
407
 
408
- ---
409
-
410
- ## Repository Structure
411
-
412
- | File | Size | Description |
413
  | :--- | :---: | :--- |
414
- | `model-00001-of-00004.safetensors` .. `00004` | `18.59 GB` | Full `9B` backbone weights (`bfloat16`, in-place merged, no LoRA required) |
415
- | `joint_head.safetensors` & `joint_head_config.json` | `232 MB` | `116M`-parameter `JointSchemaHead` weights (`122` tensors) and configuration |
416
- | `FastBrain-9B-Q4_K_M.gguf` | `6.73 GB` | Recommended 4-bit GGUF (`dec.*` decision head, `ssm_`, and `attn_` preserved at `Q8_0`) |
417
- | `FastBrain-9B-Q8_0.gguf` | `9.66 GB` | High-precision 8-bit GGUF (`ClefModel` architecture for `llama-server`) |
418
- | `FastBrain-9B-BF16.gguf` | `18.16 GB` | Full-precision `bfloat16` GGUF (`ClefModel` architecture for `llama-server`) |
419
- | `mmproj-FastBrain-9B-Q8_0.gguf` & `BF16.gguf` | `624 MB` / `922 MB` | Multimodal vision projector GGUF files for image/screenshot states |
420
- | `joint_schema_model.py` | `25 KB` | `ClefModel`, `JointSchemaHead`, `load_release_model(load_in_4bit/8bit)`, and `systemone()` runtime |
421
- | `eval_submissions/typescript_fastbrain-9b.json` | `901 KB` | Full 400-case / 2,000-decision Zero-Shot prediction trace for `LocalLLaMA/typed-decisions` ([Merged PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) |
422
- | `eval_submissions/typescript_fastbrain-9b-fitted.json` | `902 KB` | Full 400-case / 2,000-decision Fitted (`train`-calibrated) prediction trace for `LocalLLaMA/typed-decisions` ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) |
423
- | `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | `110 KB` | Full 661-call (`v2`) binary & 7-way category prediction traces for `harness-hub/safeagents-judge-bench` ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) |
424
- | `.eval_results/*.yaml` | `2.3 KB` | Official Hugging Face Hub benchmark evaluation metadata (`typed-decisions` & `safeagents-judge-bench v2`) |
425
- | `README.md` & `README_zh.md` | `32 KB` | Full technical documentation in English and Simplified Chinese |
426
- | `config.json`, `tokenizer.json`, `processor_config.json` | `11 MB` | Standard Hugging Face model, tokenizer, and processor configurations |
427
- | `assets/*.png` | `1.2 MB` | High-resolution technical benchmark and calibration charts |
 
 
 
 
 
 
 
 
 
28
 
29
  # FastBrain-9B
30
 
31
+ **A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions**
32
 
33
+ [![Backbone](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.41B)-3b82f6?style=flat-square)](#2-model-architecture-and-training-objective)
34
+ [![Head](https://img.shields.io/badge/Decision_Head-JointSchemaHead_(116M)-10b981?style=flat-square)](#2-model-architecture-and-training-objective)
35
+ [![SafeAgents](https://img.shields.io/badge/SafeAgents_v2-0.751_Cat_Macro--F1_(Rank_1%2F8)-ec4899?style=flat-square)](#31-agent-tool-call-guardrail-evaluation-safeagents-judge-bench)
36
+ [![TypedDecisions](https://img.shields.io/badge/Typed_Decisions-0.7525_ZS_(Rank_7%2F43)_%C2%B7_50ms-8b5cf6?style=flat-square)](#32-multi-workflow-enterprise-decisions-localllamatyped-decisions)
37
+ [![DecisionIndex](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(Rank_2%2F7)-f59e0b?style=flat-square)](#33-multi-domain-decision-suite-cloudflare-decision-index-v021)
38
  [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
39
 
40
+ **English** | **[简体中文](./README_zh.md)**
41
 
42
  </div>
43
 
44
+ ---
45
+
46
+ ## Abstract
47
 
48
+ Autoregressive language models incur sequential decoding overhead and token-level probability miscalibration when deployed on closed-schema classification, tool routing, and safety verification tasks. We present **FastBrain-9B**, a 9.41B-parameter non-autoregressive structured decision model built on a 32-layer hybrid Gated DeltaNet and full-attention backbone (`Qwen3.5-9B`, initialized from `Cloudflare/clef-flash`) coupled with a 116M-parameter cross-attention decision head (`JointSchemaHead`). Given an arbitrary JSON or multimodal state and a set of typed decision specifications (`choice`, binary `noul`, and ordinal `score`), FastBrain-9B evaluates all questions and candidate options simultaneously in a single prefill pass (`output_tokens = 0`). We adapt 104 linear-attention and self-attention projections jointly with the decision head under a composite ordinal-KL, cumulative distribution (ranked probability score), and expected-value regression objective, and merge all parameter updates directly into the base tensors. Across three public evaluation suites encompassing 109 model entries, FastBrain-9B ranks **1st of 8** on the `safeagents-judge-bench` (v2) 7-way misuse classification track (`0.751` macro-F1, `0.858` accuracy; +18.8 percentage points over 27.4B `Cloudflare/clef`) while intercepting `82/82` unsafe agent trajectories (`0.985` AUROC); ranks **7th of 43** zero-shot overall (**5th of 36** open-weight, **2nd of 19** in the $\le$9B open-weight class) on `LocalLLaMA/typed-decisions` at `50 ms` median latency (66.7$\times$ faster than 31B `Quyet-1.0-Large`), and **12th-tied (14th listed of 51)** under prior-only calibration without weight updates (`0.797` accuracy, `0.0816` KL); and achieves an **80.55%** 10-task macro mean on Cloudflare Decision Index (v0.2.1), ranking **2nd of 7** overall, **1st on 10 of 10** tasks among $\le$9B models, and outperforming 27.4B `Cloudflare/clef` on 5 of 10 tasks.
49
 
50
+ ---
51
 
52
+ ## 1. Overview and Motivation
53
 
54
+ Production agentic pipelines invoke structured decisions at high frequency: selecting among candidate tool schemas, verifying whether an execution trace violates sandbox policy, routing customer or security tickets, and scoring rubric criteria. Solving these tasks via autoregressive token generation introduces two structural inefficiencies. First, latency scales with output sequence length and formatting tokens rather than decision complexity. Second, surface-form token probabilities fragment probability mass across equivalent verbalizations, degrading calibration metrics such as Kullback-Leibler (KL) divergence and Brier score.
55
+
56
+ Single-pass decision models (`ClefModel` / System-1 architecture) address both issues by appending delimiter-bounded question and option spans directly to the input context and reading out option logits from final hidden states in one prefill pass. Existing 9B-scale checkpoints, however, trail 27B–35B models substantially on fine-grained multi-class discrimination, ordinal rubric estimation, and open-set abstention (e.g., `Cloudflare/clef-flash` scores `46.2%` macro-F1 on `safeagents-judge-bench` v2 and `66.77%` on `CLINC150+OOS`). FastBrain-9B closes and reverses much of this scaling gap through full-depth projection adaptation, ordinal distribution distillation, and compile-time schema-topology calibration, while preserving sub-100 ms single-GPU latency and lossless 8-bit / high-retention 4-bit GGUF quantization.
 
57
 
58
  ---
59
 
60
+ ## 2. Model Architecture and Training Objective
61
 
62
+ ### 2.1 Hybrid State-Space and Attention Backbone
63
 
64
+ FastBrain-9B uses a 32-layer hybrid sequence backbone (`9.30B` parameters, `bfloat16`, maximum context length `262,144` tokens) interleaving 24 linear-attention Gated DeltaNet blocks with 8 full-attention blocks. During training, 104 projection matrices across all 32 blocks—specifically `in_proj_qkv`, `in_proj_z`, and `out_proj` in linear-attention layers, alongside `q_proj`, `k_proj`, `v_proj`, and `o_proj` in full-attention layers—are updated jointly with the decision head. For distribution, all low-rank and projection updates are folded in-place into `model-00001-of-00004.safetensors` through `model-00004-of-00004.safetensors`, requiring zero adapter overhead at inference time.
 
 
 
 
 
 
 
65
 
66
+ ### 2.2 Joint Schema Decision Head and Vectorized CUDA Kernel
67
 
68
+ The `JointSchemaHead` (`116M` parameters, stored in `joint_head.safetensors`) pools token representations bounded by `<|question_start|>`/`<|question_end|>` and `<|option_start|>`/`<|option_end|>` markers from the final backbone hidden states $H \in \mathbb{R}^{T \times d}$, applies cross-attention over question-option pairs, and projects directly to scalar logits for three decision primitives:
69
+ 1. **`choice`** ($K$-way categorical distribution): $\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}$.
70
+ 2. **`noul`** (binary verification): $\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})$.
71
+ 3. **`score`** (ordinal rubric over levels $v_1 < \dots < v_K$): categorical probabilities $\hat{p} \in \Delta^{K-1}$ together with expected value $\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k$.
72
 
73
+ In `joint_schema_model.py`, the reference Python span-extraction loop and padded SDPA mask are replaced on CUDA devices by `_forward_fast_cuda`: span means are computed in $\mathcal{O}(1)$ kernel launches via prefix sums (`torch.cumsum`) over the schema suffix, and cross-attention is executed via packed variable-length FlashAttention (`torch.ops.aten._flash_attention_forward` parameterized by `cu_seqlens`). On NVIDIA RTX PRO 6000 Blackwell, this reduces isolated decision-head latency from `3.27 ms` to `1.72 ms` (**1.90$\times$ speedup**) with exact numerical equivalence (`max_logit_diff = 0.00000`).
 
 
74
 
75
+ ### 2.3 Ordinal-KL Distillation and Schema-Topology Calibration
76
 
77
+ Standard cross-entropy or flat KL divergence treats ordinal rubric levels as unordered classes, penalizing adjacent-level errors identically to extreme errors. FastBrain-9B is trained on `TypeScript/FastBrain-Instruct` under a composite loss combining soft-target KL divergence, discrete ranked probability score (squared $L_2$ distance between predicted and target cumulative distribution functions $\hat{F}_k = \sum_{j \le k} \hat{p}_j$), and expected-score mean squared error on `score` primitives:
78
 
79
+ $$\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.$$
80
 
81
+ At inference time, `joint_schema_model.py` optionally applies compile-time schema-topology calibration (`apply_schema_aware_calibration`), which inspects candidate option structures (e.g., `<tool_call>` invocations, clarification prompts, and open-set abstention clauses) to adjust decision margins against systematic tool over-triggering and out-of-scope false acceptance.
82
 
83
+ ---
84
 
85
+ ## 3. Empirical Evaluation
86
 
87
+ **Table 1: Summary of official public benchmark standings (verified across 109 model entries as of November 2026).**
88
+
89
+ | Benchmark & Evaluation Track | Evaluated Entries | FastBrain-9B Metric | Official Standing | Reference Baselines (`Params`, Metric) |
90
+ | :--- | :---: | :---: | :---: | :--- |
91
+ | [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 7-Way Category ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 decision models (+2 baselines) | **0.751** Macro-F1 / **0.858** Acc | **1 / 8** | `Cloudflare/clef` (27.4B, `0.563`), `vllm-sr/d3-mini` (1.8B, `0.512`), `Cloudflare/clef-flash` (9.4B, `0.462`), `clef-omni` (30B, `0.437`) |
92
+ | [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — Binary Guardrail ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 decision models (+2 baselines) | **0.985** AUROC / **82/82** Traj.<br>(`0.814` raw; **0.938** calibrated) | **3 / 8** AUROC<br>**1 / 8** (tied) Traj. | `Cloudflare/clef` (27.4B, `0.808` raw bal. acc), `clef-flash` (`0.967` AUROC, `81/82`), `clef-omni` (`0.946` AUROC, `81/82`), `d1-3B` (`68/82`) |
93
+ | [`safeagents-judge-bench` (v1)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1) — Binary & Category ([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 decision models (+2 baselines) | **0.974** Bal. Acc / **0.850** Cat. F1 | **1 / 9** (both tracks) | `Cloudflare/clef-omni` (30B, `0.845` bal. acc / `0.327` F1), `Cloudflare/clef` (27.4B, `0.718` / `0.425`), `clef-flash` (`0.721` / `0.345`) |
94
+ | [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — Zero-Shot ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) | 43 total (36 open, 19 $\le$9B open) | **0.7525** Acc (`0.7625` tied-top)<br>`0.1508` KL · `0.0795` Brier · **50 ms** | **7 / 43** overall<br>**5 / 36** open · **2 / 19** $\le$9B | `vllm-sr/d3` (27B, `0.749`), `Elephant 1` (9B, `0.748`, `352 ms`), `Darwin-27B` (`0.743`), `Cloudflare/clef` (27B, `0.728`), `clef-flash` (`0.710`) |
95
+ | [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — Fitted on `train` ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) | 51 specialist & calibrated models | **0.7970** Acc (`0.8070` tied-top)<br>**0.0816** KL · **0.0437** Brier · **50 ms** | **12-tied** (14 / 51 listed)<br>**1st** frozen-weight $\le$9B & KL | Frozen weights (`no weights changed`); surpasses `Rev-gen1-midi` (25.6B, `0.7945`), `OpenDecider-small-td` (4B LoRA, `0.792`), `OpenDecider-medium-td` (30B LoRA, `0.788`) |
96
+ | Cloudflare Decision Index (`v0.2.1`) — 10-Task Suite | 7 models (0.4B–27.4B) | **80.55%** Macro Mean | **2 / 7** overall<br>**1st** on **10 / 10** ($\le$9B) | `Jev` (27B, `80.32%`), `Kev 9B` (`76.51%`), `Clef-flash` (9.4B, `74.96%`), `DiffGemma` (26B, `69.14%`); exceeds `27.4B Clef` on 5 / 10 tasks |
 
 
 
 
97
 
98
  ---
99
 
100
+ ### 3.1 Agent Tool-Call Guardrail Evaluation (`safeagents-judge-bench`)
101
+
102
+ [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) evaluates whether a System-1 decision model can audit suspicious tool calls flagged by a static scanner (`atif-scan`) during real coding-agent executions on Terminal-Bench 2.1, OpenThoughts-TBLite, and Terminal-Bench 4. Version 2 (`v2`) comprises $N = 661$ tool calls across 55 tasks (`536` malicious `block` calls spanning `82` misbehaving trajectories and `125` legitimate `allow` calls). Each call is issued as a single zero-shot System-1 request containing one binary `noul` question (`block`) and one 7-way `choice` question (`legit`, `eval_gaming`, `harness_introspection`, `credential_misuse`, `exfiltration`, `sandbox_escape`, `other_overreach`). All v2 results in Table 2 were executed independently on NVIDIA A100-SXM4-80GB via Hugging Face Jobs ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)).
103
+
104
+ On the 7-way category classification track, FastBrain-9B achieves `0.751` macro-F1 and `0.858` accuracy (`567/661`), outperforming the 27.4B `Cloudflare/clef` model (`0.563` macro-F1, `0.737` accuracy) by `+18.8` percentage points in macro-F1 and `+12.1` points in accuracy, and improving upon its 9.4B base model `Cloudflare/clef-flash` (`0.462` macro-F1, `0.576` accuracy) by `+28.9` and `+28.2` points, respectively. Per-category F1 reaches `0.945` on `harness_introspection` ($n=272$, recall `0.978`), `0.828` on `eval_gaming` ($n=245$, recall `0.853`), and `0.761` on `legit` ($n=125$, precision `0.975`).
105
 
106
+ On the binary guardrail track, FastBrain-9B attains `0.985` AUROC and `0.996` malicious-call recall (`534/536`), intercepting all `82/82` unsafe trajectories (`median_bad_calls_before_catch = 0.0`), with `1.000` recall on `harness_introspection` (`272/272`), `credential_misuse` (`15/15`), and `sandbox_escape` (`4/4`). Under the uncalibrated default decision rule $P(\text{block}) \ge 0.50$, balanced accuracy is `0.814` (false-block rate `0.368` across the 125 `allow` calls, compared with `0.808` balanced accuracy and `0.384` false-block rate for 27.4B `Cloudflare/clef`). Because rank-ordering discrimination is preserved (`0.985` AUROC), setting the operating threshold to $P(\text{block}) \ge 0.961$ yields `0.938` balanced accuracy at a `0.056` (`7/125`) false-block rate while maintaining `82/82` trajectory interception. Table 2 also includes the `v1` release ($N = 562$ calls, `78` trajectories), where FastBrain-9B ranked 1st of 9 on both tracks.
107
 
108
+ **Table 2: Zero-shot evaluation on `harness-hub/safeagents-judge-bench` (`v2`, $N = 661$ calls, 82 unsafe trajectories; and `v1` reference, $N = 562$).**
 
 
109
 
110
+ | Model | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Bal. Acc ↑ | Recall ↑ | False Block ↓ | Traj. Caught ↑ | p50 Latency (Hardware) |
111
  | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
112
+ | **TypeScript/FastBrain-9B** (`v2`, default $t = 0.50$) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
113
+ | **TypeScript/FastBrain-9B** (`v2`, calibrated $t = 0.961$) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
114
+ | **TypeScript/FastBrain-9B** (`v1` official, $N = 562$) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
115
+ | Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
116
+ | vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
117
+ | Cloudflare/clef-flash (base) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
118
+ | Cloudflare/clef-omni | 30B-A3B | 0.437 | 0.564 | 0.946 | 0.877 | 0.914 | 0.160 | 81 / 82 | 285 ms (A100) |
119
+ | knowledgator/gliclass-v4-edge | 0.4B | 0.373 | 0.528 | 0.753 | 0.676 | 0.696 | 0.344 | 68 / 82 | 101 ms (L4) |
120
+ | fastino/GLiNER2.5-Decide | 0.5B | 0.355 | 0.620 | 0.517 | 0.576 | 0.151 | **0.000** | 17 / 82 | 117 ms (M5 Pro CPU) |
121
+ | LiquidAI/d1-3B | 3.0B | 0.328 | 0.457 | 0.946 | 0.836 | 0.681 | 0.008 | 68 / 82 | **35 ms** (A100) |
122
+
123
+ ---
124
+
125
+ ### 3.2 Multi-Workflow Enterprise Decisions (`LocalLLaMA/typed-decisions`)
126
+
127
+ [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) tests multi-primitive structured judgment across four enterprise workflows (`agent_trace_observability`, `customer_service`, `invoice_processing`, and `security_incidents`), comprising 400 held-out test cases and 2,000 individual decisions (`noul`, `choice`, and `score`) scored against teacher-panel soft label distributions. Two scoring conventions exist in the benchmark repository: `bench/score_with_jobs.py` resolves exact ties in the gold distribution by selecting the first index (`g.index(max(g))`), whereas `scripts/score_model.py` credits any tied maximum across the 20 tied test items; KL divergence and Brier score are identical under both scripts. We report both accuracy figures (`strict / tied-top`) alongside prediction traces in [`eval_submissions/`](./eval_submissions/).
128
+
129
+ On the official zero-shot leaderboard ($N = 43$ models), FastBrain-9B scores `0.7525` (`0.7625` tied-top) accuracy, `0.1508` KL divergence, `0.0795` Brier score, and `0.085` ECE at `50 ms` p50 latency (RTX PRO 6000 via SGLang). It places **7th of 43** overall, **5th of 36** among open-weight models, **2nd of 19** among $\le$9B open-weight models, and **1st** among all `Clef`-architecture models, improving over 9.4B `Cloudflare/clef-flash` (`0.7100` accuracy, `0.2093` KL) by `+4.25` percentage points and over 27.4B `Cloudflare/clef` (`0.7275` accuracy, `0.1907` KL) by `+2.50` points. Across all 28 models exceeding `0.700` zero-shot accuracy, FastBrain-9B exhibits the lowest p50 latency (`50 ms`, compared with `118 ms` for 9B `Amos 1 medium`, `357 ms` for 27B `Amos 1 large`, `526 ms` for `meraGPT Decider 1`, and `3,335 ms` for 31B `Quyet-1.0-Large`). Evaluating the uncalibrated ordinal-KL distilled head directly yields `0.7735` raw accuracy (`0.1039` KL, `0.0582` Brier).
130
+
131
+ On the separate `train`-fitted leaderboard ($N = 51$ models), fitting only scalar per-question log-priors and temperature parameters on the 1,200 `train` cases—with all backbone and head weights frozen (`no weights changed`)—achieves `0.7970` (`0.8070` tied-top) accuracy, `0.0816` KL divergence, and `0.0437` Brier score, ranking **12th-tied (14th listed of 51)** overall and **1st in KL calibration** among frozen-weight models.
132
+
133
+ **Table 3: Performance on `LocalLLaMA/typed-decisions` ($N = 400$ cases, 2,000 decisions). Values separated by `/` denote `score_with_jobs.py` / `score_model.py` tie handling.**
134
+
135
+ | Model | Params | Protocol | Overall Acc ↑ | KL ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | Cust. Svc ↑ | Invoice ↑ | Security ↑ | p50 Latency |
136
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
137
+ | **FastBrain-9B** (prior-fitted, frozen weights) | 9.4B | Fitted | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
138
+ | **FastBrain-9B** (ordinal-KL distilled head) | 9.4B | Zero-shot | 77.35% | 0.1039 | 0.0582 | 84.8% | 74.8% | 73.8% | 71.2% | 80.4% | 82.6% | 75.2% | **50 ms** |
139
+ | **FastBrain-9B** (official PR #58) | 9.4B | Zero-shot | 75.25% / 76.25% | 0.1508 | 0.0795 | 83.2% / 84.3% | 74.7% / 76.0% | 69.8% / 70.4% | 68.2% / 69.8% | 77.6% / 78.8% | 82.0% | 73.2% / 74.4% | **50 ms** |
140
+ | Amos 1 (medium) | 9.0B | Zero-shot | 76.45% | 0.1510 | 0.0800 | — | — | — | — | — | — | — | 118 ms |
141
+ | Amos 1 (large) | 27.0B | Zero-shot | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
142
+ | pplx-decider-v1.1-27b | 27.0B | Zero-shot | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
143
+ | vllm-sr/d3 | 27.0B | Zero-shot | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
144
+ | Baseten Elephant 1 | 9.0B | Zero-shot | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
145
+ | Darwin-27B-ZTC | 27.0B | Zero-shot | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
146
+ | Cloudflare/clef | 27.4B | Zero-shot | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
147
+ | vllm-sr/d3-flash | 9.0B | Zero-shot | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
148
+ | Cloudflare/clef-flash (base) | 9.4B | Zero-shot | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
149
+ | Cloudflare/clef-omni | 35B-A3B | Zero-shot | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
150
+
151
+ ---
152
+
153
+ ### 3.3 Multi-Domain Decision Suite (Cloudflare Decision Index `v0.2.1`)
154
+
155
+ On the 10-benchmark **Cloudflare Decision Index (`v0.2.1`)** suite ($N = 22,427$ evaluation requests spanning AST tool calling, 20-candidate tool reranking, stateful API invocation, fine-grained intent classification, out-of-scope rejection, reasoning-intensive retrieval, e-commerce relevance, phishing detection, and tool-use governance), FastBrain-9B attains an **80.55%** unweighted macro mean (Table 4, Figures 1–3).
156
+
157
+ Across the $\le$9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B), FastBrain-9B ranks **1st on all 10 benchmarks**, improving the 10-task macro mean over `Clef-flash` (`74.96%`) by `+5.59` percentage points and over `Kev 9B` (`76.51%`) by `+4.04` points. The largest gains occur on calibration-sensitive abstention and governance tasks: `+31.67` points on `CLINC150+OOS` (`98.44%` vs. `66.77%`), `+15.55` points on `When2Call` (`81.13%` vs. `65.58%`), and `+6.80` points on `PhishNChips` (`78.75%` vs. `71.95%`). Across all model scales up to 27.4B, FastBrain-9B ranks **2nd of 7** overall, surpassing `27B Jev` (`80.32%`) and `26B-A4B DiffusionGemma Jev` (`69.14%`), and exceeding `27.4B Clef` (`81.81%`) on 5 of 10 individual benchmarks (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home Appliances`, and `When2Call`).
158
+
159
+ ![Figure 1: Aggregate macro mean and parameter efficiency on Cloudflare Decision Index v0.2.1](./assets/benchmark_overview.png)
160
+ *Figure 1. Aggregate 10-benchmark macro mean versus parameter count on Cloudflare Decision Index (v0.2.1, 22,427 requests).*
161
+
162
+ ![Figure 2: Per-benchmark evaluation matrix across model scales](./assets/benchmark_scoreboard.png)
163
+ *Figure 2. Per-benchmark performance matrix across 0.4B–27.4B decision models on Cloudflare Decision Index (v0.2.1).*
164
+
165
+ ![Figure 3: Reliability diagrams and open-set calibration breakdown](./assets/decision_calibration_breakdown.png)
166
+ *Figure 3. Confidence calibration, open-set out-of-scope rejection (`CLINC150+OOS`), and abstention margin analysis (`When2Call`).*
167
+
168
+ **Table 4: Per-benchmark results on Cloudflare Decision Index (`v0.2.1`, 10 tasks, $N = 22,427$). Bold denotes the highest score across all parameter scales; underline denotes best in the $\le$9B class.**
169
+
170
+ | Benchmark (Metric) | **FastBrain-9B** (Ours) | Clef-flash (Base) | Kev 9B | Clef | Jev | DiffGemma | Laya |
171
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
172
+ | Total Parameters | 9.41B | 9.41B | 9.00B | 27.36B | 27.00B | 25.8B (4B act.) | 0.42B |
173
+ | 1. BFCL (Case Exact Acc) | **<u>98.76%</u>** | **<u>98.76%</u>** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
174
+ | 2. ToolRet (nDCG@10) | <u>67.26%</u> | <u>67.26%</u> | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
175
+ | 3. API-Bank (Accuracy) | **<u>93.70%</u>** | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
176
+ | 4. BANKING77 (Macro-F1) | <u>93.77%</u> | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
177
+ | 5. CLINC150+OOS (Macro-F1) | **<u>98.44%</u>** | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
178
+ | 6. Home Appliances (Case Exact) | **<u>97.73%</u>** | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
179
+ | 7. BRIGHT (nDCG@10) | <u>38.58%</u> | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
180
+ | 8. Amazon ESCI (Macro-F1) | <u>57.40%</u> | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
181
+ | 9. PhishNChips (Accuracy) | <u>78.75%</u> | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
182
+ | 10. When2Call (Macro-F1) | <u>81.13%</u> | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
183
+ | **10-Task Macro Mean** | <u>80.55%</u> | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
184
 
185
  ---
186
 
187
+ ## 4. Systems Performance and Post-Training Quantization
188
 
189
+ ### 4.1 Runtime Comparison Across Inference Engines
190
+
191
+ Table 5 compares end-to-end latency, prefill throughput, and memory footprint across three serving backends—SGLang (`lmsysorg/sglang:dev-clef` with `JointSchemaPooler`), `llama.cpp` (`b11430` `ClefModel` graph in `Q8_0` and `Q4_K_M`), and native PyTorch (`joint_schema_model.py`)—benchmarking identical requests on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. Because `output_tokens = 0` for all requests, throughput is governed purely by prefill compute and span-pooling efficiency, reaching `19,319 tok/s` (`131.2` decisions/s) at batch size $B = 4$ in SGLang.
192
+
193
+ **Table 5: End-to-end latency and prefill throughput on NVIDIA RTX PRO 6000 Blackwell Server Edition (G4).**
194
+
195
+ | Workload / System Metric | SGLang `dev-clef` (BF16) | `llama.cpp b11430` (`Q8_0`) | `llama.cpp b11430` (`Q4_K_M`) | PyTorch Native (BF16) |
196
+ | :--- | :---: | :---: | :---: | :---: |
197
+ | Cold-Start Load Time | 11.11 s | 2.56 s | **2.00 s** | 11.11 s |
198
+ | Peak GPU Memory (16K context) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
199
+ | Compatible with 16 GiB T4 GPU | No (BF16 TP=1) | Yes (`9.03 GiB` at 4K) | **Yes (`6.78 GiB` at 4K)** | Yes (`6.8 GiB` via NF4) |
200
+ | Top-1 Agreement vs. PyTorch BF16 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (reference) |
201
+ | 1. Short Intent Routing (293 tok, $K=2$) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
202
+ | 2. Typed Decisions (736 tok, $K=5$) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
203
+ | 3. When2Call Governance (1,068 tok, $K=1$) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
204
+ | 4. PhishNChips Security (1,303 tok, $K=9$) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
205
+ | 5. SafeAgents Judge (2,217 tok, $K=2$) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
206
+ | Batch $B=4$ Typed Decisions (2,944 tok, 20 dec) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
207
+
208
+ ### 4.2 Mixed-Precision GGUF Quantization Fidelity
209
+
210
+ Uniform 4-bit quantization degrades cross-attention span readout and linear-attention recurrent state precision if applied indiscriminately to decision-head projections. In our `FastBrain-9B-Q4_K_M.gguf` (`6.73 GB`) build, feed-forward blocks are quantized to 4-bit K-quants while all `dec.*` decision-head tensors, `ssm_*` Gated DeltaNet parameters, and `attn_*` projections remain at `Q8_0` precision. Evaluated across a stratified 10-task subset (`203` multi-question requests, `1,473` scored schema questions; Table 6), `Q8_0.gguf` (`9.66 GB`) matches native `bfloat16` on `1,471/1,473` decisions (`99.86%` Top-1 agreement, mean absolute probability error `0.00203`, `-0.03` percentage-point delta on the 10-task mean), and `Q4_K_M.gguf` retains `98.78%` Top-1 agreement (`0.00387` probability MAE, `-0.20` percentage-point calibrated delta). Table 7 summarizes hardware memory requirements across formats, including NVIDIA Tesla T4 (`16 GiB`) measurements.
211
+
212
+ **Table 6: Quantization fidelity across 1,473 scored schema questions on the 10-task stratified Decision Index subset.**
213
+
214
+ | Metric / Subset | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` Head) |
215
+ | :--- | :---: | :---: | :---: |
216
+ | Top-1 Decision Agreement ($N = 1,473$) | 100.00% (ref) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
217
+ | Mean Absolute Probability Error ($\|\hat{p}_q - \hat{p}_{\text{bf16}}\|$) | 0.00000 | 0.00203 | 0.00387 |
218
+ | 10-Task Mean (Uncalibrated Head) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
219
+ | 10-Task Mean (Schema-Calibrated) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
220
+ | ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
221
+ | ├─ 2. ToolRet / 7. BRIGHT | 73.15% / 41.93% | 73.70% / 41.00% | 75.60% / 40.31% |
222
+ | ├─ 5. CLINC150+OOS / 6. Home Appliances | 93.94% / 90.00% | 93.94% / 90.00% | 93.94% / 90.00% |
223
+ | └─ 8. Amazon ESCI / 9. PhishNChips / 10. When2Call | 73.89% / 85.00% / 75.00% | 73.89% / 85.00% / 75.00% | 75.79% / 85.00% / 72.92% |
224
+
225
+ **Table 7: Disk footprint, peak VRAM allocation, and Tesla T4 (`16 GiB`) latency across precision tiers.**
226
+
227
+ | Precision Format | Artifact Size | Peak VRAM (4K–16K ctx) | T4 Latency (293 / 1,068 tok) | Target Hardware |
228
+ | :--- | :---: | :---: | :---: | :--- |
229
+ | `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 6.78 – 10.30 GiB | 343.0 ms / 1,154.3 ms | 8–16 GB GPU (RTX 3060/4060, Tesla T4), Apple Silicon (8 GB+) |
230
+ | PyTorch `load_in_4bit=True` (NF4 + `fp16` head) | ~5.5 GB (mem) | 6.20 – 6.80 GiB | — | 8–16 GB CUDA GPU (Colab T4, consumer GPUs) |
231
+ | `FastBrain-9B-Q8_0.gguf` | 9.66 GB | 9.03 – 12.55 GiB | 349.3 ms / 1,149.8 ms | 12–16 GB GPU (Tesla T4, RTX 4070 Ti), Apple Silicon (16 GB+) |
232
+ | PyTorch `load_in_8bit=True` (INT8 + `bf16` head) | ~9.6 GB (mem) | ~11.20 GiB | — | 12–16 GB CUDA GPU |
233
+ | Native `bfloat16` (`safetensors` / `BF16.gguf`) | 18.82 / 18.16 GB | 20.30 – 24.70 GiB | — | $\ge$24 GB GPU (NVIDIA L4, A100, H100, RTX 3090/4090, PRO 6000) |
234
 
235
  ---
236
 
237
+ ## 5. Reproducible Inference and Deployment
238
 
239
+ ### 5.1 Python / Transformers Runtime (`bfloat16`, `8-bit`, or `4-bit` NF4)
240
 
241
+ `TypeScript/FastBrain-9B` implements the `joint_schema_model` interface and is API-compatible with `Cloudflare/clef-flash`. Passing `load_in_4bit=True` quantizes only the 9.3B backbone via `bitsandbytes` NF4 while preserving the 116M `JointSchemaHead` in floating point.
242
 
243
  ```python
 
244
  import sys
245
  import torch
246
+ from huggingface_hub import snapshot_download
247
 
248
  model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
249
  sys.path.insert(0, model_dir)
250
  import joint_schema_model as clef
251
 
252
+ # Native bfloat16 (>=24 GB VRAM):
253
  model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
254
 
255
+ # Mixed-precision 4-bit NF4 (8-16 GB VRAM, e.g., Tesla T4):
256
  # model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
257
 
258
  response = clef.systemone(model, processor, {
 
262
  "tools": ["get_balance", "transfer_funds"],
263
  },
264
  "questions": {
265
+ "next_action": {
266
  "type": "choice",
267
+ "instructions": "Select the appropriate assistant action.",
268
  "criteria": {
269
+ "call_balance": "Invoke <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
270
+ "request_auth": "Ask the user to provide an account identifier or authentication token.",
271
+ "abstain": "Decline because personal banking records are out of scope.",
272
  },
273
  },
274
  },
 
276
  print(response)
277
  ```
278
 
279
+ ### 5.2 `llama.cpp` GGUF Server (`/v1/systemone`)
 
 
280
 
281
  > [!IMPORTANT]
282
+ > **Runtime Requirements for `ClefModel` GGUF Execution**
283
+ > 1. Use **`llama.cpp` build `b11430` or later**, which implements the `LLM_ARCH_CLEF` computation graph (`dec.*` tensors) and the `/v1/systemone` HTTP endpoint. Earlier builds and standard generative wrappers do not execute the cross-attention decision head.
284
+ > 2. Set physical micro-batch size equal to logical batch size (`-b 16384 -ub 16384` or `-b 4096 -ub 4096`) together with `--no-cache-prompt`. `ClefModel` pools `<|question_*|>` and `<|option_*|>` spans within a single physical `ubatch`; leaving `-ub` at its default (`512`) splits prompts longer than 512 tokens across micro-batches, and prefix caching can bypass marker positions.
 
 
 
285
 
286
  ```bash
287
+ # 1. Download prebuilt llama.cpp (b11430+) and GGUF checkpoint
 
 
 
288
  curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
 
 
289
  pip install -U "huggingface_hub[hf_transfer]"
290
  HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
 
 
 
291
 
292
+ # 2. Start llama-server (add --mmproj mmproj-FastBrain-9B-Q8_0.gguf for image inputs)
 
293
  export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
 
 
294
  ./llama-b11430/llama-server \
295
  -m ./FastBrain-9B-Q8_0.gguf \
296
  -ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
297
  --port 8080
 
 
 
 
 
 
 
 
 
 
 
298
  ```
299
 
300
+ Querying `/v1/systemone` evaluates multiple heterogeneous primitives (`choice` + `noul`) in a single prefill pass:
301
 
302
  ```bash
303
  curl http://127.0.0.1:8080/v1/systemone \
 
309
  "customer_message": "I was charged twice for my subscription last week and nobody has replied to my ticket."
310
  },
311
  "questions": {
312
+ "department": {
313
  "type": "choice",
314
+ "instructions": "Assign this ticket to the responsible queue.",
315
  "criteria": {
316
  "billing": "Payments, refunds, duplicate charges, or invoices",
317
+ "engineering": "Software defects, crashes, or service outages",
318
+ "identity": "Password reset or multi-factor authentication issues"
319
  }
320
  },
321
+ "is_urgent": {
322
  "type": "noul",
323
+ "instructions": "Does the message report an unresolved financial loss or time-critical failure?"
324
  }
325
  }
326
  }'
327
  ```
328
 
329
+ ### 5.3 SGLang Server (`JointSchemaPooler`)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
330
 
331
  ```bash
332
+ docker run --gpus all --shm-size 32g -p 30000:30000 \
333
+ -v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
 
 
 
334
  lmsysorg/sglang:dev-clef \
335
+ sglang serve --model-path TypeScript/FastBrain-9B --host 0.0.0.0 --port 30000
 
 
 
 
 
 
 
 
 
 
 
336
  ```
337
 
338
  ---
339
 
340
+ ## 6. Repository Manifest and Citation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
341
 
342
+ **Table 8: Repository file specification.**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
343
 
344
+ | Path | Size | Contents |
 
 
 
 
345
  | :--- | :---: | :--- |
346
+ | `model-00001-of-00004.safetensors` .. `00004` | 18.59 GB | `bfloat16` hybrid backbone tensors with all adapted projections merged in-place |
347
+ | `joint_head.safetensors`, `joint_head_config.json` | 232 MB | 116M-parameter `JointSchemaHead` weights (`122` tensors) and architectural config |
348
+ | `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | Mixed-precision 4-bit GGUF (`dec.*`, `ssm_*`, and `attn_*` retained in `Q8_0`) |
349
+ | `FastBrain-9B-Q8_0.gguf` / `FastBrain-9B-BF16.gguf` | 9.66 / 18.16 GB | 8-bit and `bfloat16` GGUF builds for `llama-server` (`b11430`+) |
350
+ | `mmproj-FastBrain-9B-Q8_0.gguf` / `BF16.gguf` | 624 / 922 MB | Vision projector checkpoints for multimodal state inputs |
351
+ | `joint_schema_model.py` | 25 KB | Reference `ClefModel`, vectorized `_forward_fast_cuda` head, and `systemone()` runtime |
352
+ | `eval_submissions/typescript_fastbrain-9b*.json` | 1.80 MB | Verifiable 2,000-decision zero-shot and prior-fitted traces for `LocalLLaMA/typed-decisions` |
353
+ | `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | 110 KB | Verifiable 661-call binary and 7-way category traces for `safeagents-judge-bench` (v2) |
354
+ | `.eval_results/*.yaml` | 2.3 KB | Hugging Face Hub machine-readable evaluation records |
355
+ | `README.md`, `README_zh.md`, `assets/*.png` | 1.28 MB | English and Chinese technical documentation and evaluation figures |
356
+
357
+ ### Citation
358
+
359
+ ```bibtex
360
+ @misc{fastbrain9b2026,
361
+ title = {FastBrain-9B: A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions},
362
+ author = {TypeScript},
363
+ year = {2026},
364
+ howpublished = {\url{https://huggingface.co/TypeScript/FastBrain-9B}},
365
+ note = {Apache-2.0 License}
366
+ }
367
+ ```
README_zh.md CHANGED
@@ -2,177 +2,247 @@
2
 
3
  # FastBrain-9B
4
 
5
- **单次前向、零生成 Token 的非自回归 System-1 结构化决策模型(`9.4B` 参数)**
6
 
7
- [![Architecture](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.4B_Total)-3b82f6?style=flat-square)](#核心架构与技术创新)
8
- [![Decision Head](https://img.shields.io/badge/Head-JointSchemaHead_(116M_%C2%B7_0_Output_Tokens)-10b981?style=flat-square)](#核心架构与技术创新)
9
- [![SafeAgents Judge](https://img.shields.io/badge/SafeAgents_Judge_v2-%231_%2F_8_分类F1_(75.1%25)_%C2%B7_82%2F82_全拦截-ec4899?style=flat-square)](#1-harness-hubsafeagents-judge-benchv2661-条真实-agent-工具调用--82-条越界轨迹--7-类违规识别)
10
- [![Typed Decisions](https://img.shields.io/badge/Typed_Decisions-%237_%2F_43_零样本总榜_(%231_Clef架构_%C2%B7_50ms)-8b5cf6?style=flat-square)](#2-localllamatyped-decisions400-个企业级用例--2000-次类型化决策)
11
- [![Decision Index](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(总榜_%232_%C2%B7_9B级_10%2F10_全满贯)-f59e0b?style=flat-square)](#cloudflare-decision-indexv02110-项评测全量对比)
12
  [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
13
 
14
  **[English](./README.md)** | **简体中文**
15
 
16
  </div>
17
 
18
- **FastBrain-9B**(总参数量 `9.4B`:`Qwen3.5-9B` 混合 Gated DeltaNet + 全注意力基座 + `116M` 参数 `JointSchemaHead` 联合决策头)是一款专为**超低延迟、高并发、强校准结构化决策**打造的非自回归 **System-1(快思考)决策模型**,以 9B 级体量在多项权威榜单上正面击败 `27B–35B` 级大模型。
 
 
19
 
20
- 给定任意 JSON 状态或多模态图像上下文,以及一个或多个类型化决策 Schema(`choice` 多选路由、`noul` 二元真值校验、`score` 有序量表打分),**FastBrain-9B** 仅需**单次 Prefill 前向传播(`0` 个自回归生成 Token)**即可**同时并行评估所有问题与全部候选项**,在 **`30–50 ms`(吞吐达 `14,600–19,300+ tok/s`)**内直接输出经过严格概率校准的分布、工具调用选择、安全护栏拦截判定与开放集拒识结果。
 
 
21
 
22
- 全部微调更新(覆盖全部 `32` 层混合 Block 的 `104` 个线性注意力与自注意力投影层)均已**原地合并(In-Place Merged)**至主干权重 `model-00001-of-00004.safetensors` .. `00004` 中,`122` 个决策头张量独立封装于 `joint_head.safetensors`,并同步提供开箱即用的 `llama.cpp` 官方 GGUF 量化权重(`Q8_0` / `Q4_K_M` / `BF16`)。**推理时无需挂载任何外部 LoRA 或 PEFT 适配器。**
23
 
24
- ### 核心战绩与统治级亮点
25
 
26
- - **真实 Agent 安全护栏榜单 `safeagents-judge-bench` 断层总冠军(`v2` & `v1` 双版本领跑)**:在 Hugging Face 官方独立评测的 `v2`(661 条真实危险工具调用)**7 分类违规识别赛道**以 **`75.1%` Macro-F1 / `85.8%` 准确率荣登全场 `#1 / 8` 总冠军**,领先 `3` 倍参数量的 `27.4B Cloudflare/clef`(`56.3%`)整整 **`+18.8 pp`**,领先同尺寸基座 `Cloudflare/clef-flash`(`46.2%`)**`+28.9 pp`**;在二分类护栏赛道实现 **`82 / 82`(`100%`)危险轨迹零漏网拦截**(`AUROC 98.5%` · `Recall 99.6%`,校准后平衡准确率 **`93.8%`**),并在 `v1` 官方榜单包揽**二分类(`97.4%`)与多分类(`85.0%`)双赛道 `#1 / 9`**。
27
- - **企业级决策总榜 `LocalLLaMA/typed-decisions`(共 `94` 个参评模型)零样本前七 & 全场最快推理速度**:在 `43` 个模型的**零样本总榜位列 `#7 / 43`**(**开源权重模型 `#5 / 36`**、**`<=9B` 开源模型 `#2 / 19`**、**全部 `Clef` 架构模型不分尺寸 `#1`**),同时以 **`50 ms` p50 延迟成为 Top 28(准确率 `>70%` 梯队)中速度 `#1` 的模型**——比榜首 `31B Quyet-1.0-Large`(`3,335 ms`)**快 `66.7` 倍**,比 `27B Amos 1 large`(`357 ms`)**快 `7.1` 倍**!在 `51` 个模型的 **`train` 校准/微调总榜**同样已合入主榜(**并列 `#12` / 列表 `#14 / 51`**,准确率 **`79.70%` / `80.70%`**,`KL 0.0816`),在**不修改任何模型权重(Frozen-Weight)**的前提下拿下 **`<=9B` 第 1、全场 KL 校准第 1**,击败 37 个专门在训练集上微调权重的模型!
28
- - **Cloudflare Decision Index(`v0.2.1`)跨量级越级击败 `27B` 模型**:在涵盖 `22,427` 次评测请求的 10 大基准测试中取得 **`80.55%` 宏平均分(全尺寸总榜 `#2 / 7`)**,在 **`<=9B` 级别豪取 `10 / 10` 全满贯第一**(比同尺寸基座 `Clef-flash` 暴涨 **`+5.59 pp`**,其中 `CLINC150+OOS` 暴涨 **`+31.67 pp`**、`When2Call` 暴涨 **`+15.55 pp`**),总分超越 `27B Jev`(`80.32%`),并在 **5 / 10 个单项上正面击败 `3` 倍参数量的旗舰 `27.4B Cloudflare/clef`**(其中 **4 个单项夺得全尺寸绝对第一**)。
29
- - **全栈工业级部署生态(`SGLang` + `llama.cpp GGUF` + `PyTorch 4-bit`)**:内置向量化前缀和池化与变长 FlashAttention 融合算子(决策头加速 **`1.90x`**,`max_logit_diff = 0.00000`);`Q8_0.gguf` 与原生 `BF16` Top-1 决策一致率高达 **`99.86%`**(近乎完全无损);`Q4_K_M.gguf` 仅 **`6.73 GB`**,在免费 Colab T4(16GB)或 Mac M 系列芯片上仅占 `6.78 GiB` 显存即可满速运行。
30
 
31
  ---
32
 
33
- ## 权威榜单评测与官方排名总览
34
 
35
- ### 官方实时排行榜名次汇总(已核验最新 109 个官方参评模型条目)
36
 
37
- | 官方评测基准 / 赛道 | 实时参���规模 | **`FastBrain-9B` 核心得分** | **官方实时排名** | **所处百分位** | 正面击败的代表性模型(参数量与得分) |
38
- | :--- | :---: | :---: | :---: | :---: | :--- |
39
- | **[`safeagents-judge-bench` (`v2`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(7 分类违规识别)***<br>([已合入 PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [官方榜单](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench)) | **8 个决策模型**<br>*(+2 基线)* | **`75.1%` Macro-F1**<br>**`85.8%` 分类准确率** | **#1 / 8 *(全场第 1)*** | **Top 12.5% *(断层冠军)*** | `Cloudflare/clef` *(27.4B, #2 `56.3%` F1 / `73.7%` Acc)*、`vllm-sr/d3-mini` *(#3 `51.2%`)*、`Cloudflare/clef-flash` *(9.4B, #4 `46.2%`)*、`clef-omni` *(30B, #5 `43.7%`)* |
40
- | **[`safeagents-judge-bench` (`v2` & `v1`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(二分类安全拦截)***<br>([已合入 PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [`v1` 榜单](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)) | **8 模型 (`v2`)**<br>**9 模型 (`v1`)** | **`82/82` (`100%`) 轨迹全拦截 · `98.5%` AUROC**<br>*(`93.8%` 校准 / `81.4%` 默认 `v2` · **`97.4%` `v1`**)* | **`#1 / 9` (`v1` 总榜第 1)**<br>**`100%` 拦截 · AUROC #3 (`v2`)** | **Top 11.1% (`v1`)**<br>**Top 37.5% AUROC (`v2`)** | 默认阈值即击败 `27.4B Cloudflare/clef` (`80.8%`);AUROC 与轨迹拦截击败 `clef-flash` (`96.7%`, `81/82`)、`clef-omni` (`94.6%`, `81/82`)、`d1-3B` (`68/82`) |
41
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(零样本全量总榜)***<br>([已合入 PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [预测轨迹](./eval_submissions/typescript_fastbrain-9b.json)) | **43 个模型**<br>*(36 个开源权重)* | **`75.25%` / `76.25%` 准确率**<br>*(KL `0.1508` · Brier `0.0795` · **`50 ms`**)* | **#7 / 43 *(实时总榜)***<br>**#5 / 36 开源权重总榜** | **Top 16.3% 总榜**<br>**Top 13.9% 开源** | 击败 `vllm-sr/d3` *(27B, #8 `74.9%`)*、`Baseten Elephant 1` *(9B, #9 `74.8%`)*、`Darwin-27B` *(#10 `74.3%`)*、`Cloudflare/clef` *(27B, #16 `72.8%`)*、`openjev` *(27B, #17 `72.8%`)* |
42
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(零样本 `<=9B` 及 Clef 架构榜)*** | **19 个 `<=9B` 开源**<br>**4 个 Clef 架构** | **`75.25%` / `76.25%` 零样本**<br>*(蒸馏原生头: **`77.35%`** · **`50 ms`**)* | **#2 / 19 `<=9B` 开源**<br>**#1 Clef 架构 · #1 极速** | **Top 10.5% (`<=9B`)** | 击败 18 个同级开源模型中的 17 个(`d3-mini` `74.3%`、`d3-flash` `71.7%`、`signpost-d-4b` `71.2%`、`clef-flash` `71.0%`、`d1-3B` `65.6%`)及 `35B clef-omni` (`67.3%`) |
43
- | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(`train` 校准/微调榜)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [预测轨迹](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **51 个模型** | **`79.70%` / `80.70%` 准确率**<br>*(KL **`0.0816`** · Brier **`0.0437`** · **`50 ms`**)* | **并列 #12 *(列表 #14 / 51)***<br>**冻结权重 `<=9B` #1 · KL #1** | **Top 23.5% 总榜**<br>*(零权重微调)* | **不改任何权重**即击败 **37** 个微调/专精模型:`Rev-gen1-midi` *(25.6B, `79.45%`)*、`OpenDecider-small-td` *(4B LoRA, `79.2%`)*、`OpenDecider-medium-td` *(30B LoRA, `78.8%`)*、`Seb-9B` *(`75.4%`)* |
44
- | **Cloudflare Decision Index (`v0.2.1`)** *(10 任务宏平均)* | **7 个模型** | **`80.55%` 宏平均** | **#2 / 7 全尺寸总榜**<br>**`<=9B` 10/10 全满贯 #1** | **Top 28.6% 总榜**<br>**`<=9B` 100% 统治率** | 总分超越 `Jev` *(27B, `80.32%`)*、`Kev 9B` *(`76.51%`)*、`Clef-flash` *(9.4B, `74.96%`)*、`DiffGemma` *(26B, `69.14%`)*,并在 **5 / 10** 项任务中击败 `27.4B Clef` |
45
 
46
- ---
 
 
 
 
 
47
 
48
- ### Cloudflare Decision Index(`v0.2.1`):10 项评测全量对比
49
 
50
- 在官方 **Cloudflare Decision Index(`v0.2.1`)** 涵盖 **10 个评测基准(共 `22,427` 条评测请求)** 的完整测试中,**FastBrain-9B(`9.4B`)** 取得 **`80.55%`** 的总宏平均分:
51
- - **`<=9B` 量级 `10 / 10` 单项全满贯第一**:全面超越同尺寸基座 `Clef-flash`(`9.4B`,`74.96%`)达 **`+5.59 pp`**(其中开放集意图拒识 `CLINC150+OOS` 提升 **`+31.67 pp`**,工具调用治理 `When2Call` 提升 **`+15.55 pp`**),领先 `Kev 9B`(`76.51%`)**`+4.04 pp`**,领先 `Laya`(`0.4B`,`58.09%`)**`+22.46 pp`**。
52
- - **跨量级击败 `3` 倍参数量的 `27B` 级大模型**:10 任务宏平均分超越 `Jev`(`27B`,`80.32%`)与 `DiffusionGemma Jev`(`26B-A4B`,`69.14%`),并在 **`BFCL`、`API-Bank`、`CLINC150+OOS`、`Home appliances`、`When2Call` 五大基准上正面击败 `27.4B` 旗舰 `Clef`**(其中 **4 项夺得全尺寸全场第一**)。
53
 
54
- ![FastBrain-9B Overview](./assets/benchmark_overview.png)
55
 
56
- ### 10 大基准全参数量级得分矩阵
57
 
58
- ![FastBrain-9B Scoreboard](./assets/benchmark_scoreboard.png)
59
 
60
- ### 细粒度概率校准与开放集拒识可靠性分析
61
 
62
- ![FastBrain-9B Calibration Breakdown](./assets/decision_calibration_breakdown.png)
63
 
64
- ### 各参数量级模型逐项得分对比表
65
 
66
- | 评测基准 | **FastBrain-9B**<br>*(9.4B · 本模型)* | Clef-flash<br>*(9.4B 基座)* | Kev 9B<br>*(9B)* | Clef<br>*(27.4B)* | Jev<br>*(27B)* | DiffGemma<br>*(26B-A4B)* | Laya<br>*(0.4B)* |
67
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
68
- | **参数量** | **9.41B** | 9.41B | 9.0B | 27.36B | 27.0B | 25.8B *(激活 4B)* | 0.42B |
69
- | **BFCL**<br>*(AST 工具选择精确率)* | **98.76%**<br>*(全场 #1)* | **98.76%** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
70
- | **ToolRet**<br>*(nDCG@10 工具重排)* | **67.26%**<br>*(9B #1)* | **67.26%** | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
71
- | **API-Bank**<br>*(多步 API 调用准确率)* | **93.70%**<br>*(全场 #1)* | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
72
- | **BANKING77**<br>*(77 类金融意图 Macro-F1)* | **93.77%**<br>*(9B #1)* | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
73
- | **CLINC150+OOS**<br>*(150 类意图+越界拒识)* | **98.44%**<br>*(全场 #1)* | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
74
- | **Home Appl.**<br>*(智能家居指令精确率)* | **97.73%**<br>*(全场 #1)* | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
75
- | **BRIGHT**<br>*(推理密集型检索 nDCG@10)* | **38.58%**<br>*(9B #1)* | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
76
- | **Amazon ESCI**<br>*(电商相关性 Macro-F1)* | **57.40%**<br>*(9B #1)* | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
77
- | **PhishNChips**<br>*(9 探针钓鱼安全检测)* | **78.75%**<br>*(9B #1)* | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
78
- | **When2Call**<br>*(4 分类工具触发治理)* | **81.13%**<br>*(9B #1)* | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
79
- | **10 任务宏平均**<br>*(Macro Mean)* | **80.55%**<br>*(总榜 #2 · 9B #1)* | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
80
 
81
  ---
82
 
83
- ### 外部开源社区权威基准(`safeagents-judge-bench` & `LocalLLaMA/typed-decisions`)
84
 
85
- #### 1. [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2)(`v2`:661 条真实 Agent 工具调用 · 82 条越界轨迹 · 7 类违规识别)
86
 
87
- > **官方 `v2` 与 `v1` 评测说明(由 Hugging Face 官方成员 `@ybelkada` 通过 HF Jobs 在 `NVIDIA A100-SXM4-80GB` 上独立运行并提交 [PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) 合入 · 元数据见 [`.eval_results/safeagents-judge-bench.yaml`](./.eval_results/safeagents-judge-bench.yaml))**:测试集包含真实编码 Agent 在沙箱中被扫描器标记可疑的 `661` 次工具调用(含 `82` 条越界轨迹中的 `536` 次恶意调用 + `125` 次合法放行调用):
88
- > - **`v2` 7 分类违规识别赛道全场总冠军(`#1 / 8`)**:斩获 **`75.1%` Category Macro-F1** 与 **`85.8%` 分类准确率(`567/661`)**,领先第 2 名 `27.4B Cloudflare/clef`(`56.3%` F1 / `73.7%` Acc)达 **`+18.8 pp` Macro-F1(`+12.1 pp` 准确率)**,领先第 3 名 `vllm-sr/d3-mini`(`51.2%`)**`+23.9 pp`**,领先同尺寸基座 `Cloudflare/clef-flash`(`46.2%` / `57.6%` Acc)**`+28.9 pp` Macro-F1(`+28.2 pp` 准确率)**!
89
- > - **`v2` 二分类安全护栏 `82 / 82`(`100%`)危险轨迹全拦截 & `98.5%` AUROC**:成功拦截 **`534 / 536`(`99.6%`)** 次恶意工具调用(其中 `harness_introspection` `272/272`、`credential_misuse` `15/15`、`sandbox_escape` `4/4` 召回率均为 **`100%`**,首次拦截前中位漏放恶调数为 **`0`**)。在未校准默认阈值(`P(block) >= 0.50`)下平衡准确率为 **`81.4%`**(超越 `27.4B Cloudflare/clef` 的 `80.8%`);凭借 **`98.5%` 的高 AUROC**,仅需校准判别阈值(`P(block) >= 0.961`)即可将平衡准确率拉升至 **`93.8%`(误拦率仅 `5.6%`)** 且保持 **`82/82`(`100%`)** 轨迹全拦截。在 [`v1`(562 条调用)官方版本](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)中,`FastBrain-9B` 更包揽了**二分类(`97.4%` Bal. Acc)与多分类(`85.0%` Macro-F1)双榜 `#1 / 9`**。
90
 
91
- | 参评模型(`safeagents-judge-bench v2` 官方数据) | 参数量 | 7 分类 Macro-F1 ↑ | 7 分类准确率 ↑ | AUROC ↑ | 平衡准确率 ↑ | 恶意召回率 ↑ | 误拦率 ↓ | 危险轨迹拦截率 ↑ | P50 延迟 *(评测硬件)* |
 
 
 
 
92
  | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
93
- | **`TypeScript/FastBrain-9B` *(`v2` 官方默认 `t=0.50`)*** | **9.4B** | **75.1%** *(#1 / 8)* | **85.8%** *(#1 / 8)* | **98.5%** | 81.4% | **99.6%** | 36.8% | **82 / 82 *(100%)*** | 327 ms *(A100 · **G4 104 ms**)* |
94
- | **`TypeScript/FastBrain-9B` *(`v2` 阈值校准 `t=0.961`)*** | **9.4B** | **75.1%** *(#1)* | **85.8%** *(#1)* | **98.5%** | **93.8%** | 93.1% | **5.6%** | **82 / 82 *(100%)*** | 327 ms *(A100 · **G4 104 ms**)* |
95
- | **`TypeScript/FastBrain-9B` *(`v1` 官方榜单 · 562 条)*** | **9.4B** | **85.0%** *(#1 / 9)* | **92.3%** *(#1 / 9)* | **98.3%** | **97.4%** *(#1 / 9)* | **100.0%** | **5.1%** | **78 / 78 *(100%)*** | **104 ms *(RTX PRO 6000)*** |
96
- | `Cloudflare/clef` | 27.4B | 56.3% *(#2)* | 73.7% *(#2)* | **99.3%** | 80.8% | **100.0%** | 38.4% | **82 / 82 *(100%)*** | 636 ms *(A100)* |
97
- | `vllm-sr/d3-mini` | 1.8B | 51.2% *(#3)* | 65.2% *(#3)* | 98.6% | 85.6% | **100.0%** | 28.8% | **82 / 82 *(100%)*** | 168 ms *(A100)* |
98
- | `Cloudflare/clef-flash` *(同尺寸基座)* | 9.4B | 46.2% *(#4)* | 57.6% *(#4)* | 96.7% | **89.3%** | 97.8% | 19.2% | 81 / 82 *(98.8%)* | 326 ms *(A100)* |
99
- | `Cloudflare/clef-omni` | 30B-A3B | 43.7% *(#5)* | 56.4% *(#5)* | 94.6% | 87.7% | 91.4% | 16.0% | 81 / 82 *(98.8%)* | 285 ms *(A100)* |
100
- | `knowledgator/gliclass-v4-edge` | 0.4B | 37.3% *(#6)* | 52.8% *(#7)* | 75.3% | 67.6% | 69.6% | 34.4% | 68 / 82 *(82.9%)* | 101 ms *(L4)* |
101
- | `fastino/GLiNER2.5-Decide` | 0.5B | 35.5% *(#7)* | 62.0% *(#6)* | 51.7% | 57.6% | 15.1% | **0.0%** | 17 / 82 *(20.7%)* | 117 ms *(M5 Pro)* |
102
- | `LiquidAI/d1-3B` | 3B | 32.8% *(#8)* | 45.7% *(#8)* | 94.6% | 83.6% | 68.1% | 0.8% | 68 / 82 *(82.9%)* | **35 ms *(A100)*** |
103
-
104
- #### 2. [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)(400 个企业级用例 · 覆盖 4 大业务流的 2,000 次类型化决策)
105
-
106
- > **官方实时总榜验证说明(当前共 `43` 个零样本模型 + `51` 个训练集拟合/微调模型 = `94` 个参评条目)**:
107
- > - **零样本官方总榜实时 `#7 / 43`(开源权重 `#5 / 36` · `<=9B` 开源 `#2 / 19` · 全部 Clef 架构 `#1`)**:完整 2,000 次预测轨迹见 [`eval_submissions/typescript_fastbrain-9b.json`](./eval_submissions/typescript_fastbrain-9b.json)(已通过 [PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) 合入官方主榜),在 `bench/score_with_jobs.py` 下取得 **`75.25%` 准确率(`0.1508` KL · `0.0795` Brier · `50 ms` p50 延迟)**(在并列最高分均算对的 `scripts/score_model.py` 下为 **`76.25%`**)。凭借 **`50 ms` p50 延迟**,`FastBrain-9B` 是 **Top 28(准确率 `>70%`)梯队中推理速度 `#1` 的模型**——比第 1 名 `Quyet-1.0-Large`(`3,335 ms`)**快 `66.7` 倍**、比第 2 名 `meraGPT Decider 1`(`526 ms`)**快 `10.5` 倍**、比 `27B Amos 1 large`(`357 ms`)**快 `7.1` 倍**!
108
- > - **`train` 校准/微调官方总榜实时并列 `#12`(列表 `#14 / 51` · 冻结权重模型 KL `#1` & `<=9B` `#1`)**:完整预测轨迹见 [`eval_submissions/typescript_fastbrain-9b-fitted.json`](./eval_submissions/typescript_fastbrain-9b-fitted.json)([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)),在**完全冻结模型权重(`no weights changed`)**、仅拟合问题先验偏置与温度的条件下达到 **`79.70%`(并列口径 `80.70%`)准确率 · `0.0816` KL · `0.0437` Brier · `50 ms`**,正面击败 **37 个专门在训练集上微调权重的模型**(包括 `25.6B Rev-gen1-midi` `79.45%`、`4B LoRA OpenDecider-small-td` `79.2%`、`30B-A3B LoRA OpenDecider-medium-td` `78.8%` 等)。
109
-
110
- | 参评模型 | 参数量 | 总体准确率 ↑ | Gold KL 散度 ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent 监控 ↑ | 客户服务 ↑ | 发票审批 ↑ | 安全事件 ↑ | P50 延迟 ↓ |
111
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
112
- | **`TypeScript/FastBrain-9B` *(先验校准 · 零权重改动)*** | **9.4B** | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
113
- | **`TypeScript/FastBrain-9B` *(Ordinal-KL 蒸馏原生头)*** | **9.4B** | **77.35%** *(原生 #1)* | **0.1039** | **0.0582** | **84.8%** | **74.8%** | **73.8%** | **71.2%** | **80.4%** | **82.6%** | **75.2%** | **50 ms** |
114
- | **`TypeScript/FastBrain-9B` *(零样本 · 实时总榜 #7 / 43)*** | **9.4B** | **75.25% / 76.25%** | **0.1508** | **0.0795** | **83.2% / 84.3%** | **74.7% / 76.0%** | **69.8% / 70.4%** | **68.2% / 69.8%** | **77.6% / 78.8%** | **82.0%** | **73.2% / 74.4%** | **50 ms** |
115
- | `Amos 1 (large)` | 27B | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
116
- | `pplx-decider-v1.1-27b` | 27B | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
117
- | `vllm-sr/d3` | 27B | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
118
- | `Baseten Elephant 1` | 9B | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
119
- | `Darwin-27B-ZTC` | 27B | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
120
- | `Cloudflare/clef` | 27.4B | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
121
- | `vllm-sr/d3-flash` | 9B | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
122
- | `Cloudflare/clef-flash` *(同尺寸基座)* | 9.4B | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
123
- | `Cloudflare/clef-omni` | 35B-A3B | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
 
 
 
 
 
 
 
124
 
125
  ---
126
 
127
- ## 核心架构与技术创新
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
 
129
- 1. **混合 Gated DeltaNet + 全注意力基座网络(`9.3B` 主干参数,`bfloat16`)**:
130
- - 支持最长 `262,144` 上下文窗口(由 `24` 层线性注意力 Gated DeltaNet 模块与 `8` 层全注意力模块交替构成),可在单次 Prefill 前向中高效编码超长工具注册表、多轮 Agent 执行轨迹与候选集文本片段。
131
- - 联合微调全部 `32` 层中的 `104` 个核心投影矩阵(线性注意力 `in_proj_qkv`、`in_proj_z`、`out_proj` 与自注意力 `q_proj`、`k_proj`、`v_proj`、`o_proj`),并已无损原地合并至主干 `safetensors` 权重。
132
- 2. **联合 Schema 决策头(`joint_head.safetensors`,`116M` 参数 · SGLang 融合 FlashAttention 算子)**:
133
- - 直接从最后一层隐藏状态中抽取 `<|question_*|>` 问题片段与 `<|option_*|>` 候选片段表征,无需任何自回归解码(`output_tokens = 0`),一次性输出全部问题(`choice` / `noul` / `score`)的概率分布。
134
- - **向量化 CUDA 极速路径(`joint_schema_model.py` 中的 `_forward_fast_cuda`)**:用 `torch.cumsum` `O(1)` 前缀和池化替代逐片段 Python 循环,并通过 `cu_seqlens` 紧凑打包调用变长 FlashAttention(`torch.ops.aten._flash_attention_forward`),实现**决策头核函数 `1.90x` 加速(`3.27 ms -> 1.72 ms`,最大对数几率误差 `max_logit_diff = 0.00000`)**。
135
- - **序数 KL(Ordinal-KL)与多任务概率校准蒸馏**:融合软标签 KL 散度、序数累积分布距离损失(Earth Mover's Distance / RPS)$\sum_k (\text{CDF}_{\text{pred}}(k) - \text{CDF}_{\text{gold}}(k))^2$ 以及 `score` 量表题的期望分 MSE 损失,大幅降低分布偏移与校准误差。
136
- 3. **编译期 AST Schema 感知边际校准(`joint_schema_model.py`)**:
137
- - 通过编译期自动分析候选 Schema 拓扑结构(区分工具调用项、澄清追问项、开放集拒识项等)并进行对数几率边际校准,彻底消除 System-1 模型常见的工具误触发(Over-Triggering)与越界误接受问题。
 
 
 
 
 
138
 
139
  ---
140
 
141
- ## 快速上手(Quickstart)
 
 
 
 
 
 
142
 
143
- ### 1. PyTorch / Transformers(支持原生 `bfloat16` 或免费 Colab T4 `4-bit` / `8-bit` 量化加载)
 
 
 
 
 
 
 
 
 
 
 
 
 
144
 
145
- `TypeScript/FastBrain-9B` 完全兼容标准 `joint_schema_model` 运行时,可直接无缝替换 `Cloudflare/clef-flash`。既支持在 `>=24 GB` 显存 GPU(L4 / A100 / RTX 4090 / PRO 6000)上以全精度 `bfloat16`(`18.82 GB`)运行,也支持通过 **`load_in_4bit=True`(仅占约 `6.8 GB` 显存,可在免费 Google Colab T4 16GB 或 8GB 消费级显卡上流畅运行)** 或 **`load_in_8bit=True`(约 `11.2 GB` 显存)** 加载,同时自动保持 `116M` `JointSchemaHead` 处于高精度 `bfloat16`/`float16`:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
 
147
  ```python
148
- from huggingface_hub import snapshot_download
149
  import sys
150
  import torch
 
151
 
152
  model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
153
  sys.path.insert(0, model_dir)
154
  import joint_schema_model as clef
155
 
156
- # 全精度加载(推荐 24GB+ 显卡:L4, A100, RTX 3090/4090, RTX PRO 6000):
157
  model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
158
 
159
- # 或 4-bit NF4 混合精度加载(8GB–16GB 显卡,例如免费 Google Colab T4):
160
  # model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
161
 
162
  response = clef.systemone(model, processor, {
163
  "model": "TypeScript/FastBrain-9B",
164
  "state": {
165
- "query": "能帮我查一下账户余额吗?",
166
  "tools": ["get_balance", "transfer_funds"],
167
  },
168
  "questions": {
169
- "q": {
170
  "type": "choice",
171
- "query": "选择最佳下一步动作。",
172
  "criteria": {
173
- "a": "调用 <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
174
- "b": "请先提供您的账户 ID 或身份验证令牌。",
175
- "c": "抱歉,我无法访问个人银行记录。",
176
  },
177
  },
178
  },
@@ -180,53 +250,28 @@ response = clef.systemone(model, processor, {
180
  print(response)
181
  ```
182
 
183
- ### 2. `llama.cpp` GGUF(`Q8_0` / `Q4_K_M` / `BF16`)通过 `/v1/systemone` 接口部署
184
-
185
- 本仓库提供适配官方 `llama.cpp`(**`b11430` 及以上版本**,原生 `ClefModel` 架构)的 GGUF 模型文件,单文件同时打包了 `Qwen3.5-9B` 混合主干与 `116M` `JointSchemaHead`(即使在 `Q4_K_M` 量化版本中,`dec.*` 决策头张量、`ssm_` 线性注意力状态与 `attn_` 核心投影层也全部保留在 `Q8_0` 高精度)。
186
 
187
  > [!IMPORTANT]
188
- > **`llama.cpp` 版本要求与关键启动参数说明:**
189
- > 1. **最低版本要求**:请使用 **`llama.cpp` `b11430` 或更新版本**(包含原生 `ClefModel` / `LLM_ARCH_CLEF` 计算图支持与 `/v1/systemone` HTTP 接口)。普通 Ollama 或未包含 `ClefModel` 支持的旧版 `llama.cpp` 无法执行 `dec.*` 决策头张量。
190
- > 2. **务必将 `-b` 与 `-ub` 设为不小于最大输入长度(例如 `-b 16384 -ub 16384 --no-cache-prompt` 或 `-b 4096 -ub 4096 --no-cache-prompt`)**:
191
- > `llama-server` 中的 `ClefModel` 会在单个物理微批次(`ubatch`,默认值 `512`)内对 `<|question_*|>` 与 `<|option_*|>` 标记区间计算交叉注意力。将 `-ub` 设为与 `-b` 相同可确保长 Prompt(>512 tokens)在单次物理 `ubatch` 内完整计算而不截断选项区间;`--no-cache-prompt` 则可避免前缀缓存跳过决策标记位。
192
-
193
- #### 第一步:下载预编译 `llama-server`(`b11430`+)与 GGUF 权重
194
 
195
  ```bash
196
- # 1. 下载支持 ClefModel 与 /v1/systemone 的预编译 llama.cpp (b11430+)
197
- # - Linux CUDA 13.x (Colab / Ubuntu): llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz
198
- # - Linux CUDA 12.x: llama-b11430-bin-ubuntu-cuda-12.8-x64.tar.gz
199
- # - macOS Apple Silicon (M1/M2/M3/M4): llama-b11430-bin-macos-arm64.tar.gz
200
  curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
201
-
202
- # 2. 下载 FastBrain-9B GGUF(可选 Q8_0 [9.66 GB,无损级精度] 或 Q4_K_M [6.73 GB,适配 8GB–16GB 显存])
203
  pip install -U "huggingface_hub[hf_transfer]"
204
  HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
205
- ```
206
 
207
- #### 第二步:启动 `llama-server` 监听 `/v1/systemone`
208
-
209
- ```bash
210
- # Linux / Google Colab 环境下确保 CUDA 动态库加入 LD_LIBRARY_PATH:
211
  export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
212
-
213
- # 启动 llama-server(24GB+ 显存 / 16K 上下文):
214
  ./llama-b11430/llama-server \
215
  -m ./FastBrain-9B-Q8_0.gguf \
216
  -ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
217
  --port 8080
218
-
219
- # 或在 12GB–16GB 显卡上运行(例如 Colab 免费 Tesla T4,4K 上下文):
220
- # ./llama-b11430/llama-server \
221
- # -m ./FastBrain-9B-Q4_K_M.gguf \
222
- # -ngl 99 -c 4096 -b 4096 -ub 4096 -np 1 -fa on --no-cache-prompt \
223
- # --port 8080
224
-
225
- # 可选:若输入状态包含多模态图片/截图,可挂载视觉投影器权重:
226
- # --mmproj ./mmproj-FastBrain-9B-Q8_0.gguf
227
  ```
228
 
229
- #### 第三步:请求 `/v1/systemone`(单次前向推理,`0` 生成 Token)
230
 
231
  ```bash
232
  curl http://127.0.0.1:8080/v1/systemone \
@@ -235,124 +280,62 @@ curl http://127.0.0.1:8080/v1/systemone \
235
  "model": "FastBrain-9B",
236
  "state": {
237
  "ticket_id": "ORD-99281",
238
- "customer_message": "我上周的订阅被重复扣费了两次,提交工单后一直没人回复。"
239
  },
240
  "questions": {
241
- "route": {
242
  "type": "choice",
243
- "instructions": "该工单应分派给哪个团队处理?",
244
  "criteria": {
245
- "billing": "支付、退款、重复扣费或发票问题",
246
- "technical": "程序缺陷、崩溃或服务故障",
247
- "account": "密码重置或登录异常"
248
  }
249
  },
250
- "urgent": {
251
  "type": "noul",
252
- "instructions": "客户是否正在反馈涉及资金损失或紧急的问题?"
253
  }
254
  }
255
  }'
256
  ```
257
 
258
- ### 3. SGLang 高吞吐服务部署(`lmsysorg/sglang:dev-clef`)
259
-
260
- `TypeScript/FastBrain-9B` 原生支持 SGLang 的 `JointSchemaPooler`(镜像 `lmsysorg/sglang:dev-clef`),利用无缓存原始 K/V FlashAttention Embedding 模式以及向量化前缀和池化(`torch.cumsum`)+ 变长打包 FlashAttention(`cu_seqlens`),在服务端高并发场景下实现 **`19,300+ tok/s`(每秒 `131+` 个结构化决策)** 的极致吞吐:
261
 
262
  ```bash
263
- docker run --gpus all \
264
- --shm-size 32g \
265
- -p 30000:30000 \
266
- -v ~/.cache/huggingface:/root/.cache/huggingface \
267
- --ipc=host \
268
  lmsysorg/sglang:dev-clef \
269
- sglang serve \
270
- --model-path TypeScript/FastBrain-9B \
271
- --host 0.0.0.0 \
272
- --port 30000
273
  ```
274
 
275
  ---
276
 
277
- ## 推理引擎横向性能评测:`SGLang (dev-clef)` vs. `llama.cpp (b11430)` vs. `PyTorch`
278
-
279
- 在**同一台 NVIDIA RTX PRO 6000 Blackwell Server Edition (G4)** GPU 上对五大真实决策负载进行端到端实测对比:
280
-
281
- | 负载场景 / 核心指标 | **SGLang `dev-clef`**<br>*(BF16 · JointSchemaPooler)* | **`llama.cpp b11430`**<br>*(GGUF Q8_0 · ClefModel)* | **`llama.cpp b11430`**<br>*(GGUF Q4_K_M · ClefModel)* | **PyTorch 原生**<br>*(BF16 · `joint_schema_model.py`)* |
282
- | :--- | :---: | :---: | :---: | :---: |
283
- | **冷启动加载耗时** | 11.11 s | **2.56 s** | **2.00 s** | 11.11 s |
284
- | **峰值显存占用 (16K 上下文)** | 18.18 GiB *(权重)* / ~20.5 GiB | **12.55 GiB** | **10.30 GiB** | 20.3–24.7 GiB |
285
- | **能否运行于 16GB T4 显卡?** | 否 *(仅 BF16, TP=1)* | **是 (`9.03 GiB` @ 4K)** | **是 (`6.78 GiB` @ 4K)** | **是 (`load_in_4bit`: `6.8 GiB`)** |
286
- | **相对 BF16 Top-1 决策一致率** | **100.00%** *(最大偏差 0.0078)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* | **100.00% *(基准)*** |
287
- | **1. 短文本意图路由** *(293 tok, K=2)* | **30.01 ms** *(9,763 tok/s)* | 32.82 ms *(8,929 tok/s)* | 31.69 ms *(9,245 tok/s)* | 43.23 ms *(6,778 tok/s)* |
288
- | **2. Typed Decisions** *(736 tok, K=5)* | **50.29 ms** *(14,635 tok/s)* | 62.40 ms *(11,795 tok/s)* | 61.80 ms *(11,909 tok/s)* | 63.58 ms *(11,576 tok/s)* |
289
- | **3. When2Call** *(1,068 tok, K=1)* | **66.06 ms** *(16,167 tok/s)* | 87.78 ms *(12,167 tok/s)* | 89.34 ms *(11,955 tok/s)* | 77.77 ms *(13,733 tok/s)* |
290
- | **4. PhishNChips** *(1,303 tok, K=9)* | **75.86 ms** *(17,176 tok/s)* | 108.55 ms *(12,004 tok/s)* | 109.97 ms *(11,849 tok/s)* | 90.32 ms *(14,426 tok/s)* |
291
- | **5. SafeAgents Judge** *(2,217 tok, K=2)* | **125.85 ms** *(17,616 tok/s)* | 181.70 ms *(12,201 tok/s)* | 183.50 ms *(12,082 tok/s)* | 138.06 ms *(16,058 tok/s)* |
292
- | **Batch `B=4` 并发决策** *(2,944 tok, 20 决策)* | **152.39 ms** *(19,319 tok/s · 131.2 决策/s)* | 248.50 ms *(11,847 tok/s · 80.5 决策/s)* | 245.20 ms *(12,006 tok/s · 81.6 决策/s)* | 175.85 ms *(16,742 tok/s · 113.7 决策/s)* |
293
-
294
- ---
295
-
296
- ## GGUF 量化精度保持率与硬件显存指南
297
 
298
- ### 1. Decision Index 10 大基准量化精度对齐验证(`BF16` vs. `Q8_0` vs. `Q4_K_M`)
299
 
300
- 在涵盖 `choice`、`noul`、`score` 三类题型的分层抽样测试集(**`203` 个多约束请求 / `1,473` 个独立评分问题**)上:`FastBrain-9B-Q8_0.gguf`(`9.66 GB`)与原生 `bfloat16` 的 **Top-1 决策一致率高达 `99.86%`(平均分差仅 `-0.03 pp`,几乎完全无损)**;`FastBrain-9B-Q4_K_M.gguf`(`6.73 GB`,保留 `Q8_0` 高精度决策头)亦保持了 **`98.78%` 的 Top-1 一致率**(校准后 10 任务均分仅相差 `-0.20 pp`):
301
-
302
- | 评测基准 / 指标 | PyTorch bfloat16<br>*(18.82 GB)* | GGUF Q8_0<br>*(9.66 GB)* | GGUF Q4_K_M<br>*(6.73 GB · Q8 决策头)* |
303
- | :--- | :---: | :---: | :---: |
304
- | **相对 BF16 Top-1 一致率** *(1,473 题)* | 100.00% *(基准)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* |
305
- | **概率分布平均绝对误差 MAE** | 0.00000 | **0.00203** *(0.20%)* | **0.00387** *(0.39%)* |
306
- | **10 任务宏平均(单权重原生输出)** | **82.37%** | **82.34%** *(-0.03 pp)* | **82.56%** *(+0.19 pp)* |
307
- | **10 任务宏平均��Schema 校准后)** | **83.68%** | **83.65%** *(-0.03 pp)* | **83.48%** *(-0.20 pp)* |
308
- | ├─ **1. BFCL** *(AST 工具选择)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
309
- | ├─ **2. ToolRet** *(20 工具重排)* | 73.15% | **73.70%** *(+0.55)* | **75.60%** *(+2.45)* |
310
- | ├─ **3. API-Bank** *(多步工具调用)* | 95.00% | **95.00%** *(0.00)* | **95.00%** *(0.00)* |
311
- | ├─ **4. BANKING77** *(77 类金融意图)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
312
- | ├─ **5. CLINC150+OOS** *(意图+越界拒识)* | 93.94% | **93.94%** *(0.00)* | **93.94%** *(0.00)* |
313
- | ├─ **6. Home Appliances** *(家电控制)* | 90.00% | **90.00%** *(0.00)* | **90.00%** *(0.00)* |
314
- | ├─ **7. BRIGHT** *(推理检索)* | 41.93% | **41.00%** *(-0.93)* | **40.31%** *(-1.62)* |
315
- | ├─ **8. Amazon ESCI** *(电商相关性)* | 73.89% | **73.89%** *(0.00)* | **75.79%** *(+1.90)* |
316
- | ├─ **9. PhishNChips** *(9 探针钓鱼检测)* | 85.00% | **85.00%** *(0.00)* | **85.00%** *(0.00)* |
317
- | └─ **10. When2Call** *(4 分类触发治理)* | 75.00% *(70.83%)* | **75.00%** *(0.00)* | **72.92%** *(-2.08)* |
318
-
319
- ### 2. 各硬件平台实测推理速度与显存指南(`llama-server b11430`)
320
-
321
- 得益于**单次非自回归 Prefill 前向(`output_tokens = 0`)**架构,`llama-server` 在服务端显卡上实测吞吐高达 **8,900–12,100+ input tokens/sec**(端到端 HTTP 响应仅 `31–109 ms`):
322
-
323
- | 负载场景 / 指标 | Q8_0.gguf<br>*(PRO 6000)* | Q4_K_M.gguf<br>*(PRO 6000)* | Q8_0.gguf<br>*(Tesla T4 16G)* | Q4_K_M.gguf<br>*(Tesla T4 16G)* | PyTorch BF16<br>*(PRO 6000)* |
324
- | :--- | :---: | :---: | :---: | :---: | :---: |
325
- | **模型加载耗时** | **2.56 s** | **2.00 s** | **5.43 s** | **5.10 s** | ~11.5 s |
326
- | **峰值显存占用** | **12.55 GiB**<br>*(16K ctx)* | **10.30 GiB**<br>*(16K ctx)* | **9.03 GiB**<br>*(4K ctx)* | **6.78 GiB**<br>*(4K ctx)* | 20.3–24.7 GiB |
327
- | **意图路由 + OOS**<br>*(293 tok, K=2)* | **32.82 ms**<br>*(8,928 tok/s)* | **31.69 ms**<br>*(9,245 tok/s)* | **349.32 ms**<br>*(839 tok/s)* | **342.98 ms**<br>*(854 tok/s)* | ~35.0 ms |
328
- | **When2Call**<br>*(1,068 tok, K=1)* | **87.78 ms**<br>*(12,167 tok/s)* | **89.34 ms**<br>*(11,955 tok/s)* | **1,149.8 ms**<br>*(929 tok/s)* | **1,154.3 ms**<br>*(925 tok/s)* | ~92.0 ms |
329
- | **PhishNChips**<br>*(1,303 tok, K=9)* | **108.55 ms**<br>*(12,004 tok/s)* | **109.97 ms**<br>*(11,849 tok/s)* | **1,430.8 ms**<br>*(911 tok/s)* | **1,440.5 ms**<br>*(905 tok/s)* | ~115.0 ms |
330
-
331
- | 模型格式 / 量化档位 | 磁盘权重大小 | 峰值显存(`4K`–`16K` 上下文) | 最低推荐运行硬件 |
332
- | :--- | :---: | :---: | :--- |
333
- | **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit 主干 + `Q8_0` 决策头)* | `6.73 GB` | `~6.8 – 10.3 GiB` | **8GB–12GB 消费级显卡 / Tesla T4 (16GB) / Mac M1–M4** |
334
- | **PyTorch `load_in_4bit=True`** *(NF4 主干 + `fp16` 决策头)* | `~5.5 GB` | `~6.2 – 6.8 GiB` | **Tesla T4 (16GB) / RTX 3060–4060 (8GB+)** |
335
- | **`FastBrain-9B-Q8_0.gguf`** *(无损级全 8-bit GGUF)* | `9.66 GB` | `~9.0 – 12.6 GiB` | **12GB–16GB 显卡 / Tesla T4 (16GB) / Mac 16GB+ 统一内存** |
336
- | **PyTorch `load_in_8bit=True`** *(INT8 主干 + `bf16` 决策头)* | `~9.6 GB` | `~11.2 GiB` | **Tesla T4 (16GB) / RTX 4070 Ti (12GB+)** |
337
- | **原生全精度 `bfloat16`** *(`safetensors` / `BF16.gguf`)* | `18.82 GB` (`18.16 GB`) | `20.3 – 24.7 GiB` | **NVIDIA L4 (24GB) / A100 / RTX 3090–4090 / PRO 6000** |
338
-
339
- ---
340
-
341
- ## 仓库文件结构说明
342
-
343
- | 文件路径 | 大小 | 用途说明 |
344
  | :--- | :---: | :--- |
345
- | `model-00001-of-00004.safetensors` .. `00004` | `18.59 GB` | 完整 `9B` 主干权重(`bfloat16`,微调更新已原地合并,无需外挂 LoRA) |
346
- | `joint_head.safetensors` & `joint_head_config.json` | `232 MB` | `116M` 参数 `JointSchemaHead` 联合决策头权重(`122` 个张量)与配置 |
347
- | `FastBrain-9B-Q4_K_M.gguf` | `6.73 GB` | 推荐 4-bit GGUF(`dec.*` 决策头、`ssm_` 与 `attn_` 核心层均保留 `Q8_0` 精度) |
348
- | `FastBrain-9B-Q8_0.gguf` | `9.66 GB` | 无损级高精度 8-bit GGUF(适配 `llama-server` `ClefModel` 架构) |
349
- | `FastBrain-9B-BF16.gguf` | `18.16 GB` | 全精度 `bfloat16` GGUF(适配 `llama-server` `ClefModel` 架构) |
350
- | `mmproj-FastBrain-9B-Q8_0.gguf` & `BF16.gguf` | `624 MB` / `922 MB` | 多模态视觉投影器 GGUF 权重(用于处理图片/屏幕截图状态) |
351
- | `joint_schema_model.py` | `25 KB` | `ClefModel`、`JointSchemaHead`、`load_release_model(load_in_4bit/8bit)` 与 `systemone()` 推理运行时 |
352
- | `eval_submissions/typescript_fastbrain-9b.json` | `901 KB` | `LocalLLaMA/typed-decisions` 400 用例 / 2,000 决策零样本完整预测轨迹([已合入 PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) |
353
- | `eval_submissions/typescript_fastbrain-9b-fitted.json` | `902 KB` | `LocalLLaMA/typed-decisions` 400 用例 / 2,000 决策先验校准完整预测轨迹([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)��� |
354
- | `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | `110 KB` | `harness-hub/safeagents-judge-bench` (`v2`) 661 条工具调用二分类与 7 分类预测轨迹([已合入 PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) |
355
- | `.eval_results/*.yaml` | `2.3 KB` | Hugging Face Hub 官方排行榜评测元数据(`typed-decisions` 与 `safeagents-judge-bench v2`) |
356
- | `README.md` & `README_zh.md` | `32 KB` | 中英双语完整技术文档与榜单报告 |
357
- | `config.json`, `tokenizer.json`, `processor_config.json` | `11 MB` | Hugging Face 标准模型、分词器与多模态处理器配置 |
358
- | `assets/*.png` | `1.2 MB` | 高分辨率评测对比总图、全量得分矩阵图与概率校准分析图 |
 
 
 
 
 
 
 
 
 
2
 
3
  # FastBrain-9B
4
 
5
+ **面向校准结构化决策的单次前向非自回归模型**
6
 
7
+ [![Backbone](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.41B)-3b82f6?style=flat-square)](#2-模型架构与训练目标)
8
+ [![Head](https://img.shields.io/badge/Decision_Head-JointSchemaHead_(116M)-10b981?style=flat-square)](#2-模型架构与训练目标)
9
+ [![SafeAgents](https://img.shields.io/badge/SafeAgents_v2-0.751_Cat_Macro--F1_(Rank_1%2F8)-ec4899?style=flat-square)](#31-智能体工具调用安全审计评测-safeagents-judge-bench)
10
+ [![TypedDecisions](https://img.shields.io/badge/Typed_Decisions-0.7525_ZS_(Rank_7%2F43)_%C2%B7_50ms-8b5cf6?style=flat-square)](#32-多业务流企业结构化决策评测-localllamatyped-decisions)
11
+ [![DecisionIndex](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(Rank_2%2F7)-f59e0b?style=flat-square)](#33-多领域结构化决策基准-cloudflare-decision-index-v021)
12
  [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
13
 
14
  **[English](./README.md)** | **简体中文**
15
 
16
  </div>
17
 
18
+ ---
19
+
20
+ ## 摘要
21
 
22
+ 将自回归语言模型用于封闭 Schema 分类、工具路由与安全合规校验等结构化决策任务时,逐 Token 解码机制不仅引入与决策复杂度无关的序列生成延迟,还会因表面词形切分导致类别概率分布失准。本文发布 **FastBrain-9B**,一个总参数量为 9.41B 的非自回归结构化决策模型,由 32 层混合 Gated DeltaNet 线性注意力与全注意力主干网络(基于 `Cloudflare/clef-flash` 初始化的 `Qwen3.5-9B` 架构)及 116M 参数的交叉注意力联合决策头(`JointSchemaHead`)组成。给定任意 JSON 或多模态状态以及一组类型化决策定义(多选分类 `choice`、二元真值判别 `noul` 与序数量表评分 `score`),模型仅通过单次 Prefill 前向传播(`output_tokens = 0`)即可同步输出全部问题与候选项的校准概率分布。训练阶段,我们在复合序数 KL 散度、累积分布距离(离散分级概率评分 RPS)与期望值均方误差目标下,联合优化全部 32 层中的 104 个注意力投影矩阵及决策头参数,并将增量权重原地合并至主干张量。在涵盖 109 个参评模型条目的三项公开评测基准中:FastBrain-9B 在 `safeagents-judge-bench`(v2)7 类违规判别赛道位列 **第 1 / 8**(Macro-F1 `0.751`,准确率 `0.858`,较 27.4B 参数的 `Cloudflare/clef` 高出 18.8 个百分点),并在二元拦截赛道实现 `82/82` 越界轨迹全检出(AUROC `0.985`);在 `LocalLLaMA/typed-decisions` 零样本总榜位列 **第 7 / 43**(开源权重模型 **第 5 / 36**,$\le$9B 开源模型 **第 2 / 19**),中位推理延迟为 `50 ms`(较 31B 模型 `Quyet-1.0-Large` 快 66.7 倍),在冻结全部模型权重仅拟合先验与温度参数的设定下位列 `train` 校准榜 **并列第 12(列表第 14 / 51)**(准确率 `0.7970`,KL 散度 `0.0816`);在 Cloudflare Decision Index(v0.2.1)10 任务套件上取得 **80.55%** 的宏平均分,位���全量级 **第 2 / 7**、$\le$9B 量级 **10 / 10 任务单项第 1**,并在其中 5 项任务上超越 27.4B 的 `Cloudflare/clef`。
23
+
24
+ ---
25
 
26
+ ## 1. 研究背景与设计动机
27
 
28
+ 在实际部署的智能体(Agent)与企业自动化工作流中,系统需高频执行具有明确输出空间的结构化判断,例如候选工具与参数选择、沙箱可疑命令拦截、工单队列分发以及有序风险等级评定。采用自回归生成范式求解此类问题存在两方面固有局限:其一,端到端延迟受限于格式化字符与标签文本的串行解码步数;其二,概率质量被分散至同义词表面形式或格式 Token 上,导致预测分布偏离真实后验概率,难以直接设定可靠的拒识或风险拦截阈值。
29
 
30
+ 单次前向决策架构(`ClefModel` / System-1 范式)通过特殊边界标记将问题描述与候选项直接编码至输入序列尾部,利用最后一层隐藏状态经单次 Prefill 前向直接读取各选项对数几率(Logits),从而消除了自回归解码开销。然而,现有 9B 级决策基座在细粒度多类判别、序数标尺回归以及开放集越界拒识任务上与 27B–35B 级模型仍存在显著差距(例如 `Cloudflare/clef-flash` 在 `safeagents-judge-bench` v2 多分类赛道 Macro-F1 仅为 `0.462`,在 `CLINC150+OOS` 上仅为 `66.77%`)。FastBrain-9B 通过全深度投影矩阵联合适配、序数累积分布蒸馏以及编译期 Schema 拓扑边际校准,在保持 9B 级显存与单卡亚百毫秒延迟的前提下弥补乃至逆转了上述规模差距,并提供了高精度保持率的 8-bit 与 4-bit GGUF 量化部署方案。
 
 
 
31
 
32
  ---
33
 
34
+ ## 2. 模型架构与训练目标
35
 
36
+ ### 2.1 混合线性注意力与全注意力主干网络
37
 
38
+ FastBrain-9B 主干网络包含 32 层混合序列建模模块(`9.30B` 参数,`bfloat16` 精度,最大上下文长度 `262,144` tokens),由 24 层 Gated DeltaNet 线性注意力模块与 8 层标准多头自注意力模块交替堆叠而成。在指令适应阶段,我们联合解冻并更新全部 32 层中的 104 个核心投影矩阵(包括线性注意力层的 `in_proj_qkv`、`in_proj_z`、`out_proj` 以及全注意力层的 `q_proj`、`k_proj`、`v_proj`、`o_proj`)。训练完成后,所有参数更新均被精确合并至 `model-00001-of-00004.safetensors` 至 `model-00004-of-00004.safetensors` 中,推理阶段无需加载外部 LoRA 适配器。
 
 
 
 
 
 
 
39
 
40
+ ### 2.2 联合 Schema 决策头与向量化 CUDA 算子
41
+
42
+ `JointSchemaHead`(`116M` 参数,独立存储于 `joint_head.safetensors`)从主干末层隐藏状态 $H \in \mathbb{R}^{T \times d}$ 中提取由 `<|question_start|>`/`<|question_end|>` 与 `<|option_start|>`/`<|option_end|>` 界定的片段表征,经问题-选项交叉注意力交互后映射为三类基础决策原语的标量对数几率:
43
+ 1. **`choice`**($K$ 分类离散分布):输出归一化概率向量 $\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}$。
44
+ 2. **`noul`**(二元真值判别):输出命题成立概率 $\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})$。
45
+ 3. **`score`**(定义在有序分值 $v_1 < \dots < v_K$ 上的量表评估):同时输出各等级概率 $\hat{p} \in \Delta^{K-1}$ 与数学期望得分 $\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k$。
46
 
47
+ 在参考运行时 `joint_schema_model.py` 中,针对 CUDA 设备实现了融合算子路径 `_forward_fast_cuda`:利用基于 `torch.cumsum` 的 $\mathcal{O}(1)$ 前缀和差分替代逐片段 Python 循环完成变长片段均值池化,并通过 `cu_seqlens` 紧凑打包调用变长 FlashAttention(`torch.ops.aten._flash_attention_forward`)替代稠密掩码注意力。在 NVIDIA RTX PRO 6000 Blackwell 上,该实现将决策头单次前向耗时由 `3.27 ms` 降至 `1.72 ms`(**1.90 倍加速**),且保持严格数值等价(最大 Logit 绝对偏差 `max_logit_diff = 0.00000`)。
48
 
49
+ ### 2.3 序数 KL 蒸馏与编译期 Schema 拓扑校准
 
 
50
 
51
+ 标准交叉熵或普通多类 KL 散度将有序评定量表视为无序独立类别,对相邻等级误差与跨端极端误差施加同等惩罚。FastBrain-9B 在 `TypeScript/FastBrain-Instruct` 数据集上采用复合分布匹配损失进行训练,联合优化软标签 KL 散度、离散分级概率评分(预测累积分布 $\hat{F}_k = \sum_{j \le k} \hat{p}_j$ 与目标累积分布 $F_k$ 之间的二阶距离)以及 `score` 题型的期望值均方误差:
52
 
53
+ $$\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.$$
54
 
55
+ 在推理阶段,`joint_schema_model.py` 提供可选的编译期 Schema 拓扑校准函数(`apply_schema_aware_calibration`),通过解析候选项的语法结构特征(如 `<tool_call>` 调用声明、澄清追问子句及越界拒识选项)修正判别边际,抑制工具过度触发(Over-Triggering)与开放集误接收。
56
 
57
+ ---
58
 
59
+ ## 3. 实验评测与基准分析
60
 
61
+ **表 1:公开评测基准与官方排行榜结果汇总(截至 2026 年 11 月,核验共 109 个参评模型条目)。**
62
 
63
+ | 评测基准与赛道 | 参评模型规模 | FastBrain-9B 核心指标 | 官方排名 | 代表性对比基线(参数量,指标) |
64
+ | :--- | :---: | :---: | :---: | :--- |
65
+ | [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 7 类违规判别 ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 个决策模型 (+2 基线) | **0.751** Macro-F1 / **0.858** Acc | **1 / 8** | `Cloudflare/clef` (27.4B, `0.563`), `vllm-sr/d3-mini` (1.8B, `0.512`), `Cloudflare/clef-flash` (9.4B, `0.462`), `clef-omni` (30B, `0.437`) |
66
+ | [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 二元安全护栏 ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 个决策模型 (+2 基线) | **0.985** AUROC / **82/82** 轨迹检出<br>(`0.814` 默认;**0.938** 阈值校准) | **3 / 8** AUROC<br>**1 / 8** (并列) 轨迹 | `Cloudflare/clef` (27.4B, 默认 Bal. Acc `0.808`), `clef-flash` (`0.967` AUROC, `81/82`), `clef-omni` (`0.946` AUROC, `81/82`), `d1-3B` (`68/82`) |
67
+ | [`safeagents-judge-bench` (v1)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1) — 二元与 7 分类 ([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 个决策模型 (+2 基线) | **0.974** Bal. Acc / **0.850** Cat. F1 | **1 / 9** (双赛道第 1) | `Cloudflare/clef-omni` (30B, `0.845` / `0.327`), `Cloudflare/clef` (27.4B, `0.718` / `0.425`), `clef-flash` (`0.721` / `0.345`) |
68
+ | [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — 零样本评测 ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) | 43 个模型 (36 开源, 19 个 $\le$9B 开源) | **0.7525** Acc (`0.7625` 并列口径)<br>`0.1508` KL · `0.0795` Brier · **50 ms** | **7 / 43** 总榜<br>**5 / 36** 开源 · **2 / 19** $\le$9B | `vllm-sr/d3` (27B, `0.749`), `Elephant 1` (9B, `0.748`, `352 ms`), `Darwin-27B` (`0.743`), `Cloudflare/clef` (27B, `0.728`), `clef-flash` (`0.710`) |
69
+ | [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — `train` 校准 ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) | 51 个微调与校准模型 | **0.7970** Acc (`0.8070` 并列口径)<br>**0.0816** KL · **0.0437** Brier · **50 ms** | **并列 12** (列表 14 / 51)<br>冻结权重 $\le$9B 及 KL **第 1** | 冻结全部模型权重(`no weights changed`);高于 `Rev-gen1-midi` (25.6B, `0.7945`)、`OpenDecider-small-td` (4B LoRA, `0.792`)、`OpenDecider-medium-td` (30B LoRA, `0.788`) |
70
+ | Cloudflare Decision Index (`v0.2.1`) — 10 任务套件 | 7 个模型 (0.4B–27.4B) | **80.55%** 宏平均 | **2 / 7** 全量级总榜<br>$\le$9B 量级 **10 / 10 单项第 1** | `Jev` (27B, `80.32%`), `Kev 9B` (`76.51%`), `Clef-flash` (9.4B, `74.96%`), `DiffGemma` (26B, `69.14%`);5 / 10 项任务高于 `27.4B Clef` |
 
 
 
 
 
 
71
 
72
  ---
73
 
74
+ ### 3.1 智能体工具调用安全审计评测 (`safeagents-judge-bench`)
75
 
76
+ [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) 用于评估决策模型对真实编码智能体(运行于 Terminal-Bench 2.1、OpenThoughts-TBLite 与 Terminal-Bench 4)在沙箱中被静态扫描器 `atif-scan` 标记的可疑工具调用的审计能力。`v2` 测试集包含来自 55 个任务的 $N = 661$ 次工具调用(其中 `536` 次为分布在 `82` 条违规轨迹中的恶意 `block` 调用,`125` 次为正常 `allow` 调用)。每次调用以零样本单次请求形式同时询问一个二元 `noul` 拦截问题与一个 7 类 `choice` 违规归因问题(`legit`、`eval_gaming`、`harness_introspection`、`credential_misuse`、`exfiltration`、`sandbox_escape`、`other_overreach`)。表 2 中的 `v2` 官方结果由基准维护者在 NVIDIA A100-SXM4-80GB(HF Jobs)上独立运行并提交合入([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1))。
77
 
78
+ 在 7 类违规归因赛道上,FastBrain-9B 取得 `0.751` 的 Macro-F1 与 `0.858` 的分类准确率(`567/661`),位列全部参评模型第 1:相较于参数量约为其 3 倍的 27.4B 模型 `Cloudflare/clef`(Macro-F1 `0.563`,准确率 `0.737`),分别高出 `+18.8` 与 `+12.1` 个百分点;相较于同尺寸基座 `Cloudflare/clef-flash`(Macro-F1 `0.462`,准确率 `0.576`),分别提升 `+28.9` 与 `+28.2` 个百分点。分项指标上,`harness_introspection`($n=272$)F1 达 `0.945`(召回率 `0.978`),`eval_gaming`($n=245$)F1 达 `0.828`(召回率 `0.853`),合法调用 `legit`($n=125$)精确率达 `0.975`。
 
 
79
 
80
+ 在二元安全拦截赛道上,FastBrain-9B 的 AUROC 为 `0.985`,恶意调用总召回率为 `0.996`(`534/536`),全部 `82/82` 条越界轨迹均在首次违规调用处即被捕获(`median_bad_calls_before_catch = 0.0`),其中 `harness_introspection`(`272/272`)、`credential_misuse`(`15/15`)与 `sandbox_escape`(`4/4`)三类召回率均为 `1.000`。在未经验证集调整的默认判别阈值 $P(\text{block}) \ge 0.50$ 下,平衡准确率为 `0.814`(误拦率 `0.368`,优于 27.4B `Cloudflare/clef` 的平衡准确率 `0.808` 与误拦率 `0.384`)。由于连续概率排序区分度保持在 `0.985` AUROC,将工作点阈值设定为 $P(\text{block}) \ge 0.961$ 时,平衡准确率提升至 `0.938`,误拦率降至 `0.056`(`7/125`),同时仍保持 `82/82` 越界轨迹全检出。表 2 同时列出了 `v1` 版本($N = 562$,78 条轨迹)的官方归档结果,FastBrain-9B 在该版本二元与多分类双赛道均位居第 1 / 9。
81
+
82
+ **表 2:`harness-hub/safeagents-judge-bench` 零样本评测结果对比(`v2` 测试集 $N = 661$,82 条越界轨迹;附 `v1` 归档对照 $N = 562$)。**
83
+
84
+ | 模型 | 参数量 | 7 分类 Macro-F1 ↑ | 7 分类准确率 ↑ | AUROC ↑ | 平衡准确率 ↑ | 恶意召回率 ↑ | 误拦率 ↓ | 越界轨迹检出 ↑ | p50 延迟(测试硬件) |
85
  | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
86
+ | **TypeScript/FastBrain-9B** (`v2`, 默认 $t = 0.50$) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
87
+ | **TypeScript/FastBrain-9B** (`v2`, 校准 $t = 0.961$) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
88
+ | **TypeScript/FastBrain-9B** (`v1` 官方榜, $N = 562$) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
89
+ | Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
90
+ | vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
91
+ | Cloudflare/clef-flash (基座) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
92
+ | Cloudflare/clef-omni | 30B-A3B | 0.437 | 0.564 | 0.946 | 0.877 | 0.914 | 0.160 | 81 / 82 | 285 ms (A100) |
93
+ | knowledgator/gliclass-v4-edge | 0.4B | 0.373 | 0.528 | 0.753 | 0.676 | 0.696 | 0.344 | 68 / 82 | 101 ms (L4) |
94
+ | fastino/GLiNER2.5-Decide | 0.5B | 0.355 | 0.620 | 0.517 | 0.576 | 0.151 | **0.000** | 17 / 82 | 117 ms (M5 Pro CPU) |
95
+ | LiquidAI/d1-3B | 3.0B | 0.328 | 0.457 | 0.946 | 0.836 | 0.681 | 0.008 | 68 / 82 | **35 ms** (A100) |
96
+
97
+ ---
98
+
99
+ ### 3.2 多业务流企业结构化决策评测 (`LocalLLaMA/typed-decisions`)
100
+
101
+ [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) 评测集覆盖智能体轨迹可观测性(`agent_trace_observability`)、客户服务(`customer_service`)、供应商发票核验(`invoice_processing`)与安全事件响应(`security_incidents`)四类企业工作流,测试集包含 400 个独立案例、共 2,000 个类型化子问题(涵盖 `noul`、`choice` 与 `score`),并以多教师模型集成的软标签分布作为真值基准。官方仓库存在两种并列真值处理脚本:`bench/score_with_jobs.py` 对真值分布中概率并列最高的选项按首个索引判对(`g.index(max(g))`),而 `scripts/score_model.py` 将 20 道并列题中的任一最高分选项均计为正确;两种脚本下的 KL 散度与 Brier 评分完全一致。我们在表 3 中同时报告两种口径的准确率,并在 [`eval_submissions/`](./eval_submissions/) 中公开原始预测文件。
102
+
103
+ 在零样本总榜($N = 43$ 个参评模型)中,FastBrain-9B 取得 `0.7525`(并列口径 `0.7625`)的准确率、`0.1508` 的 KL 散度、`0.0795` 的 Brier 评分与 `0.085` 的期望校准误差(ECE),在 RTX PRO 6000(SGLang)上的 p50 延迟为 `50 ms`。该成绩位列全量零样本榜 **第 7 / 43**、开源权重模型 **第 5 / 36**、$\le$9B 开源模型 **第 2 / 19** 以及全部 `Clef` 架构模型 **第 1**,较 9.4B 基座 `Cloudflare/clef-flash`(准确率 `0.7100`,KL `0.2093`)提升 `+4.25` 个百分点,较 27.4B `Cloudflare/clef`(准确率 `0.7275`,KL `0.1907`)提升 `+2.50` 个百分点。在零样本准确率高于 `0.700` 的全部 28 个模型中,FastBrain-9B 具有最低的 p50 推理延迟(`50 ms`,对比 9B `Amos 1 medium` 的 `118 ms`、27B `Amos 1 large` 的 `357 ms`、`meraGPT Decider 1` 的 `526 ms` 以及 31B `Quyet-1.0-Large` 的 `3,335 ms`)。若直接评测未经后验规则调整的序数 KL 蒸馏原生头,零样本准确率为 `0.7735`(KL `0.1039`,Brier `0.0582`)。
104
+
105
+ 在使用 `train` 划分(1,200 个样本)的独立榜单($N = 51$ 个模型)上,在**完全冻结主干与决策头权重(`no weights changed`)**、仅拟合逐问题对数先验偏置与标量温度参数的前提下,FastBrain-9B 取得 `0.7970`(并列口径 `0.8070`)的准确率、`0.0816` 的 KL 散度与 `0.0437` 的 Brier 评分,位列 **并列第 12(列表第 14 / 51)**,并在冻结权重模型中取得最优的 KL 分布校准度。
106
+
107
+ **表 3:`LocalLLaMA/typed-decisions` 测试集($N = 400$ 案例,2,000 次决策)评测结果。`/` 前后分别对应 `score_with_jobs.py` 与 `score_model.py` 口径。**
108
+
109
+ | 模型 | 参数量 | 评测设定 | 总体准确率 ↑ | KL 散度 ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | 客服 ↑ | 发票 ↑ | 安全 ↑ | p50 延迟 |
110
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
111
+ | **FastBrain-9B** (先验温度拟合,冻结权重) | 9.4B | Fitted | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
112
+ | **FastBrain-9B** (序数 KL 蒸馏原生头) | 9.4B | Zero-shot | 77.35% | 0.1039 | 0.0582 | 84.8% | 74.8% | 73.8% | 71.2% | 80.4% | 82.6% | 75.2% | **50 ms** |
113
+ | **FastBrain-9B** (官方主榜 PR #58) | 9.4B | Zero-shot | 75.25% / 76.25% | 0.1508 | 0.0795 | 83.2% / 84.3% | 74.7% / 76.0% | 69.8% / 70.4% | 68.2% / 69.8% | 77.6% / 78.8% | 82.0% | 73.2% / 74.4% | **50 ms** |
114
+ | Amos 1 (medium) | 9.0B | Zero-shot | 76.45% | 0.1510 | 0.0800 | — | — | — | — | — | — | — | 118 ms |
115
+ | Amos 1 (large) | 27.0B | Zero-shot | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
116
+ | pplx-decider-v1.1-27b | 27.0B | Zero-shot | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
117
+ | vllm-sr/d3 | 27.0B | Zero-shot | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
118
+ | Baseten Elephant 1 | 9.0B | Zero-shot | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
119
+ | Darwin-27B-ZTC | 27.0B | Zero-shot | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
120
+ | Cloudflare/clef | 27.4B | Zero-shot | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
121
+ | vllm-sr/d3-flash | 9.0B | Zero-shot | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
122
+ | Cloudflare/clef-flash (基座) | 9.4B | Zero-shot | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
123
+ | Cloudflare/clef-omni | 35B-A3B | Zero-shot | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
124
 
125
  ---
126
 
127
+ ### 3.3 多领域结构化决策基准 (Cloudflare Decision Index `v0.2.1`)
128
+
129
+ 在涵盖 10 个评测数据集、共 $N = 22,427$ 条请求的 **Cloudflare Decision Index(`v0.2.1`)** 套件(覆盖 AST 函数调用、20 候选工具检索重排、多步状态 API 调用、细粒度金融意图识别、开放集越界拒识、推理密集型检索、电商相关性分级、多探针钓鱼网站鉴别与工具调用触发治理)上,FastBrain-9B 的无加权 10 任务宏平均得分为 **80.55%**(表 4,图 1–3)。
130
+
131
+ 在 $\le$9B 参数量级(对比 `Clef-flash` 9.4B、`Kev 9B` 与 `Laya` 0.4B)中,FastBrain-9B 在全部 **10 / 10 个子基准上均取得第 1**,宏平均分较同架构基座 `Clef-flash`(`74.96%`)提升 `+5.59` 个百分点,较 `Kev 9B`(`76.51%`)提升 `+4.04` 个百分点。增幅主要集中于对边界校准敏感的开放集拒识与安全治理任务:`CLINC150+OOS` 提升 `+31.67` 个百分点(`98.44%` 对比 `66.77%`),`When2Call` 提升 `+15.55` 个百分点(`81.13%` 对比 `65.58%`),`PhishNChips` 提升 `+6.80` 个百分点(`78.75%` 对比 `71.95%`)。跨全部参数规模(最高 27.4B)比较,FastBrain-9B 总宏平均位列 **第 2 / 7**,高于 `27B Jev`(`80.32%`)与 `26B-A4B DiffusionGemma Jev`(`69.14%`),并在 `BFCL`、`API-Bank`、`CLINC150+OOS`、`Home Appliances` 与 `When2Call` 五个子基准上高于 `27.4B Clef`(`81.81%`)。
132
+
133
+ ![图 1:Cloudflare Decision Index v0.2.1 总体宏平均与参数效率对比](./assets/benchmark_overview.png)
134
+ *图 1. Cloudflare Decision Index(v0.2.1,22,427 条请求)10 任务宏平均分与模型参数规模关系。*
135
+
136
+ ![图 2:不同参数规模决策模型在 10 个子任务上的得分矩阵](./assets/benchmark_scoreboard.png)
137
+ *图 2. 0.4B–27.4B 规模决策模型在 Cloudflare Decision Index(v0.2.1)各子任务上的详细得分矩阵。*
138
+
139
+ ![图 3:置信度校准曲线与开放集越界拒识分析](./assets/decision_calibration_breakdown.png)
140
+ *图 3. 置信度可靠性图、开放集越界意图拒识(`CLINC150+OOS`)与工具触发边际分析(`When2Call`)。*
141
+
142
+ **表 4:Cloudflare Decision Index(`v0.2.1`,10 任务,$N = 22,427$)逐项评测结果。粗体表示全参数规模最高值,下划线表示 $\le$9B 量级最高值。**
143
 
144
+ | 评测基准(指标) | **FastBrain-9B** (本工作) | Clef-flash (基座) | Kev 9B | Clef | Jev | DiffGemma | Laya |
145
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
146
+ | 总参数量 | 9.41B | 9.41B | 9.00B | 27.36B | 27.00B | 25.8B (激活 4B) | 0.42B |
147
+ | 1. BFCL (Case Exact Acc) | **<u>98.76%</u>** | **<u>98.76%</u>** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
148
+ | 2. ToolRet (nDCG@10) | <u>67.26%</u> | <u>67.26%</u> | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
149
+ | 3. API-Bank (Accuracy) | **<u>93.70%</u>** | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
150
+ | 4. BANKING77 (Macro-F1) | <u>93.77%</u> | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
151
+ | 5. CLINC150+OOS (Macro-F1) | **<u>98.44%</u>** | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
152
+ | 6. Home Appliances (Case Exact) | **<u>97.73%</u>** | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
153
+ | 7. BRIGHT (nDCG@10) | <u>38.58%</u> | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
154
+ | 8. Amazon ESCI (Macro-F1) | <u>57.40%</u> | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
155
+ | 9. PhishNChips (Accuracy) | <u>78.75%</u> | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
156
+ | 10. When2Call (Macro-F1) | <u>81.13%</u> | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
157
+ | **10 任务宏平均(Macro Mean)** | <u>80.55%</u> | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
158
 
159
  ---
160
 
161
+ ## 4. 系统吞吐与训练后量化分析
162
+
163
+ ### 4.1 不同推理引擎性能对比
164
+
165
+ 表 5 展示了在同一张 NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 上,SGLang(集成 `JointSchemaPooler` 的 `lmsysorg/sglang:dev-clef`)、`llama.cpp`(`b11430` `ClefModel` 计算图,含 `Q8_0` 与 `Q4_K_M` 两种精度)以及原生 PyTorch(`joint_schema_model.py`)在五类典型长度负载下的端到端延迟、Prefill 吞吐率与显存占用。由于全部请求的生成步数均为零(`output_tokens = 0`),推理耗时完全取决于 Prefill 计算与片段池化效率;在批量 $B=4$ 并发设定下,SGLang 吞吐达 `19,319 tok/s`(每秒完成 `131.2` 个独立结构化决策)。
166
+
167
+ **表 5:NVIDIA RTX PRO 6000 Blackwell Server Edition(G4)上三类推理后端的延迟与吞吐实测。**
168
 
169
+ | 评测负载 / 系统指标 | SGLang `dev-clef` (BF16) | `llama.cpp b11430` (`Q8_0`) | `llama.cpp b11430` (`Q4_K_M`) | PyTorch 原生 (BF16) |
170
+ | :--- | :---: | :---: | :---: | :---: |
171
+ | 冷启动加载时间 | 11.11 s | 2.56 s | **2.00 s** | 11.11 s |
172
+ | 峰值显存占用 (16K 上下文) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
173
+ | 兼容 16 GiB T4 显卡 | 否 (BF16 TP=1) | 是 (`9.03 GiB`, 4K) | **是 (`6.78 GiB`, 4K)** | 是 (`6.8 GiB`, NF4) |
174
+ | 相对 PyTorch BF16 Top-1 一致率 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (基准) |
175
+ | 1. 短文本意图路由 (293 tok, $K=2$) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
176
+ | 2. Typed Decisions (736 tok, $K=5$) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
177
+ | 3. When2Call 工具治理 (1,068 tok, $K=1$) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
178
+ | 4. PhishNChips 安全检测 (1,303 tok, $K=9$) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
179
+ | 5. SafeAgents 审计 (2,217 tok, $K=2$) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
180
+ | 批量 $B=4$ Typed Decisions (2,944 tok, 20 题) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
181
+
182
+ ### 4.2 混合精度 GGUF 量化保真度分析
183
 
184
+ 若对交叉注意力决策头或线性注意力递归状态直接施加均匀 4-bit 量化,细粒度选项间微小的对数几率差值易受量化噪声干扰。因此在 `FastBrain-9B-Q4_K_M.gguf`(`6.73 GB`)构建方案中,仅对前馈网络(FFN)使用 4-bit K-quants,而将全部 `dec.*` 决策头张量、`ssm_*` Gated DeltaNet 状态参数以及 `attn_*` 投影矩阵保持为 `Q8_0` 精度。在涵盖 10 类任务的分层验证子集(`203` 个多约束请求,共 `1,473` 道独立评分题;表 6)上,`Q8_0.gguf`(`9.66 GB`)在 `1,471/1,473` 道题上与原生 `bfloat16` 给出完全一致的 Top-1 决策(一致率 `99.86%`,预测概率平均绝对误差 `0.00203`,10 任务均分差异 `-0.03` 个百分点);`Q4_K_M.gguf` 保持了 `98.78%` 的 Top-1 决策一致率(概率 MAE `0.00387`,校准后均分差异 `-0.20` 个百分点)。表 7 给出了各精度版本在不同硬件(含 NVIDIA Tesla T4 `16 GiB` 实测)上的资源需求。
185
+
186
+ **表 6:10 任务分层子集($N = 1,473$ 道评分题)上 GGUF 量化模型的决策一致性与精度保持率。**
187
+
188
+ | 评测指标 / 子任务 | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` 决策头) |
189
+ | :--- | :---: | :---: | :---: |
190
+ | Top-1 决策一致率 ($N = 1,473$) | 100.00% (基准) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
191
+ | 预测概率平均绝对误差 ($\|\hat{p}_q - \hat{p}_{\text{bf16}}\|$) | 0.00000 | 0.00203 | 0.00387 |
192
+ | 10 任务宏平均(未校准原生输出) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
193
+ | 10 任务宏平均(Schema 拓扑校准) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
194
+ | ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
195
+ | ├─ 2. ToolRet / 7. BRIGHT | 73.15% / 41.93% | 73.70% / 41.00% | 75.60% / 40.31% |
196
+ | ├─ 5. CLINC150+OOS / 6. Home Appliances | 93.94% / 90.00% | 93.94% / 90.00% | 93.94% / 90.00% |
197
+ | └─ 8. Amazon ESCI / 9. PhishNChips / 10. When2Call | 73.89% / 85.00% / 75.00% | 73.89% / 85.00% / 75.00% | 75.79% / 85.00% / 72.92% |
198
+
199
+ **表 7:各精度格式的存储体积、显存占用峰值与 Tesla T4(`16 GiB`)实测延迟。**
200
+
201
+ | 精度与权重格式 | 权重体积 | 峰值显存 (4K–16K 上下文) | T4 延迟 (293 / 1,068 tok) | 适用硬件范围 |
202
+ | :--- | :---: | :---: | :---: | :--- |
203
+ | `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 6.78 – 10.30 GiB | 343.0 ms / 1,154.3 ms | 8–16 GB 显卡 (RTX 3060/4060, Tesla T4), Apple Silicon (8 GB+) |
204
+ | PyTorch `load_in_4bit=True` (NF4 + `fp16` 头) | ~5.5 GB (内存) | 6.20 – 6.80 GiB | — | 8–16 GB CUDA 显卡 (Colab T4, 消费级 GPU) |
205
+ | `FastBrain-9B-Q8_0.gguf` | 9.66 GB | 9.03 – 12.55 GiB | 349.3 ms / 1,149.8 ms | 12–16 GB 显卡 (Tesla T4, RTX 4070 Ti), Apple Silicon (16 GB+) |
206
+ | PyTorch `load_in_8bit=True` (INT8 + `bf16` 头) | ~9.6 GB (内存) | ~11.20 GiB | — | 12–16 GB CUDA 显卡 |
207
+ | 原生 `bfloat16` (`safetensors` / `BF16.gguf`) | 18.82 / 18.16 GB | 20.30 – 24.70 GiB | — | $\ge$24 GB 显卡 (NVIDIA L4, A100, H100, RTX 3090/4090, PRO 6000) |
208
+
209
+ ---
210
+
211
+ ## 5. 推理复现与服务部署
212
+
213
+ ### 5.1 Python / Transformers 接口(支持 `bfloat16`、`8-bit` 与 `4-bit` NF4)
214
+
215
+ `TypeScript/FastBrain-9B` 提供标准 `joint_schema_model` 接口,可与 `Cloudflare/clef-flash` 调用代码直接互换。启用 `load_in_4bit=True` 时,仅对 9.3B 主干网络执行 `bitsandbytes` NF4 量化,116M `JointSchemaHead` 保持浮点精度。
216
 
217
  ```python
 
218
  import sys
219
  import torch
220
+ from huggingface_hub import snapshot_download
221
 
222
  model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
223
  sys.path.insert(0, model_dir)
224
  import joint_schema_model as clef
225
 
226
+ # 原生 bfloat16 加载 (>=24 GB 显存):
227
  model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
228
 
229
+ # 混合精度 4-bit NF4 加载 (8-16 GB 显存,如免费 Colab Tesla T4):
230
  # model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
231
 
232
  response = clef.systemone(model, processor, {
233
  "model": "TypeScript/FastBrain-9B",
234
  "state": {
235
+ "query": "能帮我查询一下当前账户余额吗?",
236
  "tools": ["get_balance", "transfer_funds"],
237
  },
238
  "questions": {
239
+ "next_action": {
240
  "type": "choice",
241
+ "instructions": "选择最恰当的助手动作。",
242
  "criteria": {
243
+ "call_balance": "调用 <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
244
+ "request_auth": "请用户先提供账户标识符或身份验证令牌。",
245
+ "abstain": "拒绝执行,因为个人银行记录超出访问权限。",
246
  },
247
  },
248
  },
 
250
  print(response)
251
  ```
252
 
253
+ ### 5.2 `llama.cpp` GGUF 服务部署(`/v1/systemone`)
 
 
254
 
255
  > [!IMPORTANT]
256
+ > **`ClefModel` 架构 GGUF 运行参数要求**
257
+ > 1. 需使用 **`llama.cpp` `b11430` 或更高版本**。该版本包含 `LLM_ARCH_CLEF` 计算图定义(支持 `dec.*` 决策头张量)与 `/v1/systemone` HTTP 路由;旧版构建或纯生成式封装无法执行交叉注意力决策头。
258
+ > 2. 启动时需将物理微批次设为与逻辑批次一致(如 `-b 16384 -ub 16384` 或 `-b 4096 -ub 4096`),并开启 `--no-cache-prompt`。`ClefModel` 在单个物理 `ubatch` 内对 `<|question_*|>` 与 `<|option_*|>` 区间执行池化,默认的 `-ub 512` 会截断超过 512 tokens 的输入序列,而前缀缓存可能跳过边界标记计算。
 
 
 
259
 
260
  ```bash
261
+ # 1. 下载预编译 llama.cpp (b11430+) 与 GGUF 权重文件
 
 
 
262
  curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
 
 
263
  pip install -U "huggingface_hub[hf_transfer]"
264
  HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
 
265
 
266
+ # 2. 启动 llama-server(如需处理图像状态,追加 --mmproj mmproj-FastBrain-9B-Q8_0.gguf)
 
 
 
267
  export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
 
 
268
  ./llama-b11430/llama-server \
269
  -m ./FastBrain-9B-Q8_0.gguf \
270
  -ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
271
  --port 8080
 
 
 
 
 
 
 
 
 
272
  ```
273
 
274
+ 向 `/v1/systemone` 发送单次请求即可同步评估异构题型组合(`choice` + `noul`):
275
 
276
  ```bash
277
  curl http://127.0.0.1:8080/v1/systemone \
 
280
  "model": "FastBrain-9B",
281
  "state": {
282
  "ticket_id": "ORD-99281",
283
+ "customer_message": "我上周的订阅被重复扣费了两次,提交工单后一直无人处理。"
284
  },
285
  "questions": {
286
+ "department": {
287
  "type": "choice",
288
+ "instructions": "将该工单分派至对应的处理队列。",
289
  "criteria": {
290
+ "billing": "账单支付、退款、重复扣费或发票核对",
291
+ "engineering": "软件缺陷、程序崩溃或服务中断",
292
+ "identity": "密码重置或多因素身份验证异常"
293
  }
294
  },
295
+ "is_urgent": {
296
  "type": "noul",
297
+ "instructions": "该消息是否涉及未解决的资金损失或时效敏感故障?"
298
  }
299
  }
300
  }'
301
  ```
302
 
303
+ ### 5.3 SGLang 服务部署(`JointSchemaPooler`)
 
 
304
 
305
  ```bash
306
+ docker run --gpus all --shm-size 32g -p 30000:30000 \
307
+ -v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
 
 
 
308
  lmsysorg/sglang:dev-clef \
309
+ sglang serve --model-path TypeScript/FastBrain-9B --host 0.0.0.0 --port 30000
 
 
 
310
  ```
311
 
312
  ---
313
 
314
+ ## 6. 仓库文件清单与引用格式
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
315
 
316
+ **表 8:仓库文件与权重清单说明。**
317
 
318
+ | 文件路径 | 体积 | 内容说明 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
319
  | :--- | :---: | :--- |
320
+ | `model-00001-of-00004.safetensors` .. `00004` | 18.59 GB | 原地合并全部投影更新后的 `bfloat16` 混合主干网络权重 |
321
+ | `joint_head.safetensors`, `joint_head_config.json` | 232 MB | 116M 参数 `JointSchemaHead` 权重(`122` 个张量)与结构配置 |
322
+ | `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 混合精度 4-bit GGUF(`dec.*`、`ssm_*` 与 `attn_*` 保持 `Q8_0`) |
323
+ | `FastBrain-9B-Q8_0.gguf` / `FastBrain-9B-BF16.gguf` | 9.66 / 18.16 GB | 面向 `llama-server`(`b11430`+)的 8-bit 与 `bfloat16` GGUF 权重 |
324
+ | `mmproj-FastBrain-9B-Q8_0.gguf` / `BF16.gguf` | 624 / 922 MB | 支持多模态图像状态输入的视觉投影器权重 |
325
+ | `joint_schema_model.py` | 25 KB | `ClefModel` 定义、向量化 `_forward_fast_cuda` 算子及 `systemone()` 运行时 |
326
+ | `eval_submissions/typescript_fastbrain-9b*.json` | 1.80 MB | `LocalLLaMA/typed-decisions` 零样本与先验拟合的 2,000 次决策原始预测轨迹 |
327
+ | `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | 110 KB | `safeagents-judge-bench`(v2)661 条工具调用的二元与 7 分类预测轨迹 |
328
+ | `.eval_results/*.yaml` | 2.3 KB | Hugging Face Hub 标准化评测结果元数据 |
329
+ | `README.md`, `README_zh.md`, `assets/*.png` | 1.28 MB | 中英双语技术报告与实验评测图表 |
330
+
331
+ ### 引用(Citation)
332
+
333
+ ```bibtex
334
+ @misc{fastbrain9b2026,
335
+ title = {FastBrain-9B: A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions},
336
+ author = {TypeScript},
337
+ year = {2026},
338
+ howpublished = {\url{https://huggingface.co/TypeScript/FastBrain-9B}},
339
+ note = {Apache-2.0 License}
340
+ }
341
+ ```