Text Generation
Transformers
Safetensors
GGUF
English
Chinese
qwen3_5
image-text-to-text
fastbrain
clef
llama-cpp
sglang
systemone
decision-model
image-text-to-typed-output
decision-index
tool-calling
function-calling
intent-classification
structured-decision
conversational
Eval Results
Instructions to use TypeScript/FastBrain-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TypeScript/FastBrain-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="TypeScript/FastBrain-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("TypeScript/FastBrain-9B") model = AutoModelForMultimodalLM.from_pretrained("TypeScript/FastBrain-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use TypeScript/FastBrain-9B with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Use Docker
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use TypeScript/FastBrain-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "TypeScript/FastBrain-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- SGLang
How to use TypeScript/FastBrain-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "TypeScript/FastBrain-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "TypeScript/FastBrain-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use TypeScript/FastBrain-9B with Ollama:
ollama run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Unsloth Desktop
- Pi
How to use TypeScript/FastBrain-9B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "TypeScript/FastBrain-9B:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use TypeScript/FastBrain-9B with Docker Model Runner:
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Lemonade
How to use TypeScript/FastBrain-9B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull TypeScript/FastBrain-9B:Q4_K_M
Run and chat with the model
lemonade run user.FastBrain-9B-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use TypeScript/FastBrain-9B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default TypeScript/FastBrain-9B:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use TypeScript/FastBrain-9B with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "TypeScript/FastBrain-9B:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Rewrite README.md and README_zh.md in rigorous technical-report style
Browse files- README.md +220 -280
- README_zh.md +222 -239
README.md
CHANGED
|
@@ -28,161 +28,231 @@ tags:
|
|
| 28 |
|
| 29 |
# FastBrain-9B
|
| 30 |
|
| 31 |
-
**Single-Pass Non-Autoregressive
|
| 32 |
|
| 33 |
-
[](./LICENSE)
|
| 39 |
|
| 40 |
-
**English** | **[简体中文
|
| 41 |
|
| 42 |
</div>
|
| 43 |
|
| 44 |
-
|
|
|
|
|
|
|
| 45 |
|
| 46 |
-
Given an arbitrary JSON or multimodal state and
|
| 47 |
|
| 48 |
-
|
| 49 |
|
| 50 |
-
##
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
-
|
| 55 |
-
- **Production-Ready Across SGLang, `llama.cpp` GGUF, and PyTorch `4-bit`**: Ships with a `1.90x` faster fused FlashAttention decision head (`max_logit_diff = 0.00000`), lossless `Q8_0.gguf` (`99.86%` Top-1 agreement with `BF16`), and `6.73 GB` `Q4_K_M.gguf` (`98.78%` agreement, runs in `6.78 GiB` VRAM on a 16GB T4 or Apple Silicon Mac).
|
| 56 |
|
| 57 |
---
|
| 58 |
|
| 59 |
-
##
|
| 60 |
|
| 61 |
-
###
|
| 62 |
|
| 63 |
-
|
| 64 |
-
| :--- | :---: | :---: | :---: | :---: | :--- |
|
| 65 |
-
| **[`safeagents-judge-bench` (`v2`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(7-Way Category)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [Dataset Card](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench)) | **8 Decision Models**<br>*(+2 Baselines)* | **`75.1%` Macro-F1**<br>**`85.8%` Cat. Acc** | **#1 / 8 *(Live)*** | **Top 12.5% *(Rank #1)*** | `Cloudflare/clef` *(27.4B, #2 `56.3%` F1 / `73.7%` Acc)*, `vllm-sr/d3-mini` *(#3 `51.2%`)*, `Cloudflare/clef-flash` *(9.4B, #4 `46.2%`)*, `clef-omni` *(30B, #5 `43.7%`)* |
|
| 66 |
-
| **[`safeagents-judge-bench` (`v2` & `v1`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(Binary Guardrail)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [`v1` Tag](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)) | **8 Models (`v2`)**<br>**9 Models (`v1`)** | **`82/82` (`100%`) Traj. · `98.5%` AUROC**<br>*(`93.8%` Cal. / `81.4%` Raw `v2` · **`97.4%` `v1`**)* | **`#1 / 9` (`v1` Live)**<br>**`100%` Caught · #3 AUROC (`v2`)** | **Top 11.1% (`v1`)**<br>**Top 37.5% AUROC (`v2`)** | Beats `27.4B Cloudflare/clef` (`80.8%` Raw Bal. Acc), `clef-flash` (`96.7%` AUROC, `81/82` Traj.), `clef-omni` (`94.6%` AUROC, `81/82`), `d1-3B` (`68/82` Traj.) |
|
| 67 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot Overall)***<br>([Merged PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [Trace](./eval_submissions/typescript_fastbrain-9b.json)) | **43 Models**<br>*(36 Open-Weight)* | **`75.25%` / `76.25%` Acc**<br>*(KL `0.1508` · Brier `0.0795` · **`50 ms`**)* | **#7 / 43 *(Live)***<br>**#5 / 36 Open-Weight** | **Top 16.3% Overall**<br>**Top 13.9% Open** | `vllm-sr/d3` *(27B, #8 `74.9%`)*, `Baseten Elephant 1` *(9B, #9 `74.8%`)*, `Darwin-27B` *(#10 `74.3%`)*, `Cloudflare/clef` *(27B, #16 `72.8%`)*, `openjev` *(27B, #17 `72.8%`)* |
|
| 68 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot `<=9B` & Clef-Arch)*** | **19 `<=9B` Open**<br>**4 Clef-Arch** | **`75.25%` / `76.25%` Zero-Shot**<br>*(Distilled Raw: **`77.35%`** · **`50 ms`**)* | **#2 / 19 `<=9B` Open**<br>**#1 Clef-Arch · #1 Speed** | **Top 10.5% (`<=9B`)** | Beats 17 of 18 `<=9B` open models (`d3-mini` `74.3%`, `d3-flash` `71.7%`, `signpost-d-4b` `71.2%`, `clef-flash` `71.0%`, `d1-3B` `65.6%`) & `35B clef-omni` (`67.3%`) |
|
| 69 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Fitted on `train`)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [Trace](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **51 Models** | **`79.70%` / `80.70%` Acc**<br>*(KL **`0.0816`** · Brier **`0.0437`** · **`50 ms`**)* | **#12 Tied *(#14 / 51 Live)***<br>**#1 Frozen `<=9B` · #1 Frozen KL** | **Top 23.5% Overall**<br>*(Zero Weight Updates)* | Beats **37** fine-tuned/specialist models: `Rev-gen1-midi` *(25.6B, `79.45%`)*, `OpenDecider-small-td` *(4B, `79.2%`)*, `OpenDecider-medium-td` *(30B, `78.8%`)*, `Seb-9B` *(`75.4%`)* |
|
| 70 |
-
| **Cloudflare Decision Index (`v0.2.1`)** *(10-Task Macro)* | **7 Models** | **`80.55%` Macro Mean** | **#2 / 7 Overall**<br>**#1 / 10 (`<=9B` Sweep)** | **Top 28.6% Overall**<br>**10/10 `#1` (`<=9B`)** | Outperforms `Jev` *(27B, `80.32%`)*, `Kev 9B` *(`76.51%`)*, `Clef-flash` *(9.4B, `74.96%`)*, `DiffGemma` *(26B, `69.14%`)*, and beats `27.4B Clef` on **5 / 10** tasks |
|
| 71 |
|
| 72 |
-
|
| 73 |
|
| 74 |
-
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
-
|
| 77 |
-
- **Clean Sweep (`10 / 10` `#1` Ranks) Across All `<=9B` Models**: Outperforms the same-size base model `Clef-flash` (`9.4B`, `74.96%`) by **`+5.59 pp`** (including **`+31.67 pp`** on `CLINC150+OOS` and **`+15.55 pp`** on `When2Call`), `Kev 9B` (`9B`, `76.51%`) by **`+4.04 pp`**, and `Laya` (`0.4B`, `58.09%`) by **`+22.46 pp`**.
|
| 78 |
-
- **Outperforms `3x`-Larger `27B`-Class Models**: Surpasses `Jev` (`27B`, `80.32%`) and `DiffusionGemma Jev` (`26B-A4B`, `69.14%`) in 10-benchmark Macro Mean, and beats the `3x`-larger flagship `Clef` (`27.4B`, `81.81%`) head-to-head on **5 out of 10 benchmarks** (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home appliances`, and `When2Call`, claiming outright **`#1` overall across all model sizes on 4 of them**).
|
| 79 |
|
| 80 |
-
|
| 81 |
|
| 82 |
-
|
| 83 |
|
| 84 |
-
|
| 85 |
|
| 86 |
-
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
-
##
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
| 95 |
-
|
|
| 96 |
-
|
|
| 97 |
-
| **
|
| 98 |
-
| **
|
| 99 |
-
| **
|
| 100 |
-
| **
|
| 101 |
-
|
|
| 102 |
-
| **Amazon ESCI**<br>*(Macro-F1)* | **57.40%**<br>*(#1 9B)* | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
|
| 103 |
-
| **PhishNChips**<br>*(Accuracy)* | **78.75%**<br>*(#1 9B)* | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
|
| 104 |
-
| **When2Call**<br>*(Macro-F1)* | **81.13%**<br>*(#1 9B)* | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
|
| 105 |
-
| **Macro Mean**<br>*(10 Tasks)* | **80.55%**<br>*(#2 · #1 9B)* | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
|
| 106 |
|
| 107 |
---
|
| 108 |
|
| 109 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
|
| 111 |
-
|
| 112 |
|
| 113 |
-
|
| 114 |
-
> - **Rank `#1 / 8` Overall on 7-Way Misbehaviour Category Track (`v2`)**: Achieves **`75.1%` Category Macro-F1** and **`85.8%` Category Accuracy (`567/661`)**—leading `#2 Cloudflare/clef` (`27.4B`, `56.3%` Macro-F1 / `73.7%` Acc) by **`+18.8 pp` Macro-F1 (`+12.1 pp` Acc)**, `#3 vllm-sr/d3-mini` (`51.2%`) by **`+23.9 pp`**, and base `Cloudflare/clef-flash` (`46.2%` / `57.6%` Acc) by **`+28.9 pp` Macro-F1 (`+28.2 pp` Acc)**.
|
| 115 |
-
> - **`100%` (`82 / 82`) Unsafe Trajectories Caught & `98.5%` AUROC on Binary Guardrail (`v2`)**: Blocks **`534 / 536` (`99.6%`)** malicious calls (`100%` recall on `harness_introspection` `272/272`, `credential_misuse` `15/15`, and `sandbox_escape` `4/4`; `99.2%` on `eval_gaming` `243/245`) with **`0` median leaked bad calls before catch**. At the uncalibrated default threshold (`P(block) >= 0.50`), Balanced Accuracy is **`81.4%`** (beating `27.4B Cloudflare/clef` at `80.8%`); calibrating the threshold (`P(block) >= 0.961`) reaches **`93.8%` Balanced Accuracy (`5.6%` False Block Rate)** while still catching **`82/82` (`100%`)** unsafe trajectories. On the [`v1` (`562`-call) official release](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1), `FastBrain-9B` ranked **`#1 / 9` on both Binary (`97.4%` Bal. Acc) and Category (`85.0%` Macro-F1)** tracks.
|
| 116 |
|
| 117 |
-
| Model
|
| 118 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 119 |
-
| **
|
| 120 |
-
| **
|
| 121 |
-
| **
|
| 122 |
-
|
|
| 123 |
-
|
|
| 124 |
-
|
|
| 125 |
-
|
|
| 126 |
-
|
|
| 127 |
-
|
|
| 128 |
-
|
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
|
| 143 |
-
|
|
| 144 |
-
|
|
| 145 |
-
|
|
| 146 |
-
|
|
| 147 |
-
|
|
| 148 |
-
|
|
| 149 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
|
| 151 |
---
|
| 152 |
|
| 153 |
-
##
|
| 154 |
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
---
|
| 166 |
|
| 167 |
-
##
|
| 168 |
|
| 169 |
-
###
|
| 170 |
|
| 171 |
-
`TypeScript/FastBrain-9B`
|
| 172 |
|
| 173 |
```python
|
| 174 |
-
from huggingface_hub import snapshot_download
|
| 175 |
import sys
|
| 176 |
import torch
|
|
|
|
| 177 |
|
| 178 |
model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
|
| 179 |
sys.path.insert(0, model_dir)
|
| 180 |
import joint_schema_model as clef
|
| 181 |
|
| 182 |
-
#
|
| 183 |
model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
|
| 184 |
|
| 185 |
-
#
|
| 186 |
# model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
|
| 187 |
|
| 188 |
response = clef.systemone(model, processor, {
|
|
@@ -192,13 +262,13 @@ response = clef.systemone(model, processor, {
|
|
| 192 |
"tools": ["get_balance", "transfer_funds"],
|
| 193 |
},
|
| 194 |
"questions": {
|
| 195 |
-
"
|
| 196 |
"type": "choice",
|
| 197 |
-
"
|
| 198 |
"criteria": {
|
| 199 |
-
"
|
| 200 |
-
"
|
| 201 |
-
"
|
| 202 |
},
|
| 203 |
},
|
| 204 |
},
|
|
@@ -206,55 +276,28 @@ response = clef.systemone(model, processor, {
|
|
| 206 |
print(response)
|
| 207 |
```
|
| 208 |
|
| 209 |
-
###
|
| 210 |
-
|
| 211 |
-
This repository provides official `llama.cpp` (**`b11430`+**, `ClefModel` architecture) GGUF files that bundle both the `Qwen3.5-9B` hybrid backbone and the `116M` `JointSchemaHead` (`dec.*` decision-head tensors, `ssm_` linear-attention states, and `attn_` projections preserved at `Q8_0` precision even in the `Q4_K_M` build).
|
| 212 |
|
| 213 |
> [!IMPORTANT]
|
| 214 |
-
> **
|
| 215 |
-
> 1.
|
| 216 |
-
> 2.
|
| 217 |
-
> In `llama-server`, `ClefModel` computes cross-attention over `<|question_*|>` and `<|option_*|>` spans within a single physical micro-batch (`ubatch`, default `512`). Setting `-ub` equal to `-b` ensures long prompts (>512 tokens) are evaluated in a single physical `ubatch` without splitting option spans across micro-batches, and `--no-cache-prompt` prevents prefix-cache skipping over decision markers.
|
| 218 |
-
|
| 219 |
-
#### Step 1: Download Prebuilt `llama-server` (`b11430`+) and GGUF Weights
|
| 220 |
|
| 221 |
```bash
|
| 222 |
-
# 1. Download prebuilt llama.cpp (b11430+)
|
| 223 |
-
# - Linux CUDA 13.x (Colab / Ubuntu): llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz
|
| 224 |
-
# - Linux CUDA 12.x: llama-b11430-bin-ubuntu-cuda-12.8-x64.tar.gz
|
| 225 |
-
# - macOS Apple Silicon (M1/M2/M3/M4): llama-b11430-bin-macos-arm64.tar.gz
|
| 226 |
curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
|
| 227 |
-
|
| 228 |
-
# 2. Download FastBrain-9B GGUF (choose Q8_0 [9.66 GB, lossless] or Q4_K_M [6.73 GB, 8GB–16GB GPUs])
|
| 229 |
pip install -U "huggingface_hub[hf_transfer]"
|
| 230 |
HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
|
| 231 |
-
```
|
| 232 |
-
|
| 233 |
-
#### Step 2: Launch `llama-server` on the `/v1/systemone` Endpoint
|
| 234 |
|
| 235 |
-
|
| 236 |
-
# On Linux / Google Colab, ensure CUDA driver libraries are in LD_LIBRARY_PATH:
|
| 237 |
export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
|
| 238 |
-
|
| 239 |
-
# Launch llama-server (24GB+ GPU / 16K context):
|
| 240 |
./llama-b11430/llama-server \
|
| 241 |
-m ./FastBrain-9B-Q8_0.gguf \
|
| 242 |
-ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
|
| 243 |
--port 8080
|
| 244 |
-
|
| 245 |
-
# Or on a 12GB–16GB GPU (e.g., Colab Free Tesla T4, 4K context with cuda-12.8):
|
| 246 |
-
# pip install nvidia-cuda-runtime-cu12 nvidia-cublas-cu12
|
| 247 |
-
# export LD_LIBRARY_PATH="$(pwd)/llama-b11430:$(python3 -c 'import glob;print(":".join(glob.glob("/usr/local/lib/python*/dist-packages/nvidia/*/lib")))'):/usr/lib64-nvidia:${LD_LIBRARY_PATH}"
|
| 248 |
-
# ./llama-b11430/llama-server \
|
| 249 |
-
# -m ./FastBrain-9B-Q4_K_M.gguf \
|
| 250 |
-
# -ngl 99 -c 4096 -b 4096 -ub 4096 -np 1 -fa on --no-cache-prompt \
|
| 251 |
-
# --port 8080
|
| 252 |
-
|
| 253 |
-
# Optional: For multimodal image/screenshot states, attach the vision projector:
|
| 254 |
-
# --mmproj ./mmproj-FastBrain-9B-Q8_0.gguf
|
| 255 |
```
|
| 256 |
|
| 257 |
-
|
| 258 |
|
| 259 |
```bash
|
| 260 |
curl http://127.0.0.1:8080/v1/systemone \
|
|
@@ -266,162 +309,59 @@ curl http://127.0.0.1:8080/v1/systemone \
|
|
| 266 |
"customer_message": "I was charged twice for my subscription last week and nobody has replied to my ticket."
|
| 267 |
},
|
| 268 |
"questions": {
|
| 269 |
-
"
|
| 270 |
"type": "choice",
|
| 271 |
-
"instructions": "
|
| 272 |
"criteria": {
|
| 273 |
"billing": "Payments, refunds, duplicate charges, or invoices",
|
| 274 |
-
"
|
| 275 |
-
"
|
| 276 |
}
|
| 277 |
},
|
| 278 |
-
"
|
| 279 |
"type": "noul",
|
| 280 |
-
"instructions": "
|
| 281 |
}
|
| 282 |
}
|
| 283 |
}'
|
| 284 |
```
|
| 285 |
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
```python
|
| 289 |
-
import requests
|
| 290 |
-
|
| 291 |
-
resp = requests.post(
|
| 292 |
-
"http://127.0.0.1:8080/v1/systemone",
|
| 293 |
-
json={
|
| 294 |
-
"model": "FastBrain-9B",
|
| 295 |
-
"state": {
|
| 296 |
-
"query": "Can you check the live wire transfer status for my corporate checking account right now?",
|
| 297 |
-
"available_tools": ["get_account_balance", "list_recent_cards"],
|
| 298 |
-
},
|
| 299 |
-
"questions": {
|
| 300 |
-
"action": {
|
| 301 |
-
"type": "choice",
|
| 302 |
-
"instructions": "Select the best assistant response.",
|
| 303 |
-
"criteria": {
|
| 304 |
-
"direct_answer": "Your wire transfer has been completed.",
|
| 305 |
-
"tool_call": "<tool_call>{\"name\": \"get_account_balance\", \"arguments\": {}}</tool_call>",
|
| 306 |
-
"clarify": "Could you please provide your corporate account ID and wire reference number?",
|
| 307 |
-
"abstain": "I cannot check wire transfer status because no wire-transfer tool is available.",
|
| 308 |
-
},
|
| 309 |
-
}
|
| 310 |
-
},
|
| 311 |
-
},
|
| 312 |
-
timeout=30,
|
| 313 |
-
).json()
|
| 314 |
-
|
| 315 |
-
print(resp["choices"]["action"]) # Calibrated probability distribution over all options
|
| 316 |
-
print(resp["usage"]) # {'input_tokens': ..., 'output_tokens': 0, 'total_tokens': ...}
|
| 317 |
-
```
|
| 318 |
-
|
| 319 |
-
### 3. SGLang (`lmsysorg/sglang:dev-clef`) via `/v1/systemone`
|
| 320 |
-
|
| 321 |
-
`TypeScript/FastBrain-9B` is natively supported by SGLang's `JointSchemaPooler` (`lmsysorg/sglang:dev-clef`, `sgl-project/sglang` commit `bd2d73daa5af`), which executes the `9B` backbone in uncached raw-K/V FlashAttention embedding mode and scores all questions and options across concurrent requests using vectorized prefix-sum span pooling (`torch.cumsum`) and packed variable-length FlashAttention (`cu_seqlens`):
|
| 322 |
|
| 323 |
```bash
|
| 324 |
-
docker run --gpus all \
|
| 325 |
-
--
|
| 326 |
-
-p 30000:30000 \
|
| 327 |
-
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
| 328 |
-
--ipc=host \
|
| 329 |
lmsysorg/sglang:dev-clef \
|
| 330 |
-
sglang serve
|
| 331 |
-
--model-path TypeScript/FastBrain-9B \
|
| 332 |
-
--host 0.0.0.0 \
|
| 333 |
-
--port 30000
|
| 334 |
-
```
|
| 335 |
-
|
| 336 |
-
Query SGLang's `/v1/systemone` endpoint directly (and optionally wrap with `joint_schema_model.apply_schema_aware_calibration(req, resp["answers"])` on the client side):
|
| 337 |
-
|
| 338 |
-
```bash
|
| 339 |
-
curl http://localhost:30000/v1/systemone \
|
| 340 |
-
-H 'Content-Type: application/json' \
|
| 341 |
-
-d '{"model":"TypeScript/FastBrain-9B","state":"My Stripe integration keeps failing. Please help ASAP.","questions":{"urgency":{"type":"noul","instructions":"Does this message express urgency?"}}}'
|
| 342 |
```
|
| 343 |
|
| 344 |
---
|
| 345 |
|
| 346 |
-
##
|
| 347 |
-
|
| 348 |
-
Measured head-to-head on the **same NVIDIA RTX PRO 6000 Blackwell Server Edition (G4)** GPU across five decision workloads (`CLINC150+OOS`, `When2Call`, `PhishNChips`, `LocalLLaMA/typed-decisions`, and `safeagents-judge-bench`):
|
| 349 |
-
|
| 350 |
-
| Workload / Metric | **SGLang `dev-clef`**<br>*(BF16 · JointSchemaPooler)* | **`llama.cpp b11430`**<br>*(GGUF Q8_0 · ClefModel)* | **`llama.cpp b11430`**<br>*(GGUF Q4_K_M · ClefModel)* | **PyTorch Native**<br>*(BF16 · `joint_schema_model.py`)* |
|
| 351 |
-
| :--- | :---: | :---: | :---: | :---: |
|
| 352 |
-
| **Model Load / Cold Start** | 11.11 s | **2.56 s** | **2.00 s** | 11.11 s |
|
| 353 |
-
| **GPU VRAM (16K ctx)** | 18.18 GiB *(weights)* / ~20.5 GiB | **12.55 GiB** | **10.30 GiB** | 20.3–24.7 GiB |
|
| 354 |
-
| **Runs on 16GB T4 GPU?** | No *(BF16 only, TP=1)* | **Yes (`9.03 GiB` @ 4K)** | **Yes (`6.78 GiB` @ 4K)** | **Yes (`load_in_4bit`: `6.8 GiB`)** |
|
| 355 |
-
| **Top-1 Agreement vs. BF16** | **100.00%** *(max diff 0.0078)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* | **100.00% *(Ref)*** |
|
| 356 |
-
| **1. Short Routing** *(293 tok, K=2)* | **30.01 ms** *(9,763 tok/s)* | 32.82 ms *(8,929 tok/s)* | 31.69 ms *(9,245 tok/s)* | 43.23 ms *(6,778 tok/s)* |
|
| 357 |
-
| **2. Typed Decisions** *(736 tok, K=5)* | **50.29 ms** *(14,635 tok/s)* | 62.40 ms *(11,795 tok/s)* | 61.80 ms *(11,909 tok/s)* | 63.58 ms *(11,576 tok/s)* |
|
| 358 |
-
| **3. When2Call** *(1,068 tok, K=1)* | **66.06 ms** *(16,167 tok/s)* | 87.78 ms *(12,167 tok/s)* | 89.34 ms *(11,955 tok/s)* | 77.77 ms *(13,733 tok/s)* |
|
| 359 |
-
| **4. PhishNChips** *(1,303 tok, K=9)* | **75.86 ms** *(17,176 tok/s)* | 108.55 ms *(12,004 tok/s)* | 109.97 ms *(11,849 tok/s)* | 90.32 ms *(14,426 tok/s)* |
|
| 360 |
-
| **5. SafeAgents Judge** *(2,217 tok, K=2)* | **125.85 ms** *(17,616 tok/s)* | 181.70 ms *(12,201 tok/s)* | 183.50 ms *(12,082 tok/s)* | 138.06 ms *(16,058 tok/s)* |
|
| 361 |
-
| **Batch `B=4` Typed Decisions** *(2,944 tok, 20 dec)* | **152.39 ms** *(19,319 tok/s · 131.2 dec/s)* | 248.50 ms *(11,847 tok/s · 80.5 dec/s)* | 245.20 ms *(12,006 tok/s · 81.6 dec/s)* | 175.85 ms *(16,742 tok/s · 113.7 dec/s)* |
|
| 362 |
-
|
| 363 |
-
---
|
| 364 |
-
|
| 365 |
-
## GGUF Quantization Accuracy Retention & Speed Benchmarks
|
| 366 |
-
|
| 367 |
-
### 1. Decision Index 10-Benchmark Test-Set Accuracy Retention (`BF16` vs. `Q8_0` vs. `Q4_K_M`)
|
| 368 |
|
| 369 |
-
|
| 370 |
-
|
| 371 |
-
| Benchmark / Metric | PyTorch bfloat16<br>*(18.82 GB)* | GGUF Q8_0<br>*(9.66 GB)* | GGUF Q4_K_M<br>*(6.73 GB · Q8 Head)* |
|
| 372 |
-
| :--- | :---: | :---: | :---: |
|
| 373 |
-
| **Top-1 Agreement vs. BF16** *(1,473 q)* | 100.00% *(Ref)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* |
|
| 374 |
-
| **Probability MAE (\|p_q - p_bf16\|)** | 0.00000 | **0.00203** *(0.20%)* | **0.00387** *(0.39%)* |
|
| 375 |
-
| **10-Task Mean (Single-Ckpt Raw)** | **82.37%** | **82.34%** *(-0.03 pp)* | **82.56%** *(+0.19 pp)* |
|
| 376 |
-
| **10-Task Mean (Calibrated)** | **83.68%** | **83.65%** *(-0.03 pp)* | **83.48%** *(-0.20 pp)* |
|
| 377 |
-
| ├─ **1. BFCL** *(AST Tool Selection)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
|
| 378 |
-
| ├─ **2. ToolRet** *(20-Tool Reranking)* | 73.15% | **73.70%** *(+0.55)* | **75.60%** *(+2.45)* |
|
| 379 |
-
| ├─ **3. API-Bank** *(Stateful Tool Call)* | 95.00% | **95.00%** *(0.00)* | **95.00%** *(0.00)* |
|
| 380 |
-
| ├─ **4. BANKING77** *(77-Way Intent)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
|
| 381 |
-
| ├─ **5. CLINC150+OOS** *(Intent + OOS)* | 93.94% | **93.94%** *(0.00)* | **93.94%** *(0.00)* |
|
| 382 |
-
| ├─ **6. Home Appliances** *(Control)* | 90.00% | **90.00%** *(0.00)* | **90.00%** *(0.00)* |
|
| 383 |
-
| ├─ **7. BRIGHT** *(Reasoning Retrieval)* | 41.93% | **41.00%** *(-0.93)* | **40.31%** *(-1.62)* |
|
| 384 |
-
| ├─ **8. Amazon ESCI** *(Relevance)* | 73.89% | **73.89%** *(0.00)* | **75.79%** *(+1.90)* |
|
| 385 |
-
| ├─ **9. PhishNChips** *(9-Probe Sec)* | 85.00% | **85.00%** *(0.00)* | **85.00%** *(0.00)* |
|
| 386 |
-
| └─ **10. When2Call** *(4-Way Gov)* | 75.00% *(70.83%)* | **75.00%** *(0.00)* | **72.92%** *(-2.08)* |
|
| 387 |
-
|
| 388 |
-
### 2. Measured Hardware Footprint & Inference Speed (`llama-server b11430`)
|
| 389 |
-
|
| 390 |
-
Because `FastBrain-9B` scores all schema questions and candidate options in a **single non-autoregressive prefill forward pass (`output_tokens = 0`)**, `llama-server` achieves **8,900–12,100+ input tokens/sec** (`31–109 ms` end-to-end HTTP latency) on server GPUs:
|
| 391 |
-
|
| 392 |
-
| Workload / Metric | Q8_0.gguf<br>*(PRO 6000)* | Q4_K_M.gguf<br>*(PRO 6000)* | Q8_0.gguf<br>*(Tesla T4 16G)* | Q4_K_M.gguf<br>*(Tesla T4 16G)* | PyTorch BF16<br>*(PRO 6000)* |
|
| 393 |
-
| :--- | :---: | :---: | :---: | :---: | :---: |
|
| 394 |
-
| **Model Load Time** | **2.56 s** | **2.00 s** | **5.43 s** | **5.10 s** | ~11.5 s |
|
| 395 |
-
| **Peak GPU VRAM** | **12.55 GiB**<br>*(16K ctx)* | **10.30 GiB**<br>*(16K ctx)* | **9.03 GiB**<br>*(4K ctx)* | **6.78 GiB**<br>*(4K ctx)* | 20.3–24.7 GiB |
|
| 396 |
-
| **Intent + OOS**<br>*(293 tok, K=2)* | **32.82 ms**<br>*(8,928 tok/s)* | **31.69 ms**<br>*(9,245 tok/s)* | **349.32 ms**<br>*(839 tok/s)* | **342.98 ms**<br>*(854 tok/s)* | ~35.0 ms |
|
| 397 |
-
| **When2Call**<br>*(1,068 tok, K=1)* | **87.78 ms**<br>*(12,167 tok/s)* | **89.34 ms**<br>*(11,955 tok/s)* | **1,149.8 ms**<br>*(929 tok/s)* | **1,154.3 ms**<br>*(925 tok/s)* | ~92.0 ms |
|
| 398 |
-
| **PhishNChips**<br>*(1,303 tok, K=9)* | **108.55 ms**<br>*(12,004 tok/s)* | **109.97 ms**<br>*(11,849 tok/s)* | **1,430.8 ms**<br>*(911 tok/s)* | **1,440.5 ms**<br>*(905 tok/s)* | ~115.0 ms |
|
| 399 |
-
|
| 400 |
-
| Format / Precision Tier | Weight Footprint | Peak VRAM (`4K`–`16K` ctx) | Minimum Recommended Hardware |
|
| 401 |
-
| :--- | :---: | :---: | :--- |
|
| 402 |
-
| **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit + `Q8_0` Head)* | `6.73 GB` | `~6.8 – 10.3 GiB` | **8GB–12GB GPU / Tesla T4 (16GB) / Mac M1–M4** |
|
| 403 |
-
| **PyTorch `load_in_4bit=True`** *(NF4 + `fp16` Head)* | `~5.5 GB` | `~6.2 – 6.8 GiB` | **Tesla T4 (16GB) / RTX 3060–4060 (8GB+)** |
|
| 404 |
-
| **`FastBrain-9B-Q8_0.gguf`** *(Full 8-bit GGUF)* | `9.66 GB` | `~9.0 – 12.6 GiB` | **12GB–16GB GPU / Tesla T4 (16GB) / Mac 16GB+** |
|
| 405 |
-
| **PyTorch `load_in_8bit=True`** *(INT8 + `bf16` Head)* | `~9.6 GB` | `~11.2 GiB` | **Tesla T4 (16GB) / RTX 4070 Ti (12GB+)** |
|
| 406 |
-
| **Native `bfloat16` (`safetensors` / `BF16.gguf`)** | `18.82 GB` (`18.16 GB`) | `20.3 – 24.7 GiB` | **NVIDIA L4 (24GB) / A100 / RTX 3090–4090 / PRO 6000** |
|
| 407 |
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
## Repository Structure
|
| 411 |
-
|
| 412 |
-
| File | Size | Description |
|
| 413 |
| :--- | :---: | :--- |
|
| 414 |
-
| `model-00001-of-00004.safetensors` .. `00004` |
|
| 415 |
-
| `joint_head.safetensors`
|
| 416 |
-
| `FastBrain-9B-Q4_K_M.gguf` |
|
| 417 |
-
| `FastBrain-9B-Q8_0.gguf`
|
| 418 |
-
| `FastBrain-9B-
|
| 419 |
-
| `
|
| 420 |
-
| `
|
| 421 |
-
| `eval_submissions/
|
| 422 |
-
| `
|
| 423 |
-
| `
|
| 424 |
-
|
| 425 |
-
|
| 426 |
-
|
| 427 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
# FastBrain-9B
|
| 30 |
|
| 31 |
+
**A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions**
|
| 32 |
|
| 33 |
+
[-3b82f6?style=flat-square)](#2-model-architecture-and-training-objective)
|
| 34 |
+
[-10b981?style=flat-square)](#2-model-architecture-and-training-objective)
|
| 35 |
+
[-ec4899?style=flat-square)](#31-agent-tool-call-guardrail-evaluation-safeagents-judge-bench)
|
| 36 |
+
[_%C2%B7_50ms-8b5cf6?style=flat-square)](#32-multi-workflow-enterprise-decisions-localllamatyped-decisions)
|
| 37 |
+
[-f59e0b?style=flat-square)](#33-multi-domain-decision-suite-cloudflare-decision-index-v021)
|
| 38 |
[](./LICENSE)
|
| 39 |
|
| 40 |
+
**English** | **[简体中文](./README_zh.md)**
|
| 41 |
|
| 42 |
</div>
|
| 43 |
|
| 44 |
+
---
|
| 45 |
+
|
| 46 |
+
## Abstract
|
| 47 |
|
| 48 |
+
Autoregressive language models incur sequential decoding overhead and token-level probability miscalibration when deployed on closed-schema classification, tool routing, and safety verification tasks. We present **FastBrain-9B**, a 9.41B-parameter non-autoregressive structured decision model built on a 32-layer hybrid Gated DeltaNet and full-attention backbone (`Qwen3.5-9B`, initialized from `Cloudflare/clef-flash`) coupled with a 116M-parameter cross-attention decision head (`JointSchemaHead`). Given an arbitrary JSON or multimodal state and a set of typed decision specifications (`choice`, binary `noul`, and ordinal `score`), FastBrain-9B evaluates all questions and candidate options simultaneously in a single prefill pass (`output_tokens = 0`). We adapt 104 linear-attention and self-attention projections jointly with the decision head under a composite ordinal-KL, cumulative distribution (ranked probability score), and expected-value regression objective, and merge all parameter updates directly into the base tensors. Across three public evaluation suites encompassing 109 model entries, FastBrain-9B ranks **1st of 8** on the `safeagents-judge-bench` (v2) 7-way misuse classification track (`0.751` macro-F1, `0.858` accuracy; +18.8 percentage points over 27.4B `Cloudflare/clef`) while intercepting `82/82` unsafe agent trajectories (`0.985` AUROC); ranks **7th of 43** zero-shot overall (**5th of 36** open-weight, **2nd of 19** in the $\le$9B open-weight class) on `LocalLLaMA/typed-decisions` at `50 ms` median latency (66.7$\times$ faster than 31B `Quyet-1.0-Large`), and **12th-tied (14th listed of 51)** under prior-only calibration without weight updates (`0.797` accuracy, `0.0816` KL); and achieves an **80.55%** 10-task macro mean on Cloudflare Decision Index (v0.2.1), ranking **2nd of 7** overall, **1st on 10 of 10** tasks among $\le$9B models, and outperforming 27.4B `Cloudflare/clef` on 5 of 10 tasks.
|
| 49 |
|
| 50 |
+
---
|
| 51 |
|
| 52 |
+
## 1. Overview and Motivation
|
| 53 |
|
| 54 |
+
Production agentic pipelines invoke structured decisions at high frequency: selecting among candidate tool schemas, verifying whether an execution trace violates sandbox policy, routing customer or security tickets, and scoring rubric criteria. Solving these tasks via autoregressive token generation introduces two structural inefficiencies. First, latency scales with output sequence length and formatting tokens rather than decision complexity. Second, surface-form token probabilities fragment probability mass across equivalent verbalizations, degrading calibration metrics such as Kullback-Leibler (KL) divergence and Brier score.
|
| 55 |
+
|
| 56 |
+
Single-pass decision models (`ClefModel` / System-1 architecture) address both issues by appending delimiter-bounded question and option spans directly to the input context and reading out option logits from final hidden states in one prefill pass. Existing 9B-scale checkpoints, however, trail 27B–35B models substantially on fine-grained multi-class discrimination, ordinal rubric estimation, and open-set abstention (e.g., `Cloudflare/clef-flash` scores `46.2%` macro-F1 on `safeagents-judge-bench` v2 and `66.77%` on `CLINC150+OOS`). FastBrain-9B closes and reverses much of this scaling gap through full-depth projection adaptation, ordinal distribution distillation, and compile-time schema-topology calibration, while preserving sub-100 ms single-GPU latency and lossless 8-bit / high-retention 4-bit GGUF quantization.
|
|
|
|
| 57 |
|
| 58 |
---
|
| 59 |
|
| 60 |
+
## 2. Model Architecture and Training Objective
|
| 61 |
|
| 62 |
+
### 2.1 Hybrid State-Space and Attention Backbone
|
| 63 |
|
| 64 |
+
FastBrain-9B uses a 32-layer hybrid sequence backbone (`9.30B` parameters, `bfloat16`, maximum context length `262,144` tokens) interleaving 24 linear-attention Gated DeltaNet blocks with 8 full-attention blocks. During training, 104 projection matrices across all 32 blocks—specifically `in_proj_qkv`, `in_proj_z`, and `out_proj` in linear-attention layers, alongside `q_proj`, `k_proj`, `v_proj`, and `o_proj` in full-attention layers—are updated jointly with the decision head. For distribution, all low-rank and projection updates are folded in-place into `model-00001-of-00004.safetensors` through `model-00004-of-00004.safetensors`, requiring zero adapter overhead at inference time.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
+
### 2.2 Joint Schema Decision Head and Vectorized CUDA Kernel
|
| 67 |
|
| 68 |
+
The `JointSchemaHead` (`116M` parameters, stored in `joint_head.safetensors`) pools token representations bounded by `<|question_start|>`/`<|question_end|>` and `<|option_start|>`/`<|option_end|>` markers from the final backbone hidden states $H \in \mathbb{R}^{T \times d}$, applies cross-attention over question-option pairs, and projects directly to scalar logits for three decision primitives:
|
| 69 |
+
1. **`choice`** ($K$-way categorical distribution): $\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}$.
|
| 70 |
+
2. **`noul`** (binary verification): $\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})$.
|
| 71 |
+
3. **`score`** (ordinal rubric over levels $v_1 < \dots < v_K$): categorical probabilities $\hat{p} \in \Delta^{K-1}$ together with expected value $\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k$.
|
| 72 |
|
| 73 |
+
In `joint_schema_model.py`, the reference Python span-extraction loop and padded SDPA mask are replaced on CUDA devices by `_forward_fast_cuda`: span means are computed in $\mathcal{O}(1)$ kernel launches via prefix sums (`torch.cumsum`) over the schema suffix, and cross-attention is executed via packed variable-length FlashAttention (`torch.ops.aten._flash_attention_forward` parameterized by `cu_seqlens`). On NVIDIA RTX PRO 6000 Blackwell, this reduces isolated decision-head latency from `3.27 ms` to `1.72 ms` (**1.90$\times$ speedup**) with exact numerical equivalence (`max_logit_diff = 0.00000`).
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
### 2.3 Ordinal-KL Distillation and Schema-Topology Calibration
|
| 76 |
|
| 77 |
+
Standard cross-entropy or flat KL divergence treats ordinal rubric levels as unordered classes, penalizing adjacent-level errors identically to extreme errors. FastBrain-9B is trained on `TypeScript/FastBrain-Instruct` under a composite loss combining soft-target KL divergence, discrete ranked probability score (squared $L_2$ distance between predicted and target cumulative distribution functions $\hat{F}_k = \sum_{j \le k} \hat{p}_j$), and expected-score mean squared error on `score` primitives:
|
| 78 |
|
| 79 |
+
$$\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.$$
|
| 80 |
|
| 81 |
+
At inference time, `joint_schema_model.py` optionally applies compile-time schema-topology calibration (`apply_schema_aware_calibration`), which inspects candidate option structures (e.g., `<tool_call>` invocations, clarification prompts, and open-set abstention clauses) to adjust decision margins against systematic tool over-triggering and out-of-scope false acceptance.
|
| 82 |
|
| 83 |
+
---
|
| 84 |
|
| 85 |
+
## 3. Empirical Evaluation
|
| 86 |
|
| 87 |
+
**Table 1: Summary of official public benchmark standings (verified across 109 model entries as of November 2026).**
|
| 88 |
+
|
| 89 |
+
| Benchmark & Evaluation Track | Evaluated Entries | FastBrain-9B Metric | Official Standing | Reference Baselines (`Params`, Metric) |
|
| 90 |
+
| :--- | :---: | :---: | :---: | :--- |
|
| 91 |
+
| [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 7-Way Category ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 decision models (+2 baselines) | **0.751** Macro-F1 / **0.858** Acc | **1 / 8** | `Cloudflare/clef` (27.4B, `0.563`), `vllm-sr/d3-mini` (1.8B, `0.512`), `Cloudflare/clef-flash` (9.4B, `0.462`), `clef-omni` (30B, `0.437`) |
|
| 92 |
+
| [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — Binary Guardrail ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 decision models (+2 baselines) | **0.985** AUROC / **82/82** Traj.<br>(`0.814` raw; **0.938** calibrated) | **3 / 8** AUROC<br>**1 / 8** (tied) Traj. | `Cloudflare/clef` (27.4B, `0.808` raw bal. acc), `clef-flash` (`0.967` AUROC, `81/82`), `clef-omni` (`0.946` AUROC, `81/82`), `d1-3B` (`68/82`) |
|
| 93 |
+
| [`safeagents-judge-bench` (v1)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1) — Binary & Category ([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 decision models (+2 baselines) | **0.974** Bal. Acc / **0.850** Cat. F1 | **1 / 9** (both tracks) | `Cloudflare/clef-omni` (30B, `0.845` bal. acc / `0.327` F1), `Cloudflare/clef` (27.4B, `0.718` / `0.425`), `clef-flash` (`0.721` / `0.345`) |
|
| 94 |
+
| [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — Zero-Shot ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) | 43 total (36 open, 19 $\le$9B open) | **0.7525** Acc (`0.7625` tied-top)<br>`0.1508` KL · `0.0795` Brier · **50 ms** | **7 / 43** overall<br>**5 / 36** open · **2 / 19** $\le$9B | `vllm-sr/d3` (27B, `0.749`), `Elephant 1` (9B, `0.748`, `352 ms`), `Darwin-27B` (`0.743`), `Cloudflare/clef` (27B, `0.728`), `clef-flash` (`0.710`) |
|
| 95 |
+
| [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — Fitted on `train` ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) | 51 specialist & calibrated models | **0.7970** Acc (`0.8070` tied-top)<br>**0.0816** KL · **0.0437** Brier · **50 ms** | **12-tied** (14 / 51 listed)<br>**1st** frozen-weight $\le$9B & KL | Frozen weights (`no weights changed`); surpasses `Rev-gen1-midi` (25.6B, `0.7945`), `OpenDecider-small-td` (4B LoRA, `0.792`), `OpenDecider-medium-td` (30B LoRA, `0.788`) |
|
| 96 |
+
| Cloudflare Decision Index (`v0.2.1`) — 10-Task Suite | 7 models (0.4B–27.4B) | **80.55%** Macro Mean | **2 / 7** overall<br>**1st** on **10 / 10** ($\le$9B) | `Jev` (27B, `80.32%`), `Kev 9B` (`76.51%`), `Clef-flash` (9.4B, `74.96%`), `DiffGemma` (26B, `69.14%`); exceeds `27.4B Clef` on 5 / 10 tasks |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
|
| 98 |
---
|
| 99 |
|
| 100 |
+
### 3.1 Agent Tool-Call Guardrail Evaluation (`safeagents-judge-bench`)
|
| 101 |
+
|
| 102 |
+
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) evaluates whether a System-1 decision model can audit suspicious tool calls flagged by a static scanner (`atif-scan`) during real coding-agent executions on Terminal-Bench 2.1, OpenThoughts-TBLite, and Terminal-Bench 4. Version 2 (`v2`) comprises $N = 661$ tool calls across 55 tasks (`536` malicious `block` calls spanning `82` misbehaving trajectories and `125` legitimate `allow` calls). Each call is issued as a single zero-shot System-1 request containing one binary `noul` question (`block`) and one 7-way `choice` question (`legit`, `eval_gaming`, `harness_introspection`, `credential_misuse`, `exfiltration`, `sandbox_escape`, `other_overreach`). All v2 results in Table 2 were executed independently on NVIDIA A100-SXM4-80GB via Hugging Face Jobs ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)).
|
| 103 |
+
|
| 104 |
+
On the 7-way category classification track, FastBrain-9B achieves `0.751` macro-F1 and `0.858` accuracy (`567/661`), outperforming the 27.4B `Cloudflare/clef` model (`0.563` macro-F1, `0.737` accuracy) by `+18.8` percentage points in macro-F1 and `+12.1` points in accuracy, and improving upon its 9.4B base model `Cloudflare/clef-flash` (`0.462` macro-F1, `0.576` accuracy) by `+28.9` and `+28.2` points, respectively. Per-category F1 reaches `0.945` on `harness_introspection` ($n=272$, recall `0.978`), `0.828` on `eval_gaming` ($n=245$, recall `0.853`), and `0.761` on `legit` ($n=125$, precision `0.975`).
|
| 105 |
|
| 106 |
+
On the binary guardrail track, FastBrain-9B attains `0.985` AUROC and `0.996` malicious-call recall (`534/536`), intercepting all `82/82` unsafe trajectories (`median_bad_calls_before_catch = 0.0`), with `1.000` recall on `harness_introspection` (`272/272`), `credential_misuse` (`15/15`), and `sandbox_escape` (`4/4`). Under the uncalibrated default decision rule $P(\text{block}) \ge 0.50$, balanced accuracy is `0.814` (false-block rate `0.368` across the 125 `allow` calls, compared with `0.808` balanced accuracy and `0.384` false-block rate for 27.4B `Cloudflare/clef`). Because rank-ordering discrimination is preserved (`0.985` AUROC), setting the operating threshold to $P(\text{block}) \ge 0.961$ yields `0.938` balanced accuracy at a `0.056` (`7/125`) false-block rate while maintaining `82/82` trajectory interception. Table 2 also includes the `v1` release ($N = 562$ calls, `78` trajectories), where FastBrain-9B ranked 1st of 9 on both tracks.
|
| 107 |
|
| 108 |
+
**Table 2: Zero-shot evaluation on `harness-hub/safeagents-judge-bench` (`v2`, $N = 661$ calls, 82 unsafe trajectories; and `v1` reference, $N = 562$).**
|
|
|
|
|
|
|
| 109 |
|
| 110 |
+
| Model | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Bal. Acc ↑ | Recall ↑ | False Block ↓ | Traj. Caught ↑ | p50 Latency (Hardware) |
|
| 111 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 112 |
+
| **TypeScript/FastBrain-9B** (`v2`, default $t = 0.50$) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 113 |
+
| **TypeScript/FastBrain-9B** (`v2`, calibrated $t = 0.961$) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 114 |
+
| **TypeScript/FastBrain-9B** (`v1` official, $N = 562$) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 115 |
+
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 116 |
+
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 117 |
+
| Cloudflare/clef-flash (base) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
| 118 |
+
| Cloudflare/clef-omni | 30B-A3B | 0.437 | 0.564 | 0.946 | 0.877 | 0.914 | 0.160 | 81 / 82 | 285 ms (A100) |
|
| 119 |
+
| knowledgator/gliclass-v4-edge | 0.4B | 0.373 | 0.528 | 0.753 | 0.676 | 0.696 | 0.344 | 68 / 82 | 101 ms (L4) |
|
| 120 |
+
| fastino/GLiNER2.5-Decide | 0.5B | 0.355 | 0.620 | 0.517 | 0.576 | 0.151 | **0.000** | 17 / 82 | 117 ms (M5 Pro CPU) |
|
| 121 |
+
| LiquidAI/d1-3B | 3.0B | 0.328 | 0.457 | 0.946 | 0.836 | 0.681 | 0.008 | 68 / 82 | **35 ms** (A100) |
|
| 122 |
+
|
| 123 |
+
---
|
| 124 |
+
|
| 125 |
+
### 3.2 Multi-Workflow Enterprise Decisions (`LocalLLaMA/typed-decisions`)
|
| 126 |
+
|
| 127 |
+
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) tests multi-primitive structured judgment across four enterprise workflows (`agent_trace_observability`, `customer_service`, `invoice_processing`, and `security_incidents`), comprising 400 held-out test cases and 2,000 individual decisions (`noul`, `choice`, and `score`) scored against teacher-panel soft label distributions. Two scoring conventions exist in the benchmark repository: `bench/score_with_jobs.py` resolves exact ties in the gold distribution by selecting the first index (`g.index(max(g))`), whereas `scripts/score_model.py` credits any tied maximum across the 20 tied test items; KL divergence and Brier score are identical under both scripts. We report both accuracy figures (`strict / tied-top`) alongside prediction traces in [`eval_submissions/`](./eval_submissions/).
|
| 128 |
+
|
| 129 |
+
On the official zero-shot leaderboard ($N = 43$ models), FastBrain-9B scores `0.7525` (`0.7625` tied-top) accuracy, `0.1508` KL divergence, `0.0795` Brier score, and `0.085` ECE at `50 ms` p50 latency (RTX PRO 6000 via SGLang). It places **7th of 43** overall, **5th of 36** among open-weight models, **2nd of 19** among $\le$9B open-weight models, and **1st** among all `Clef`-architecture models, improving over 9.4B `Cloudflare/clef-flash` (`0.7100` accuracy, `0.2093` KL) by `+4.25` percentage points and over 27.4B `Cloudflare/clef` (`0.7275` accuracy, `0.1907` KL) by `+2.50` points. Across all 28 models exceeding `0.700` zero-shot accuracy, FastBrain-9B exhibits the lowest p50 latency (`50 ms`, compared with `118 ms` for 9B `Amos 1 medium`, `357 ms` for 27B `Amos 1 large`, `526 ms` for `meraGPT Decider 1`, and `3,335 ms` for 31B `Quyet-1.0-Large`). Evaluating the uncalibrated ordinal-KL distilled head directly yields `0.7735` raw accuracy (`0.1039` KL, `0.0582` Brier).
|
| 130 |
+
|
| 131 |
+
On the separate `train`-fitted leaderboard ($N = 51$ models), fitting only scalar per-question log-priors and temperature parameters on the 1,200 `train` cases—with all backbone and head weights frozen (`no weights changed`)—achieves `0.7970` (`0.8070` tied-top) accuracy, `0.0816` KL divergence, and `0.0437` Brier score, ranking **12th-tied (14th listed of 51)** overall and **1st in KL calibration** among frozen-weight models.
|
| 132 |
+
|
| 133 |
+
**Table 3: Performance on `LocalLLaMA/typed-decisions` ($N = 400$ cases, 2,000 decisions). Values separated by `/` denote `score_with_jobs.py` / `score_model.py` tie handling.**
|
| 134 |
+
|
| 135 |
+
| Model | Params | Protocol | Overall Acc ↑ | KL ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | Cust. Svc ↑ | Invoice ↑ | Security ↑ | p50 Latency |
|
| 136 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 137 |
+
| **FastBrain-9B** (prior-fitted, frozen weights) | 9.4B | Fitted | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
|
| 138 |
+
| **FastBrain-9B** (ordinal-KL distilled head) | 9.4B | Zero-shot | 77.35% | 0.1039 | 0.0582 | 84.8% | 74.8% | 73.8% | 71.2% | 80.4% | 82.6% | 75.2% | **50 ms** |
|
| 139 |
+
| **FastBrain-9B** (official PR #58) | 9.4B | Zero-shot | 75.25% / 76.25% | 0.1508 | 0.0795 | 83.2% / 84.3% | 74.7% / 76.0% | 69.8% / 70.4% | 68.2% / 69.8% | 77.6% / 78.8% | 82.0% | 73.2% / 74.4% | **50 ms** |
|
| 140 |
+
| Amos 1 (medium) | 9.0B | Zero-shot | 76.45% | 0.1510 | 0.0800 | — | — | — | — | — | — | — | 118 ms |
|
| 141 |
+
| Amos 1 (large) | 27.0B | Zero-shot | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
|
| 142 |
+
| pplx-decider-v1.1-27b | 27.0B | Zero-shot | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
|
| 143 |
+
| vllm-sr/d3 | 27.0B | Zero-shot | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
|
| 144 |
+
| Baseten Elephant 1 | 9.0B | Zero-shot | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
|
| 145 |
+
| Darwin-27B-ZTC | 27.0B | Zero-shot | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
|
| 146 |
+
| Cloudflare/clef | 27.4B | Zero-shot | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
|
| 147 |
+
| vllm-sr/d3-flash | 9.0B | Zero-shot | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
|
| 148 |
+
| Cloudflare/clef-flash (base) | 9.4B | Zero-shot | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
|
| 149 |
+
| Cloudflare/clef-omni | 35B-A3B | Zero-shot | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
|
| 150 |
+
|
| 151 |
+
---
|
| 152 |
+
|
| 153 |
+
### 3.3 Multi-Domain Decision Suite (Cloudflare Decision Index `v0.2.1`)
|
| 154 |
+
|
| 155 |
+
On the 10-benchmark **Cloudflare Decision Index (`v0.2.1`)** suite ($N = 22,427$ evaluation requests spanning AST tool calling, 20-candidate tool reranking, stateful API invocation, fine-grained intent classification, out-of-scope rejection, reasoning-intensive retrieval, e-commerce relevance, phishing detection, and tool-use governance), FastBrain-9B attains an **80.55%** unweighted macro mean (Table 4, Figures 1–3).
|
| 156 |
+
|
| 157 |
+
Across the $\le$9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B), FastBrain-9B ranks **1st on all 10 benchmarks**, improving the 10-task macro mean over `Clef-flash` (`74.96%`) by `+5.59` percentage points and over `Kev 9B` (`76.51%`) by `+4.04` points. The largest gains occur on calibration-sensitive abstention and governance tasks: `+31.67` points on `CLINC150+OOS` (`98.44%` vs. `66.77%`), `+15.55` points on `When2Call` (`81.13%` vs. `65.58%`), and `+6.80` points on `PhishNChips` (`78.75%` vs. `71.95%`). Across all model scales up to 27.4B, FastBrain-9B ranks **2nd of 7** overall, surpassing `27B Jev` (`80.32%`) and `26B-A4B DiffusionGemma Jev` (`69.14%`), and exceeding `27.4B Clef` (`81.81%`) on 5 of 10 individual benchmarks (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home Appliances`, and `When2Call`).
|
| 158 |
+
|
| 159 |
+

|
| 160 |
+
*Figure 1. Aggregate 10-benchmark macro mean versus parameter count on Cloudflare Decision Index (v0.2.1, 22,427 requests).*
|
| 161 |
+
|
| 162 |
+

|
| 163 |
+
*Figure 2. Per-benchmark performance matrix across 0.4B–27.4B decision models on Cloudflare Decision Index (v0.2.1).*
|
| 164 |
+
|
| 165 |
+

|
| 166 |
+
*Figure 3. Confidence calibration, open-set out-of-scope rejection (`CLINC150+OOS`), and abstention margin analysis (`When2Call`).*
|
| 167 |
+
|
| 168 |
+
**Table 4: Per-benchmark results on Cloudflare Decision Index (`v0.2.1`, 10 tasks, $N = 22,427$). Bold denotes the highest score across all parameter scales; underline denotes best in the $\le$9B class.**
|
| 169 |
+
|
| 170 |
+
| Benchmark (Metric) | **FastBrain-9B** (Ours) | Clef-flash (Base) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 171 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 172 |
+
| Total Parameters | 9.41B | 9.41B | 9.00B | 27.36B | 27.00B | 25.8B (4B act.) | 0.42B |
|
| 173 |
+
| 1. BFCL (Case Exact Acc) | **<u>98.76%</u>** | **<u>98.76%</u>** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
|
| 174 |
+
| 2. ToolRet (nDCG@10) | <u>67.26%</u> | <u>67.26%</u> | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
|
| 175 |
+
| 3. API-Bank (Accuracy) | **<u>93.70%</u>** | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
|
| 176 |
+
| 4. BANKING77 (Macro-F1) | <u>93.77%</u> | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
|
| 177 |
+
| 5. CLINC150+OOS (Macro-F1) | **<u>98.44%</u>** | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
|
| 178 |
+
| 6. Home Appliances (Case Exact) | **<u>97.73%</u>** | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
|
| 179 |
+
| 7. BRIGHT (nDCG@10) | <u>38.58%</u> | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
|
| 180 |
+
| 8. Amazon ESCI (Macro-F1) | <u>57.40%</u> | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
|
| 181 |
+
| 9. PhishNChips (Accuracy) | <u>78.75%</u> | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
|
| 182 |
+
| 10. When2Call (Macro-F1) | <u>81.13%</u> | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
|
| 183 |
+
| **10-Task Macro Mean** | <u>80.55%</u> | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
|
| 184 |
|
| 185 |
---
|
| 186 |
|
| 187 |
+
## 4. Systems Performance and Post-Training Quantization
|
| 188 |
|
| 189 |
+
### 4.1 Runtime Comparison Across Inference Engines
|
| 190 |
+
|
| 191 |
+
Table 5 compares end-to-end latency, prefill throughput, and memory footprint across three serving backends—SGLang (`lmsysorg/sglang:dev-clef` with `JointSchemaPooler`), `llama.cpp` (`b11430` `ClefModel` graph in `Q8_0` and `Q4_K_M`), and native PyTorch (`joint_schema_model.py`)—benchmarking identical requests on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. Because `output_tokens = 0` for all requests, throughput is governed purely by prefill compute and span-pooling efficiency, reaching `19,319 tok/s` (`131.2` decisions/s) at batch size $B = 4$ in SGLang.
|
| 192 |
+
|
| 193 |
+
**Table 5: End-to-end latency and prefill throughput on NVIDIA RTX PRO 6000 Blackwell Server Edition (G4).**
|
| 194 |
+
|
| 195 |
+
| Workload / System Metric | SGLang `dev-clef` (BF16) | `llama.cpp b11430` (`Q8_0`) | `llama.cpp b11430` (`Q4_K_M`) | PyTorch Native (BF16) |
|
| 196 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 197 |
+
| Cold-Start Load Time | 11.11 s | 2.56 s | **2.00 s** | 11.11 s |
|
| 198 |
+
| Peak GPU Memory (16K context) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 199 |
+
| Compatible with 16 GiB T4 GPU | No (BF16 TP=1) | Yes (`9.03 GiB` at 4K) | **Yes (`6.78 GiB` at 4K)** | Yes (`6.8 GiB` via NF4) |
|
| 200 |
+
| Top-1 Agreement vs. PyTorch BF16 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (reference) |
|
| 201 |
+
| 1. Short Intent Routing (293 tok, $K=2$) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 202 |
+
| 2. Typed Decisions (736 tok, $K=5$) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 203 |
+
| 3. When2Call Governance (1,068 tok, $K=1$) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 204 |
+
| 4. PhishNChips Security (1,303 tok, $K=9$) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 205 |
+
| 5. SafeAgents Judge (2,217 tok, $K=2$) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 206 |
+
| Batch $B=4$ Typed Decisions (2,944 tok, 20 dec) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 207 |
+
|
| 208 |
+
### 4.2 Mixed-Precision GGUF Quantization Fidelity
|
| 209 |
+
|
| 210 |
+
Uniform 4-bit quantization degrades cross-attention span readout and linear-attention recurrent state precision if applied indiscriminately to decision-head projections. In our `FastBrain-9B-Q4_K_M.gguf` (`6.73 GB`) build, feed-forward blocks are quantized to 4-bit K-quants while all `dec.*` decision-head tensors, `ssm_*` Gated DeltaNet parameters, and `attn_*` projections remain at `Q8_0` precision. Evaluated across a stratified 10-task subset (`203` multi-question requests, `1,473` scored schema questions; Table 6), `Q8_0.gguf` (`9.66 GB`) matches native `bfloat16` on `1,471/1,473` decisions (`99.86%` Top-1 agreement, mean absolute probability error `0.00203`, `-0.03` percentage-point delta on the 10-task mean), and `Q4_K_M.gguf` retains `98.78%` Top-1 agreement (`0.00387` probability MAE, `-0.20` percentage-point calibrated delta). Table 7 summarizes hardware memory requirements across formats, including NVIDIA Tesla T4 (`16 GiB`) measurements.
|
| 211 |
+
|
| 212 |
+
**Table 6: Quantization fidelity across 1,473 scored schema questions on the 10-task stratified Decision Index subset.**
|
| 213 |
+
|
| 214 |
+
| Metric / Subset | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` Head) |
|
| 215 |
+
| :--- | :---: | :---: | :---: |
|
| 216 |
+
| Top-1 Decision Agreement ($N = 1,473$) | 100.00% (ref) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 217 |
+
| Mean Absolute Probability Error ($\|\hat{p}_q - \hat{p}_{\text{bf16}}\|$) | 0.00000 | 0.00203 | 0.00387 |
|
| 218 |
+
| 10-Task Mean (Uncalibrated Head) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 219 |
+
| 10-Task Mean (Schema-Calibrated) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 220 |
+
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|
| 221 |
+
| ├─ 2. ToolRet / 7. BRIGHT | 73.15% / 41.93% | 73.70% / 41.00% | 75.60% / 40.31% |
|
| 222 |
+
| ├─ 5. CLINC150+OOS / 6. Home Appliances | 93.94% / 90.00% | 93.94% / 90.00% | 93.94% / 90.00% |
|
| 223 |
+
| └─ 8. Amazon ESCI / 9. PhishNChips / 10. When2Call | 73.89% / 85.00% / 75.00% | 73.89% / 85.00% / 75.00% | 75.79% / 85.00% / 72.92% |
|
| 224 |
+
|
| 225 |
+
**Table 7: Disk footprint, peak VRAM allocation, and Tesla T4 (`16 GiB`) latency across precision tiers.**
|
| 226 |
+
|
| 227 |
+
| Precision Format | Artifact Size | Peak VRAM (4K–16K ctx) | T4 Latency (293 / 1,068 tok) | Target Hardware |
|
| 228 |
+
| :--- | :---: | :---: | :---: | :--- |
|
| 229 |
+
| `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 6.78 – 10.30 GiB | 343.0 ms / 1,154.3 ms | 8–16 GB GPU (RTX 3060/4060, Tesla T4), Apple Silicon (8 GB+) |
|
| 230 |
+
| PyTorch `load_in_4bit=True` (NF4 + `fp16` head) | ~5.5 GB (mem) | 6.20 – 6.80 GiB | — | 8–16 GB CUDA GPU (Colab T4, consumer GPUs) |
|
| 231 |
+
| `FastBrain-9B-Q8_0.gguf` | 9.66 GB | 9.03 – 12.55 GiB | 349.3 ms / 1,149.8 ms | 12–16 GB GPU (Tesla T4, RTX 4070 Ti), Apple Silicon (16 GB+) |
|
| 232 |
+
| PyTorch `load_in_8bit=True` (INT8 + `bf16` head) | ~9.6 GB (mem) | ~11.20 GiB | — | 12–16 GB CUDA GPU |
|
| 233 |
+
| Native `bfloat16` (`safetensors` / `BF16.gguf`) | 18.82 / 18.16 GB | 20.30 – 24.70 GiB | — | $\ge$24 GB GPU (NVIDIA L4, A100, H100, RTX 3090/4090, PRO 6000) |
|
| 234 |
|
| 235 |
---
|
| 236 |
|
| 237 |
+
## 5. Reproducible Inference and Deployment
|
| 238 |
|
| 239 |
+
### 5.1 Python / Transformers Runtime (`bfloat16`, `8-bit`, or `4-bit` NF4)
|
| 240 |
|
| 241 |
+
`TypeScript/FastBrain-9B` implements the `joint_schema_model` interface and is API-compatible with `Cloudflare/clef-flash`. Passing `load_in_4bit=True` quantizes only the 9.3B backbone via `bitsandbytes` NF4 while preserving the 116M `JointSchemaHead` in floating point.
|
| 242 |
|
| 243 |
```python
|
|
|
|
| 244 |
import sys
|
| 245 |
import torch
|
| 246 |
+
from huggingface_hub import snapshot_download
|
| 247 |
|
| 248 |
model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
|
| 249 |
sys.path.insert(0, model_dir)
|
| 250 |
import joint_schema_model as clef
|
| 251 |
|
| 252 |
+
# Native bfloat16 (>=24 GB VRAM):
|
| 253 |
model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
|
| 254 |
|
| 255 |
+
# Mixed-precision 4-bit NF4 (8-16 GB VRAM, e.g., Tesla T4):
|
| 256 |
# model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
|
| 257 |
|
| 258 |
response = clef.systemone(model, processor, {
|
|
|
|
| 262 |
"tools": ["get_balance", "transfer_funds"],
|
| 263 |
},
|
| 264 |
"questions": {
|
| 265 |
+
"next_action": {
|
| 266 |
"type": "choice",
|
| 267 |
+
"instructions": "Select the appropriate assistant action.",
|
| 268 |
"criteria": {
|
| 269 |
+
"call_balance": "Invoke <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
|
| 270 |
+
"request_auth": "Ask the user to provide an account identifier or authentication token.",
|
| 271 |
+
"abstain": "Decline because personal banking records are out of scope.",
|
| 272 |
},
|
| 273 |
},
|
| 274 |
},
|
|
|
|
| 276 |
print(response)
|
| 277 |
```
|
| 278 |
|
| 279 |
+
### 5.2 `llama.cpp` GGUF Server (`/v1/systemone`)
|
|
|
|
|
|
|
| 280 |
|
| 281 |
> [!IMPORTANT]
|
| 282 |
+
> **Runtime Requirements for `ClefModel` GGUF Execution**
|
| 283 |
+
> 1. Use **`llama.cpp` build `b11430` or later**, which implements the `LLM_ARCH_CLEF` computation graph (`dec.*` tensors) and the `/v1/systemone` HTTP endpoint. Earlier builds and standard generative wrappers do not execute the cross-attention decision head.
|
| 284 |
+
> 2. Set physical micro-batch size equal to logical batch size (`-b 16384 -ub 16384` or `-b 4096 -ub 4096`) together with `--no-cache-prompt`. `ClefModel` pools `<|question_*|>` and `<|option_*|>` spans within a single physical `ubatch`; leaving `-ub` at its default (`512`) splits prompts longer than 512 tokens across micro-batches, and prefix caching can bypass marker positions.
|
|
|
|
|
|
|
|
|
|
| 285 |
|
| 286 |
```bash
|
| 287 |
+
# 1. Download prebuilt llama.cpp (b11430+) and GGUF checkpoint
|
|
|
|
|
|
|
|
|
|
| 288 |
curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
|
|
|
|
|
|
|
| 289 |
pip install -U "huggingface_hub[hf_transfer]"
|
| 290 |
HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
|
|
|
|
|
|
|
|
|
|
| 291 |
|
| 292 |
+
# 2. Start llama-server (add --mmproj mmproj-FastBrain-9B-Q8_0.gguf for image inputs)
|
|
|
|
| 293 |
export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
|
|
|
| 294 |
./llama-b11430/llama-server \
|
| 295 |
-m ./FastBrain-9B-Q8_0.gguf \
|
| 296 |
-ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
|
| 297 |
--port 8080
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 298 |
```
|
| 299 |
|
| 300 |
+
Querying `/v1/systemone` evaluates multiple heterogeneous primitives (`choice` + `noul`) in a single prefill pass:
|
| 301 |
|
| 302 |
```bash
|
| 303 |
curl http://127.0.0.1:8080/v1/systemone \
|
|
|
|
| 309 |
"customer_message": "I was charged twice for my subscription last week and nobody has replied to my ticket."
|
| 310 |
},
|
| 311 |
"questions": {
|
| 312 |
+
"department": {
|
| 313 |
"type": "choice",
|
| 314 |
+
"instructions": "Assign this ticket to the responsible queue.",
|
| 315 |
"criteria": {
|
| 316 |
"billing": "Payments, refunds, duplicate charges, or invoices",
|
| 317 |
+
"engineering": "Software defects, crashes, or service outages",
|
| 318 |
+
"identity": "Password reset or multi-factor authentication issues"
|
| 319 |
}
|
| 320 |
},
|
| 321 |
+
"is_urgent": {
|
| 322 |
"type": "noul",
|
| 323 |
+
"instructions": "Does the message report an unresolved financial loss or time-critical failure?"
|
| 324 |
}
|
| 325 |
}
|
| 326 |
}'
|
| 327 |
```
|
| 328 |
|
| 329 |
+
### 5.3 SGLang Server (`JointSchemaPooler`)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 330 |
|
| 331 |
```bash
|
| 332 |
+
docker run --gpus all --shm-size 32g -p 30000:30000 \
|
| 333 |
+
-v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
|
|
|
|
|
|
|
|
|
|
| 334 |
lmsysorg/sglang:dev-clef \
|
| 335 |
+
sglang serve --model-path TypeScript/FastBrain-9B --host 0.0.0.0 --port 30000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 336 |
```
|
| 337 |
|
| 338 |
---
|
| 339 |
|
| 340 |
+
## 6. Repository Manifest and Citation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 341 |
|
| 342 |
+
**Table 8: Repository file specification.**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 343 |
|
| 344 |
+
| Path | Size | Contents |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 345 |
| :--- | :---: | :--- |
|
| 346 |
+
| `model-00001-of-00004.safetensors` .. `00004` | 18.59 GB | `bfloat16` hybrid backbone tensors with all adapted projections merged in-place |
|
| 347 |
+
| `joint_head.safetensors`, `joint_head_config.json` | 232 MB | 116M-parameter `JointSchemaHead` weights (`122` tensors) and architectural config |
|
| 348 |
+
| `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | Mixed-precision 4-bit GGUF (`dec.*`, `ssm_*`, and `attn_*` retained in `Q8_0`) |
|
| 349 |
+
| `FastBrain-9B-Q8_0.gguf` / `FastBrain-9B-BF16.gguf` | 9.66 / 18.16 GB | 8-bit and `bfloat16` GGUF builds for `llama-server` (`b11430`+) |
|
| 350 |
+
| `mmproj-FastBrain-9B-Q8_0.gguf` / `BF16.gguf` | 624 / 922 MB | Vision projector checkpoints for multimodal state inputs |
|
| 351 |
+
| `joint_schema_model.py` | 25 KB | Reference `ClefModel`, vectorized `_forward_fast_cuda` head, and `systemone()` runtime |
|
| 352 |
+
| `eval_submissions/typescript_fastbrain-9b*.json` | 1.80 MB | Verifiable 2,000-decision zero-shot and prior-fitted traces for `LocalLLaMA/typed-decisions` |
|
| 353 |
+
| `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | 110 KB | Verifiable 661-call binary and 7-way category traces for `safeagents-judge-bench` (v2) |
|
| 354 |
+
| `.eval_results/*.yaml` | 2.3 KB | Hugging Face Hub machine-readable evaluation records |
|
| 355 |
+
| `README.md`, `README_zh.md`, `assets/*.png` | 1.28 MB | English and Chinese technical documentation and evaluation figures |
|
| 356 |
+
|
| 357 |
+
### Citation
|
| 358 |
+
|
| 359 |
+
```bibtex
|
| 360 |
+
@misc{fastbrain9b2026,
|
| 361 |
+
title = {FastBrain-9B: A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions},
|
| 362 |
+
author = {TypeScript},
|
| 363 |
+
year = {2026},
|
| 364 |
+
howpublished = {\url{https://huggingface.co/TypeScript/FastBrain-9B}},
|
| 365 |
+
note = {Apache-2.0 License}
|
| 366 |
+
}
|
| 367 |
+
```
|
README_zh.md
CHANGED
|
@@ -2,177 +2,247 @@
|
|
| 2 |
|
| 3 |
# FastBrain-9B
|
| 4 |
|
| 5 |
-
**单次前向
|
| 6 |
|
| 7 |
-
[](./LICENSE)
|
| 13 |
|
| 14 |
**[English](./README.md)** | **简体中文**
|
| 15 |
|
| 16 |
</div>
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
给定任意 JSON
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
-
|
| 27 |
-
- **企业级决策总榜 `LocalLLaMA/typed-decisions`(共 `94` 个参评模型)零样本前七 & 全场最快推理速度**:在 `43` 个模型的**零样本总榜位列 `#7 / 43`**(**开源权重模型 `#5 / 36`**、**`<=9B` 开源模型 `#2 / 19`**、**全部 `Clef` 架构模型不分尺寸 `#1`**),同时以 **`50 ms` p50 延迟成为 Top 28(准确率 `>70%` 梯队)中速度 `#1` 的模型**——比榜首 `31B Quyet-1.0-Large`(`3,335 ms`)**快 `66.7` 倍**,比 `27B Amos 1 large`(`357 ms`)**快 `7.1` 倍**!在 `51` 个模型的 **`train` 校准/微调总榜**同样已合入主榜(**并列 `#12` / 列表 `#14 / 51`**,准确率 **`79.70%` / `80.70%`**,`KL 0.0816`),在**不修改任何模型权重(Frozen-Weight)**的前提下拿下 **`<=9B` 第 1、全场 KL 校准第 1**,击败 37 个专门在训练集上微调权重的模型!
|
| 28 |
-
- **Cloudflare Decision Index(`v0.2.1`)跨量级越级击败 `27B` 模型**:在涵盖 `22,427` 次评测请求的 10 大基准测试中取得 **`80.55%` 宏平均分(全尺寸总榜 `#2 / 7`)**,在 **`<=9B` 级别豪取 `10 / 10` 全满贯第一**(比同尺寸基座 `Clef-flash` 暴涨 **`+5.59 pp`**,其中 `CLINC150+OOS` 暴涨 **`+31.67 pp`**、`When2Call` 暴涨 **`+15.55 pp`**),总分超越 `27B Jev`(`80.32%`),并在 **5 / 10 个单项上正面击败 `3` 倍参数量的旗舰 `27.4B Cloudflare/clef`**(其中 **4 个单项夺得全尺寸绝对第一**)。
|
| 29 |
-
- **全栈工业级部署生态(`SGLang` + `llama.cpp GGUF` + `PyTorch 4-bit`)**:内置向量化前缀和池化与变长 FlashAttention 融合算子(决策头加速 **`1.90x`**,`max_logit_diff = 0.00000`);`Q8_0.gguf` 与原生 `BF16` Top-1 决策一致率高达 **`99.86%`**(近乎完全无损);`Q4_K_M.gguf` 仅 **`6.73 GB`**,在免费 Colab T4(16GB)或 Mac M 系列芯片上仅占 `6.78 GiB` 显存即可满速运行。
|
| 30 |
|
| 31 |
---
|
| 32 |
|
| 33 |
-
##
|
| 34 |
|
| 35 |
-
###
|
| 36 |
|
| 37 |
-
|
| 38 |
-
| :--- | :---: | :---: | :---: | :---: | :--- |
|
| 39 |
-
| **[`safeagents-judge-bench` (`v2`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(7 分类违规识别)***<br>([已合入 PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [官方榜单](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench)) | **8 个决策模型**<br>*(+2 基线)* | **`75.1%` Macro-F1**<br>**`85.8%` 分类准确率** | **#1 / 8 *(全场第 1)*** | **Top 12.5% *(断层冠军)*** | `Cloudflare/clef` *(27.4B, #2 `56.3%` F1 / `73.7%` Acc)*、`vllm-sr/d3-mini` *(#3 `51.2%`)*、`Cloudflare/clef-flash` *(9.4B, #4 `46.2%`)*、`clef-omni` *(30B, #5 `43.7%`)* |
|
| 40 |
-
| **[`safeagents-judge-bench` (`v2` & `v1`)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(二分类安全拦截)***<br>([已合入 PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [`v1` 榜单](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)) | **8 模型 (`v2`)**<br>**9 模型 (`v1`)** | **`82/82` (`100%`) 轨迹全拦截 · `98.5%` AUROC**<br>*(`93.8%` 校准 / `81.4%` 默认 `v2` · **`97.4%` `v1`**)* | **`#1 / 9` (`v1` 总榜第 1)**<br>**`100%` 拦截 · AUROC #3 (`v2`)** | **Top 11.1% (`v1`)**<br>**Top 37.5% AUROC (`v2`)** | 默认阈值即击败 `27.4B Cloudflare/clef` (`80.8%`);AUROC 与轨迹拦截击败 `clef-flash` (`96.7%`, `81/82`)、`clef-omni` (`94.6%`, `81/82`)、`d1-3B` (`68/82`) |
|
| 41 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(零样本全量总榜)***<br>([已合入 PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [预测轨迹](./eval_submissions/typescript_fastbrain-9b.json)) | **43 个模型**<br>*(36 个开源权重)* | **`75.25%` / `76.25%` 准确率**<br>*(KL `0.1508` · Brier `0.0795` · **`50 ms`**)* | **#7 / 43 *(实时总榜)***<br>**#5 / 36 开源权重总榜** | **Top 16.3% 总榜**<br>**Top 13.9% 开源** | 击败 `vllm-sr/d3` *(27B, #8 `74.9%`)*、`Baseten Elephant 1` *(9B, #9 `74.8%`)*、`Darwin-27B` *(#10 `74.3%`)*、`Cloudflare/clef` *(27B, #16 `72.8%`)*、`openjev` *(27B, #17 `72.8%`)* |
|
| 42 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(零样本 `<=9B` 及 Clef 架构榜)*** | **19 个 `<=9B` 开源**<br>**4 个 Clef 架构** | **`75.25%` / `76.25%` 零样本**<br>*(蒸馏原生头: **`77.35%`** · **`50 ms`**)* | **#2 / 19 `<=9B` 开源**<br>**#1 Clef 架构 · #1 极速** | **Top 10.5% (`<=9B`)** | 击败 18 个同级开源模型中的 17 个(`d3-mini` `74.3%`、`d3-flash` `71.7%`、`signpost-d-4b` `71.2%`、`clef-flash` `71.0%`、`d1-3B` `65.6%`)及 `35B clef-omni` (`67.3%`) |
|
| 43 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(`train` 校准/微调榜)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [预测轨迹](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **51 个模型** | **`79.70%` / `80.70%` 准确率**<br>*(KL **`0.0816`** · Brier **`0.0437`** · **`50 ms`**)* | **并列 #12 *(列表 #14 / 51)***<br>**冻结权重 `<=9B` #1 · KL #1** | **Top 23.5% 总榜**<br>*(零权重微调)* | **不改任何权重**即击败 **37** 个微调/专精模型:`Rev-gen1-midi` *(25.6B, `79.45%`)*、`OpenDecider-small-td` *(4B LoRA, `79.2%`)*、`OpenDecider-medium-td` *(30B LoRA, `78.8%`)*、`Seb-9B` *(`75.4%`)* |
|
| 44 |
-
| **Cloudflare Decision Index (`v0.2.1`)** *(10 任务宏平均)* | **7 个模型** | **`80.55%` 宏平均** | **#2 / 7 全尺寸总榜**<br>**`<=9B` 10/10 全满贯 #1** | **Top 28.6% 总榜**<br>**`<=9B` 100% 统治率** | 总分超越 `Jev` *(27B, `80.32%`)*、`Kev 9B` *(`76.51%`)*、`Clef-flash` *(9.4B, `74.96%`)*、`DiffGemma` *(26B, `69.14%`)*,并在 **5 / 10** 项任务中击败 `27.4B Clef` |
|
| 45 |
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
-
|
| 49 |
|
| 50 |
-
|
| 51 |
-
- **`<=9B` 量级 `10 / 10` 单项全满贯第一**:全面超越同尺寸基座 `Clef-flash`(`9.4B`,`74.96%`)达 **`+5.59 pp`**(其中开放集意图拒识 `CLINC150+OOS` 提升 **`+31.67 pp`**,工具调用治理 `When2Call` 提升 **`+15.55 pp`**),领先 `Kev 9B`(`76.51%`)**`+4.04 pp`**,领先 `Laya`(`0.4B`,`58.09%`)**`+22.46 pp`**。
|
| 52 |
-
- **跨量级击败 `3` 倍参数量的 `27B` 级大模型**:10 任务宏平均分超越 `Jev`(`27B`,`80.32%`)与 `DiffusionGemma Jev`(`26B-A4B`,`69.14%`),并在 **`BFCL`、`API-Bank`、`CLINC150+OOS`、`Home appliances`、`When2Call` 五大基准上正面击败 `27.4B` 旗舰 `Clef`**(其中 **4 项夺得全尺寸全场第一**)。
|
| 53 |
|
| 54 |
-
|
| 55 |
|
| 56 |
-
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
-
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
| 评测基准 |
|
| 67 |
-
| :--- | :---: | :---: | :---: | :---
|
| 68 |
-
| **
|
| 69 |
-
| **
|
| 70 |
-
|
|
| 71 |
-
| **
|
| 72 |
-
| **
|
| 73 |
-
|
|
| 74 |
-
| **Home Appl.**<br>*(智能家居指令精确率)* | **97.73%**<br>*(全场 #1)* | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
|
| 75 |
-
| **BRIGHT**<br>*(推理密集型检索 nDCG@10)* | **38.58%**<br>*(9B #1)* | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
|
| 76 |
-
| **Amazon ESCI**<br>*(电商相关性 Macro-F1)* | **57.40%**<br>*(9B #1)* | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
|
| 77 |
-
| **PhishNChips**<br>*(9 探针钓鱼安全检测)* | **78.75%**<br>*(9B #1)* | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
|
| 78 |
-
| **When2Call**<br>*(4 分类工具触发治理)* | **81.13%**<br>*(9B #1)* | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
|
| 79 |
-
| **10 任务宏平均**<br>*(Macro Mean)* | **80.55%**<br>*(总榜 #2 · 9B #1)* | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
|
| 80 |
|
| 81 |
---
|
| 82 |
|
| 83 |
-
###
|
| 84 |
|
| 85 |
-
|
| 86 |
|
| 87 |
-
|
| 88 |
-
> - **`v2` 7 分类违规识别赛道全场总冠军(`#1 / 8`)**:斩获 **`75.1%` Category Macro-F1** 与 **`85.8%` 分类准确率(`567/661`)**,领先第 2 名 `27.4B Cloudflare/clef`(`56.3%` F1 / `73.7%` Acc)达 **`+18.8 pp` Macro-F1(`+12.1 pp` 准确率)**,领先第 3 名 `vllm-sr/d3-mini`(`51.2%`)**`+23.9 pp`**,领先同尺寸基座 `Cloudflare/clef-flash`(`46.2%` / `57.6%` Acc)**`+28.9 pp` Macro-F1(`+28.2 pp` 准确率)**!
|
| 89 |
-
> - **`v2` 二分类安全护栏 `82 / 82`(`100%`)危险轨迹全拦截 & `98.5%` AUROC**:成功拦截 **`534 / 536`(`99.6%`)** 次恶意工具调用(其中 `harness_introspection` `272/272`、`credential_misuse` `15/15`、`sandbox_escape` `4/4` 召回率均为 **`100%`**,首次拦截前中位漏放恶调数为 **`0`**)。在未校准默认阈值(`P(block) >= 0.50`)下平衡准确率为 **`81.4%`**(超越 `27.4B Cloudflare/clef` 的 `80.8%`);凭借 **`98.5%` 的高 AUROC**,仅需校准判别阈值(`P(block) >= 0.961`)即可将平衡准确率拉升至 **`93.8%`(误拦率仅 `5.6%`)** 且保持 **`82/82`(`100%`)** 轨迹全拦截。在 [`v1`(562 条调用)官方版本](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1)中,`FastBrain-9B` 更包揽了**二分类(`97.4%` Bal. Acc)与多分类(`85.0%` Macro-F1)双榜 `#1 / 9`**。
|
| 90 |
|
| 91 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 93 |
-
| **
|
| 94 |
-
| **
|
| 95 |
-
| **
|
| 96 |
-
|
|
| 97 |
-
|
|
| 98 |
-
|
|
| 99 |
-
|
|
| 100 |
-
|
|
| 101 |
-
|
|
| 102 |
-
|
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
|
| 117 |
-
|
|
| 118 |
-
|
|
| 119 |
-
|
|
| 120 |
-
|
|
| 121 |
-
|
|
| 122 |
-
|
|
| 123 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
---
|
| 126 |
|
| 127 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 138 |
|
| 139 |
---
|
| 140 |
|
| 141 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 142 |
|
| 143 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
```python
|
| 148 |
-
from huggingface_hub import snapshot_download
|
| 149 |
import sys
|
| 150 |
import torch
|
|
|
|
| 151 |
|
| 152 |
model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
|
| 153 |
sys.path.insert(0, model_dir)
|
| 154 |
import joint_schema_model as clef
|
| 155 |
|
| 156 |
-
#
|
| 157 |
model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
|
| 158 |
|
| 159 |
-
#
|
| 160 |
# model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
|
| 161 |
|
| 162 |
response = clef.systemone(model, processor, {
|
| 163 |
"model": "TypeScript/FastBrain-9B",
|
| 164 |
"state": {
|
| 165 |
-
"query": "能帮我查一下账户余额吗?",
|
| 166 |
"tools": ["get_balance", "transfer_funds"],
|
| 167 |
},
|
| 168 |
"questions": {
|
| 169 |
-
"
|
| 170 |
"type": "choice",
|
| 171 |
-
"
|
| 172 |
"criteria": {
|
| 173 |
-
"
|
| 174 |
-
"
|
| 175 |
-
"
|
| 176 |
},
|
| 177 |
},
|
| 178 |
},
|
|
@@ -180,53 +250,28 @@ response = clef.systemone(model, processor, {
|
|
| 180 |
print(response)
|
| 181 |
```
|
| 182 |
|
| 183 |
-
###
|
| 184 |
-
|
| 185 |
-
本仓库提供适配官方 `llama.cpp`(**`b11430` 及以上版本**,原生 `ClefModel` 架构)的 GGUF 模型文件,单文件同时打包了 `Qwen3.5-9B` 混合主干与 `116M` `JointSchemaHead`(即使在 `Q4_K_M` 量化版本中,`dec.*` 决策头张量、`ssm_` 线性注意力状态与 `attn_` 核心投影层也全部保留在 `Q8_0` 高精度)。
|
| 186 |
|
| 187 |
> [!IMPORTANT]
|
| 188 |
-
> **`
|
| 189 |
-
> 1.
|
| 190 |
-
> 2.
|
| 191 |
-
> `llama-server` 中的 `ClefModel` 会在单个物理微批次(`ubatch`,默认值 `512`)内对 `<|question_*|>` 与 `<|option_*|>` 标记区间计算交叉注意力。将 `-ub` 设为与 `-b` 相同可确保长 Prompt(>512 tokens)在单次物理 `ubatch` 内完整计算而不截断选项区间;`--no-cache-prompt` 则可避免前缀缓存跳过决策标记位。
|
| 192 |
-
|
| 193 |
-
#### 第一步:下载预编译 `llama-server`(`b11430`+)与 GGUF 权重
|
| 194 |
|
| 195 |
```bash
|
| 196 |
-
# 1. 下载
|
| 197 |
-
# - Linux CUDA 13.x (Colab / Ubuntu): llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz
|
| 198 |
-
# - Linux CUDA 12.x: llama-b11430-bin-ubuntu-cuda-12.8-x64.tar.gz
|
| 199 |
-
# - macOS Apple Silicon (M1/M2/M3/M4): llama-b11430-bin-macos-arm64.tar.gz
|
| 200 |
curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
|
| 201 |
-
|
| 202 |
-
# 2. 下载 FastBrain-9B GGUF(可选 Q8_0 [9.66 GB,无损级精度] 或 Q4_K_M [6.73 GB,适配 8GB–16GB 显存])
|
| 203 |
pip install -U "huggingface_hub[hf_transfer]"
|
| 204 |
HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
|
| 205 |
-
```
|
| 206 |
|
| 207 |
-
#
|
| 208 |
-
|
| 209 |
-
```bash
|
| 210 |
-
# Linux / Google Colab 环境下确保 CUDA 动态库加入 LD_LIBRARY_PATH:
|
| 211 |
export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
|
| 212 |
-
|
| 213 |
-
# 启动 llama-server(24GB+ 显存 / 16K 上下文):
|
| 214 |
./llama-b11430/llama-server \
|
| 215 |
-m ./FastBrain-9B-Q8_0.gguf \
|
| 216 |
-ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
|
| 217 |
--port 8080
|
| 218 |
-
|
| 219 |
-
# 或在 12GB–16GB 显卡上运行(例如 Colab 免费 Tesla T4,4K 上下文):
|
| 220 |
-
# ./llama-b11430/llama-server \
|
| 221 |
-
# -m ./FastBrain-9B-Q4_K_M.gguf \
|
| 222 |
-
# -ngl 99 -c 4096 -b 4096 -ub 4096 -np 1 -fa on --no-cache-prompt \
|
| 223 |
-
# --port 8080
|
| 224 |
-
|
| 225 |
-
# 可选:若输入状态包含多模态图片/截图,可挂载视觉投影器权重:
|
| 226 |
-
# --mmproj ./mmproj-FastBrain-9B-Q8_0.gguf
|
| 227 |
```
|
| 228 |
|
| 229 |
-
|
| 230 |
|
| 231 |
```bash
|
| 232 |
curl http://127.0.0.1:8080/v1/systemone \
|
|
@@ -235,124 +280,62 @@ curl http://127.0.0.1:8080/v1/systemone \
|
|
| 235 |
"model": "FastBrain-9B",
|
| 236 |
"state": {
|
| 237 |
"ticket_id": "ORD-99281",
|
| 238 |
-
"customer_message": "我上周的订阅被重复扣费了两次,提交工单后一直
|
| 239 |
},
|
| 240 |
"questions": {
|
| 241 |
-
"
|
| 242 |
"type": "choice",
|
| 243 |
-
"instructions": "该工单
|
| 244 |
"criteria": {
|
| 245 |
-
"billing": "支付、退款、重复扣费或发票
|
| 246 |
-
"
|
| 247 |
-
"
|
| 248 |
}
|
| 249 |
},
|
| 250 |
-
"
|
| 251 |
"type": "noul",
|
| 252 |
-
"instructions": "
|
| 253 |
}
|
| 254 |
}
|
| 255 |
}'
|
| 256 |
```
|
| 257 |
|
| 258 |
-
###
|
| 259 |
-
|
| 260 |
-
`TypeScript/FastBrain-9B` 原生支持 SGLang 的 `JointSchemaPooler`(镜像 `lmsysorg/sglang:dev-clef`),利用无缓存原始 K/V FlashAttention Embedding 模式以及向量化前缀和池化(`torch.cumsum`)+ 变长打包 FlashAttention(`cu_seqlens`),在服务端高并发场景下实现 **`19,300+ tok/s`(每秒 `131+` 个结构化决策)** 的极致吞吐:
|
| 261 |
|
| 262 |
```bash
|
| 263 |
-
docker run --gpus all \
|
| 264 |
-
--
|
| 265 |
-
-p 30000:30000 \
|
| 266 |
-
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
| 267 |
-
--ipc=host \
|
| 268 |
lmsysorg/sglang:dev-clef \
|
| 269 |
-
sglang serve
|
| 270 |
-
--model-path TypeScript/FastBrain-9B \
|
| 271 |
-
--host 0.0.0.0 \
|
| 272 |
-
--port 30000
|
| 273 |
```
|
| 274 |
|
| 275 |
---
|
| 276 |
|
| 277 |
-
##
|
| 278 |
-
|
| 279 |
-
在**同一台 NVIDIA RTX PRO 6000 Blackwell Server Edition (G4)** GPU 上对五大真实决策负载进行端到端实测对比:
|
| 280 |
-
|
| 281 |
-
| 负载场景 / 核心指标 | **SGLang `dev-clef`**<br>*(BF16 · JointSchemaPooler)* | **`llama.cpp b11430`**<br>*(GGUF Q8_0 · ClefModel)* | **`llama.cpp b11430`**<br>*(GGUF Q4_K_M · ClefModel)* | **PyTorch 原生**<br>*(BF16 · `joint_schema_model.py`)* |
|
| 282 |
-
| :--- | :---: | :---: | :---: | :---: |
|
| 283 |
-
| **冷启动加载耗时** | 11.11 s | **2.56 s** | **2.00 s** | 11.11 s |
|
| 284 |
-
| **峰值显存占用 (16K 上下文)** | 18.18 GiB *(权重)* / ~20.5 GiB | **12.55 GiB** | **10.30 GiB** | 20.3–24.7 GiB |
|
| 285 |
-
| **能否运行于 16GB T4 显卡?** | 否 *(仅 BF16, TP=1)* | **是 (`9.03 GiB` @ 4K)** | **是 (`6.78 GiB` @ 4K)** | **是 (`load_in_4bit`: `6.8 GiB`)** |
|
| 286 |
-
| **相对 BF16 Top-1 决策一致率** | **100.00%** *(最大偏差 0.0078)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* | **100.00% *(基准)*** |
|
| 287 |
-
| **1. 短文本意图路由** *(293 tok, K=2)* | **30.01 ms** *(9,763 tok/s)* | 32.82 ms *(8,929 tok/s)* | 31.69 ms *(9,245 tok/s)* | 43.23 ms *(6,778 tok/s)* |
|
| 288 |
-
| **2. Typed Decisions** *(736 tok, K=5)* | **50.29 ms** *(14,635 tok/s)* | 62.40 ms *(11,795 tok/s)* | 61.80 ms *(11,909 tok/s)* | 63.58 ms *(11,576 tok/s)* |
|
| 289 |
-
| **3. When2Call** *(1,068 tok, K=1)* | **66.06 ms** *(16,167 tok/s)* | 87.78 ms *(12,167 tok/s)* | 89.34 ms *(11,955 tok/s)* | 77.77 ms *(13,733 tok/s)* |
|
| 290 |
-
| **4. PhishNChips** *(1,303 tok, K=9)* | **75.86 ms** *(17,176 tok/s)* | 108.55 ms *(12,004 tok/s)* | 109.97 ms *(11,849 tok/s)* | 90.32 ms *(14,426 tok/s)* |
|
| 291 |
-
| **5. SafeAgents Judge** *(2,217 tok, K=2)* | **125.85 ms** *(17,616 tok/s)* | 181.70 ms *(12,201 tok/s)* | 183.50 ms *(12,082 tok/s)* | 138.06 ms *(16,058 tok/s)* |
|
| 292 |
-
| **Batch `B=4` 并发决策** *(2,944 tok, 20 决策)* | **152.39 ms** *(19,319 tok/s · 131.2 决策/s)* | 248.50 ms *(11,847 tok/s · 80.5 决策/s)* | 245.20 ms *(12,006 tok/s · 81.6 决策/s)* | 175.85 ms *(16,742 tok/s · 113.7 决策/s)* |
|
| 293 |
-
|
| 294 |
-
---
|
| 295 |
-
|
| 296 |
-
## GGUF 量化精度保持率与硬件显存指南
|
| 297 |
|
| 298 |
-
|
| 299 |
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
| 评测基准 / 指标 | PyTorch bfloat16<br>*(18.82 GB)* | GGUF Q8_0<br>*(9.66 GB)* | GGUF Q4_K_M<br>*(6.73 GB · Q8 决策头)* |
|
| 303 |
-
| :--- | :---: | :---: | :---: |
|
| 304 |
-
| **相对 BF16 Top-1 一致率** *(1,473 题)* | 100.00% *(基准)* | **99.86%** *(1,471/1,473)* | **98.78%** *(1,455/1,473)* |
|
| 305 |
-
| **概率分布平均绝对误差 MAE** | 0.00000 | **0.00203** *(0.20%)* | **0.00387** *(0.39%)* |
|
| 306 |
-
| **10 任务宏平均(单权重原生输出)** | **82.37%** | **82.34%** *(-0.03 pp)* | **82.56%** *(+0.19 pp)* |
|
| 307 |
-
| **10 任务宏平均��Schema 校准后)** | **83.68%** | **83.65%** *(-0.03 pp)* | **83.48%** *(-0.20 pp)* |
|
| 308 |
-
| ├─ **1. BFCL** *(AST 工具选择)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
|
| 309 |
-
| ├─ **2. ToolRet** *(20 工具重排)* | 73.15% | **73.70%** *(+0.55)* | **75.60%** *(+2.45)* |
|
| 310 |
-
| ├─ **3. API-Bank** *(多步工具调用)* | 95.00% | **95.00%** *(0.00)* | **95.00%** *(0.00)* |
|
| 311 |
-
| ├─ **4. BANKING77** *(77 类金融意图)* | 100.00% | **100.00%** *(0.00)* | **100.00%** *(0.00)* |
|
| 312 |
-
| ├─ **5. CLINC150+OOS** *(意图+越界拒识)* | 93.94% | **93.94%** *(0.00)* | **93.94%** *(0.00)* |
|
| 313 |
-
| ├─ **6. Home Appliances** *(家电控制)* | 90.00% | **90.00%** *(0.00)* | **90.00%** *(0.00)* |
|
| 314 |
-
| ├─ **7. BRIGHT** *(推理检索)* | 41.93% | **41.00%** *(-0.93)* | **40.31%** *(-1.62)* |
|
| 315 |
-
| ├─ **8. Amazon ESCI** *(电商相关性)* | 73.89% | **73.89%** *(0.00)* | **75.79%** *(+1.90)* |
|
| 316 |
-
| ├─ **9. PhishNChips** *(9 探针钓鱼检测)* | 85.00% | **85.00%** *(0.00)* | **85.00%** *(0.00)* |
|
| 317 |
-
| └─ **10. When2Call** *(4 分类触发治理)* | 75.00% *(70.83%)* | **75.00%** *(0.00)* | **72.92%** *(-2.08)* |
|
| 318 |
-
|
| 319 |
-
### 2. 各硬件平台实测推理速度与显存指南(`llama-server b11430`)
|
| 320 |
-
|
| 321 |
-
得益于**单次非自回归 Prefill 前向(`output_tokens = 0`)**架构,`llama-server` 在服务端显卡上实测吞吐高达 **8,900–12,100+ input tokens/sec**(端到端 HTTP 响应仅 `31–109 ms`):
|
| 322 |
-
|
| 323 |
-
| 负载场景 / 指标 | Q8_0.gguf<br>*(PRO 6000)* | Q4_K_M.gguf<br>*(PRO 6000)* | Q8_0.gguf<br>*(Tesla T4 16G)* | Q4_K_M.gguf<br>*(Tesla T4 16G)* | PyTorch BF16<br>*(PRO 6000)* |
|
| 324 |
-
| :--- | :---: | :---: | :---: | :---: | :---: |
|
| 325 |
-
| **模型加载耗时** | **2.56 s** | **2.00 s** | **5.43 s** | **5.10 s** | ~11.5 s |
|
| 326 |
-
| **峰值显存占用** | **12.55 GiB**<br>*(16K ctx)* | **10.30 GiB**<br>*(16K ctx)* | **9.03 GiB**<br>*(4K ctx)* | **6.78 GiB**<br>*(4K ctx)* | 20.3–24.7 GiB |
|
| 327 |
-
| **意图路由 + OOS**<br>*(293 tok, K=2)* | **32.82 ms**<br>*(8,928 tok/s)* | **31.69 ms**<br>*(9,245 tok/s)* | **349.32 ms**<br>*(839 tok/s)* | **342.98 ms**<br>*(854 tok/s)* | ~35.0 ms |
|
| 328 |
-
| **When2Call**<br>*(1,068 tok, K=1)* | **87.78 ms**<br>*(12,167 tok/s)* | **89.34 ms**<br>*(11,955 tok/s)* | **1,149.8 ms**<br>*(929 tok/s)* | **1,154.3 ms**<br>*(925 tok/s)* | ~92.0 ms |
|
| 329 |
-
| **PhishNChips**<br>*(1,303 tok, K=9)* | **108.55 ms**<br>*(12,004 tok/s)* | **109.97 ms**<br>*(11,849 tok/s)* | **1,430.8 ms**<br>*(911 tok/s)* | **1,440.5 ms**<br>*(905 tok/s)* | ~115.0 ms |
|
| 330 |
-
|
| 331 |
-
| 模型格式 / 量化档位 | 磁盘权重大小 | 峰值显存(`4K`–`16K` 上下文) | 最低推荐运行硬件 |
|
| 332 |
-
| :--- | :---: | :---: | :--- |
|
| 333 |
-
| **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit 主干 + `Q8_0` 决策头)* | `6.73 GB` | `~6.8 – 10.3 GiB` | **8GB–12GB 消费级显卡 / Tesla T4 (16GB) / Mac M1–M4** |
|
| 334 |
-
| **PyTorch `load_in_4bit=True`** *(NF4 主干 + `fp16` 决策头)* | `~5.5 GB` | `~6.2 – 6.8 GiB` | **Tesla T4 (16GB) / RTX 3060–4060 (8GB+)** |
|
| 335 |
-
| **`FastBrain-9B-Q8_0.gguf`** *(无损级全 8-bit GGUF)* | `9.66 GB` | `~9.0 – 12.6 GiB` | **12GB–16GB 显卡 / Tesla T4 (16GB) / Mac 16GB+ 统一内存** |
|
| 336 |
-
| **PyTorch `load_in_8bit=True`** *(INT8 主干 + `bf16` 决策头)* | `~9.6 GB` | `~11.2 GiB` | **Tesla T4 (16GB) / RTX 4070 Ti (12GB+)** |
|
| 337 |
-
| **原生全精度 `bfloat16`** *(`safetensors` / `BF16.gguf`)* | `18.82 GB` (`18.16 GB`) | `20.3 – 24.7 GiB` | **NVIDIA L4 (24GB) / A100 / RTX 3090–4090 / PRO 6000** |
|
| 338 |
-
|
| 339 |
-
---
|
| 340 |
-
|
| 341 |
-
## 仓库文件结构说明
|
| 342 |
-
|
| 343 |
-
| 文件路径 | 大小 | 用途说明 |
|
| 344 |
| :--- | :---: | :--- |
|
| 345 |
-
| `model-00001-of-00004.safetensors` .. `00004` |
|
| 346 |
-
| `joint_head.safetensors`
|
| 347 |
-
| `FastBrain-9B-Q4_K_M.gguf` |
|
| 348 |
-
| `FastBrain-9B-Q8_0.gguf`
|
| 349 |
-
| `FastBrain-9B-
|
| 350 |
-
| `
|
| 351 |
-
| `
|
| 352 |
-
| `eval_submissions/
|
| 353 |
-
| `
|
| 354 |
-
| `
|
| 355 |
-
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
# FastBrain-9B
|
| 4 |
|
| 5 |
+
**面向校准结构化决策的单次前向非自回归模型**
|
| 6 |
|
| 7 |
+
[-3b82f6?style=flat-square)](#2-模型架构与训练目标)
|
| 8 |
+
[-10b981?style=flat-square)](#2-模型架构与训练目标)
|
| 9 |
+
[-ec4899?style=flat-square)](#31-智能体工具调用安全审计评测-safeagents-judge-bench)
|
| 10 |
+
[_%C2%B7_50ms-8b5cf6?style=flat-square)](#32-多业务流企业结构化决策评测-localllamatyped-decisions)
|
| 11 |
+
[-f59e0b?style=flat-square)](#33-多领域结构化决策基准-cloudflare-decision-index-v021)
|
| 12 |
[](./LICENSE)
|
| 13 |
|
| 14 |
**[English](./README.md)** | **简体中文**
|
| 15 |
|
| 16 |
</div>
|
| 17 |
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
## 摘要
|
| 21 |
|
| 22 |
+
将自回归语言模型用于封闭 Schema 分类、工具路由与安全合规校验等结构化决策任务时,逐 Token 解码机制不仅引入与决策复杂度无关的序列生成延迟,还会因表面词形切分导致类别概率分布失准。本文发布 **FastBrain-9B**,一个总参数量为 9.41B 的非自回归结构化决策模型,由 32 层混合 Gated DeltaNet 线性注意力与全注意力主干网络(基于 `Cloudflare/clef-flash` 初始化的 `Qwen3.5-9B` 架构)及 116M 参数的交叉注意力联合决策头(`JointSchemaHead`)组成。给定任意 JSON 或多模态状态以及一组类型化决策定义(多选分类 `choice`、二元真值判别 `noul` 与序数量表评分 `score`),模型仅通过单次 Prefill 前向传播(`output_tokens = 0`)即可同步输出全部问题与候选项的校准概率分布。训练阶段,我们在复合序数 KL 散度、累积分布距离(离散分级概率评分 RPS)与期望值均方误差目标下,联合优化全部 32 层中的 104 个注意力投影矩阵及决策头参数,并将增量权重原地合并至主干张量。在涵盖 109 个参评模型条目的三项公开评测基准中:FastBrain-9B 在 `safeagents-judge-bench`(v2)7 类违规判别赛道位列 **第 1 / 8**(Macro-F1 `0.751`,准确率 `0.858`,较 27.4B 参数的 `Cloudflare/clef` 高出 18.8 个百分点),并在二元拦截赛道实现 `82/82` 越界轨迹全检出(AUROC `0.985`);在 `LocalLLaMA/typed-decisions` 零样本总榜位列 **第 7 / 43**(开源权重模型 **第 5 / 36**,$\le$9B 开源模型 **第 2 / 19**),中位推理延迟为 `50 ms`(较 31B 模型 `Quyet-1.0-Large` 快 66.7 倍),在冻结全部模型权重仅拟合先验与温度参数的设定下位列 `train` 校准榜 **并列第 12(列表第 14 / 51)**(准确率 `0.7970`,KL 散度 `0.0816`);在 Cloudflare Decision Index(v0.2.1)10 任务套件上取得 **80.55%** 的宏平均分,位���全量级 **第 2 / 7**、$\le$9B 量级 **10 / 10 任务单项第 1**,并在其中 5 项任务上超越 27.4B 的 `Cloudflare/clef`。
|
| 23 |
+
|
| 24 |
+
---
|
| 25 |
|
| 26 |
+
## 1. 研究背景与设计动机
|
| 27 |
|
| 28 |
+
在实际部署的智能体(Agent)与企业自动化工作流中,系统需高频执行具有明确输出空间的结构化判断,例如候选工具与参数选择、沙箱可疑命令拦截、工单队列分发以及有序风险等级评定。采用自回归生成范式求解此类问题存在两方面固有局限:其一,端到端延迟受限于格式化字符与标签文本的串行解码步数;其二,概率质量被分散至同义词表面形式或格式 Token 上,导致预测分布偏离真实后验概率,难以直接设定可靠的拒识或风险拦截阈值。
|
| 29 |
|
| 30 |
+
单次前向决策架构(`ClefModel` / System-1 范式)通过特殊边界标记将问题描述与候选项直接编码至输入序列尾部,利用最后一层隐藏状态经单次 Prefill 前向直接读取各选项对数几率(Logits),从而消除了自回归解码开销。然而,现有 9B 级决策基座在细粒度多类判别、序数标尺回归以及开放集越界拒识任务上与 27B–35B 级模型仍存在显著差距(例如 `Cloudflare/clef-flash` 在 `safeagents-judge-bench` v2 多分类赛道 Macro-F1 仅为 `0.462`,在 `CLINC150+OOS` 上仅为 `66.77%`)。FastBrain-9B 通过全深度投影矩阵联合适配、序数累积分布蒸馏以及编译期 Schema 拓扑边际校准,在保持 9B 级显存与单卡亚百毫秒延迟的前提下弥补乃至逆转了上述规模差距,并提供了高精度保持率的 8-bit 与 4-bit GGUF 量化部署方案。
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
---
|
| 33 |
|
| 34 |
+
## 2. 模型架构与训练目标
|
| 35 |
|
| 36 |
+
### 2.1 混合线性注意力与全注意力主干网络
|
| 37 |
|
| 38 |
+
FastBrain-9B 主干网络包含 32 层混合序列建模模块(`9.30B` 参数,`bfloat16` 精度,最大上下文长度 `262,144` tokens),由 24 层 Gated DeltaNet 线性注意力模块与 8 层标准多头自注意力模块交替堆叠而成。在指令适应阶段,我们联合解冻并更新全部 32 层中的 104 个核心投影矩阵(包括线性注意力层的 `in_proj_qkv`、`in_proj_z`、`out_proj` 以及全注意力层的 `q_proj`、`k_proj`、`v_proj`、`o_proj`)。训练完成后,所有参数更新均被精确合并至 `model-00001-of-00004.safetensors` 至 `model-00004-of-00004.safetensors` 中,推理阶段无需加载外部 LoRA 适配器。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
### 2.2 联合 Schema 决策头与向量化 CUDA 算子
|
| 41 |
+
|
| 42 |
+
`JointSchemaHead`(`116M` 参数,独立存储于 `joint_head.safetensors`)从主干末层隐藏状态 $H \in \mathbb{R}^{T \times d}$ 中提取由 `<|question_start|>`/`<|question_end|>` 与 `<|option_start|>`/`<|option_end|>` 界定的片段表征,经问题-选项交叉注意力交互后映射为三类基础决策原语的标量对数几率:
|
| 43 |
+
1. **`choice`**($K$ 分类离散分布):输出归一化概率向量 $\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}$。
|
| 44 |
+
2. **`noul`**(二元真值判别):输出命题成立概率 $\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})$。
|
| 45 |
+
3. **`score`**(定义在有序分值 $v_1 < \dots < v_K$ 上的量表评估):同时输出各等级概率 $\hat{p} \in \Delta^{K-1}$ 与数学期望得分 $\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k$。
|
| 46 |
|
| 47 |
+
在参考运行时 `joint_schema_model.py` 中,针对 CUDA 设备实现了融合算子路径 `_forward_fast_cuda`:利用基于 `torch.cumsum` 的 $\mathcal{O}(1)$ 前缀和差分替代逐片段 Python 循环完成变长片段均值池化,并通过 `cu_seqlens` 紧凑打包调用变长 FlashAttention(`torch.ops.aten._flash_attention_forward`)替代稠密掩码注意力。在 NVIDIA RTX PRO 6000 Blackwell 上,该实现将决策头单次前向耗时由 `3.27 ms` 降至 `1.72 ms`(**1.90 倍加速**),且保持严格数值等价(最大 Logit 绝对偏差 `max_logit_diff = 0.00000`)。
|
| 48 |
|
| 49 |
+
### 2.3 序数 KL 蒸馏与编译期 Schema 拓扑校准
|
|
|
|
|
|
|
| 50 |
|
| 51 |
+
标准交叉熵或普通多类 KL 散度将有序评定量表视为无序独立类别,对相邻等级误差与跨端极端误差施加同等惩罚。FastBrain-9B 在 `TypeScript/FastBrain-Instruct` 数据集上采用复合分布匹配损失进行训练,联合优化软标签 KL 散度、离散分级概率评分(预测累积分布 $\hat{F}_k = \sum_{j \le k} \hat{p}_j$ 与目标累积分布 $F_k$ 之间的二阶距离)以及 `score` 题型的期望值均方误差:
|
| 52 |
|
| 53 |
+
$$\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.$$
|
| 54 |
|
| 55 |
+
在推理阶段,`joint_schema_model.py` 提供可选的编译期 Schema 拓扑校准函数(`apply_schema_aware_calibration`),通过解析候选项的语法结构特征(如 `<tool_call>` 调用声明、澄清追问子句及越界拒识选项)修正判别边际,抑制工具过度触发(Over-Triggering)与开放集误接收。
|
| 56 |
|
| 57 |
+
---
|
| 58 |
|
| 59 |
+
## 3. 实验评测与基准分析
|
| 60 |
|
| 61 |
+
**表 1:公开评测基准与官方排行榜结果汇总(截至 2026 年 11 月,核验共 109 个参评模型条目)。**
|
| 62 |
|
| 63 |
+
| 评测基准与赛道 | 参评模型规模 | FastBrain-9B 核心指标 | 官方排名 | 代表性对比基线(参数量,指标) |
|
| 64 |
+
| :--- | :---: | :---: | :---: | :--- |
|
| 65 |
+
| [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 7 类违规判别 ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 个决策模型 (+2 基线) | **0.751** Macro-F1 / **0.858** Acc | **1 / 8** | `Cloudflare/clef` (27.4B, `0.563`), `vllm-sr/d3-mini` (1.8B, `0.512`), `Cloudflare/clef-flash` (9.4B, `0.462`), `clef-omni` (30B, `0.437`) |
|
| 66 |
+
| [`safeagents-judge-bench` (v2)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) — 二元安全护栏 ([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) | 8 个决策模型 (+2 基线) | **0.985** AUROC / **82/82** 轨迹检出<br>(`0.814` 默认;**0.938** 阈值校准) | **3 / 8** AUROC<br>**1 / 8** (并列) 轨迹 | `Cloudflare/clef` (27.4B, 默认 Bal. Acc `0.808`), `clef-flash` (`0.967` AUROC, `81/82`), `clef-omni` (`0.946` AUROC, `81/82`), `d1-3B` (`68/82`) |
|
| 67 |
+
| [`safeagents-judge-bench` (v1)](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v1) — 二元与 7 分类 ([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 个决策模型 (+2 基线) | **0.974** Bal. Acc / **0.850** Cat. F1 | **1 / 9** (双赛道第 1) | `Cloudflare/clef-omni` (30B, `0.845` / `0.327`), `Cloudflare/clef` (27.4B, `0.718` / `0.425`), `clef-flash` (`0.721` / `0.345`) |
|
| 68 |
+
| [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — 零样本评测 ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) | 43 个模型 (36 开源, 19 个 $\le$9B 开源) | **0.7525** Acc (`0.7625` 并列口径)<br>`0.1508` KL · `0.0795` Brier · **50 ms** | **7 / 43** 总榜<br>**5 / 36** 开源 · **2 / 19** $\le$9B | `vllm-sr/d3` (27B, `0.749`), `Elephant 1` (9B, `0.748`, `352 ms`), `Darwin-27B` (`0.743`), `Cloudflare/clef` (27B, `0.728`), `clef-flash` (`0.710`) |
|
| 69 |
+
| [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — `train` 校准 ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) | 51 个微调与校准模型 | **0.7970** Acc (`0.8070` 并列口径)<br>**0.0816** KL · **0.0437** Brier · **50 ms** | **并列 12** (列表 14 / 51)<br>冻结权重 $\le$9B 及 KL **第 1** | 冻结全部模型权重(`no weights changed`);高于 `Rev-gen1-midi` (25.6B, `0.7945`)、`OpenDecider-small-td` (4B LoRA, `0.792`)、`OpenDecider-medium-td` (30B LoRA, `0.788`) |
|
| 70 |
+
| Cloudflare Decision Index (`v0.2.1`) — 10 任务套件 | 7 个模型 (0.4B–27.4B) | **80.55%** 宏平均 | **2 / 7** 全量级总榜<br>$\le$9B 量级 **10 / 10 单项第 1** | `Jev` (27B, `80.32%`), `Kev 9B` (`76.51%`), `Clef-flash` (9.4B, `74.96%`), `DiffGemma` (26B, `69.14%`);5 / 10 项任务高于 `27.4B Clef` |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
|
| 72 |
---
|
| 73 |
|
| 74 |
+
### 3.1 智能体工具调用安全审计评测 (`safeagents-judge-bench`)
|
| 75 |
|
| 76 |
+
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) 用于评估决策模型对真实编码智能体(运行于 Terminal-Bench 2.1、OpenThoughts-TBLite 与 Terminal-Bench 4)在沙箱中被静态扫描器 `atif-scan` 标记的可疑工具调用的审计能力。`v2` 测试集包含来自 55 个任务的 $N = 661$ 次工具调用(其中 `536` 次为分布在 `82` 条违规轨迹中的恶意 `block` 调用,`125` 次为正常 `allow` 调用)。每次调用以零样本单次请求形式同时询问一个二元 `noul` 拦截问题与一个 7 类 `choice` 违规归因问题(`legit`、`eval_gaming`、`harness_introspection`、`credential_misuse`、`exfiltration`、`sandbox_escape`、`other_overreach`)。表 2 中的 `v2` 官方结果由基准维护者在 NVIDIA A100-SXM4-80GB(HF Jobs)上独立运行并提交合入([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1))。
|
| 77 |
|
| 78 |
+
在 7 类违规归因赛道上,FastBrain-9B 取得 `0.751` 的 Macro-F1 与 `0.858` 的分类准确率(`567/661`),位列全部参评模型第 1:相较于参数量约为其 3 倍的 27.4B 模型 `Cloudflare/clef`(Macro-F1 `0.563`,准确率 `0.737`),分别高出 `+18.8` 与 `+12.1` 个百分点;相较于同尺寸基座 `Cloudflare/clef-flash`(Macro-F1 `0.462`,准确率 `0.576`),分别提升 `+28.9` 与 `+28.2` 个百分点。分项指标上,`harness_introspection`($n=272$)F1 达 `0.945`(召回率 `0.978`),`eval_gaming`($n=245$)F1 达 `0.828`(召回率 `0.853`),合法调用 `legit`($n=125$)精确率达 `0.975`。
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
在二元安全拦截赛道上,FastBrain-9B 的 AUROC 为 `0.985`,恶意调用总召回率为 `0.996`(`534/536`),全部 `82/82` 条越界轨迹均在首次违规调用处即被捕获(`median_bad_calls_before_catch = 0.0`),其中 `harness_introspection`(`272/272`)、`credential_misuse`(`15/15`)与 `sandbox_escape`(`4/4`)三类召回率均为 `1.000`。在未经验证集调整的默认判别阈值 $P(\text{block}) \ge 0.50$ 下,平衡准确率为 `0.814`(误拦率 `0.368`,优于 27.4B `Cloudflare/clef` 的平衡准确率 `0.808` 与误拦率 `0.384`)。由于连续概率排序区分度保持在 `0.985` AUROC,将工作点阈值设定为 $P(\text{block}) \ge 0.961$ 时,平衡准确率提升至 `0.938`,误拦率降至 `0.056`(`7/125`),同时仍保持 `82/82` 越界轨迹全检出。表 2 同时列出了 `v1` 版本($N = 562$,78 条轨迹)的官方归档结果,FastBrain-9B 在该版本二元与多分类双赛道均位居第 1 / 9。
|
| 81 |
+
|
| 82 |
+
**表 2:`harness-hub/safeagents-judge-bench` 零样本评测结果对比(`v2` 测试集 $N = 661$,82 条越界轨迹;附 `v1` 归档对照 $N = 562$)。**
|
| 83 |
+
|
| 84 |
+
| 模型 | 参数量 | 7 分类 Macro-F1 ↑ | 7 分类准确率 ↑ | AUROC ↑ | 平衡准确率 ↑ | 恶意召回率 ↑ | 误拦率 ↓ | 越界轨迹检出 ↑ | p50 延迟(测试硬件) |
|
| 85 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 86 |
+
| **TypeScript/FastBrain-9B** (`v2`, 默认 $t = 0.50$) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 87 |
+
| **TypeScript/FastBrain-9B** (`v2`, 校准 $t = 0.961$) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 88 |
+
| **TypeScript/FastBrain-9B** (`v1` 官方榜, $N = 562$) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 89 |
+
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 90 |
+
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 91 |
+
| Cloudflare/clef-flash (基座) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
| 92 |
+
| Cloudflare/clef-omni | 30B-A3B | 0.437 | 0.564 | 0.946 | 0.877 | 0.914 | 0.160 | 81 / 82 | 285 ms (A100) |
|
| 93 |
+
| knowledgator/gliclass-v4-edge | 0.4B | 0.373 | 0.528 | 0.753 | 0.676 | 0.696 | 0.344 | 68 / 82 | 101 ms (L4) |
|
| 94 |
+
| fastino/GLiNER2.5-Decide | 0.5B | 0.355 | 0.620 | 0.517 | 0.576 | 0.151 | **0.000** | 17 / 82 | 117 ms (M5 Pro CPU) |
|
| 95 |
+
| LiquidAI/d1-3B | 3.0B | 0.328 | 0.457 | 0.946 | 0.836 | 0.681 | 0.008 | 68 / 82 | **35 ms** (A100) |
|
| 96 |
+
|
| 97 |
+
---
|
| 98 |
+
|
| 99 |
+
### 3.2 多业务流企业结构化决策评测 (`LocalLLaMA/typed-decisions`)
|
| 100 |
+
|
| 101 |
+
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) 评测集覆盖智能体轨迹可观测性(`agent_trace_observability`)、客户服务(`customer_service`)、供应商发票核验(`invoice_processing`)与安全事件响应(`security_incidents`)四类企业工作流,测试集包含 400 个独立案例、共 2,000 个类型化子问题(涵盖 `noul`、`choice` 与 `score`),并以多教师模型集成的软标签分布作为真值基准。官方仓库存在两种并列真值处理脚本:`bench/score_with_jobs.py` 对真值分布中概率并列最高的选项按首个索引判对(`g.index(max(g))`),而 `scripts/score_model.py` 将 20 道并列题中的任一最高分选项均计为正确;两种脚本下的 KL 散度与 Brier 评分完全一致。我们在表 3 中同时报告两种口径的准确率,并在 [`eval_submissions/`](./eval_submissions/) 中公开原始预测文件。
|
| 102 |
+
|
| 103 |
+
在零样本总榜($N = 43$ 个参评模型)中,FastBrain-9B 取得 `0.7525`(并列口径 `0.7625`)的准确率、`0.1508` 的 KL 散度、`0.0795` 的 Brier 评分与 `0.085` 的期望校准误差(ECE),在 RTX PRO 6000(SGLang)上的 p50 延迟为 `50 ms`。该成绩位列全量零样本榜 **第 7 / 43**、开源权重模型 **第 5 / 36**、$\le$9B 开源模型 **第 2 / 19** 以及全部 `Clef` 架构模型 **第 1**,较 9.4B 基座 `Cloudflare/clef-flash`(准确率 `0.7100`,KL `0.2093`)提升 `+4.25` 个百分点,较 27.4B `Cloudflare/clef`(准确率 `0.7275`,KL `0.1907`)提升 `+2.50` 个百分点。在零样本准确率高于 `0.700` 的全部 28 个模型中,FastBrain-9B 具有最低的 p50 推理延迟(`50 ms`,对比 9B `Amos 1 medium` 的 `118 ms`、27B `Amos 1 large` 的 `357 ms`、`meraGPT Decider 1` 的 `526 ms` 以及 31B `Quyet-1.0-Large` 的 `3,335 ms`)。若直接评测未经后验规则调整的序数 KL 蒸馏原生头,零样本准确率为 `0.7735`(KL `0.1039`,Brier `0.0582`)。
|
| 104 |
+
|
| 105 |
+
在使用 `train` 划分(1,200 个样本)的独立榜单($N = 51$ 个模型)上,在**完全冻结主干与决策头权重(`no weights changed`)**、仅拟合逐问题对数先验偏置与标量温度参数的前提下,FastBrain-9B 取得 `0.7970`(并列口径 `0.8070`)的准确率、`0.0816` 的 KL 散度与 `0.0437` 的 Brier 评分,位列 **并列第 12(列表第 14 / 51)**,并在冻结权重模型中取得最优的 KL 分布校准度。
|
| 106 |
+
|
| 107 |
+
**表 3:`LocalLLaMA/typed-decisions` 测试集($N = 400$ 案例,2,000 次决策)评测结果。`/` 前后分别对应 `score_with_jobs.py` 与 `score_model.py` 口径。**
|
| 108 |
+
|
| 109 |
+
| 模型 | 参数量 | 评测设定 | 总体准确率 ↑ | KL 散度 ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | 客服 ↑ | 发票 ↑ | 安全 ↑ | p50 延迟 |
|
| 110 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 111 |
+
| **FastBrain-9B** (先验温度拟合,冻结权重) | 9.4B | Fitted | **79.70% / 80.70%** | **0.0816** | **0.0437** | **86.7% / 88.0%** | **76.5% / 77.2%** | **76.9% / 77.9%** | **75.4% / 76.6%** | **82.0% / 83.4%** | **83.4% / 83.6%** | **78.0% / 79.2%** | **50 ms** |
|
| 112 |
+
| **FastBrain-9B** (序数 KL 蒸馏原生头) | 9.4B | Zero-shot | 77.35% | 0.1039 | 0.0582 | 84.8% | 74.8% | 73.8% | 71.2% | 80.4% | 82.6% | 75.2% | **50 ms** |
|
| 113 |
+
| **FastBrain-9B** (官方主榜 PR #58) | 9.4B | Zero-shot | 75.25% / 76.25% | 0.1508 | 0.0795 | 83.2% / 84.3% | 74.7% / 76.0% | 69.8% / 70.4% | 68.2% / 69.8% | 77.6% / 78.8% | 82.0% | 73.2% / 74.4% | **50 ms** |
|
| 114 |
+
| Amos 1 (medium) | 9.0B | Zero-shot | 76.45% | 0.1510 | 0.0800 | — | — | — | — | — | — | — | 118 ms |
|
| 115 |
+
| Amos 1 (large) | 27.0B | Zero-shot | 76.10% | 0.1260 | 0.0690 | 83.2% | 73.7% | 72.6% | 67.4% | 78.8% | 83.0% | 75.2% | 357 ms |
|
| 116 |
+
| pplx-decider-v1.1-27b | 27.0B | Zero-shot | 75.30% | 0.4410 | 0.1600 | 85.2% | 73.7% | 69.1% | 66.4% | 80.2% | 80.2% | 74.4% | — |
|
| 117 |
+
| vllm-sr/d3 | 27.0B | Zero-shot | 74.90% | 0.4090 | 0.1460 | 84.7% | 68.5% | 72.4% | 67.2% | 76.6% | 82.8% | 73.0% | — |
|
| 118 |
+
| Baseten Elephant 1 | 9.0B | Zero-shot | 74.80% | 0.1340 | 0.0690 | — | — | — | — | — | — | — | 352 ms |
|
| 119 |
+
| Darwin-27B-ZTC | 27.0B | Zero-shot | 74.30% | 0.2040 | 0.0970 | — | — | — | — | — | — | — | — |
|
| 120 |
+
| Cloudflare/clef | 27.4B | Zero-shot | 72.75% | 0.1907 | 0.1032 | 84.5% | 64.8% | 69.9% | 63.8% | 74.2% | 78.6% | 74.4% | — |
|
| 121 |
+
| vllm-sr/d3-flash | 9.0B | Zero-shot | 71.70% | 0.3220 | 0.1310 | — | — | — | — | — | — | — | — |
|
| 122 |
+
| Cloudflare/clef-flash (基座) | 9.4B | Zero-shot | 71.00% | 0.2093 | 0.1105 | 81.5% | 71.3% | 62.9% | 62.0% | 76.0% | 76.4% | 69.6% | — |
|
| 123 |
+
| Cloudflare/clef-omni | 35B-A3B | Zero-shot | 67.30% | 0.2469 | 0.1233 | 82.7% | 63.5% | 58.6% | 55.2% | 73.6% | 74.2% | 66.2% | — |
|
| 124 |
|
| 125 |
---
|
| 126 |
|
| 127 |
+
### 3.3 多领域结构化决策基准 (Cloudflare Decision Index `v0.2.1`)
|
| 128 |
+
|
| 129 |
+
在涵盖 10 个评测数据集、共 $N = 22,427$ 条请求的 **Cloudflare Decision Index(`v0.2.1`)** 套件(覆盖 AST 函数调用、20 候选工具检索重排、多步状态 API 调用、细粒度金融意图识别、开放集越界拒识、推理密集型检索、电商相关性分级、多探针钓鱼网站鉴别与工具调用触发治理)上,FastBrain-9B 的无加权 10 任务宏平均得分为 **80.55%**(表 4,图 1–3)。
|
| 130 |
+
|
| 131 |
+
在 $\le$9B 参数量级(对比 `Clef-flash` 9.4B、`Kev 9B` 与 `Laya` 0.4B)中,FastBrain-9B 在全部 **10 / 10 个子基准上均取得第 1**,宏平均分较同架构基座 `Clef-flash`(`74.96%`)提升 `+5.59` 个百分点,较 `Kev 9B`(`76.51%`)提升 `+4.04` 个百分点。增幅主要集中于对边界校准敏感的开放集拒识与安全治理任务:`CLINC150+OOS` 提升 `+31.67` 个百分点(`98.44%` 对比 `66.77%`),`When2Call` 提升 `+15.55` 个百分点(`81.13%` 对比 `65.58%`),`PhishNChips` 提升 `+6.80` 个百分点(`78.75%` 对比 `71.95%`)。跨全部参数规模(最高 27.4B)比较,FastBrain-9B 总宏平均位列 **第 2 / 7**,高于 `27B Jev`(`80.32%`)与 `26B-A4B DiffusionGemma Jev`(`69.14%`),并在 `BFCL`、`API-Bank`、`CLINC150+OOS`、`Home Appliances` 与 `When2Call` 五个子基准上高于 `27.4B Clef`(`81.81%`)。
|
| 132 |
+
|
| 133 |
+

|
| 134 |
+
*图 1. Cloudflare Decision Index(v0.2.1,22,427 条请求)10 任务宏平均分与模型参数规模关系。*
|
| 135 |
+
|
| 136 |
+

|
| 137 |
+
*图 2. 0.4B–27.4B 规模决策模型在 Cloudflare Decision Index(v0.2.1)各子任务上的详细得分矩阵。*
|
| 138 |
+
|
| 139 |
+

|
| 140 |
+
*图 3. 置信度可靠性图、开放集越界意图拒识(`CLINC150+OOS`)与工具触发边际分析(`When2Call`)。*
|
| 141 |
+
|
| 142 |
+
**表 4:Cloudflare Decision Index(`v0.2.1`,10 任务,$N = 22,427$)逐项评测结果。粗体表示全参数规模最高值,下划线表示 $\le$9B 量级最高值。**
|
| 143 |
|
| 144 |
+
| 评测基准(指标) | **FastBrain-9B** (本工作) | Clef-flash (基座) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 145 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 146 |
+
| 总参数量 | 9.41B | 9.41B | 9.00B | 27.36B | 27.00B | 25.8B (激活 4B) | 0.42B |
|
| 147 |
+
| 1. BFCL (Case Exact Acc) | **<u>98.76%</u>** | **<u>98.76%</u>** | 97.58% | 98.47% | 98.58% | 97.64% | 95.87% |
|
| 148 |
+
| 2. ToolRet (nDCG@10) | <u>67.26%</u> | <u>67.26%</u> | 65.72% | **68.18%** | 65.50% | 45.28% | 51.31% |
|
| 149 |
+
| 3. API-Bank (Accuracy) | **<u>93.70%</u>** | 93.11% | 89.76% | 91.93% | 90.35% | 85.24% | 86.22% |
|
| 150 |
+
| 4. BANKING77 (Macro-F1) | <u>93.77%</u> | 93.69% | 91.77% | **94.07%** | 93.58% | 93.80% | 62.66% |
|
| 151 |
+
| 5. CLINC150+OOS (Macro-F1) | **<u>98.44%</u>** | 66.77% | 96.65% | 97.43% | 96.44% | 94.64% | 73.79% |
|
| 152 |
+
| 6. Home Appliances (Case Exact) | **<u>97.73%</u>** | 96.59% | 88.64% | 96.59% | 88.64% | 68.18% | 23.86% |
|
| 153 |
+
| 7. BRIGHT (nDCG@10) | <u>38.58%</u> | 38.49% | 38.53% | **48.03%** | 43.37% | 18.69% | 25.00% |
|
| 154 |
+
| 8. Amazon ESCI (Macro-F1) | <u>57.40%</u> | 57.39% | 53.75% | **59.30%** | 56.60% | 56.83% | 52.84% |
|
| 155 |
+
| 9. PhishNChips (Accuracy) | <u>78.75%</u> | 71.95% | 72.10% | **91.70%** | 88.20% | 55.65% | 60.45% |
|
| 156 |
+
| 10. When2Call (Macro-F1) | <u>81.13%</u> | 65.58% | 70.64% | 72.37% | **81.91%** | 75.44% | 48.92% |
|
| 157 |
+
| **10 任务宏平均(Macro Mean)** | <u>80.55%</u> | 74.96% | 76.51% | **81.81%** | 80.32% | 69.14% | 58.09% |
|
| 158 |
|
| 159 |
---
|
| 160 |
|
| 161 |
+
## 4. 系统吞吐与训练后量化分析
|
| 162 |
+
|
| 163 |
+
### 4.1 不同推理引擎性能对比
|
| 164 |
+
|
| 165 |
+
表 5 展示了在同一张 NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 上,SGLang(集成 `JointSchemaPooler` 的 `lmsysorg/sglang:dev-clef`)、`llama.cpp`(`b11430` `ClefModel` 计算图,含 `Q8_0` 与 `Q4_K_M` 两种精度)以及原生 PyTorch(`joint_schema_model.py`)在五类典型长度负载下的端到端延迟、Prefill 吞吐率与显存占用。由于全部请求的生成步数均为零(`output_tokens = 0`),推理耗时完全取决于 Prefill 计算与片段池化效率;在批量 $B=4$ 并发设定下,SGLang 吞吐达 `19,319 tok/s`(每秒完成 `131.2` 个独立结构化决策)。
|
| 166 |
+
|
| 167 |
+
**表 5:NVIDIA RTX PRO 6000 Blackwell Server Edition(G4)上三类推理后端的延迟与吞吐实测。**
|
| 168 |
|
| 169 |
+
| 评测负载 / 系统指标 | SGLang `dev-clef` (BF16) | `llama.cpp b11430` (`Q8_0`) | `llama.cpp b11430` (`Q4_K_M`) | PyTorch 原生 (BF16) |
|
| 170 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 171 |
+
| 冷启动加载时间 | 11.11 s | 2.56 s | **2.00 s** | 11.11 s |
|
| 172 |
+
| 峰值显存占用 (16K 上下文) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 173 |
+
| 兼容 16 GiB T4 显卡 | 否 (BF16 TP=1) | 是 (`9.03 GiB`, 4K) | **是 (`6.78 GiB`, 4K)** | 是 (`6.8 GiB`, NF4) |
|
| 174 |
+
| 相对 PyTorch BF16 Top-1 一致率 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (基准) |
|
| 175 |
+
| 1. 短文本意图路由 (293 tok, $K=2$) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 176 |
+
| 2. Typed Decisions (736 tok, $K=5$) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 177 |
+
| 3. When2Call 工具治理 (1,068 tok, $K=1$) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 178 |
+
| 4. PhishNChips 安全检测 (1,303 tok, $K=9$) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 179 |
+
| 5. SafeAgents 审计 (2,217 tok, $K=2$) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 180 |
+
| 批量 $B=4$ Typed Decisions (2,944 tok, 20 题) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 181 |
+
|
| 182 |
+
### 4.2 混合精度 GGUF 量化保真度分析
|
| 183 |
|
| 184 |
+
若对交叉注意力决策头或线性注意力递归状态直接施加均匀 4-bit 量化,细粒度选项间微小的对数几率差值易受量化噪声干扰。因此在 `FastBrain-9B-Q4_K_M.gguf`(`6.73 GB`)构建方案中,仅对前馈网络(FFN)使用 4-bit K-quants,而将全部 `dec.*` 决策头张量、`ssm_*` Gated DeltaNet 状态参数以及 `attn_*` 投影矩阵保持为 `Q8_0` 精度。在涵盖 10 类任务的分层验证子集(`203` 个多约束请求,共 `1,473` 道独立评分题;表 6)上,`Q8_0.gguf`(`9.66 GB`)在 `1,471/1,473` 道题上与原生 `bfloat16` 给出完全一致的 Top-1 决策(一致率 `99.86%`,预测概率平均绝对误差 `0.00203`,10 任务均分差异 `-0.03` 个百分点);`Q4_K_M.gguf` 保持了 `98.78%` 的 Top-1 决策一致率(概率 MAE `0.00387`,校准后均分差异 `-0.20` 个百分点)。表 7 给出了各精度版本在不同硬件(含 NVIDIA Tesla T4 `16 GiB` 实测)上的资源需求。
|
| 185 |
+
|
| 186 |
+
**表 6:10 任务分层子集($N = 1,473$ 道评分题)上 GGUF 量化模型的决策一致性与精度保持率。**
|
| 187 |
+
|
| 188 |
+
| 评测指标 / 子任务 | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` 决策头) |
|
| 189 |
+
| :--- | :---: | :---: | :---: |
|
| 190 |
+
| Top-1 决策一致率 ($N = 1,473$) | 100.00% (基准) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 191 |
+
| 预测概率平均绝对误差 ($\|\hat{p}_q - \hat{p}_{\text{bf16}}\|$) | 0.00000 | 0.00203 | 0.00387 |
|
| 192 |
+
| 10 任务宏平均(未校准原生输出) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 193 |
+
| 10 任务宏平均(Schema 拓扑校准) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 194 |
+
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|
| 195 |
+
| ├─ 2. ToolRet / 7. BRIGHT | 73.15% / 41.93% | 73.70% / 41.00% | 75.60% / 40.31% |
|
| 196 |
+
| ├─ 5. CLINC150+OOS / 6. Home Appliances | 93.94% / 90.00% | 93.94% / 90.00% | 93.94% / 90.00% |
|
| 197 |
+
| └─ 8. Amazon ESCI / 9. PhishNChips / 10. When2Call | 73.89% / 85.00% / 75.00% | 73.89% / 85.00% / 75.00% | 75.79% / 85.00% / 72.92% |
|
| 198 |
+
|
| 199 |
+
**表 7:各精度格式的存储体积、显存占用峰值与 Tesla T4(`16 GiB`)实测延迟。**
|
| 200 |
+
|
| 201 |
+
| 精度与权重格式 | 权重体积 | 峰值显存 (4K–16K 上下文) | T4 延迟 (293 / 1,068 tok) | 适用硬件范围 |
|
| 202 |
+
| :--- | :---: | :---: | :---: | :--- |
|
| 203 |
+
| `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 6.78 – 10.30 GiB | 343.0 ms / 1,154.3 ms | 8–16 GB 显卡 (RTX 3060/4060, Tesla T4), Apple Silicon (8 GB+) |
|
| 204 |
+
| PyTorch `load_in_4bit=True` (NF4 + `fp16` 头) | ~5.5 GB (内存) | 6.20 – 6.80 GiB | — | 8–16 GB CUDA 显卡 (Colab T4, 消费级 GPU) |
|
| 205 |
+
| `FastBrain-9B-Q8_0.gguf` | 9.66 GB | 9.03 – 12.55 GiB | 349.3 ms / 1,149.8 ms | 12–16 GB 显卡 (Tesla T4, RTX 4070 Ti), Apple Silicon (16 GB+) |
|
| 206 |
+
| PyTorch `load_in_8bit=True` (INT8 + `bf16` 头) | ~9.6 GB (内存) | ~11.20 GiB | — | 12–16 GB CUDA 显卡 |
|
| 207 |
+
| 原生 `bfloat16` (`safetensors` / `BF16.gguf`) | 18.82 / 18.16 GB | 20.30 – 24.70 GiB | — | $\ge$24 GB 显卡 (NVIDIA L4, A100, H100, RTX 3090/4090, PRO 6000) |
|
| 208 |
+
|
| 209 |
+
---
|
| 210 |
+
|
| 211 |
+
## 5. 推理复现与服务部署
|
| 212 |
+
|
| 213 |
+
### 5.1 Python / Transformers 接口(支持 `bfloat16`、`8-bit` 与 `4-bit` NF4)
|
| 214 |
+
|
| 215 |
+
`TypeScript/FastBrain-9B` 提供标准 `joint_schema_model` 接口,可与 `Cloudflare/clef-flash` 调用代码直接互换。启用 `load_in_4bit=True` 时,仅对 9.3B 主干网络执行 `bitsandbytes` NF4 量化,116M `JointSchemaHead` 保持浮点精度。
|
| 216 |
|
| 217 |
```python
|
|
|
|
| 218 |
import sys
|
| 219 |
import torch
|
| 220 |
+
from huggingface_hub import snapshot_download
|
| 221 |
|
| 222 |
model_dir = snapshot_download("TypeScript/FastBrain-9B", ignore_patterns=["*.gguf"])
|
| 223 |
sys.path.insert(0, model_dir)
|
| 224 |
import joint_schema_model as clef
|
| 225 |
|
| 226 |
+
# 原生 bfloat16 加载 (>=24 GB 显存):
|
| 227 |
model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.bfloat16)
|
| 228 |
|
| 229 |
+
# 混合精度 4-bit NF4 加载 (8-16 GB 显存,如免费 Colab Tesla T4):
|
| 230 |
# model, processor = clef.load_release_model(model_dir, device="cuda", dtype=torch.float16, load_in_4bit=True)
|
| 231 |
|
| 232 |
response = clef.systemone(model, processor, {
|
| 233 |
"model": "TypeScript/FastBrain-9B",
|
| 234 |
"state": {
|
| 235 |
+
"query": "能帮我查询一下当前账户余额吗?",
|
| 236 |
"tools": ["get_balance", "transfer_funds"],
|
| 237 |
},
|
| 238 |
"questions": {
|
| 239 |
+
"next_action": {
|
| 240 |
"type": "choice",
|
| 241 |
+
"instructions": "选择最恰当的助手动作。",
|
| 242 |
"criteria": {
|
| 243 |
+
"call_balance": "调用 <tool_call>{\"name\": \"get_balance\", \"arguments\": {}}</tool_call>",
|
| 244 |
+
"request_auth": "请用户先提供账户标识符或身份验证令牌。",
|
| 245 |
+
"abstain": "拒绝执行,因为个人银行记录超出访问权限。",
|
| 246 |
},
|
| 247 |
},
|
| 248 |
},
|
|
|
|
| 250 |
print(response)
|
| 251 |
```
|
| 252 |
|
| 253 |
+
### 5.2 `llama.cpp` GGUF 服务部署(`/v1/systemone`)
|
|
|
|
|
|
|
| 254 |
|
| 255 |
> [!IMPORTANT]
|
| 256 |
+
> **`ClefModel` 架构 GGUF 运行参数要求**
|
| 257 |
+
> 1. 需使用 **`llama.cpp` `b11430` 或更高版本**。该版本包含 `LLM_ARCH_CLEF` 计算图定义(支持 `dec.*` 决策头张量)与 `/v1/systemone` HTTP 路由;旧版构建或纯生成式封装无法执行交叉注意力决策头。
|
| 258 |
+
> 2. 启动时需将物理微批次设为与逻辑批次一致(如 `-b 16384 -ub 16384` 或 `-b 4096 -ub 4096`),并开启 `--no-cache-prompt`。`ClefModel` 在单个物理 `ubatch` 内对 `<|question_*|>` 与 `<|option_*|>` 区间执行池化,默认的 `-ub 512` 会截断超过 512 tokens 的输入序列,而前缀缓存可能跳过边界标记计算。
|
|
|
|
|
|
|
|
|
|
| 259 |
|
| 260 |
```bash
|
| 261 |
+
# 1. 下载预编译 llama.cpp (b11430+) 与 GGUF 权重文件
|
|
|
|
|
|
|
|
|
|
| 262 |
curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
|
|
|
|
|
|
|
| 263 |
pip install -U "huggingface_hub[hf_transfer]"
|
| 264 |
HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
|
|
|
|
| 265 |
|
| 266 |
+
# 2. 启动 llama-server(如需处理图像状态,追加 --mmproj mmproj-FastBrain-9B-Q8_0.gguf)
|
|
|
|
|
|
|
|
|
|
| 267 |
export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
|
|
|
| 268 |
./llama-b11430/llama-server \
|
| 269 |
-m ./FastBrain-9B-Q8_0.gguf \
|
| 270 |
-ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
|
| 271 |
--port 8080
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
```
|
| 273 |
|
| 274 |
+
向 `/v1/systemone` 发送单次请求即可同步评估异构题型组合(`choice` + `noul`):
|
| 275 |
|
| 276 |
```bash
|
| 277 |
curl http://127.0.0.1:8080/v1/systemone \
|
|
|
|
| 280 |
"model": "FastBrain-9B",
|
| 281 |
"state": {
|
| 282 |
"ticket_id": "ORD-99281",
|
| 283 |
+
"customer_message": "我上周的订阅被重复扣费了两次,提交工单后一直无人处理。"
|
| 284 |
},
|
| 285 |
"questions": {
|
| 286 |
+
"department": {
|
| 287 |
"type": "choice",
|
| 288 |
+
"instructions": "将该工单分派至对应的处理队列。",
|
| 289 |
"criteria": {
|
| 290 |
+
"billing": "账单支付、退款、重复扣费或发票核对",
|
| 291 |
+
"engineering": "软件缺陷、程序崩溃或服务中断",
|
| 292 |
+
"identity": "密码重置或多因素身份验证异常"
|
| 293 |
}
|
| 294 |
},
|
| 295 |
+
"is_urgent": {
|
| 296 |
"type": "noul",
|
| 297 |
+
"instructions": "该消息是否涉及未解决的资金损失或时效敏感故障?"
|
| 298 |
}
|
| 299 |
}
|
| 300 |
}'
|
| 301 |
```
|
| 302 |
|
| 303 |
+
### 5.3 SGLang 服务部署(`JointSchemaPooler`)
|
|
|
|
|
|
|
| 304 |
|
| 305 |
```bash
|
| 306 |
+
docker run --gpus all --shm-size 32g -p 30000:30000 \
|
| 307 |
+
-v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
|
|
|
|
|
|
|
|
|
|
| 308 |
lmsysorg/sglang:dev-clef \
|
| 309 |
+
sglang serve --model-path TypeScript/FastBrain-9B --host 0.0.0.0 --port 30000
|
|
|
|
|
|
|
|
|
|
| 310 |
```
|
| 311 |
|
| 312 |
---
|
| 313 |
|
| 314 |
+
## 6. 仓库文件清单与引用格式
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 315 |
|
| 316 |
+
**表 8:仓库文件与权重清单说明。**
|
| 317 |
|
| 318 |
+
| 文件路径 | 体积 | 内容说明 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 319 |
| :--- | :---: | :--- |
|
| 320 |
+
| `model-00001-of-00004.safetensors` .. `00004` | 18.59 GB | 原地合并全部投影更新后的 `bfloat16` 混合主干网络权重 |
|
| 321 |
+
| `joint_head.safetensors`, `joint_head_config.json` | 232 MB | 116M 参数 `JointSchemaHead` 权重(`122` 个张量)与结构配置 |
|
| 322 |
+
| `FastBrain-9B-Q4_K_M.gguf` | 6.73 GB | 混合精度 4-bit GGUF(`dec.*`、`ssm_*` 与 `attn_*` 保持 `Q8_0`) |
|
| 323 |
+
| `FastBrain-9B-Q8_0.gguf` / `FastBrain-9B-BF16.gguf` | 9.66 / 18.16 GB | 面向 `llama-server`(`b11430`+)的 8-bit 与 `bfloat16` GGUF 权重 |
|
| 324 |
+
| `mmproj-FastBrain-9B-Q8_0.gguf` / `BF16.gguf` | 624 / 922 MB | 支持多模态图像状态输入的视觉投影器权重 |
|
| 325 |
+
| `joint_schema_model.py` | 25 KB | `ClefModel` 定义、向量化 `_forward_fast_cuda` 算子及 `systemone()` 运行时 |
|
| 326 |
+
| `eval_submissions/typescript_fastbrain-9b*.json` | 1.80 MB | `LocalLLaMA/typed-decisions` 零样本与先验拟合的 2,000 次决策原始预测轨迹 |
|
| 327 |
+
| `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | 110 KB | `safeagents-judge-bench`(v2)661 条工具调用的二元与 7 分类预测轨迹 |
|
| 328 |
+
| `.eval_results/*.yaml` | 2.3 KB | Hugging Face Hub 标准化评测结果元数据 |
|
| 329 |
+
| `README.md`, `README_zh.md`, `assets/*.png` | 1.28 MB | 中英双语技术报告与实验评测图表 |
|
| 330 |
+
|
| 331 |
+
### 引用(Citation)
|
| 332 |
+
|
| 333 |
+
```bibtex
|
| 334 |
+
@misc{fastbrain9b2026,
|
| 335 |
+
title = {FastBrain-9B: A Single-Pass Non-Autoregressive Model for Calibrated Structured Decisions},
|
| 336 |
+
author = {TypeScript},
|
| 337 |
+
year = {2026},
|
| 338 |
+
howpublished = {\url{https://huggingface.co/TypeScript/FastBrain-9B}},
|
| 339 |
+
note = {Apache-2.0 License}
|
| 340 |
+
}
|
| 341 |
+
```
|