mkzero commited on
Commit
d194699
·
verified ·
1 Parent(s): d32bab5

docs: sync README with official GitHub repo https://github.com/mkeco/fast-decider-149m

Browse files
Files changed (1) hide show
  1. README.md +142 -70
README.md CHANGED
@@ -16,83 +16,155 @@ library_name: transformers
16
  metrics:
17
  - accuracy
18
  - brier_score
19
- model-index:
20
- - name: FastDecider-149M
21
- results:
22
- - task:
23
- type: text-classification
24
- name: Agent Decision Ranking
25
- dataset:
26
- name: JevBench v1.2.11
27
- type: jevbench
28
- metrics:
29
- - type: accuracy
30
- value: 100.0
31
- name: Easy Tier Accuracy
32
- - type: accuracy
33
- value: 44.14
34
- name: Hard Tier Accuracy
35
- - type: accuracy
36
- value: 68.06
37
- name: Original Tier Accuracy
38
  ---
39
 
40
- # FastDecider-149M (v1.3.0): Sub-15ms System-1 Neural Decision Engine
41
 
42
- **FastDecider-149M** is a high-speed (14.93ms P50 latency), 149M-parameter neural Cross-Encoder designed for autonomous agents, browser DOM interaction, API routing, and high-throughput operational decisions.
 
 
43
 
44
- ## Benchmark Performance
 
 
 
 
 
 
 
45
 
46
- - **JevBench Easy Tier**: **100.0%** (48/48)
47
- - **JevBench Hard Tier**: **44.14%** (49/111)
48
- - **JevBench Original Tier**: **68.06%** (49/72)
49
- - **JevBench Composite Score**: **78.15 / 100** (Rank #1 Composite)
50
- - **P50 Latency**: **14.93 ms** on GPU
51
- - **Cost**: **$0.0012** per 1,000 decisions
52
- - **6 Core Business Pillars**:
53
- - Browser DOM Control: 100.0% (50/50)
54
- - API Tool Dispatch: 98.0% (49/50)
55
- - E-Commerce Brands: 100.0% (50/50)
56
- - E-Commerce Specs: 100.0% (50/50)
57
- - E-Commerce Category: 94.0% (47/50)
58
- - Legal Contracts: 96.0% (48/50)
59
 
60
- ## Usage with Transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
 
62
  ```python
63
- import torch
64
- import torch.nn.functional as F
65
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
66
-
67
- model_id = "mkzero/FastDecider-149M"
68
- tokenizer = AutoTokenizer.from_pretrained(model_id)
69
- model = AutoModelForSequenceClassification.from_pretrained(
70
- model_id, dtype=torch.bfloat16
71
- ).to("cuda")
72
- model.eval()
73
-
74
- context = "User wishes to cancel order #4821 within 2 hours of checkout."
75
- instruction = "Select the appropriate API handler."
76
- options = {
77
- "cancel_order": "orders.cancel_immediate",
78
- "request_return": "returns.create_ticket",
79
- "contact_support": "support.live_agent"
80
- }
81
-
82
- pairs = [(f"Context:
83
- {context}
84
-
85
- Instruction:
86
- {instruction}", f"Option {k}: {v}") for k, v in options.items()]
87
- inputs = tokenizer(pairs, padding=True, truncation=True, max_length=2048, return_tensors="pt").to("cuda")
88
-
89
- with torch.no_grad():
90
- logits = model(**inputs).logits.squeeze(-1).float()
91
- probs = F.softmax(logits, dim=-1).cpu().numpy()
92
-
93
- keys = list(options.keys())
94
- print("Decision:", keys[probs.argmax()])
95
  ```
96
 
97
- ## GitHub Repository
98
- Source code, benchmarks, and microservice serving: [https://github.com/mkzero/FastDecider-149M](https://github.com/mkzero/FastDecider-149M)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  metrics:
17
  - accuracy
18
  - brier_score
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  ---
20
 
21
+ # FastDecider-149M (v1.3.0)
22
 
23
+ <p align="center">
24
+ <strong>Sub-15ms Neural Decision Engine for Autonomous Agents & Industrial Automation</strong>
25
+ </p>
26
 
27
+ <p align="center">
28
+ <a href="https://github.com/mkeco/fast-decider-149m/releases/tag/v1.3.0"><img src="https://img.shields.io/badge/Release-v1.3.0-blue.svg" alt="Release"></a>
29
+ <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License"></a>
30
+ <img src="https://img.shields.io/badge/Parameters-149M_(285MB)-success.svg" alt="Parameters">
31
+ <img src="https://img.shields.io/badge/P50_Latency-14.93ms-brightgreen.svg" alt="Latency">
32
+ <img src="https://img.shields.io/badge/JevBench_Composite-78.15-orange.svg" alt="JevBench Composite">
33
+ <img src="https://img.shields.io/badge/Framework-PyTorch_%7C_ModernBERT-blueviolet.svg" alt="Framework">
34
+ </p>
35
 
36
+ <p align="center">
37
+ <a href="README.md">English</a> | <a href="README_zh.md">简体中文</a>
38
+ </p>
 
 
 
 
 
 
 
 
 
 
39
 
40
+ ---
41
+
42
+ ## Overview
43
+
44
+ **FastDecider-149M** is a lightweight (149M parameter, ~285MB VRAM) neural cross-encoder purpose-built as a **System-1 (fast-thinking) decision engine** for autonomous AI agents, web/DOM automation, API routing, and high-frequency dispatch.
45
+
46
+ While general-purpose LLMs (7B–70B) suffer from 300–2,000ms latency and autoregressive token generation overhead, FastDecider-149M leverages full bidirectional cross-attention across context, instructions, and candidate options. It delivers deterministic, calibrated decision probabilities in **under 15 milliseconds** at **$0.0012 per 1,000 decisions**—over 50x faster and 500x cheaper than commercial generative LLMs.
47
+
48
+ ---
49
+
50
+ ## Benchmark & Performance
51
+
52
+ ### Official JevBench Leaderboard Ranking
53
+
54
+ FastDecider-149M achieves a composite score of **78.15** on the standard JevBench benchmark (231 tasks), leading the composite efficiency leaderboard by pairing solid operational accuracy with state-of-the-art speed and cost efficiency.
55
+
56
+ <p align="center">
57
+ <img src="docs/images/leaderboard_verified.png" width="880" alt="JevBench Official Benchmark Leaderboard">
58
+ </p>
59
+
60
+ ### Detailed Scorecard
61
+
62
+ | Evaluation Axis | Score / Metric | Industry Tier | Notes |
63
+ | :--- | :--- | :--- | :--- |
64
+ | **Easy Tier Accuracy** | **100.00%** (48/48) | Perfect | Full coverage on intent, extraction, tool pick |
65
+ | **Hard Tier Accuracy** | **44.14%** (49/111) | Competitive | 100% on complex routing & business tradeoffs |
66
+ | **Original Baseline** | **68.06%** (49/72) | Solid | Natural language decision benchmarks |
67
+ | **Intelligence Score** | **65.35 / 100** | High Utility | 149M bidirectional representation capacity |
68
+ | **Speed Score** | **99.22 / 100** | Top 0.1% | P50 = **14.93 ms**, P99 = 17.8 ms |
69
+ | **Cost Score** | **97.62 / 100** | Top 0.1% | **$0.0012** / 1k decisions |
70
+ | **Calibration Score** | **58.92 / 100** | Calibrated | Brier score = 0.6751, monotonic probabilities |
71
+ | ★ **JevBench Composite** | **78.15 / 100** | **Rank #1** | Geometric mean across Intel, Calib, Speed, Cost |
72
+
73
+ ### Industrial Domain Verification (6 Pillars)
74
+
75
+ FastDecider-149M was evaluated across six real-world industrial automation tasks:
76
+
77
+ ```text
78
+ ======================================================================
79
+ INDUSTRIAL DOMAIN | ACCURACY | TEST SET
80
+ ======================================================================
81
+ Browser DOM Control (XPath / Elements) | 100.0% | 50/50
82
+ API Dispatch / Function Routing | 98.0% | 49/50
83
+ E-Commerce Brand Extraction | 100.0% | 50/50
84
+ E-Commerce Specs & Attributes | 100.0% | 50/50
85
+ E-Commerce Category Classification | 94.0% | 47/50
86
+ Legal Contract Clause Matching | 96.0% | 48/50
87
+ ======================================================================
88
+ ```
89
+
90
+ ---
91
+
92
+ ## Quickstart
93
+
94
+ ### Installation
95
+
96
+ ```bash
97
+ git clone https://github.com/mkeco/fast-decider-149m.git
98
+ cd fast_decider_149m
99
+ pip install -e .
100
+ ```
101
+
102
+ ### Python API
103
 
104
  ```python
105
+ from fast_decider import FastDecider
106
+
107
+ # Initialize the 149M decision engine (BF16, ~285MB VRAM)
108
+ decider = FastDecider("mkzero/FastDecider-149M")
109
+
110
+ # Example: Real-time Web Browser DOM Action Selection
111
+ result = decider.decide(
112
+ context="User wants to complete purchase. Cart contains 2 items. Page displays modal with coupon prompt.",
113
+ instruction="Select the primary checkout button to proceed.",
114
+ options={
115
+ "checkout_btn": "button.btn-primary-checkout",
116
+ "coupon_apply": "button.btn-apply-coupon",
117
+ "continue_shop": "a.link-continue-shopping",
118
+ "clear_cart": "button.btn-danger-clear"
119
+ }
120
+ )
121
+
122
+ print(f"Selected Action : {result.best_option}")
123
+ print(f"Confidence : {result.confidence:.2%}")
124
+ print(f"Inference Time : {result.latency_ms:.2f} ms")
125
+ ```
126
+
127
+ ### High-Throughput HTTP Microservice (FastAPI)
128
+
129
+ ```bash
130
+ # Launch the ASGI microservice on port 8000
131
+ python -m uvicorn fast_decider.server:app --host 0.0.0.0 --port 8000
 
 
 
 
 
132
  ```
133
 
134
+ ---
135
+
136
+ ## Reproducibility & Benchmark Suite
137
+
138
+ All benchmark figures reported above are fully reproducible with the provided scripts:
139
+
140
+ ```bash
141
+ # 1. Run official JevBench CLI evaluation (231 tasks)
142
+ bash benchmarks/run_jevbench_official.sh
143
+
144
+ # 2. Run the 6 industrial domain verification
145
+ python benchmarks/verify_6_pillars.py
146
+
147
+ # 3. Benchmark latency and throughput
148
+ python benchmarks/benchmark_latency.py
149
+
150
+ # 4. Run test suite
151
+ pytest tests/
152
+ ```
153
+
154
+ ---
155
+
156
+ ## Support the Project / Buy Me a Coffee ☕
157
+
158
+ If FastDecider-149M is helpful to your work, agent framework, or research, feel free to sponsor or buy the author a cup of coffee!
159
+
160
+ <p align="center">
161
+ <img src="docs/images/buy_me_a_coffee.jpg" width="260" alt="Buy Me a Coffee">
162
+ <br>
163
+ <em>Thank you for supporting open-source software!</em>
164
+ </p>
165
+
166
+ ---
167
+
168
+ ## License
169
+
170
+ This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.