Update README.md
Browse files
README.md
CHANGED
|
@@ -24,41 +24,22 @@ tags:
|
|
| 24 |
- long-context
|
| 25 |
---
|
| 26 |
|
| 27 |
-
<
|
| 28 |
-
|
| 29 |
-
<
|
| 30 |
-
<
|
| 31 |
-
</
|
| 32 |
|
| 33 |
-
|
| 34 |
|
| 35 |
-
|
| 36 |
|
| 37 |
-
|
| 38 |
|
| 39 |
-
<p>
|
| 40 |
-
<img src="https://img.shields.io/badge/
|
| 41 |
-
<img src="https://img.shields.io/badge/Decoder-3.21_bpw-111111?style=for-the-badge">
|
| 42 |
-
<img src="https://img.shields.io/badge/PPL_Delta-%2B0.02%25-111111?style=for-the-badge">
|
| 43 |
-
<img src="https://img.shields.io/badge/Top--1-93.2%25-111111?style=for-the-badge">
|
| 44 |
</p>
|
| 45 |
|
| 46 |
-
<p>
|
| 47 |
-
<img src="https://img.shields.io/badge/Context-262K-111111?style=flat-square">
|
| 48 |
-
<img src="https://img.shields.io/badge/vLLM-Production_Ready-111111?style=flat-square">
|
| 49 |
-
<img src="https://img.shields.io/badge/MTP-Speculative_Decoding-111111?style=flat-square">
|
| 50 |
-
<img src="https://img.shields.io/badge/Tool_Calling-Yes-111111?style=flat-square">
|
| 51 |
-
<img src="https://img.shields.io/badge/License-Apache_2.0-111111?style=flat-square">
|
| 52 |
-
</p>
|
| 53 |
-
|
| 54 |
-
**[OrcaRouter](https://www.orcarouter.ai)** ·
|
| 55 |
-
**[Models](https://www.orcarouter.ai/models)** ·
|
| 56 |
-
**[GitHub](https://github.com/Continuum-AI-Corp)** ·
|
| 57 |
-
**[Discord](https://discord.gg/yAh6Tex6kx)** ·
|
| 58 |
-
**[X](https://x.com/OrcaRouter)**
|
| 59 |
-
|
| 60 |
-
</div>
|
| 61 |
-
|
| 62 |
---
|
| 63 |
|
| 64 |
> ## 27B reasoning. 12.3 GB.
|
|
@@ -89,7 +70,7 @@ The resulting checkpoint provides:
|
|
| 89 |
|
| 90 |
---
|
| 91 |
|
| 92 |
-
# At a glance
|
| 93 |
|
| 94 |
| Metric | BF16 | OrcaSAQ2 |
|
| 95 |
|---|---:|---:|
|
|
@@ -103,19 +84,13 @@ The resulting checkpoint provides:
|
|
| 103 |
| Mean KLD | — | **0.031** |
|
| 104 |
| Context | 262K | **262K** |
|
| 105 |
|
| 106 |
-
<
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
### The point is not 3-bit.
|
| 111 |
-
|
| 112 |
-
### **The point is what survives at 3-bit.**
|
| 113 |
-
|
| 114 |
-
</div>
|
| 115 |
|
| 116 |
---
|
| 117 |
|
| 118 |
-
# BF16 fidelity
|
| 119 |
|
| 120 |
All numbers below are measured using **these exact OrcaSAQ2 weights** against the BF16 reference through the same evaluation path.
|
| 121 |
|
|
@@ -161,9 +136,9 @@ OrcaSAQ2
|
|
| 161 |
|
| 162 |
---
|
| 163 |
|
| 164 |
-
# Long-horizon agents
|
| 165 |
|
| 166 |
-
## Low-bit fidelity matters more as the horizon grows.
|
| 167 |
|
| 168 |
Short benchmarks can hide small degradation.
|
| 169 |
|
|
@@ -225,7 +200,7 @@ For agent models, both matter.
|
|
| 225 |
|
| 226 |
---
|
| 227 |
|
| 228 |
-
# Long-horizon performance
|
| 229 |
|
| 230 |
Long-horizon evaluations should use the same:
|
| 231 |
|
|
@@ -242,7 +217,7 @@ Scores produced by materially different agent stacks should not be treated as di
|
|
| 242 |
|
| 243 |
---
|
| 244 |
|
| 245 |
-
# Architecture
|
| 246 |
|
| 247 |
| | |
|
| 248 |
|---|---|
|
|
@@ -264,9 +239,9 @@ Scores produced by materially different agent stacks should not be treated as di
|
|
| 264 |
|
| 265 |
---
|
| 266 |
|
| 267 |
-
# Production serving
|
| 268 |
|
| 269 |
-
## Up to 90.1 tok/s single-stream on a 16 GB GPU
|
| 270 |
|
| 271 |
Measured under a **15.7 GiB GPU memory cap**.
|
| 272 |
|
|
@@ -302,7 +277,7 @@ For highly batched workloads, benchmark both configurations.
|
|
| 302 |
|
| 303 |
---
|
| 304 |
|
| 305 |
-
# 27B on a 16 GB GPU
|
| 306 |
|
| 307 |
OrcaSAQ2's checkpoint is **12.3 GB**.
|
| 308 |
|
|
@@ -336,7 +311,7 @@ The model architecture supports up to **262K context**.
|
|
| 336 |
|
| 337 |
---
|
| 338 |
|
| 339 |
-
# Built for agents
|
| 340 |
|
| 341 |
### Long-horizon execution
|
| 342 |
|
|
@@ -366,9 +341,9 @@ A **12.3 GB** checkpoint designed around practical inference hardware.
|
|
| 366 |
|
| 367 |
---
|
| 368 |
|
| 369 |
-
# Quickstart
|
| 370 |
|
| 371 |
-
## Install
|
| 372 |
|
| 373 |
```bash
|
| 374 |
pip install -U vllm huggingface_hub
|
|
@@ -378,7 +353,7 @@ pip install git+https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel
|
|
| 378 |
|
| 379 |
---
|
| 380 |
|
| 381 |
-
## Download
|
| 382 |
|
| 383 |
```bash
|
| 384 |
hf download orcarouter/OrcaSAQ2-27B \
|
|
@@ -387,7 +362,7 @@ hf download orcarouter/OrcaSAQ2-27B \
|
|
| 387 |
|
| 388 |
---
|
| 389 |
|
| 390 |
-
## Serve with vLLM
|
| 391 |
|
| 392 |
```bash
|
| 393 |
vllm serve ./OrcaSAQ2-27B \
|
|
@@ -401,7 +376,7 @@ vllm serve ./OrcaSAQ2-27B \
|
|
| 401 |
|
| 402 |
---
|
| 403 |
|
| 404 |
-
# OpenAI-compatible API
|
| 405 |
|
| 406 |
```python
|
| 407 |
from openai import OpenAI
|
|
@@ -426,7 +401,7 @@ print(response.choices[0].message.content)
|
|
| 426 |
|
| 427 |
---
|
| 428 |
|
| 429 |
-
# Recommended sampling
|
| 430 |
|
| 431 |
```text
|
| 432 |
temperature = 1.0
|
|
@@ -440,23 +415,19 @@ For agent deployments, benchmark against the actual tool schema, context distrib
|
|
| 440 |
|
| 441 |
---
|
| 442 |
|
| 443 |
-
# Evaluation philosophy
|
| 444 |
|
| 445 |
A low-bit reasoning model should not be judged by checkpoint size alone.
|
| 446 |
|
| 447 |
We look at the intersection of:
|
| 448 |
|
| 449 |
-
<
|
| 450 |
-
|
| 451 |
-
### **Footprint × BF16 Fidelity × Capability × Long-Horizon Stability × Serving Performance**
|
| 452 |
-
|
| 453 |
-
</div>
|
| 454 |
|
| 455 |
A useful low-bit model must remain useful after compression.
|
| 456 |
|
| 457 |
---
|
| 458 |
|
| 459 |
-
# Why perplexity alone is not enough
|
| 460 |
|
| 461 |
Perplexity is useful and reproducible.
|
| 462 |
|
|
@@ -482,7 +453,7 @@ That is why OrcaSAQ2 reports BF16 fidelity metrics alongside downstream and long
|
|
| 482 |
|
| 483 |
---
|
| 484 |
|
| 485 |
-
# Method
|
| 486 |
|
| 487 |
OrcaSAQ2 uses a proprietary **sensitivity-aware mixed-precision quantization system** developed by OrcaRouter.
|
| 488 |
|
|
@@ -492,7 +463,7 @@ Detailed quantization methodology, calibration strategy, precision allocation an
|
|
| 492 |
|
| 493 |
---
|
| 494 |
|
| 495 |
-
# Limitations
|
| 496 |
|
| 497 |
- OrcaSAQ2 inherits the capabilities, biases and limitations of Qwen3.8-27B.
|
| 498 |
- Quantization is not mathematically lossless.
|
|
@@ -506,7 +477,7 @@ Detailed quantization methodology, calibration strategy, precision allocation an
|
|
| 506 |
|
| 507 |
---
|
| 508 |
|
| 509 |
-
# Open source from OrcaRouter
|
| 510 |
|
| 511 |
### [OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)
|
| 512 |
|
|
@@ -520,15 +491,11 @@ Record, replay, fork and debug AI-agent runs.
|
|
| 520 |
|
| 521 |
Self-hosted multi-model AI infrastructure.
|
| 522 |
|
| 523 |
-
<
|
| 524 |
-
|
| 525 |
-
### **Open model. Open harness. Open bill.**
|
| 526 |
-
|
| 527 |
-
</div>
|
| 528 |
|
| 529 |
---
|
| 530 |
|
| 531 |
-
# Citation
|
| 532 |
|
| 533 |
```bibtex
|
| 534 |
@misc{qwen38,
|
|
@@ -542,7 +509,7 @@ Self-hosted multi-model AI infrastructure.
|
|
| 542 |
|
| 543 |
---
|
| 544 |
|
| 545 |
-
# License
|
| 546 |
|
| 547 |
**Apache-2.0**
|
| 548 |
|
|
@@ -554,16 +521,14 @@ Quantization does not change the underlying license obligations.
|
|
| 554 |
|
| 555 |
---
|
| 556 |
|
| 557 |
-
<
|
| 558 |
-
|
| 559 |
-
# One Gateway. Every Model.
|
| 560 |
|
| 561 |
-
|
| 562 |
|
| 563 |
-
|
| 564 |
-
|
| 565 |
-
|
| 566 |
-
|
| 567 |
-
|
| 568 |
-
|
| 569 |
-
</
|
|
|
|
| 24 |
- long-context
|
| 25 |
---
|
| 26 |
|
| 27 |
+
<p align="center">
|
| 28 |
+
<a href="https://www.orcarouter.ai">
|
| 29 |
+
<img src="https://www.orcarouter.ai/orca-logo-classic.png" width="96" alt="OrcaRouter">
|
| 30 |
+
</a>
|
| 31 |
+
</p>
|
| 32 |
|
| 33 |
+
<h1 align="center">OrcaSAQ2 27B</h1>
|
| 34 |
|
| 35 |
+
<p align="center"><strong>High-fidelity 3-bit Qwen3.8 for long-horizon agents.</strong></p>
|
| 36 |
|
| 37 |
+
<p align="center"><strong>54 GB → 12.3 GB · +0.02% PPL · 93.2% Top-1 Agreement · 0.031 KLD · 262K Context</strong></p>
|
| 38 |
|
| 39 |
+
<p align="center">
|
| 40 |
+
<img src="https://img.shields.io/badge/12.3GB-checkpoint-FF6A00?style=flat-square"><img src="https://img.shields.io/badge/3.21bpw-decoder-111111?style=flat-square"><img src="https://img.shields.io/badge/+0.02%25-PPL-111111?style=flat-square"><img src="https://img.shields.io/badge/93.2%25-Top--1-111111?style=flat-square"><img src="https://img.shields.io/badge/262K-context-111111?style=flat-square">
|
|
|
|
|
|
|
|
|
|
| 41 |
</p>
|
| 42 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
---
|
| 44 |
|
| 45 |
> ## 27B reasoning. 12.3 GB.
|
|
|
|
| 70 |
|
| 71 |
---
|
| 72 |
|
| 73 |
+
## At a glance
|
| 74 |
|
| 75 |
| Metric | BF16 | OrcaSAQ2 |
|
| 76 |
|---|---:|---:|
|
|
|
|
| 84 |
| Mean KLD | — | **0.031** |
|
| 85 |
| Context | 262K | **262K** |
|
| 86 |
|
| 87 |
+
<p align="center"><strong>4.4× smaller. +0.02% perplexity.</strong></p>
|
| 88 |
+
<p align="center">The point is not 3-bit.</p>
|
| 89 |
+
<p align="center"><strong>The point is what survives at 3-bit.</strong></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
---
|
| 92 |
|
| 93 |
+
## BF16 fidelity
|
| 94 |
|
| 95 |
All numbers below are measured using **these exact OrcaSAQ2 weights** against the BF16 reference through the same evaluation path.
|
| 96 |
|
|
|
|
| 136 |
|
| 137 |
---
|
| 138 |
|
| 139 |
+
## Long-horizon agents
|
| 140 |
|
| 141 |
+
### Low-bit fidelity matters more as the horizon grows.
|
| 142 |
|
| 143 |
Short benchmarks can hide small degradation.
|
| 144 |
|
|
|
|
| 200 |
|
| 201 |
---
|
| 202 |
|
| 203 |
+
## Long-horizon performance
|
| 204 |
|
| 205 |
Long-horizon evaluations should use the same:
|
| 206 |
|
|
|
|
| 217 |
|
| 218 |
---
|
| 219 |
|
| 220 |
+
## Architecture
|
| 221 |
|
| 222 |
| | |
|
| 223 |
|---|---|
|
|
|
|
| 239 |
|
| 240 |
---
|
| 241 |
|
| 242 |
+
## Production serving
|
| 243 |
|
| 244 |
+
### Up to 90.1 tok/s single-stream on a 16 GB GPU
|
| 245 |
|
| 246 |
Measured under a **15.7 GiB GPU memory cap**.
|
| 247 |
|
|
|
|
| 277 |
|
| 278 |
---
|
| 279 |
|
| 280 |
+
## 27B on a 16 GB GPU
|
| 281 |
|
| 282 |
OrcaSAQ2's checkpoint is **12.3 GB**.
|
| 283 |
|
|
|
|
| 311 |
|
| 312 |
---
|
| 313 |
|
| 314 |
+
## Built for agents
|
| 315 |
|
| 316 |
### Long-horizon execution
|
| 317 |
|
|
|
|
| 341 |
|
| 342 |
---
|
| 343 |
|
| 344 |
+
## Quickstart
|
| 345 |
|
| 346 |
+
### Install
|
| 347 |
|
| 348 |
```bash
|
| 349 |
pip install -U vllm huggingface_hub
|
|
|
|
| 353 |
|
| 354 |
---
|
| 355 |
|
| 356 |
+
### Download
|
| 357 |
|
| 358 |
```bash
|
| 359 |
hf download orcarouter/OrcaSAQ2-27B \
|
|
|
|
| 362 |
|
| 363 |
---
|
| 364 |
|
| 365 |
+
### Serve with vLLM
|
| 366 |
|
| 367 |
```bash
|
| 368 |
vllm serve ./OrcaSAQ2-27B \
|
|
|
|
| 376 |
|
| 377 |
---
|
| 378 |
|
| 379 |
+
## OpenAI-compatible API
|
| 380 |
|
| 381 |
```python
|
| 382 |
from openai import OpenAI
|
|
|
|
| 401 |
|
| 402 |
---
|
| 403 |
|
| 404 |
+
## Recommended sampling
|
| 405 |
|
| 406 |
```text
|
| 407 |
temperature = 1.0
|
|
|
|
| 415 |
|
| 416 |
---
|
| 417 |
|
| 418 |
+
## Evaluation philosophy
|
| 419 |
|
| 420 |
A low-bit reasoning model should not be judged by checkpoint size alone.
|
| 421 |
|
| 422 |
We look at the intersection of:
|
| 423 |
|
| 424 |
+
<p align="center"><strong>Footprint × BF16 Fidelity × Capability × Long-Horizon Stability × Serving Performance</strong></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
|
| 426 |
A useful low-bit model must remain useful after compression.
|
| 427 |
|
| 428 |
---
|
| 429 |
|
| 430 |
+
## Why perplexity alone is not enough
|
| 431 |
|
| 432 |
Perplexity is useful and reproducible.
|
| 433 |
|
|
|
|
| 453 |
|
| 454 |
---
|
| 455 |
|
| 456 |
+
## Method
|
| 457 |
|
| 458 |
OrcaSAQ2 uses a proprietary **sensitivity-aware mixed-precision quantization system** developed by OrcaRouter.
|
| 459 |
|
|
|
|
| 463 |
|
| 464 |
---
|
| 465 |
|
| 466 |
+
## Limitations
|
| 467 |
|
| 468 |
- OrcaSAQ2 inherits the capabilities, biases and limitations of Qwen3.8-27B.
|
| 469 |
- Quantization is not mathematically lossless.
|
|
|
|
| 477 |
|
| 478 |
---
|
| 479 |
|
| 480 |
+
## Open source from OrcaRouter
|
| 481 |
|
| 482 |
### [OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)
|
| 483 |
|
|
|
|
| 491 |
|
| 492 |
Self-hosted multi-model AI infrastructure.
|
| 493 |
|
| 494 |
+
<p align="center"><strong>Open model. Open harness. Open bill.</strong></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 495 |
|
| 496 |
---
|
| 497 |
|
| 498 |
+
## Citation
|
| 499 |
|
| 500 |
```bibtex
|
| 501 |
@misc{qwen38,
|
|
|
|
| 509 |
|
| 510 |
---
|
| 511 |
|
| 512 |
+
## License
|
| 513 |
|
| 514 |
**Apache-2.0**
|
| 515 |
|
|
|
|
| 521 |
|
| 522 |
---
|
| 523 |
|
| 524 |
+
<h2 align="center">One Gateway. Every Model.</h2>
|
|
|
|
|
|
|
| 525 |
|
| 526 |
+
<p align="center"><strong>Route Smarter · Ship Safer · Spend Less</strong></p>
|
| 527 |
|
| 528 |
+
<p align="center">
|
| 529 |
+
<a href="https://www.orcarouter.ai">Website</a> ·
|
| 530 |
+
<a href="https://www.orcarouter.ai/models">Models</a> ·
|
| 531 |
+
<a href="https://github.com/Continuum-AI-Corp">GitHub</a> ·
|
| 532 |
+
<a href="https://discord.gg/yAh6Tex6kx">Discord</a> ·
|
| 533 |
+
<a href="https://x.com/OrcaRouter">X</a>
|
| 534 |
+
</p>
|