hugging-yiii commited on
Commit
df1af21
·
verified ·
1 Parent(s): 4406ad8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +47 -82
README.md CHANGED
@@ -24,41 +24,22 @@ tags:
24
  - long-context
25
  ---
26
 
27
- <div align="center">
28
-
29
- <a href="https://www.orcarouter.ai">
30
- <img src="https://www.orcarouter.ai/orca-logo-classic.png" width="96" alt="OrcaRouter">
31
- </a>
32
 
33
- # OrcaSAQ2 27B
34
 
35
- ### High-fidelity 3-bit Qwen3.8 for long-horizon agents.
36
 
37
- **54 GB → 12.3 GB · +0.02% PPL · 93.2% Top-1 Agreement · 0.031 KLD · 262K Context**
38
 
39
- <p>
40
- <img src="https://img.shields.io/badge/Checkpoint-12.3_GB-FF6A00?style=for-the-badge">
41
- <img src="https://img.shields.io/badge/Decoder-3.21_bpw-111111?style=for-the-badge">
42
- <img src="https://img.shields.io/badge/PPL_Delta-%2B0.02%25-111111?style=for-the-badge">
43
- <img src="https://img.shields.io/badge/Top--1-93.2%25-111111?style=for-the-badge">
44
  </p>
45
 
46
- <p>
47
- <img src="https://img.shields.io/badge/Context-262K-111111?style=flat-square">
48
- <img src="https://img.shields.io/badge/vLLM-Production_Ready-111111?style=flat-square">
49
- <img src="https://img.shields.io/badge/MTP-Speculative_Decoding-111111?style=flat-square">
50
- <img src="https://img.shields.io/badge/Tool_Calling-Yes-111111?style=flat-square">
51
- <img src="https://img.shields.io/badge/License-Apache_2.0-111111?style=flat-square">
52
- </p>
53
-
54
- **[OrcaRouter](https://www.orcarouter.ai)** ·
55
- **[Models](https://www.orcarouter.ai/models)** ·
56
- **[GitHub](https://github.com/Continuum-AI-Corp)** ·
57
- **[Discord](https://discord.gg/yAh6Tex6kx)** ·
58
- **[X](https://x.com/OrcaRouter)**
59
-
60
- </div>
61
-
62
  ---
63
 
64
  > ## 27B reasoning. 12.3 GB.
@@ -89,7 +70,7 @@ The resulting checkpoint provides:
89
 
90
  ---
91
 
92
- # At a glance
93
 
94
  | Metric | BF16 | OrcaSAQ2 |
95
  |---|---:|---:|
@@ -103,19 +84,13 @@ The resulting checkpoint provides:
103
  | Mean KLD | — | **0.031** |
104
  | Context | 262K | **262K** |
105
 
106
- <div align="center">
107
-
108
- ## **4.4× smaller. +0.02% perplexity.**
109
-
110
- ### The point is not 3-bit.
111
-
112
- ### **The point is what survives at 3-bit.**
113
-
114
- </div>
115
 
116
  ---
117
 
118
- # BF16 fidelity
119
 
120
  All numbers below are measured using **these exact OrcaSAQ2 weights** against the BF16 reference through the same evaluation path.
121
 
@@ -161,9 +136,9 @@ OrcaSAQ2
161
 
162
  ---
163
 
164
- # Long-horizon agents
165
 
166
- ## Low-bit fidelity matters more as the horizon grows.
167
 
168
  Short benchmarks can hide small degradation.
169
 
@@ -225,7 +200,7 @@ For agent models, both matter.
225
 
226
  ---
227
 
228
- # Long-horizon performance
229
 
230
  Long-horizon evaluations should use the same:
231
 
@@ -242,7 +217,7 @@ Scores produced by materially different agent stacks should not be treated as di
242
 
243
  ---
244
 
245
- # Architecture
246
 
247
  | | |
248
  |---|---|
@@ -264,9 +239,9 @@ Scores produced by materially different agent stacks should not be treated as di
264
 
265
  ---
266
 
267
- # Production serving
268
 
269
- ## Up to 90.1 tok/s single-stream on a 16 GB GPU
270
 
271
  Measured under a **15.7 GiB GPU memory cap**.
272
 
@@ -302,7 +277,7 @@ For highly batched workloads, benchmark both configurations.
302
 
303
  ---
304
 
305
- # 27B on a 16 GB GPU
306
 
307
  OrcaSAQ2's checkpoint is **12.3 GB**.
308
 
@@ -336,7 +311,7 @@ The model architecture supports up to **262K context**.
336
 
337
  ---
338
 
339
- # Built for agents
340
 
341
  ### Long-horizon execution
342
 
@@ -366,9 +341,9 @@ A **12.3 GB** checkpoint designed around practical inference hardware.
366
 
367
  ---
368
 
369
- # Quickstart
370
 
371
- ## Install
372
 
373
  ```bash
374
  pip install -U vllm huggingface_hub
@@ -378,7 +353,7 @@ pip install git+https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel
378
 
379
  ---
380
 
381
- ## Download
382
 
383
  ```bash
384
  hf download orcarouter/OrcaSAQ2-27B \
@@ -387,7 +362,7 @@ hf download orcarouter/OrcaSAQ2-27B \
387
 
388
  ---
389
 
390
- ## Serve with vLLM
391
 
392
  ```bash
393
  vllm serve ./OrcaSAQ2-27B \
@@ -401,7 +376,7 @@ vllm serve ./OrcaSAQ2-27B \
401
 
402
  ---
403
 
404
- # OpenAI-compatible API
405
 
406
  ```python
407
  from openai import OpenAI
@@ -426,7 +401,7 @@ print(response.choices[0].message.content)
426
 
427
  ---
428
 
429
- # Recommended sampling
430
 
431
  ```text
432
  temperature = 1.0
@@ -440,23 +415,19 @@ For agent deployments, benchmark against the actual tool schema, context distrib
440
 
441
  ---
442
 
443
- # Evaluation philosophy
444
 
445
  A low-bit reasoning model should not be judged by checkpoint size alone.
446
 
447
  We look at the intersection of:
448
 
449
- <div align="center">
450
-
451
- ### **Footprint × BF16 Fidelity × Capability × Long-Horizon Stability × Serving Performance**
452
-
453
- </div>
454
 
455
  A useful low-bit model must remain useful after compression.
456
 
457
  ---
458
 
459
- # Why perplexity alone is not enough
460
 
461
  Perplexity is useful and reproducible.
462
 
@@ -482,7 +453,7 @@ That is why OrcaSAQ2 reports BF16 fidelity metrics alongside downstream and long
482
 
483
  ---
484
 
485
- # Method
486
 
487
  OrcaSAQ2 uses a proprietary **sensitivity-aware mixed-precision quantization system** developed by OrcaRouter.
488
 
@@ -492,7 +463,7 @@ Detailed quantization methodology, calibration strategy, precision allocation an
492
 
493
  ---
494
 
495
- # Limitations
496
 
497
  - OrcaSAQ2 inherits the capabilities, biases and limitations of Qwen3.8-27B.
498
  - Quantization is not mathematically lossless.
@@ -506,7 +477,7 @@ Detailed quantization methodology, calibration strategy, precision allocation an
506
 
507
  ---
508
 
509
- # Open source from OrcaRouter
510
 
511
  ### [OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)
512
 
@@ -520,15 +491,11 @@ Record, replay, fork and debug AI-agent runs.
520
 
521
  Self-hosted multi-model AI infrastructure.
522
 
523
- <div align="center">
524
-
525
- ### **Open model. Open harness. Open bill.**
526
-
527
- </div>
528
 
529
  ---
530
 
531
- # Citation
532
 
533
  ```bibtex
534
  @misc{qwen38,
@@ -542,7 +509,7 @@ Self-hosted multi-model AI infrastructure.
542
 
543
  ---
544
 
545
- # License
546
 
547
  **Apache-2.0**
548
 
@@ -554,16 +521,14 @@ Quantization does not change the underlying license obligations.
554
 
555
  ---
556
 
557
- <div align="center">
558
-
559
- # One Gateway. Every Model.
560
 
561
- **Route Smarter · Ship Safer · Spend Less**
562
 
563
- [Website](https://www.orcarouter.ai) ·
564
- [Models](https://www.orcarouter.ai/models) ·
565
- [GitHub](https://github.com/Continuum-AI-Corp) ·
566
- [Discord](https://discord.gg/yAh6Tex6kx) ·
567
- [X](https://x.com/OrcaRouter)
568
-
569
- </div>
 
24
  - long-context
25
  ---
26
 
27
+ <p align="center">
28
+ <a href="https://www.orcarouter.ai">
29
+ <img src="https://www.orcarouter.ai/orca-logo-classic.png" width="96" alt="OrcaRouter">
30
+ </a>
31
+ </p>
32
 
33
+ <h1 align="center">OrcaSAQ2 27B</h1>
34
 
35
+ <p align="center"><strong>High-fidelity 3-bit Qwen3.8 for long-horizon agents.</strong></p>
36
 
37
+ <p align="center"><strong>54 GB → 12.3 GB · +0.02% PPL · 93.2% Top-1 Agreement · 0.031 KLD · 262K Context</strong></p>
38
 
39
+ <p align="center">
40
+ <img src="https://img.shields.io/badge/12.3GB-checkpoint-FF6A00?style=flat-square"><img src="https://img.shields.io/badge/3.21bpw-decoder-111111?style=flat-square"><img src="https://img.shields.io/badge/+0.02%25-PPL-111111?style=flat-square"><img src="https://img.shields.io/badge/93.2%25-Top--1-111111?style=flat-square"><img src="https://img.shields.io/badge/262K-context-111111?style=flat-square">
 
 
 
41
  </p>
42
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
  ---
44
 
45
  > ## 27B reasoning. 12.3 GB.
 
70
 
71
  ---
72
 
73
+ ## At a glance
74
 
75
  | Metric | BF16 | OrcaSAQ2 |
76
  |---|---:|---:|
 
84
  | Mean KLD | — | **0.031** |
85
  | Context | 262K | **262K** |
86
 
87
+ <p align="center"><strong>4.4× smaller. +0.02% perplexity.</strong></p>
88
+ <p align="center">The point is not 3-bit.</p>
89
+ <p align="center"><strong>The point is what survives at 3-bit.</strong></p>
 
 
 
 
 
 
90
 
91
  ---
92
 
93
+ ## BF16 fidelity
94
 
95
  All numbers below are measured using **these exact OrcaSAQ2 weights** against the BF16 reference through the same evaluation path.
96
 
 
136
 
137
  ---
138
 
139
+ ## Long-horizon agents
140
 
141
+ ### Low-bit fidelity matters more as the horizon grows.
142
 
143
  Short benchmarks can hide small degradation.
144
 
 
200
 
201
  ---
202
 
203
+ ## Long-horizon performance
204
 
205
  Long-horizon evaluations should use the same:
206
 
 
217
 
218
  ---
219
 
220
+ ## Architecture
221
 
222
  | | |
223
  |---|---|
 
239
 
240
  ---
241
 
242
+ ## Production serving
243
 
244
+ ### Up to 90.1 tok/s single-stream on a 16 GB GPU
245
 
246
  Measured under a **15.7 GiB GPU memory cap**.
247
 
 
277
 
278
  ---
279
 
280
+ ## 27B on a 16 GB GPU
281
 
282
  OrcaSAQ2's checkpoint is **12.3 GB**.
283
 
 
311
 
312
  ---
313
 
314
+ ## Built for agents
315
 
316
  ### Long-horizon execution
317
 
 
341
 
342
  ---
343
 
344
+ ## Quickstart
345
 
346
+ ### Install
347
 
348
  ```bash
349
  pip install -U vllm huggingface_hub
 
353
 
354
  ---
355
 
356
+ ### Download
357
 
358
  ```bash
359
  hf download orcarouter/OrcaSAQ2-27B \
 
362
 
363
  ---
364
 
365
+ ### Serve with vLLM
366
 
367
  ```bash
368
  vllm serve ./OrcaSAQ2-27B \
 
376
 
377
  ---
378
 
379
+ ## OpenAI-compatible API
380
 
381
  ```python
382
  from openai import OpenAI
 
401
 
402
  ---
403
 
404
+ ## Recommended sampling
405
 
406
  ```text
407
  temperature = 1.0
 
415
 
416
  ---
417
 
418
+ ## Evaluation philosophy
419
 
420
  A low-bit reasoning model should not be judged by checkpoint size alone.
421
 
422
  We look at the intersection of:
423
 
424
+ <p align="center"><strong>Footprint × BF16 Fidelity × Capability × Long-Horizon Stability × Serving Performance</strong></p>
 
 
 
 
425
 
426
  A useful low-bit model must remain useful after compression.
427
 
428
  ---
429
 
430
+ ## Why perplexity alone is not enough
431
 
432
  Perplexity is useful and reproducible.
433
 
 
453
 
454
  ---
455
 
456
+ ## Method
457
 
458
  OrcaSAQ2 uses a proprietary **sensitivity-aware mixed-precision quantization system** developed by OrcaRouter.
459
 
 
463
 
464
  ---
465
 
466
+ ## Limitations
467
 
468
  - OrcaSAQ2 inherits the capabilities, biases and limitations of Qwen3.8-27B.
469
  - Quantization is not mathematically lossless.
 
477
 
478
  ---
479
 
480
+ ## Open source from OrcaRouter
481
 
482
  ### [OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)
483
 
 
491
 
492
  Self-hosted multi-model AI infrastructure.
493
 
494
+ <p align="center"><strong>Open model. Open harness. Open bill.</strong></p>
 
 
 
 
495
 
496
  ---
497
 
498
+ ## Citation
499
 
500
  ```bibtex
501
  @misc{qwen38,
 
509
 
510
  ---
511
 
512
+ ## License
513
 
514
  **Apache-2.0**
515
 
 
521
 
522
  ---
523
 
524
+ <h2 align="center">One Gateway. Every Model.</h2>
 
 
525
 
526
+ <p align="center"><strong>Route Smarter · Ship Safer · Spend Less</strong></p>
527
 
528
+ <p align="center">
529
+ <a href="https://www.orcarouter.ai">Website</a> ·
530
+ <a href="https://www.orcarouter.ai/models">Models</a> ·
531
+ <a href="https://github.com/Continuum-AI-Corp">GitHub</a> ·
532
+ <a href="https://discord.gg/yAh6Tex6kx">Discord</a> ·
533
+ <a href="https://x.com/OrcaRouter">X</a>
534
+ </p>