Add pre-quantization comparison

README.md

@@ -18,7 +18,7 @@ tags:
 
 nyx is a task-specialized Qwen3.5-35B-A3B model for Choice, Score, and Noul decisions. It combines task-specific adaptation with INT4 routed-expert storage and FP8 expert compute.
 
-![Six classification benchmarks comparing nyx, Qwen/Qwen3.5-35B-A3B, and Jev](assets/nyx-vs-jev-benchmarks.png)
+![Six classification benchmarks comparing nyx and Jev](assets/nyx-vs-jev-benchmarks.png)
 
 ## Model
 
@@ -79,27 +79,40 @@ Wire compatibility does not mean behavioral identity: decisions, probabilities,
 
 ## Evaluation
 
-The frozen comparison used 2,277 identical requests across six public classification benchmarks at client concurrency 8. Every provider completed every case with zero errors and zero retries.
+The frozen comparison used 2,277 identical requests across six public classification benchmarks at client concurrency 8. Both models completed every case with zero errors and zero retries.
 
-| Benchmark | nyx INT4+FP8 | Qwen/Qwen3.5-35B-A3B | Jev 1.13 |
-|---|---:|---:|---:|
-| AG News | 84.75% | 85.75% | 86.50% |
-| TREC coarse | 92.00% | 94.00% | 93.00% |
-| CoLA | 79.00% | 79.75% | 77.75% |
-| RTE | 93.14% | 93.86% | 91.34% |
-| Emotion | 43.33% | 44.67% | 47.00% |
-| IMDb | 96.50% | 96.25% | 97.00% |
-| **Dataset-macro accuracy** | **81.45%** | **82.38%** | **82.10%** |
-| **Dataset-macro F1** | **81.06%** | **82.02%** | **81.85%** |
-| **Pooled accuracy** | **82.96%** | **83.93%** | **83.62%** |
-
-| Timing | nyx INT4+FP8 | Qwen/Qwen3.5-35B-A3B | Jev 1.13 |
+| Benchmark | nyx INT4+FP8 | Jev 1.13 |
+|---|---:|---:|
+| AG News | 84.75% | 86.50% |
+| TREC coarse | 92.00% | 93.00% |
+| CoLA | 79.00% | 77.75% |
+| RTE | 93.14% | 91.34% |
+| Emotion | 43.33% | 47.00% |
+| IMDb | 96.50% | 97.00% |
+| **Dataset-macro accuracy** | **81.45%** | **82.10%** |
+| **Dataset-macro F1** | **81.06%** | **81.85%** |
+| **Pooled accuracy** | **82.96%** | **83.62%** |
+
+| Timing | nyx INT4+FP8 | Jev 1.13 |
+|---|---:|---:|
+| Backend inference median / p95 | 62.97 / 85.30 ms | not exposed |
+| Round trip median / p95 | 66.25 / 89.22 ms | 182.33 / 245.12 ms |
+| Throughput | 112.67 req/s | 42.43 req/s |
+
+nyx ran on a local MI325X/API stack. Jev ran over remote HTTPS on unmatched infrastructure, so its latency is descriptive rather than a controlled model-speed comparison. The local runtime used continuous batching with `max-running-requests=64`; the benchmark's eight concurrent callers limited this run to eight active requests. See [the evaluation record](benchmarks/random-six-report.md).
+
+### Before quantization
+
+The shared H1 development slice below compares the official Qwen base in BF16, the pre-quantization `nyx` BF16 checkpoint, and Jev 1.13.0 on the same 1,500 scored cases with identical labels and task semantics.
+
+| H1 task | Qwen base BF16 | `nyx` BF16 | Jev 1.13.0 |
 |---|---:|---:|---:|
-| Backend inference median / p95 | 62.97 / 85.30 ms | 74.45 / 97.38 ms | not exposed |
-| Round trip median / p95 | 66.25 / 89.22 ms | 78.23 / 101.82 ms | 182.33 / 245.12 ms |
-| Throughput | 112.67 req/s | 95.58 req/s | 42.43 req/s |
+| Comment triage | 295/500 (59.00%) | **462/500 (92.40%)** | 455/500 (91.00%) |
+| Report completeness | 439/500 (87.80%) | **488/500 (97.60%)** | 468/500 (93.60%) |
+| Broad assets, exact set | 285/500 (57.00%) | **298/500 (59.60%)** | 284/500 (56.80%) |
+| **Overall** | **1,019/1,500 (67.93%)** | **1,248/1,500 (83.20%)** | **1,207/1,500 (80.47%)** |
 
-nyx and the original Qwen model ran on the same local MI325X/API stack. Jev ran over remote HTTPS on unmatched infrastructure, so its latency is descriptive rather than a controlled model-speed comparison. The local runtime used continuous batching with `max-running-requests=64`; the benchmark's eight concurrent callers limited this run to eight active requests. See [the evaluation record](benchmarks/random-six-report.md).
+This is authored development evidence, not an independent public holdout. Qwen and `nyx` ran locally in BF16 on the same MI325X/SGLang stack; Jev ran through its remote API, so latency is intentionally omitted from this table. See the [pre-quantization evaluation record](benchmarks/prequant-h1.md).
 
 ## Limitations
 

benchmarks/prequant-h1.md

@@ -0,0 +1,24 @@
+# Pre-quantization H1 comparison
+
+This record isolates the exact intersection shared by three completed evaluations: the official `Qwen/Qwen3.5-35B-A3B` base in BF16, the pre-quantization `nyx` BF16 checkpoint, and Jev 1.13.0.
+
+## Results
+
+| Task | Cases | Qwen base BF16 | `nyx` BF16 | Jev 1.13.0 |
+|---|---:|---:|---:|---:|
+| Comment triage | 500 | 295 (59.00%) | 462 (92.40%) | 455 (91.00%) |
+| Report completeness | 500 | 439 (87.80%) | 488 (97.60%) | 468 (93.60%) |
+| Broad assets, exact set | 500 | 285 (57.00%) | 298 (59.60%) | 284 (56.80%) |
+| **Overall** | **1,500** | **1,019 (67.93%)** | **1,248 (83.20%)** | **1,207 (80.47%)** |
+
+## Protocol
+
+- The intersection contains comment fixtures 501–1000 plus all 500 authored report fixtures, scored once for completeness and once for broad-asset exact-set classification.
+- Case IDs and expected labels were matched across the three result sets. All 1,500 expected labels matched exactly; no case was excluded after matching.
+- Qwen and `nyx` used BF16 before any FP8 or INT4 conversion, the same AMD Instinct MI325X, the same SGLang/API stack, and server concurrency 8.
+- The base was the official `Qwen/Qwen3.5-35B-A3B` revision `59d61f3ce65a6d9863b86d2e96597125219dc754`.
+- Jev was evaluated as `jev-1.13.0` through its remote API with the same typed requests, prompts, thresholds, and reference labels. Because the serving infrastructure differs, this record makes no speed comparison.
+
+## Scope
+
+These are synthetic, authored H1 development fixtures with known scenario-family overlap. They are useful for measuring task adaptation on this workload, but they are not an independent public holdout and do not establish general model superiority.

benchmarks/random-six-report.md

@@ -1,32 +1,32 @@
-# Six public classification benchmarks: nyx vs Qwen vs Jev
+# Six public classification benchmarks: nyx vs Jev
 
 Frozen paired cohort: **2,277 cases**; SHA-256 `046a38c24747fd724003aca7e5f1d1b6e148d599092222361c0717ecff84ecad`; client concurrency 8; every failure counts as wrong.
 
-| Dataset | Cases | nyx INT4+FP8 | Qwen/Qwen3.5-35B-A3B | Jev 1.13 |
-|---|---:|---:|---:|---:|
-| AG News | 400 | 84.75% (339/400) | 85.75% (343/400) | 86.50% (346/400) |
-| TREC coarse | 500 | 92.00% (460/500) | 94.00% (470/500) | 93.00% (465/500) |
-| CoLA | 400 | 79.00% (316/400) | 79.75% (319/400) | 77.75% (311/400) |
-| RTE | 277 | 93.14% (258/277) | 93.86% (260/277) | 91.34% (253/277) |
-| Emotion | 300 | 43.33% (130/300) | 44.67% (134/300) | 47.00% (141/300) |
-| IMDb | 400 | 96.50% (386/400) | 96.25% (385/400) | 97.00% (388/400) |
-
-| Aggregate | nyx INT4+FP8 | Qwen/Qwen3.5-35B-A3B | Jev 1.13 |
+| Dataset | Cases | nyx INT4+FP8 | Jev 1.13 |
 |---|---:|---:|---:|
-| Dataset-macro accuracy | 81.45% | 82.38% | 82.10% |
-| Dataset-macro F1 | 81.06% | 82.02% | 81.85% |
-| Pooled micro accuracy | 82.96% | 83.93% | 83.62% |
+| AG News | 400 | 84.75% (339/400) | 86.50% (346/400) |
+| TREC coarse | 500 | 92.00% (460/500) | 93.00% (465/500) |
+| CoLA | 400 | 79.00% (316/400) | 77.75% (311/400) |
+| RTE | 277 | 93.14% (258/277) | 91.34% (253/277) |
+| Emotion | 300 | 43.33% (130/300) | 47.00% (141/300) |
+| IMDb | 400 | 96.50% (386/400) | 97.00% (388/400) |
+
+| Aggregate | nyx INT4+FP8 | Jev 1.13 |
+|---|---:|---:|
+| Dataset-macro accuracy | 81.45% | 82.10% |
+| Dataset-macro F1 | 81.06% | 81.85% |
+| Pooled micro accuracy | 82.96% | 83.62% |
 
 ## Paired comparison
 
-Exact two-sided McNemar tests use identical case-level predictions. Qwen-only versus nyx-only correct cases were 63 to 41 (`p=0.03896`); Jev-only versus nyx-only were 76 to 61 (`p=0.23154`); Qwen-only versus Jev-only were 66 to 59 (`p=0.59168`). None crosses the Bonferroni-adjusted threshold of `0.0167` for three comparisons.
+The exact two-sided McNemar test uses identical case-level predictions. Jev-only versus nyx-only correct cases were 76 to 61 (`p=0.23154`).
 
-| Timing | nyx INT4+FP8 | Qwen/Qwen3.5-35B-A3B | Jev 1.13 |
-|---|---:|---:|---:|
-| Backend inference median / p95 | 62.97 / 85.30 ms | 74.45 / 97.38 ms | not exposed |
-| Round trip median / p95 | 66.25 / 89.22 ms | 78.23 / 101.82 ms | 182.33 / 245.12 ms |
-| Wall time | 20,209 ms | 23,824 ms | 53,666 ms |
-| Throughput | 112.67 req/s | 95.58 req/s | 42.43 req/s |
+| Timing | nyx INT4+FP8 | Jev 1.13 |
+|---|---:|---:|
+| Backend inference median / p95 | 62.97 / 85.30 ms | not exposed |
+| Round trip median / p95 | 66.25 / 89.22 ms | 182.33 / 245.12 ms |
+| Wall time | 20,209 ms | 53,666 ms |
+| Throughput | 112.67 req/s | 42.43 req/s |
 
 ## Method and caveats
 
@@ -34,5 +34,5 @@ Exact two-sided McNemar tests use identical case-level predictions. Qwen-only ve
 - AG News, CoLA, Emotion, and IMDb are fixed class-stratified samples; TREC test and RTE validation are complete.
 - Every request contained one Choice question. Eight asynchronous clients used persistent HTTP/1.1 connections. The local SGLang runtime used continuous batching with `max-running-requests=64`, while client concurrency limited this run to eight active requests.
 - Backend inference is parsed from the local `Server-Timing` header. Jev did not expose server timing.
-- nyx and the original Qwen model ran locally on the same AMD Instinct MI325X and API stack. Jev ran over remote HTTPS on unmatched infrastructure, so cross-service latency is descriptive only.
-- Public-benchmark pretraining contamination is unknown for all three models. This finite suite does not establish universal classification quality.
+- nyx ran locally on an AMD Instinct MI325X and API stack. Jev ran over remote HTTPS on unmatched infrastructure, so cross-service latency is descriptive only.
+- Public-benchmark pretraining contamination is unknown for both models. This finite suite does not establish universal classification quality.

benchmarks/summary.json

@@ -5,26 +5,21 @@
   "concurrency": 8,
   "errors": {
     "nyx-int4fp8": 0,
-    "qwen-qwen3.5-35b-a3b": 0,
     "jev-1.13": 0
   },
   "dataset_macro_accuracy": {
     "nyx-int4fp8": 0.8145402125952668,
-    "qwen-qwen3.5-35b-a3b": 0.8237991375852386,
     "jev-1.13": 0.8209762334536702
   },
   "pooled_accuracy": {
     "nyx-int4fp8": 0.8296003513394817,
-    "qwen-qwen3.5-35b-a3b": 0.839262187088274,
     "jev-1.13": 0.8361879666227492
   },
   "paired_mcnemar_two_sided_p": {
-    "nyx-int4fp8_vs_qwen-qwen3.5-35b-a3b": 0.03895793340131187,
-    "nyx-int4fp8_vs_jev-1.13": 0.23153716588439505,
-    "qwen-qwen3.5-35b-a3b_vs_jev-1.13": 0.591683927094163
+    "nyx-int4fp8_vs_jev-1.13": 0.23153716588439505
   },
   "limitations": [
-    "Public benchmark contamination is unknown.",
+    "Public benchmark contamination is unknown for both models.",
     "Fixed samples are not complete official-suite scores except TREC test and RTE validation.",
     "Jev timing used unmatched remote infrastructure."
   ]

release-manifest.json

@@ -14,20 +14,24 @@
       "sha256": "94d30a32935469a150faba2820dfa13bd34c67ae5d2c8adb390778666a4a8443"
     },
     "README.md": {
-      "bytes": 5924,
-      "sha256": "8ecbfcee2f782aa962f160186671417a164c28971c405105ab2868b4d276eb9f"
+      "bytes": 6627,
+      "sha256": "c37620861bbcd51d63dcc298865a7ffe8f7257617c692a1ec55280a1486099e2"
     },
     "assets/nyx-vs-jev-benchmarks.png": {
-      "bytes": 1461488,
-      "sha256": "be441ddc5889c5d625580b190d38d37038240a180a6b20dc02ba98322caac637"
+      "bytes": 1362162,
+      "sha256": "9ce0a836719964925223bc94e29a299aebfe01b81c85fc85866e05dac101ee47"
+    },
+    "benchmarks/prequant-h1.md": {
+      "bytes": 1708,
+      "sha256": "fab5d5d4db40d14a0ae6d877bbf1a5f2fa72f586d5b282d37ee2456c293c8fad"
     },
     "benchmarks/random-six-report.md": {
-      "bytes": 2708,
-      "sha256": "5b45369ebb94520103352fa321bc053228aea5d607ad84e8af132e701f75f47e"
+      "bytes": 2183,
+      "sha256": "19dff7a297bd379897b393cb3601b91fe2ed9cfabfa6e5c8eaf38601d4a12ff1"
     },
     "benchmarks/summary.json": {
-      "bytes": 984,
-      "sha256": "f181fe2989e381edb2dc4695479591e3038679dffb7726f4a877c7849b6d9f3d"
+      "bytes": 751,
+      "sha256": "90d3be0514f431979f75fba1d0ee43f47d5b7b5ce9ec3d8132a9929ca1492d33"
     },
     "chat_template.jinja": {
       "bytes": 7756,
@@ -112,5 +116,5 @@
   },
   "format_version": 1,
   "repository": "devanshbatham/nyx",
-  "total_bytes": 19679466775
+  "total_bytes": 19679369102
 }