Upload reports/amos-2.8-vs-jev-1.13-vs-laya.md with huggingface_hub

reports/amos-2.8-vs-jev-1.13-vs-laya.md

@@ -11,20 +11,20 @@
 | Invocation | All questions invoked via real production HTTP endpoints; no local direct calls, no rule-layer assistance |
 | amos-2.8 board | 500-question fresh randomized blind test × 4 rounds (independently generated question banks per round, independent seeds, zero text-level overlap with training data) |
 | jev-1.13 board | Same-board generator 500-question run (2026-10-02 batch) + two same-day live-invocation rounds (2026-10-06) |
-| laya board | 6,000 random real-task three-model same-question board (ext6kb, seed=2027, 6-domain quota) |
+| laya board | 500-question Board 1 same board, zero-shot (2026-10-09, identical request construction, same rule-based gold) + 6,000-question ext6kb board (seed=2027) as a second reference |
 
-**Comparability note**: the amos-2.8 and jev-1.13 500-question data come from the same board generator (directly comparable). The laya data come from the 6,000-question ext6kb board (different task-domain composition; provided as an order-of-magnitude reference only, not mixed with the 500-question board).
+**Comparability note**: all three models now have same-board 500-question data — amos-2.8 and jev-1.13 via the same board generator HTTP endpoint, laya zero-shot with identical request construction, all scored by the same rule-based gold. The 6,000-question ext6kb board is kept as a second, larger reference board.
 
 ## 2. Headline Comparison
 
 | Metric | **amos-2.8** | jev-1.13 | laya (zero-shot) |
 |---|---:|---:|---:|
-| 500-question blind-test raw accuracy | **96.6%** (483/500; four rounds 96.6%–97.8%) | 96.4% (482/500) | — (not run on the 500-question board) |
+| 500-question blind-test raw accuracy | **96.6%** (483/500; four rounds 96.6%–97.8%) | 96.4% (482/500) | 42.8% (214/500) |
 | 6,000-question same-board main tasks | — (not run on ext6kb) | 72.28% | 45.07% |
-| Tier-A pure decision tasks | **98.6%** (same-day two rounds) | 97.9% | — |
-| Strict OOD (task-domain overlap removed) | **96.05%** | 95.81% | — |
-| API errors | **0** (500 × 4 rounds) | 0 (500-question board) | 0 (ext6kb board) |
-| Latency p50 / p95 | ≈950–1200 ms (public end-to-end, incl. gateway) | 1161.9 ms / 2811.2 ms | 24.1 ms / 28.2 ms |
+| Tier-A pure decision tasks | **98.6%** (same-day two rounds) | 97.9% | 52.1% (146/280) |
+| Strict OOD (task-domain overlap removed) | **96.05%** | 95.81% | 37.0% (159/430) |
+| API errors | **0** (500 × 4 rounds) | 0 (500-question board) | 0 (500-question + ext6kb boards) |
+| Latency p50 / p95 | ≈950–1200 ms (public end-to-end, incl. gateway) | 1161.9 ms / 2811.2 ms | 20.7 ms / 21.8 ms (Board 1, local GPU direct) |
 | Output form | Normalized probability distribution (100% parseable) | Generated text (requires parsing) | Structured (lightweight) |
 
 ## 3. amos-2.8 Detailed Results (500-Question Blind Test × 4 Rounds)
@@ -98,7 +98,46 @@ amos-2.8 leads on NLI (+13.3pp), GSM8K word problems (+12.5pp), length band (+20
 **jev strengths**: character counting (instruction 95% vs amos-2.8 68.3% four-round mean), deduplication (100% vs 91.1%), and Tier-C generative tasks overall.
 **jev weaknesses**: NLI (80.0% vs amos-2.8 93.3%), GSM8K word problems (92.5%–95% vs amos-2.8 100%).
 
-## 5. laya Detailed Results (ext6kb 6,000-Question Three-Model Board)
+## 5. laya Detailed Results
+
+### 5.1 Board 1: 500-Question Same Board (zero-shot, run 2026-10-09)
+
+laya (convaiinnovations/laya-multilingual) was invoked zero-shot on the exact R3 500-question board (seed 1791290181) with request construction identical to the amos-2.8/jev-1.13 endpoint calls (same state / instructions / criteria fields) and scored by the same rule-based gold.
+
+| Scope | laya (zero-shot) | amos-2.8 (R3) | jev-1.13 (same board) |
+|---|---:|---:|---:|
+| Raw accuracy | 42.8% (214/500) | **96.6%** (483/500) | 96.4% (482/500) |
+| Tier-A pure decision tasks | 52.1% (146/280) | **97.9%** | 97.9% |
+| Strict OOD (sentiment+topic removed) | 37.0% (159/430) | **96.0%** | 95.8% |
+| Latency p50 / p95 (ms, local GPU direct) | **20.7 / 21.8** | ≈950–1200 (public e2e) | 1161.9 / 2811.2 |
+| Errors | 0 | 0 | 0 |
+
+Per-category (sorted by laya accuracy):
+
+| Category | n | amos-2.8 | jev-1.13 | laya |
+|---|---:|---:|---:|---:|
+| topic | 30 | 100.0% | 100.0% | **100.0%** |
+| extraction | 30 | 100.0% | 100.0% | 83.3% |
+| sentiment | 40 | 100.0% | 100.0% | 62.5% |
+| nli | 30 | **93.3%** | 80.0% | 56.7% |
+| pii | 30 | 100.0% | 100.0% | 50.0% |
+| instruction | 20 | 55.0% | **85.0%** | 50.0% |
+| duplicate | 30 | 86.7% | **100.0%** | 36.7% |
+| moderation | 30 | 100.0% | 100.0% | 36.7% |
+| math_basic | 60 | **98.3%** | 96.7% | 41.7% |
+| commonsense | 20 | 100.0% | 100.0% | 30.0% |
+| knowledge | 30 | 100.0% | 100.0% | 26.7% |
+| case_form | 20 | 100.0% | 100.0% | 25.0% |
+| math_word | 40 | 100.0% | 87.5% | 22.5% |
+| reasoning | 30 | 100.0% | 100.0% | 23.3% |
+| length_band | 10 | 100.0% | 80.0% | 20.0% |
+| lang_id | 30 | 100.0% | 100.0% | 16.7% |
+| percent | 20 | 95.0% | **100.0%** | 15.0% |
+
+- laya's only 100% category is topic (AG News); it trails amos-2.8 on 16 of 17 categories.
+- The same-board result (42.8%) is consistent with laya's ext6kb main-task accuracy (45.07%), confirming the gap is model capability rather than board composition.
+
+### 5.2 ext6kb 6,000-Question Three-Model Board (reference)
 
 | Scope | laya (zero-shot) | amos-2.6 (reference) | jev (reference) |
 |---|---:|---:|---:|
@@ -115,11 +154,11 @@ amos-2.8 leads on NLI (+13.3pp), GSM8K word problems (+12.5pp), length band (+20
 ## 6. Conclusions
 
 1. **amos-2.8 vs jev-1.13**: +0.2pp on the 500-question same board; tied (97.8% vs 97.8%) under same-day live-invocation scope; +0.7pp ahead on Tier-A pure decision tasks (the model's core job); +0.24pp ahead under strict OOD. jev retains advantages only in a few generative-benchmark categories such as character counting and deduplication.
-2. **amos-2.8 vs laya**: more than 50pp ahead on comparable task domains (ext6kb reference: amos family 86.77% vs laya 45.07%); laya's only absolute advantage is raw latency.
+2. **amos-2.8 vs laya**: +53.8pp on the same 500-question board (96.6% vs 42.8% zero-shot) and +41.7pp on the ext6kb reference board (amos family 86.77% vs laya 45.07%); laya's only absolute advantage is raw latency (p50 ≈21 ms, local GPU direct).
 3. **Stability**: amos-2.8 recorded 0 API errors across four blind-test rounds with ±0.6pp variance; outputs are normalized probability distributions, 100% parseable, suitable for production threshold decisions and human-review triage.
 
 ## 7. Reproducibility
 
 - Per-question raw responses, question banks, wrong-answer details, and run metadata (reproducible seeds) for amos-2.8 are archived in the evaluation workspace `bench_out/` directory.
 - jev-1.13 baseline: 500-question same-board generator run (third-party OpenRouter endpoint).
-- laya baseline: ext6kb three-model board (seed=2027, batch 20260929-141401).
+- laya baseline: 500-question Board 1 zero-shot run (2026-10-09, identical request construction, rule-based gold) + ext6kb three-model board (seed=2027, batch 20260929-141401).