|
Download docs/LAYA_COMPARISON.md from ken-jo/qev: direct link, hf CLI and curl.
- Browser
- Download file 8.34 kB
-
https://huggingface.co/ken-jo/qev/resolve/main/docs/LAYA_COMPARISON.md
- Command line
-
hf download hf://ken-jo/qev/docs/LAYA_COMPARISON.md
-
curl -L -o LAYA_COMPARISON.md https://huggingface.co/ken-jo/qev/resolve/main/docs/LAYA_COMPARISON.md
8.34 kB
| # QEV and LAYA: final release comparison | |
| Measured on 2026-10-01. All three frozen models answered the **same 2,800 English | |
| questions** on one RTX 4060 Ti 8 GB. No weights, temperatures or prompts were tuned | |
| after inspecting these results. Raw prediction hashes, source revisions, input audits | |
| and aggregate results are in [the evidence directory](https://github.com/ken-jo/qev/blob/archive/pre-qev/reports/release-comparison/README.md) | |
| (archived at the `archive/pre-qev` tag). | |
|  | |
| ## Accuracy: higher is better | |
| | Test | Questions | LAYA English | LAYA Typed Decisions | QEV 0.1.1 | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | Official typed-decisions | 2,000 | 36.05% | 76.95% | 77.00% | | |
| | AG News, 4 candidates | 400 | 95.00% | 95.25% | 82.50% | | |
| | DAIR Emotion, 6 candidates | 400 | 58.75% | 60.00% | 50.25% | | |
| On typed-decisions, QEV answers one more question correctly than the specialist: | |
| 1,540 versus 1,539. The paired group-bootstrap 95% interval for the difference is | |
| **-1.85 to +1.95 percentage points**. This does not demonstrate an accuracy advantage. | |
| Relative to the specialist, QEV is 12.75 points lower on news and 9.75 points lower on | |
| emotion; the respective descriptive intervals are -16.75 to -9.00 and -14.75 to -4.75 | |
| points. These intervals use 10,000 paired group resamples and are not adjusted for | |
| multiple comparisons. | |
| ## Which results are zero-shot? | |
| | Test | QEV adaptation exposure | LAYA exposure | Interpretation | | |
| | --- | --- | --- | --- | | |
| | typed-decisions | Trained on the recorded training partition | Specialist fine-tuned on this benchmark; base not specialized | Adapted regression comparison | | |
| | AG News | No source or matching test text in released adaptation snapshots | In LAYA's training mix according to its benchmark script | QEV task transfer; unequal training exposure | | |
| | DAIR Emotion | No source or matching test text in released adaptation snapshots | Reported held out by LAYA's benchmark script | Task-held-out transfer for both | | |
| Here **zero-shot means no QEV task-specific adaptation, examples in the prompt, or | |
| test-time fitting**. The audit covered 15 released corpus snapshots and 251,457 rows, | |
| including repeated stages. It found no source IDs for AG News or Emotion and no | |
| normalized test-text substring matches. This does not prove absence from upstream | |
| Qwen or ModernBERT pretraining, undistributed intermediate material, or semantic near | |
| duplicates. The two 400-example suites are a limited English transfer study. | |
| LAYA's [application benchmark source](https://github.com/NandhaKishorM/laya/blob/main/research/scripts/bench_apps.py) | |
| uses the first 400 test examples for these tasks. We retain that selection and its | |
| instructions. Emotion's null candidate descriptions are expressed as their label names | |
| for both SDKs. Both models receive the same serialized text and question dictionaries. | |
| The resulting scores are our measurements, not copied from LAYA's published tables. | |
| ## Probability quality on typed-decisions: lower is better | |
| | Metric | LAYA English | LAYA Typed Decisions | QEV 0.1.1 | | |
| | --- | ---: | ---: | ---: | | |
| | Brier / squared error against soft targets | 0.31545 | 0.06149 | 0.07460 | | |
| | NLL / cross-entropy against soft targets | 1.34337 | 0.88445 | 0.90894 | | |
| | ECE, 15 bins | 17.54% | 21.67% | 25.19% | | |
| | Ordinal expectation MAE | 0.69457 | 0.24251 | 0.30366 | | |
| The specialist has lower probability and ordinal errors than QEV on this test. | |
| Accuracy and calibration are different: a model can pick the right candidate while | |
| assigning poorly calibrated probabilities. QEV's fitted abstention accepts 30.80% of | |
| these questions, at 94.32% accepted accuracy. **The 77.00% headline includes all 2,000 | |
| argmax decisions, including abstentions.** The accepted subset is not substituted for | |
| the full-test denominator. LAYA is scored without an added abstention threshold. | |
| For completeness, QEV's news/emotion ECE is 6.68%/9.20%; the base is 2.95%/32.07% and | |
| the specialist is 15.50%/20.08%. Lower ECE on emotion does not erase QEV's accuracy gap. | |
| Full NLL and Brier values for both transfer tasks are included in the JSON reports. | |
| ### Decision types and workflows | |
| | Type | Questions | LAYA Typed Decisions | QEV 0.1.1 | | |
| | --- | ---: | ---: | ---: | | |
| | choice | 600 | 73.50% | 73.50% | | |
| | score, probability argmax | 800 | 72.88% | 74.75% | | |
| | noul | 600 | 85.83% | 83.50% | | |
| | Workflow | Questions | LAYA Typed Decisions | QEV 0.1.1 | | |
| | --- | ---: | ---: | ---: | | |
| | Agent trace observability | 500 | 73.40% | 70.80% | | |
| | Customer service | 500 | 76.80% | 81.40% | | |
| | Invoice processing | 500 | 80.60% | 83.80% | | |
| | Security incidents | 500 | 77.00% | 72.00% | | |
| ## Response time on the same GPU: lower is better | |
| Resident serial SDK calls, one question per call, four CPU threads and three warmups | |
| per suite. No concurrent model server was running. These measurements include each | |
| SDK's request processing but exclude model loading, HTTP, WAN and queueing. | |
| | Test | LAYA base p50 / p95 | LAYA specialist p50 / p95 | QEV p50 / p95 | | |
| | --- | ---: | ---: | ---: | | |
| | typed-decisions | 24.22 / 29.00 ms | 23.71 / 30.06 ms | 77.70 / 97.70 ms | | |
| | AG News | 21.50 / 24.86 ms | 27.72 / 34.59 ms | 52.34 / 61.24 ms | | |
| | Emotion | 21.61 / 25.17 ms | 22.90 / 30.60 ms | 50.72 / 56.11 ms | | |
| QEV is about 3.3 times slower at typed-decisions p50 in this run. It still falls within | |
| the project's 300 ms resident response-time target on these workloads; this is not a | |
| concurrent-service SLA. LAYA uses its released CUDA path; QEV uses BF16 backbone weights | |
| and FP32 readouts. Optional kernel acceleration was not added to either installation. | |
| ## Vision and model size | |
| The tested LAYA checkpoints accept text. QEV also accepts an image with textual criteria. | |
| Its separately measured CIFAR-10 guard accuracy is 95.83% on 600 questions, and resident | |
| local-photo HTTP p95 is 114.94 ms on the same GPU class. Those are **separate image | |
| evaluations**, not evidence of superiority over a text-only model. See | |
| [evaluation limits](https://github.com/ken-jo/qev/blob/main/docs/EVALUATION.md). | |
| LAYA's English models are reported as 421M parameters. QEV's merged inference model | |
| has 2,213,418,880 parameters, about 5.3 times as many. Its 32 MB adaptation file is not | |
| the full model. See [exact counts and dtypes](MODEL_SIZE.md). | |
| ## Frozen protocol and metric definitions | |
| - Dataset: `LocalLLaMA/typed-decisions`, revision | |
| `d51d993547ad8355b1c25157fbc1fea0649e8ffa`, all 400 official test cases / 2,000 questions. | |
| Gold labels are teacher annotations rather than independently observed outcomes. | |
| - LAYA SDK: 0.3.20. Base revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; | |
| specialist revision `1a793eb568e6718f15941d08f85432581df534e3`. | |
| - LAYA context: 1,024 tokens for both checkpoints; released option-head budgets and | |
| temperatures retained. The base's stock 512-token context would truncate 34 official | |
| questions. The actual comparison has **zero state, instruction and option truncations**. | |
| QEV uses its released 2,048-token limit and rejects over-budget requests. | |
| - Accuracy uses probability argmax against explicit hard gold. Brier sums squared | |
| candidate-probability differences against the target distribution. NLL is target | |
| cross-entropy, with probabilities clipped at 1e-12. ECE uses 15 equal-width bins | |
| against hard correctness. Score MAE compares expected zero-based ordinal levels. | |
| - The same frozen metric implementation processes all predictions. LAYA's published | |
| SDK rounds probabilities to four decimals; returned distributions are normalized | |
| before scoring. SDK defaults for criteria-free noul questions remain SDK-specific. | |
| - No multilingual router, full 77-class BANKING77, 51-language sweep or live JEV API | |
| comparison is claimed. QEV currently supports at most 16 candidates and four questions. | |
| **Conclusion:** QEV adds direct image inputs to a LAYA-inspired decision interface and | |
| matches the specialist's aggregate accuracy on this adapted text benchmark within the | |
| measured uncertainty. LAYA remains smaller and faster here, with stronger probability | |
| quality on typed-decisions and higher accuracy on both transfer suites. These findings | |
| close the current Qwen-based research release and guide the compact-model roadmap. | |