Place the full technical report on the main model page

README.md

@@ -23,7 +23,7 @@ tags:
 
 UI-TestJev uses DiffusionGemma to compare approved and current screenshots, identify UI defects, and return structured decisions for testing pipelines: pass, unknown, or a failure category.
 
-[Dataset](https://huggingface.co/datasets/Shelter/UI-testjev-data) · [Technical report](TECHNICAL_REPORT.md) · [Input format](INPUT_FORMAT.md)
+[Dataset](https://huggingface.co/datasets/Shelter/UI-testjev-data) · [Technical report](#technical-report) · [Use](#use) · [Input format](INPUT_FORMAT.md)
 
 ## What it does
 
@@ -47,7 +47,7 @@ Fine-tuning improves local-check macro-F1 by **41.70 percentage points** and ful
 
 Base DiffusionGemma is evaluated without the adapter, using the same INT8-expert/BF16-vision setup, screenshots and decision protocol. Fine-tuning increases false alarms: 1.77% → 16.31% on local checks and 0.76% → 3.05% on full-screen checks.
 
-*Luna is `gpt-6-luna` with reasoning off. These are development-test results. The weighted score combines local, full-screen and keyboard macro-F1 (40/50/10%). Jev latency is a local reference time on an RTX PRO 6000 Blackwell Server Edition; Luna includes HTTPS overhead. Jev cost uses djev's announced tariff as an estimate, not a quote for hosting this adapter. [Measurement and pricing details](TECHNICAL_REPORT.md#latency-and-cost).*
+*Luna is `gpt-6-luna` with reasoning off. These are development-test results. The weighted score combines local, full-screen and keyboard macro-F1 (40/50/10%). Jev latency is a local reference time on an RTX PRO 6000 Blackwell Server Edition; Luna includes HTTPS overhead. Jev cost uses djev's announced tariff as an estimate, not a quote for hosting this adapter. [Measurement and pricing details](#latency-and-cost).*
 
 ## Technical report
 
@@ -59,27 +59,112 @@ Synthetic examples are built as defect/harmless-change pairs. Browser measuremen
 
 The audit checked 6,536 images, removed 120 duplicate rows and excluded 21 ambiguous clipping pairs. No exact image or public-input hashes cross splits. The AdminLTE family informed earlier repairs, so the development test is not an untouched final holdout. [Dataset construction and coverage](https://huggingface.co/datasets/Shelter/UI-testjev-data).
 
+### Evaluation setup
+
+The main comparison uses 1,941 development-test decisions: 1,127 local checks, 787 full-screen checks and 27 keyboard checks. Base DiffusionGemma and UI-TestJev use the same pinned model, INT8 experts, BF16 vision, processor, prompts, option order and one-step decoding. The base arm has no LoRA adapter. This measures adaptation gains, not the effect of INT8 versus BF16 weights.
+
+GPT-6 Luna uses reasoning off and the same public requirements, ordered images and answer options. It uses its own image processing and autoregressive decoding through Codex HTTPS. All 3,098 Luna cases across validation, development test and Scry returned valid answers with zero reported reasoning tokens.
+
+Training source families are separate from validation and test. The AdminLTE test family was inspected during earlier dataset repair, however, so these are development results. Neither these test scores nor Scry selected the adapter checkpoint.
+
 ### Detection and false alarms
 
-Macro-F1 alone does not show whether the model misses defects or flags clean screens.
+**Defect recall** counts a defect as detected when the model predicts any failure category. **Precision** is the fraction of failure predictions made on actual defects. **False-alarm rate** is the fraction of passing examples assigned a failure category. A wrong defect category can therefore count as detection while still being an incorrect answer.
+
+#### Local requirement checks
 
-| Development-test diagnostic | Base | UI-TestJev | Luna |
+| Metric | Base | UI-TestJev | Luna |
 |---|---:|---:|---:|
-| Local defect recall | 21.31% | **80.99%** | 76.91% |
-| Local false-alarm rate | **1.77%** | 16.31% | 13.65% |
-| Full-screen defect recall | 14.76% | 38.17% | **59.54%** |
-| Full-screen false-alarm rate | **0.76%** | 3.05% | 2.54% |
+| Defect precision | 92.31% | 83.21% | 84.90% |
+| Defect recall | 21.31% | 80.99% | 76.91% |
+| False-alarm rate | 1.77% | 16.31% | 13.65% |
+| Exact-answer accuracy | 53.06% | 82.34% | 79.86% |
+| Answers marked unknown | 17.30% | 0.00% | 2.84% |
+
+#### Full-screen regression
+
+| Metric | Base | UI-TestJev | Luna |
+|---|---:|---:|---:|
+| Defect precision | 95.08% | 92.59% | 95.90% |
+| Defect recall | 14.76% | 38.17% | 59.54% |
+| False-alarm rate | 0.76% | 3.05% | 2.54% |
+| Exact-answer accuracy | 53.37% | 63.15% | 70.01% |
+| Answers marked unknown | 0.00% | 0.00% | 0.38% |
+
+On local checks, Jev detects **456/563 defects**, compared with **120/563** for the base. False alarms increase from **10/564** to **92/564** passing cases. On full-screen checks, detection rises from **58/393** to **150/393**, while false alarms rise from **3/394** to **12/394**.
+
+The benchmark has roughly equal numbers of defective and passing visual examples. Precision in a production pipeline will depend on how often defects actually occur.
+
+Macro-F1 in the model card averages F1 over the labels present in each task's gold answers. Visual tasks have no gold `unknown` labels: an unknown prediction counts as a miss for the true class, without adding an extra class to that average. The weighted score uses 40% local, 50% full-screen and 10% keyboard macro-F1. Missing or invalid predictions count as errors; none occurred in these runs.
+
+### Which errors remain?
+
+![Correct-category recall for base DiffusionGemma, UI-TestJev and Luna](report/category-recall.png)
+
+The chart requires the **correct defect category**, not merely any failure prediction. A dash means the category was not evaluated for that task. [Chart values](report/category-recall.csv).
+
+- **Control size:** local recall is 142/153, but control-size warnings account for 56 of 92 local false alarms. In full-screen checks, recall falls to 61/153; another 22 cases are called missing content.
+- **Contrast:** local recall is 88/103. Full-screen recall is 6/103, with 91 called pass. The model is much less reliable when it must find the affected region itself.
+- **Image aspect:** local recall is 2/12 and full-screen recall is 0/12. This remains a weak category, and the sample is small.
+- **Category confusion:** Jev flags 150 full-screen defects, but only 115 receive the right category. Reporting detection recall alone would hide 35 wrong-category answers.
+
+Local and full-screen tasks differ in their prompts, crops and category mix. Their score gap does not isolate the causal effect of cropping.
+
+#### Abstention and keyboard checks
+
+Jev gives no `unknown` answers on either visual test task. The training set has no visual unknown targets, so this does not demonstrate good handling of ambiguous screenshots. Its answer scores are not calibrated probabilities.
+
+Keyboard test accuracy is 24/27 for the base, 27/27 for Jev and 26/27 for Luna. Jev and Luna both recover all nine unknown cases; the base recovers eight. These are trace-based questions across nine scenarios, not evidence of broad keyboard-testing reliability.
+
+### Validation and uncertainty
+
+Epoch 3 is the exported checkpoint. Its weighted validation score is 67.26%, versus 39.88% for the base, but **no checkpoint met every acceptance rule**. In particular, local false alarms rose from 2.07% to 12.03%; the permitted maximum was 4.07%. Full-screen clipping recall also fell from 11/26 to 10/26. A 48-case noise probe changed decisions on 4 cases, versus 1 for the base.
+
+The export is therefore not recommended over the base under the original acceptance rules. It offers higher defect recall with a false-alarm tradeoff that requires review.
+
+For the Luna comparison, 2,000 paired bootstrap resamples keep each page or scenario together. On local checks, Luna minus Jev macro-F1 is -1.83 percentage points, with a 95% interval of [-6.17, +3.03]. On full-screen checks it is +10.27 points, [+4.46, +16.15]. Local results are close; the full-screen difference favors Luna. These intervals describe this corpus, not generalization to new repository families.
+
+### External Scry evaluation
+
+Scry contributes 311 reference/implementation pairs from 52 app groups. We ask 12 fixed category-presence questions per pair and score 505 annotated category instances.
+
+| Diagnostic | Base | UI-TestJev | Luna |
+|---|---:|---:|---:|
+| Annotated categories recovered | 266/505 | 337/505 | 410/505 |
+| Annotated-category recall | 52.67% | 66.73% | 81.19% |
+| Additional unannotated predictions | 994 | 1,214 | 1,856 |
+
+Jev recovers more annotated categories than the base, but also flags more unannotated categories. Scry does not contain exhaustive negatives, so those extra predictions cannot all be called false positives. This evaluation measures category presence, not localization or the original Scry benchmark score. Scry has been observed during development and is not a fresh blind test.
+
+### Training method
+
+| Setting | Completed run |
+|---|---|
+| Base | `google/diffusiongemma-26B-A4B-it`, revision `f7f5b7f5fa82ffc52addd066915886d497f5517b` |
+| Adapter | LoRA rank 32, alpha 64, dropout 0.05; decoder attention and dense MLP projections |
+| Frozen weights | Encoder, vision, experts, router, embeddings and output head |
+| Learning rate / epochs | 5e-5 / 3; one configuration, seed 3407 |
+| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.01, epsilon 1e-8 |
+| Batch / schedule | Microbatch 1, accumulation 16; 5% warmup, cosine decay to 10% of peak; gradient norm clipped to 1.0 |
+| Visual input | Original aspect-preserving processor; at most 1,120 visual tokens per image; 8,192 total input tokens; overflow rejected |
+
+The objective is cross-entropy over each question's allowed answer tokens. Supervised answer slots are always replaced with random vocabulary tokens before the decoder sees them. Option order changes during training and stays fixed during evaluation. This is a task-specific decision objective, not ordinary next-token chat training.
+
+All 1,590 training rows are visited once per epoch. Source families and pages are interleaved; related examples stay in sampling blocks, reshuffled each epoch. Passing visual examples receive 1.5x loss weight. No minority examples are duplicated to inflate the dataset.
+
+### Latency and cost
 
-Jev flags 150 of 393 full-screen defects, but gives the correct category for only 115. It correctly identifies 6 of 103 contrast defects and none of 12 image-aspect defects. Local checks are much stronger, but 56 of their 92 false alarms are control-size warnings.
+Jev's median development-test latency is **1.36 seconds**, with p95 **1.63 seconds**, on an RTX PRO 6000 Blackwell Server Edition. This is a **local reference time**, including preprocessing and model work with a vision-feature cache. Luna's HTTPS client median is **1.97 seconds**, p95 **3.00 seconds**, including request setup and network. The hardware and timing boundaries differ; these numbers do not establish a controlled hosted speedup.
 
-The [full report](TECHNICAL_REPORT.md) includes precision, category recall, abstention, confidence intervals and the external Scry results.
+For the same 1,941 decisions, the estimated Jev cost is **$0.2589**: 7,396,753 local input tokens priced once at [djev's announced $0.035/M input tariff](https://djev.dev/?view=usage). Its [billing contract](https://api.djev.dev/docs#billing) counts physical reads, so repeated context or provider scaffolding can change that estimate. The provider reported free preview on October 1, 2026. This adapter has not been deployed there.
 
-### Training and checkpoint selection
+Luna's estimate is **$0.7667 with the recorded cache mix**, or **$1.6112 without caching**, using [Standard API rates](https://developers.openai.com/api/docs/models/gpt-6-luna) checked on October 1: $0.10/M input, $0.01/M cached input and $0.50/M output. The run used Codex authentication, not a paid API invoice. Both cost estimates exclude retries, training and infrastructure.
 
-The run used rank-32 LoRA, alpha 64, dropout 0.05, a learning rate of 5e-5 and three epochs. Only decoder attention and dense MLP adapters were trained. The loss supervises fixed answer slots with randomly corrupted tokens; every training row is visited once per epoch, with page groups reshuffled. Visual passing examples receive 1.5x loss weight to penalize false alarms.
+### Data and reuse
 
-Epoch 3 had the highest validation score, but failed the acceptance rules: local false alarms rose from 2.07% to 12.03%, exceeding the allowed two-point increase. The release therefore supports reviewed findings rather than an automatic build-failure gate.
+See the [dataset card](https://huggingface.co/datasets/Shelter/UI-testjev-data) for source families, processing, checks and the public-input schema. The released archives preserve the original screenshots, labels and grouping records. They are dominated by controlled web-template mutations; native-app interactions, natural bugs and visual ambiguity need separate evaluation.
 
+[Diagnostic metrics](report/diagnostics.csv) · [Category-recall values](report/category-recall.csv). These tables contain aggregate results, not per-case predictions or development logs.
 
 ## Use
 

TECHNICAL_REPORT.md

@@ -1,112 +1,3 @@
 # UI-TestJev technical report
 
-UI-TestJev learns requirement-based UI decisions from paired screenshots. This report separates detection, false alarms and category errors so that a higher aggregate score can be interpreted in practical terms.
-
-[Model and usage](README.md) · [Dataset](https://huggingface.co/datasets/Shelter/UI-testjev-data)
-
-## Evaluation setup
-
-The main comparison uses 1,941 development-test decisions: 1,127 local checks, 787 full-screen checks and 27 keyboard checks. Base DiffusionGemma and UI-TestJev use the same pinned model, INT8 experts, BF16 vision, processor, prompts, option order and one-step decoding. The base arm has no LoRA adapter. This measures adaptation gains, not the effect of INT8 versus BF16 weights.
-
-GPT-6 Luna uses reasoning off and the same public requirements, ordered images and answer options. It uses its own image processing and autoregressive decoding through Codex HTTPS. All 3,098 Luna cases across validation, development test and Scry returned valid answers with zero reported reasoning tokens.
-
-Training source families are separate from validation and test. The AdminLTE test family was inspected during earlier dataset repair, however, so these are development results. Neither these test scores nor Scry selected the adapter checkpoint.
-
-## Detection and false alarms
-
-**Defect recall** counts a defect as detected when the model predicts any failure category. **Precision** is the fraction of failure predictions made on actual defects. **False-alarm rate** is the fraction of passing examples assigned a failure category. A wrong defect category can therefore count as detection while still being an incorrect answer.
-
-### Local requirement checks
-
-| Metric | Base | UI-TestJev | Luna |
-|---|---:|---:|---:|
-| Defect precision | 92.31% | 83.21% | 84.90% |
-| Defect recall | 21.31% | 80.99% | 76.91% |
-| False-alarm rate | 1.77% | 16.31% | 13.65% |
-| Exact-answer accuracy | 53.06% | 82.34% | 79.86% |
-| Answers marked unknown | 17.30% | 0.00% | 2.84% |
-
-### Full-screen regression
-
-| Metric | Base | UI-TestJev | Luna |
-|---|---:|---:|---:|
-| Defect precision | 95.08% | 92.59% | 95.90% |
-| Defect recall | 14.76% | 38.17% | 59.54% |
-| False-alarm rate | 0.76% | 3.05% | 2.54% |
-| Exact-answer accuracy | 53.37% | 63.15% | 70.01% |
-| Answers marked unknown | 0.00% | 0.00% | 0.38% |
-
-On local checks, Jev detects **456/563 defects**, compared with **120/563** for the base. False alarms increase from **10/564** to **92/564** passing cases. On full-screen checks, detection rises from **58/393** to **150/393**, while false alarms rise from **3/394** to **12/394**.
-
-The benchmark has roughly equal numbers of defective and passing visual examples. Precision in a production pipeline will depend on how often defects actually occur.
-
-Macro-F1 in the model card averages F1 over the labels present in each task's gold answers. Visual tasks have no gold `unknown` labels: an unknown prediction counts as a miss for the true class, without adding an extra class to that average. The weighted score uses 40% local, 50% full-screen and 10% keyboard macro-F1. Missing or invalid predictions count as errors; none occurred in these runs.
-
-## Which errors remain?
-
-![Correct-category recall for base DiffusionGemma, UI-TestJev and Luna](report/category-recall.png)
-
-The chart requires the **correct defect category**, not merely any failure prediction. A dash means the category was not evaluated for that task. [Chart values](report/category-recall.csv).
-
-- **Control size:** local recall is 142/153, but control-size warnings account for 56 of 92 local false alarms. In full-screen checks, recall falls to 61/153; another 22 cases are called missing content.
-- **Contrast:** local recall is 88/103. Full-screen recall is 6/103, with 91 called pass. The model is much less reliable when it must find the affected region itself.
-- **Image aspect:** local recall is 2/12 and full-screen recall is 0/12. This remains a weak category, and the sample is small.
-- **Category confusion:** Jev flags 150 full-screen defects, but only 115 receive the right category. Reporting detection recall alone would hide 35 wrong-category answers.
-
-Local and full-screen tasks differ in their prompts, crops and category mix. Their score gap does not isolate the causal effect of cropping.
-
-### Abstention and keyboard checks
-
-Jev gives no `unknown` answers on either visual test task. The training set has no visual unknown targets, so this does not demonstrate good handling of ambiguous screenshots. Its answer scores are not calibrated probabilities.
-
-Keyboard test accuracy is 24/27 for the base, 27/27 for Jev and 26/27 for Luna. Jev and Luna both recover all nine unknown cases; the base recovers eight. These are trace-based questions across nine scenarios, not evidence of broad keyboard-testing reliability.
-
-## Validation and uncertainty
-
-Epoch 3 is the exported checkpoint. Its weighted validation score is 67.26%, versus 39.88% for the base, but **no checkpoint met every acceptance rule**. In particular, local false alarms rose from 2.07% to 12.03%; the permitted maximum was 4.07%. Full-screen clipping recall also fell from 11/26 to 10/26. A 48-case noise probe changed decisions on 4 cases, versus 1 for the base.
-
-The export is therefore not recommended over the base under the original acceptance rules. It offers higher defect recall with a false-alarm tradeoff that requires review.
-
-For the Luna comparison, 2,000 paired bootstrap resamples keep each page or scenario together. On local checks, Luna minus Jev macro-F1 is -1.83 percentage points, with a 95% interval of [-6.17, +3.03]. On full-screen checks it is +10.27 points, [+4.46, +16.15]. Local results are close; the full-screen difference favors Luna. These intervals describe this corpus, not generalization to new repository families.
-
-## External Scry evaluation
-
-Scry contributes 311 reference/implementation pairs from 52 app groups. We ask 12 fixed category-presence questions per pair and score 505 annotated category instances.
-
-| Diagnostic | Base | UI-TestJev | Luna |
-|---|---:|---:|---:|
-| Annotated categories recovered | 266/505 | 337/505 | 410/505 |
-| Annotated-category recall | 52.67% | 66.73% | 81.19% |
-| Additional unannotated predictions | 994 | 1,214 | 1,856 |
-
-Jev recovers more annotated categories than the base, but also flags more unannotated categories. Scry does not contain exhaustive negatives, so those extra predictions cannot all be called false positives. This evaluation measures category presence, not localization or the original Scry benchmark score. Scry has been observed during development and is not a fresh blind test.
-
-## Training method
-
-| Setting | Completed run |
-|---|---|
-| Base | `google/diffusiongemma-26B-A4B-it`, revision `f7f5b7f5fa82ffc52addd066915886d497f5517b` |
-| Adapter | LoRA rank 32, alpha 64, dropout 0.05; decoder attention and dense MLP projections |
-| Frozen weights | Encoder, vision, experts, router, embeddings and output head |
-| Learning rate / epochs | 5e-5 / 3; one configuration, seed 3407 |
-| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.01, epsilon 1e-8 |
-| Batch / schedule | Microbatch 1, accumulation 16; 5% warmup, cosine decay to 10% of peak; gradient norm clipped to 1.0 |
-| Visual input | Original aspect-preserving processor; at most 1,120 visual tokens per image; 8,192 total input tokens; overflow rejected |
-
-The objective is cross-entropy over each question's allowed answer tokens. Supervised answer slots are always replaced with random vocabulary tokens before the decoder sees them. Option order changes during training and stays fixed during evaluation. This is a task-specific decision objective, not ordinary next-token chat training.
-
-All 1,590 training rows are visited once per epoch. Source families and pages are interleaved; related examples stay in sampling blocks, reshuffled each epoch. Passing visual examples receive 1.5x loss weight. No minority examples are duplicated to inflate the dataset.
-
-## Latency and cost
-
-Jev's median development-test latency is **1.36 seconds**, with p95 **1.63 seconds**, on an RTX PRO 6000 Blackwell Server Edition. This is a **local reference time**, including preprocessing and model work with a vision-feature cache. Luna's HTTPS client median is **1.97 seconds**, p95 **3.00 seconds**, including request setup and network. The hardware and timing boundaries differ; these numbers do not establish a controlled hosted speedup.
-
-For the same 1,941 decisions, the estimated Jev cost is **$0.2589**: 7,396,753 local input tokens priced once at [djev's announced $0.035/M input tariff](https://djev.dev/?view=usage). Its [billing contract](https://api.djev.dev/docs#billing) counts physical reads, so repeated context or provider scaffolding can change that estimate. The provider reported free preview on October 1, 2026. This adapter has not been deployed there.
-
-Luna's estimate is **$0.7667 with the recorded cache mix**, or **$1.6112 without caching**, using [Standard API rates](https://developers.openai.com/api/docs/models/gpt-6-luna) checked on October 1: $0.10/M input, $0.01/M cached input and $0.50/M output. The run used Codex authentication, not a paid API invoice. Both cost estimates exclude retries, training and infrastructure.
-
-## Data and reuse
-
-See the [dataset card](https://huggingface.co/datasets/Shelter/UI-testjev-data) for source families, processing, checks and the public-input schema. The released archives preserve the original screenshots, labels and grouping records. They are dominated by controlled web-template mutations; native-app interactions, natural bugs and visual ambiguity need separate evaluation.
-
-[Diagnostic metrics](report/diagnostics.csv) · [Category-recall values](report/category-recall.csv). These tables contain aggregate results, not per-case predictions or development logs.
+The full report is now part of the [main model page](https://huggingface.co/Shelter/UI-testjev#technical-report), including diagnostic metrics, dataset design, category errors, training, and runtime measurements.

sha256.json

@@ -13,13 +13,13 @@
   "NOTICE": "c7c1f72dae534a0513a7961c385c258d6672814c34f8ece01e0a3c9d3a898e48",
   "processor_config.json": "54943f08b029187ee4d21a6c23a270c609417791e3fcad7853a029e58394b23c",
   "quantized.py": "e8c0a9df02540fe05211949bdc88499e8a10b27c0137cf5254e9886bc6e15863",
-  "README.md": "f620faa631e26ee038bd427a7c724c8e663aa640414449d285cd8d6feab07fc2",
+  "README.md": "e730578947c096514c925888d28910f7daa4b2de58b23ac182434dd72feb5076",
   "report/category-recall.csv": "5f88b879df321a7f3e0c2aa869169689f93874e92c8316656426425bc418d63f",
   "report/category-recall.png": "0fd8787424eae860104074faac66dedb4e148e0958636e4744bea6b3c4941e50",
   "report/diagnostics.csv": "c391c71b48ddac700dd20ee341f7dd2267496346af061d141230ab2e2f009478",
   "requirements.lock.txt": "3f9351705bf608e3abf6a0065b5f7b147a8a0b8123131029a238e484d4e4771a",
   "requirements.txt": "2748c956c547e31333f537827bd8f6aa71df66ef073c566c6a9df2ef79a25538",
-  "TECHNICAL_REPORT.md": "87cfa15cf6df7734bb82fcd97bc9fb76c236e60522d71b7b6d07ad2471a8c292",
+  "TECHNICAL_REPORT.md": "0a9ee53b3fc281c632201ac548ddac1671172b3934661b77512e7a4414159d65",
   "tokenizer.json": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f",
   "tokenizer_config.json": "a1bab8c81ed15fa6ce912ec993c66cb49392e0487fb1ea5f5f11ea3618683627"
 }