Visual Verdict 2B (kinasr/visual-verdict-2b) 🎯

Visual Verdict 2B is a specialized System 1 Typed-Decision Vision-Language Model fine-tuned for Automated Web Visual Regression Testing & UI Triage.

Built on top of LiquidAI/d1-2B (powered by the LFM3-VL-2B tri-hybrid architecture), Visual Verdict 2B evaluates pairs of web screenshots (Baseline Reference vs. Current Render) alongside structured browser telemetry (app_state) and outputs temperature-calibrated probabilities in a single forward pass—without autoregressive token generation, JSON formatting errors, or pipeline hallucinations.

---

✨ Why Visual Verdict 2B?

Traditional pixel-diffing tools fail on harmless dynamic updates (timestamps, counters, benign text changes) and provide zero diagnostic context on what broke. Conversely, generative VLMs suffer from high latency and unstable outputs in CI/CD pipelines.

Visual Verdict 2B bridges both worlds:

• ⚡ Single Forward-Pass Decisions: Evaluates [Reference Image + Current Image + Browser DOM Telemetry] via SlotReader in milliseconds without generating text tokens.
• 🎯 6-Class Semantic Triage: Classifies visual diffs into 5 actionable QA outcomes plus an explicit Unknown / Needs Human Review gate:
  1. Pass (No visual regression): Includes exact re-renders and benign dynamic data changes (benign_dynamic).
  2. Layout shift, overlap, or misalignment: Flex/Grid breaks, colliding components, or unintended flow offsets.
  3. Missing interactive element or CTA: Disappeared buttons, inputs, or navigation slots (inert_navigation_slot, inert_control_slot).
  4. Broken CSS styling or unreadable contrast: Corrupted design tokens, typography breaks, or unreadable text contrast across Light & Dark themes.
  5. Broken image, icon, or media asset: Client-side media render failures, broken SVGs/icons, or unloaded thumbnails.
  6. Unknown (Ambiguous / Needs Human QA Review): Cookie consent banners (consent_interception), loading overlays/skeletons (loading_interception), incomplete captures (incomplete_capture), or full route mismatches (page_route_mismatch).
• 🌡️ Calibrated Confidence Scores: Post-trained with LBFGS temperature scaling (calibration.json) on held-out validation logits so CI/CD pipelines can safely auto-gate high-confidence verdicts and route uncertain edge cases to human QA engineers.

---

📊 Benchmark & Evaluation Results

Evaluated under matched settings (640px image cap, 4 cyclic option rotations, MPS float16, native SlotReader temperature calibration, and zero pixel short-circuiting) across both the Out-of-Distribution (OOD v3, 360 pairs) benchmark and the historical dataset-v2 (300 pairs) test split:

1. Overall 6-Class Accuracy & Latency

Model	OOD v3 (360 pairs)	Historical v2 (300 pairs)	Binary Pass/Defect (OOD v3)	Binary Pass/Defect (v2)	OOD Mean Latency (s)*	v2 Mean Latency (s)*
kinasr/visual-verdict-2b (Ours)	76.9% (277/360)	58.7% (176/300)	91.9%	97.0%	3.143s	2.888s
kinasr/imajev-2b-web-visual-testing	61.4%	76.3%	89.7%	97.0%	4.973s	7.496s
LiquidAI/d1-2B (Base Zero-Shot)	35.6%	43.0%	58.3%	—	3.081s	3.305s
LiquidAI/LFM3-VL-2B (Zero-Shot)	16.9%	18.0%	19.4%	—	3.460s	3.137s
pixel-diff (Traditional Baseline)	25.0%	33.3%	—	—	0.023s	0.020s

*Latency covers all 4 cyclic option rotations and image preprocessing on Apple Silicon MPS (float16), excluding model loading (~0.75s per single forward pass).

Key Takeaways:

• State-of-the-Art OOD Generalization: visual-verdict-2b leads the unseen OOD v3 benchmark at 76.9%, outperforming imajev-2b-web-visual-testing by +15.56 percentage points and improving over its base LiquidAI/d1-2B model by +41.39 points on OOD v3 (and +15.67 points on v2).
• 1.6x–2.6x Faster Inference: Thanks to the LFM3-VL-2B tri-hybrid backbone, visual-verdict-2b reduces end-to-end 4-rotation evaluation latency from 4.97s–7.50s down to 2.89s–3.14s.
• Note on Historical v2 Split: dataset-v2 is a historical evaluation set whose training distribution was used to train imajev-2b-web-visual-testing (in-distribution for Imajev, out-of-distribution for this v3-trained checkpoint). Even without seeing v2 domains, visual-verdict-2b achieves 97.0% binary Pass/non-Pass accuracy on v2.

2. Per-Class Recall (kinasr/visual-verdict-2b)

Canonical Class	OOD v3 Recall (60/class)	Historical v2 Recall (50/class)
Pass (No visual regression)	60.0%	100.0%
Layout (Shift, overlap, misalignment)	90.0%	60.0%
Missing (Missing interactive element / CTA)	81.7%	68.0%
Styling (Broken CSS or unreadable contrast)	70.0%	24.0%
Asset (Broken image, icon, or media)	100.0%	64.0%
Unknown (Ambiguous / Needs Human Review)	60.0%	36.0%

3. Confidence Calibration & Paired Outcomes

• High-Confidence Gating (OOD v3): At confidence $\ge 0.90$, the model covers 77.8% of the benchmark with 84.3% empirical accuracy (ECE = 0.1556, Brier = 0.3797).
• Head-to-Head vs. imajev-2b-web-visual-testing (OOD v3): Out of 360 pairs, both models agreed on 196 correct predictions; visual-verdict-2b uniquely solved 81 pairs missed by Imajev, while Imajev uniquely solved 25 (+56 net pair advantage).

---

🏗️ Model Architecture & Repository Structure

Property	Specification
Base Architecture	LiquidAI/d1-2B (LFM3-VL-2B Tri-Hybrid: 13 Gated Short Conv + 5 Sliding Window + 6 Global Attention layers)
Parameters	1.92B
Vision Encoder	164M Native-Resolution Vision Tower
Fine-Tuning Method	Merged LoRA (r=16, lora_alpha=32) over unfused language projections (q_proj, k_proj, v_proj, out_proj, in_proj, gate_proj, w1, w2, w3)
Training Dataset	dataset-v3-train (1,800 training + 240 validation browser-rendered UI pairs across Light/Dark themes and 18 fine-grained regression subtypes with strict DOM telemetry parity)
Weights Format	Standalone merged model.safetensors (bfloat16) + original LoRA adapter backup in ./lora_adapter/

---

🚀 Quickstart & Inference

1. Installation

pip install -U "transformers>=5.19.0" accelerate safetensors pillow torch torchvision

2. Running a Visual Regression Verdict in Python

import sys
import json
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "kinasr/visual-verdict-2b"
DEVICE = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
DTYPE = torch.float16 if DEVICE != "cpu" else torch.float32

# 1. Load processor and standalone fine-tuned model
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=DTYPE,
).to(DEVICE).eval()

# 2. Initialize the d1 SlotReader decision engine
pkg_name = model.__class__.__module__.rsplit(".", 1)[0]
slots_mod = sys.modules[f"{pkg_name}.slots"]
engine = slots_mod.SlotReader(model, processor)

# 3. Prepare Reference & Current Screenshots + Browser Telemetry State
ref_img = Image.open("reference.png").convert("RGB")
curr_img = Image.open("current.png").convert("RGB")

app_state = {
    "viewport": {"width": 1280, "height": 800},
    "console_error_count": 0,
    "failed_network_count": 0,
    "visible_button_count": 6
}

options = [
    "Pass (No visual regression)",
    "Layout shift, overlap, or misalignment",
    "Missing interactive element or CTA",
    "Broken CSS styling or unreadable contrast",
    "Broken image, icon, or media asset",
    "Ambiguous, consent banner, loading overlay, or incomplete capture requiring human QA review"
]

criteria = {f"opt_{i}": text for i, text in enumerate(options)}
question = {
    "type": "choice",
    "instructions": "Compare the current screenshot against the baseline reference and classify the visual regression outcome.",
    "criteria": criteria
}

# 4. Single forward-pass decision
result = engine.decide(
    state=app_state,
    questions={"visual_verdict": question},
    images=[ref_img, curr_img]
)

print(json.dumps(result, indent=2))

---

🛠️ Recommended Production Pipeline (Hybrid CI/CD Gating)

1. Fast Pixel Short-Circuit: Run a fast pixel compare first. If pixel_diff == 0%, pass immediately in <5ms without invoking the GPU.
2. Semantic AI Verdict: When pixel_diff > 0%, pass the screenshot pair and browser state to kinasr/visual-verdict-2b to filter out benign dynamic content (false positives) and diagnose the exact defect category.
3. Confidence Routing: Auto-decide test runs when calibrated confidence >= 0.75 (or >= 0.90 for strict gates), and route lower-confidence predictions or Unknown verdicts (e.g., cookie banners or loading spinners) to human QA review.

---

🔗 Related Models in the Verdict & Visual Testing Suite

• kinasr/imajev-2b-web-visual-testing — Qwen3.5-2B / Imajev-based visual regression decision model.

📜 License

This model inherits the LFM Open License v1.0 from its base model LiquidAI/d1-2B.