---
license: apache-2.0
base_model: Qwen/Qwen3.5-2B
tags:
- vision
- visual-regression-testing
- web-testing
- quality-assurance
- peft
- lora
- typed-decision
pipeline_tag: image-text-to-text
library_name: peft
---

# imajev-2b-web-visual-testing 🎯

**imajev-2b-web-visual-testing** is a specialized, typed-decision Vision-Language Model fine-tuned for **Automated Web Visual Regression Testing**. 

Unlike conventional VLMs that generate free-form text or unstable JSON (introducing latency and pipeline flakiness), this model operates as a **single-forward-pass semantic classifier** built on top of `Qwen/Qwen3.5-2B` using LoRA and a calibrated Float32 decision readout head (`decision_readout.safetensors`).

It accepts two screenshot images (Reference/Golden Baseline vs. Current Screen) along with live browser telemetry (`app_state`), evaluates the diff, and outputs calibrated probabilities across 5 regression categories plus an explicit `unknown` state for human review.

---

## 📊 Benchmark & Evaluation Results

Tested on a held-out, balanced 300-row test dataset (`dataset-v2` test split across real-world web sources including Applitools & dynamic web apps, 50 rows per class):

| Metric | Base `imajev-2b` | **`imajev-2b-web-visual-testing` (Ours)** |
| :--- | :---: | :---: |
| **Overall Accuracy (All 300)** | 36.7% | **78.7%** (+42.0%) |
| **Accuracy (250 Known Classes)** | 40.4% | **79.2%** (+38.8%) |
| **Pass Recall** | 100% *(biased)* | **100%** |
| **Layout Recall** | 10% | **52%** |
| **Missing Recall** | 30% | **86%** |
| **Styling Recall** | 0% | **90%** |
| **Asset Recall** | 62% | **68%** |
| **Unknown Recall (Needs Review)** | 18% | **76%** |
| **Decided at $\ge 0.90$ Conf** | 5% (67% right) | **52% (87.8% right)** |
| **Decided at $\ge 0.75$ Conf** | 36% (57% right) | **76% (84.3% right)** |

> **Key Takeaway:** The base model suffered from extreme confirmation bias, calling 174 of 300 samples "Pass" and missing most true regressions. This fine-tune eliminated that bias, rescued **129 rows** missed by the base model, and enables **safe automated gating on over 75% of CI/CD test runs**.

---

## 🏗️ Architecture & Model Files

This repository contains LoRA adapter weights and custom readout files designed to be loaded over `Qwen/Qwen3.5-2B`:
* `adapter_model.safetensors` & `adapter_config.json`: LoRA weights (Rank 16, Alpha 32).
* `decision_readout.safetensors`: Custom linear readout layer mapping last hidden state to decision logits.
* `calibration.json`: Temperature scaling configuration (`T = 1.2334`) for probability calibration.

---

## 🚀 Quickstart & Inference

### 1. Requirements

```bash
pip install -q transformers peft safetensors pillow torch
```

### 2. Python Inference Pipeline
```python
import json
import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

REPO_ID = "kinasr/imajev-2b-web-visual-testing"
BASE_MODEL_ID = "Qwen/Qwen3.5-2B"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Load Processor & Base Model
processor = AutoProcessor.from_pretrained(REPO_ID)
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=torch.float16 if DEVICE != "cpu" else torch.float32,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, REPO_ID).eval()

# 2. Load Decision Readout Head
readout_path = hf_hub_download(repo_id=REPO_ID, filename="decision_readout.safetensors")
readout_state = load_file(readout_path)
num_codes, hidden_dim = readout_state["weight"].shape
has_bias = "bias" in readout_state

decision_head = nn.Linear(hidden_dim, num_codes, bias=has_bias, dtype=torch.float32).to(DEVICE)
decision_head.weight.data.copy_(readout_state["weight"].to(torch.float32))
if has_bias:
    decision_head.bias.data.copy_(readout_state["bias"].to(torch.float32))
decision_head.eval()

# 3. Load Calibration Temperature
cal_path = hf_hub_download(repo_id=REPO_ID, filename="calibration.json")
with open(cal_path, "r") as f:
    temp = json.load(f).get("temperature", 1.2334)

# 4. Run Visual Regression Check
def classify_visual_diff(ref_image_path: str, curr_image_path: str, app_state: dict):
    options = [
        "Pass (No visual regression)",
        "Layout shift, overlap, or misalignment",
        "Missing interactive element or CTA",
        "Broken CSS styling or unreadable contrast",
        "Broken image, icon, or media asset"
    ]
    
    img_ref = Image.open(ref_image_path).convert("RGB")
    img_curr = Image.open(curr_image_path).convert("RGB")
    img_ref.thumbnail((640, 640))
    img_curr.thumbnail((640, 640))
    
    opts_str = "\n".join([f"[{i}] {opt}" for i, opt in enumerate(options)])
    messages = [
        {"role": "system", "content": [{"type": "text", "text": "You are a typed decision model for web visual regression testing. Output a calibrated decision over the provided options or unknown."}]},
        {"role": "user", "content": [
            {"type": "image"}, {"type": "image"},
            {"type": "text", "text": f"App State: {json.dumps(app_state)}\nQuestion: Compare the current screenshot against baseline reference and classify the regression.\nType: choice\nOptions:\n{opts_str}\nDecision:"}
        ]}
    ]
    
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = processor(text=[prompt], images=[img_ref, img_curr], return_tensors="pt").to(DEVICE)
    
    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)
        last_hidden = outputs.hidden_states[-1][:, -1, :].to(torch.float32)
        active_logits = decision_head(last_hidden)[:, :len(options) + 1]
        probs = F.softmax(active_logits / temp, dim=-1).squeeze(0).cpu().tolist()
        
    pred_idx = int(torch.argmax(active_logits, dim=-1).item())
    verdict = options[pred_idx] if pred_idx < len(options) else "Unknown (Needs Human Review)"
    
    return {
        "verdict": verdict,
        "confidence": round(probs[pred_idx], 4),
        "unknown_prob": round(probs[-1], 4),
        "needs_review": (pred_idx == len(options) or probs[pred_idx] < 0.75)
    }

```