README.md

---
license: mit
tags:
  - eye-tracking
  - gaze-estimation
  - computer-vision
  - onnx
  - ux-research
library_name: onnx
pipeline_tag: image-classification
---

# InsightUX — Webcam Eye-Tracking UX Research Browser

A webcam-based gaze tracking system that turns any consumer laptop into a UX
research tool. Browse a website, and get back a heatmap, an attention timeline,
and a ranked list of which page elements actually held your gaze.

No specialist hardware — just a webcam.

**This is a local desktop app, not a hosted demo.** It needs a physical
webcam, a native window, and full-screen capture — none of which exist on a
server, so it cannot run as a Hugging Face Space. Clone it and run it on your
own machine.

## Requirements before you start

- **Windows** (uses WebView2 via pywebview's WinForms backend)
- **Python 3.11 exactly** — mediapipe's legacy face-mesh API this project
  depends on does not exist on Python 3.12+
- A working webcam

## How to run — step by step

### 1. Clone the repo

```bash
git clone https://huggingface.co/<your-username>/insightux
cd insightux
```

### 2. Create and activate a virtual environment

```bash
python -m venv venv
venv\Scripts\activate
```

### 3. Install dependencies — use the pinned versions

```bash
pip install -r requirements.txt
```

This includes `onnx`, which `calibrate.py` needs when it exports the
fine-tuned model after training.

**Do not `pip install --upgrade` pywebview or pythonnet.** `requirements.txt`
pins `pywebview==4.4.1` and `pythonnet==3.0.3` deliberately — newer versions
have a bug in their Windows backend that freezes the app window and floods
the console with:
```
AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...
```
If you already have a newer version installed globally, this install step
will replace it with the working one.

### 4. Calibrate — required, per person, per setup

```bash
python calibrate.py
```
Sit normally at your usual distance from the screen, look at each of the 16
dots as they appear. Takes about a minute. This is **per-person and
per-setup** — your eyes, your camera, your screen size. Everyone using this
must run it themselves; it is not something you can copy from someone else.

Re-run it if your lighting, seating position, or camera position changes
noticeably.

At the end it prints an honest quality readout, including whether the model
can actually see where you're looking on each axis:
```
HORIZONTAL  yaw vs screen-X : r = +0.995
VERTICAL    pitch vs screen-Y: r = +0.883
```
If either number is low, the report will say so plainly and explain why —
that means the model isn't seeing that axis, and no amount of recalibrating
will fix it.

### 5. (Optional) Check your accuracy

```bash
python validate.py
```
Flashes 9 test dots and reports your real error in pixels. Good for knowing
what to expect before relying on a session.

### 6. Run the browser

```bash
python browser_session.py
```
- Opens on an InsightUX-branded search page. Type a search term (goes to
  real Google results) or a URL (goes straight there).
- Land on the page you want to study, **click on blank space on the page**
  (not a text field), then press **S** to start eye-tracking.
- Press **E** to stop. A full report — heatmap over real screenshots,
  ranked attention list, dwell timeline — generates and opens automatically.

## What's in this repo

| File | Purpose |
|---|---|
| `models/gaze_cnn_v4.onnx` (+ `.onnx.data`) | Binocular gaze CNN (EfficientNet-B0 backbone, dual eye patches + head pose) |
| `models/model_v4.py` | Model architecture definition, used during calibration fine-tuning |
| `checkpoints/best_model_v4.pt` | PyTorch checkpoint for fine-tuning |
| `calibrate.py` | Per-user calibration — 16-point, live blink/lighting rejection, honest quality report |
| `validate.py` | Measures real accuracy in pixels after calibration |
| `browser_session.py` | The research browser — search, track, auto-report |
| `analysis.py` | Builds the session report (heatmaps over real screenshots) |
| `inference_pipeline.py` | ONNX inference + RBF gaze→screen calibration mapping |
| `preprocessing/preprocessing_pipeline.py` | Eye patch normalization, head pose estimation, illumination correction |

## Calibration is per-person and per-setup — not included in this repo

`calibration.pkl` (generated by `calibrate.py`) encodes *your* eye geometry,
*your* camera characteristics, and *your* screen size. It is deliberately
**not** in this repo — it would be useless to anyone else and it's personal
data. Run `calibrate.py` yourself; it takes about a minute.

## Accuracy — honest numbers

This is a webcam system, not a Tobii. Expect roughly **5–10% of screen
diagonal** mean error after a good calibration. That's enough for coarse AOI
attribution (navbar vs hero vs footer) and heatmaps. It is **not** enough for
reading-level analysis (which word you're on).

Vertical accuracy is typically a bit looser than horizontal — looking down
partially occludes the iris under the eyelid, a physical limit of webcam
gaze estimation, not a bug. Calibration automatically checks whether eye
aperture (eyelid closing as you look down) tracks vertical position better
than the model's raw output, and uses whichever signal is actually stronger.

## Known limitations

- Windows-only (pywebview + WebView2 backend). Other platforms untested.
- Head movement during a session degrades accuracy — the calibration assumes
  a roughly stable head pose.
- No lens distortion correction — `solvePnP` assumes zero distortion, which
  costs some accuracy near frame edges.

## Troubleshooting

**Window freezes with `AccessibilityObject.Bounds.Empty.Empty.Empty...`
spamming the console:**
Confirm you actually have the pinned versions installed, not newer ones:
```bash
pip uninstall pywebview pythonnet -y
pip install pywebview==4.4.1 pythonnet==3.0.3
```
If it persists after that, turn off **Xbox Game Bar** (Settings → Gaming →
Xbox Game Bar) and any overlay software (Discord overlay, GeForce
Experience, OBS), then **restart your PC** — those hooks stay loaded until a
real reboot. Last resort: install `PyQt5` + `PyQtWebEngine` (see
`requirements.txt`) and switch the GUI backend as described there.

**"No calibration.pkl found" when pressing S:**
Run `python calibrate.py` first — it must exist before `browser_session.py`
can track anything.

**Can't type in the search box:**
Click directly into the search field first — this can happen if the window
just opened and hasn't fully grabbed keyboard focus yet.

## License

MIT.