InsightUX — Webcam Eye-Tracking UX Research Browser

A webcam-based gaze tracking system that turns any consumer laptop into a UX research tool. Browse a website, and get back a heatmap, an attention timeline, and a ranked list of which page elements actually held your gaze.

No specialist hardware — just a webcam.

This is a local desktop app, not a hosted demo. It needs a physical webcam, a native window, and full-screen capture — none of which exist on a server, so it cannot run as a Hugging Face Space. Clone it and run it on your own machine.

Requirements before you start

Windows (uses WebView2 via pywebview's WinForms backend)
Python 3.11 exactly — mediapipe's legacy face-mesh API this project depends on does not exist on Python 3.12+
A working webcam

How to run — step by step

1. Clone the repo

git clone https://huggingface.co/<your-username>/insightux
cd insightux

2. Create and activate a virtual environment

python -m venv venv
venv\Scripts\activate

3. Install dependencies — use the pinned versions

pip install -r requirements.txt

This includes onnx, which calibrate.py needs when it exports the fine-tuned model after training.

Do not pip install --upgrade pywebview or pythonnet. requirements.txt pins pywebview==4.4.1 and pythonnet==3.0.3 deliberately — newer versions have a bug in their Windows backend that freezes the app window and floods the console with:

AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...

If you already have a newer version installed globally, this install step will replace it with the working one.

4. Calibrate — required, per person, per setup

python calibrate.py

Sit normally at your usual distance from the screen, look at each of the 16 dots as they appear. Takes about a minute. This is per-person and per-setup — your eyes, your camera, your screen size. Everyone using this must run it themselves; it is not something you can copy from someone else.

Re-run it if your lighting, seating position, or camera position changes noticeably.

At the end it prints an honest quality readout, including whether the model can actually see where you're looking on each axis:

HORIZONTAL  yaw vs screen-X : r = +0.995
VERTICAL    pitch vs screen-Y: r = +0.883

If either number is low, the report will say so plainly and explain why — that means the model isn't seeing that axis, and no amount of recalibrating will fix it.

5. (Optional) Check your accuracy

python validate.py

Flashes 9 test dots and reports your real error in pixels. Good for knowing what to expect before relying on a session.

6. Run the browser

python browser_session.py

Opens on an InsightUX-branded search page. Type a search term (goes to real Google results) or a URL (goes straight there).
Land on the page you want to study, click on blank space on the page (not a text field), then press S to start eye-tracking.
Press E to stop. A full report — heatmap over real screenshots, ranked attention list, dwell timeline — generates and opens automatically.

What's in this repo

File	Purpose
`models/gaze_cnn_v4.onnx` (+ `.onnx.data`)	Binocular gaze CNN (EfficientNet-B0 backbone, dual eye patches + head pose)
`models/model_v4.py`	Model architecture definition, used during calibration fine-tuning
`checkpoints/best_model_v4.pt`	PyTorch checkpoint for fine-tuning
`calibrate.py`	Per-user calibration — 16-point, live blink/lighting rejection, honest quality report
`validate.py`	Measures real accuracy in pixels after calibration
`browser_session.py`	The research browser — search, track, auto-report
`analysis.py`	Builds the session report (heatmaps over real screenshots)
`inference_pipeline.py`	ONNX inference + RBF gaze→screen calibration mapping
`preprocessing/preprocessing_pipeline.py`	Eye patch normalization, head pose estimation, illumination correction

Calibration is per-person and per-setup — not included in this repo

calibration.pkl (generated by calibrate.py) encodes your eye geometry, your camera characteristics, and your screen size. It is deliberately not in this repo — it would be useless to anyone else and it's personal data. Run calibrate.py yourself; it takes about a minute.

Accuracy — honest numbers

This is a webcam system, not a Tobii. Expect roughly 5–10% of screen diagonal mean error after a good calibration. That's enough for coarse AOI attribution (navbar vs hero vs footer) and heatmaps. It is not enough for reading-level analysis (which word you're on).

Vertical accuracy is typically a bit looser than horizontal — looking down partially occludes the iris under the eyelid, a physical limit of webcam gaze estimation, not a bug. Calibration automatically checks whether eye aperture (eyelid closing as you look down) tracks vertical position better than the model's raw output, and uses whichever signal is actually stronger.

Known limitations

Windows-only (pywebview + WebView2 backend). Other platforms untested.
Head movement during a session degrades accuracy — the calibration assumes a roughly stable head pose.
No lens distortion correction — solvePnP assumes zero distortion, which costs some accuracy near frame edges.

Troubleshooting

Window freezes with AccessibilityObject.Bounds.Empty.Empty.Empty... spamming the console: Confirm you actually have the pinned versions installed, not newer ones:

pip uninstall pywebview pythonnet -y
pip install pywebview==4.4.1 pythonnet==3.0.3

If it persists after that, turn off Xbox Game Bar (Settings → Gaming → Xbox Game Bar) and any overlay software (Discord overlay, GeForce Experience, OBS), then restart your PC — those hooks stay loaded until a real reboot. Last resort: install PyQt5 + PyQtWebEngine (see requirements.txt) and switch the GUI backend as described there.

"No calibration.pkl found" when pressing S: Run python calibrate.py first — it must exist before browser_session.py can track anything.

Can't type in the search box: Click directly into the search field first — this can happen if the window just opened and hasn't fully grabbed keyboard focus yet.

License

MIT.

Downloads last month: -; Downloads are not tracked for this model. How to track