Upload README.md with huggingface_hub

72834ef verified about 2 months ago

6.08 kB

---
license: mit
pipeline_tag: video-classification
library_name: blinklinmult
tags:
  - blink-detection
  - eye-state-recognition
  - eye-tracking
  - onnx
---
# BlinkLinMulT — pretrained blink and eye-state models

Models for **eye-state recognition** (per-frame closure) and **blink
detection** (temporal events), from the
[BlinkLinMulT](https://github.com/fodorad/BlinkLinMulT) project.

Three are the published 1.x models from
[*BlinkLinMulT: Transformer-Based Eye Blink Detection*](https://www.mdpi.com/2313-433X/9/10/196)
(J. Imaging, 2023), shipped as frozen ONNX graphs. The v2 frame-wise model ships
twice: as a PyTorch checkpoint (`blinkcnn`, trainable and fine-tunable) and as
the same weights frozen into a graph (`blinkcnn-onnx`, deployable).

## Models

| id | runtime | input | output | notes |
|---|---|---|---|---|
| `densenet121-union` | ONNX | `(B, T, 3, 64, 64)` | `(B, T, 1)` | 1.x frame-wise, DenseNet121 |
| `blinklint-union` | ONNX | `(B, T, 3, 64, 64)` | `(B, T, 1)` | 1.x sequence, DenseNet121 + LinT |
| `blinklinmult-union` | ONNX | crops + `(B, T, 160)` | `(B, T, 1)` and `(B, 1)` | 1.x two-stream, the paper's headline model |
| `blinkcnn` | PyTorch | `(B, T, 3, 64, 64)` | `(B, T, 1)` | v2 frame-wise, ConvNeXt-Femto |
| `blinkcnn-onnx` | ONNX | `(B, T, 3, 64, 64)` | `(B, T, 1)` | the same weights, frozen — **4.9x faster** |

Both axes are dynamic: `T=1` is a single frame, any larger `T` is video.

`blinkcnn` and `blinkcnn-onnx` are the *same weights*. The graph was exported
behind a numerical-parity gate — three window lengths, three input
distributions, worst deviation **2.6e-06**, and an end-to-end check through the
public API at **4.8e-07** — so it is a drop-in replacement, not an
approximation. Measured on CPU it runs **20.19 → 4.15 ms/frame** and loads in
560 ms rather than 2481. Prefer `blinkcnn-onnx` unless you are fine-tuning.

The ONNX models need no PyTorch at all: `pip install blinklinmult[onnx]` is a
~100 MB install against ~600 MB for the training stack.

## Usage

```python
from blinklinmult import BlinkDetector
model = BlinkDetector.from_pretrained("blinklint-union")
signal = model.score(crops)  # (T,) closure probability per frame
blinks = model.detect(crops)  # [(start, end), ...] frame indices
```

Weights download and cache on first use. Crops are `(T, 3, 64, 64)` float32 in
`[0, 1]`, channel-first — **`BlinkDetector` applies each model's own
normalisation**, so pass raw `[0, 1]` values and do not pre-normalise.

Install: `pip install blinklinmult[onnx]` for the ONNX models, plus `torch` for
`blinkcnn`.

## Normalisation — the one thing to get right

The two generations were trained on **different input conventions**, and applying
the wrong one costs accuracy *without raising an error*:

| generation | normalisation |
|---|---|
| 1.x (`*-union`) | ImageNet, mean `(0.485, 0.456, 0.406)` / std `(0.229, 0.224, 0.225)` |
| v2 (`blinkcnn`) | plain `/255` — no shift, no scale |

Each file's `.json` sidecar records its own constants. Use `BlinkDetector` and
this is handled for you.

## Events, and the operating points

Blink intervals come from **hysteresis extraction** over the per-frame signal: a
run must peak above a high threshold to count as a blink, but extends while it
stays above a low one. The high threshold decides *whether*, the low one decides
*where* — a single cut has to be low enough to catch a closure's shallow onset
and high enough to ignore noise, and no one value does both.

| model | threshold | low ratio | fitted? |
|---|---:|---:|---|
| `blinkcnn` | 0.53 | 0.25 | **yes**, on validation |
| the three `*-union` models | 0.50 | — | **no** — see below |

**The 1.x thresholds are placeholders, not measurements.** Those models were
released as weights alone, with no operating point, and refitting one would mean
re-running their validation splits under this codebase. So:

* frame-level `score()` output is **directly comparable** across generations;
* event-level counts from `detect()` are **not** — `blinkcnn`'s threshold was
  tuned and theirs was not.

To close that gap, fit on your own validation data:

```python
from blinklinmult.fit import fit_operating_point
point = fit_operating_point(signals, annotations)  # validation only
model.calibrate(point)
```

## Provenance

Each ONNX graph was exported from the original published PyTorch weights and
**verified against them before release** at window lengths 1, 15, 30 and 45 —
including lengths the export never traced, which is what proves the time axis is
genuinely dynamic rather than unrolled. Maximum observed deviation across all
three models and all lengths: **3.4e-05**, i.e. float32 rounding.

Every `.json` sidecar carries the source checkpoint's filename and SHA-256, the
opset, the normalisation, and the parity tolerances.

## Limitations

* **Eye crops in, not faces.** These models score a cropped eye region, not a
  full frame. Crop geometry matters: v2 was trained on a square `2.0 ×` the eye's
  corner-to-corner span. A different crop convention degrades accuracy.
* **`blinklinmult-union` needs the 160-d descriptor stream** (iris landmarks,
  eyelid distances, EAR, head pose) alongside the crops. Producing it requires
  the `[preprocess]` extra.
* **Frozen.** The 1.x models ship as graphs and cannot be fine-tuned. That is
  deliberate: they are published artefacts.
* Trained on public blink corpora that skew toward frontal, indoor webcam
  footage. Expect degradation on strong profile views, heavy occlusion, or
  infrared imagery.

## Citation

```bibtex
@article{fodor2023blinklinmult,
  title   = {BlinkLinMulT: Transformer-Based Eye Blink Detection},
  author  = {Fodor, {\'A}d{\'a}m and Fenech, Kristian and L{\H{o}}rincz, Andr{\'a}s},
  journal = {Journal of Imaging},
  volume  = {9},
  number  = {10},
  pages   = {196},
  year    = {2023},
  doi     = {10.3390/jimaging9100196}
}
```

## License

MIT, matching the source repository. The corpora these models were trained on
each carry their own licence and are **not** redistributed here.

	---
	license: mit
	pipeline_tag: video-classification
	library_name: blinklinmult
	tags:
	- blink-detection
	- eye-state-recognition
	- eye-tracking
	- onnx
	---
	# BlinkLinMulT — pretrained blink and eye-state models

	Models for eye-state recognition (per-frame closure) and **blink
	detection** (temporal events), from the
	[BlinkLinMulT](https://github.com/fodorad/BlinkLinMulT) project.

	Three are the published 1.x models from
	[BlinkLinMulT: Transformer-Based Eye Blink Detection](https://www.mdpi.com/2313-433X/9/10/196)
	(J. Imaging, 2023), shipped as frozen ONNX graphs. The v2 frame-wise model ships
	twice: as a PyTorch checkpoint (`blinkcnn`, trainable and fine-tunable) and as
	the same weights frozen into a graph (`blinkcnn-onnx`, deployable).

	## Models

	\| id \| runtime \| input \| output \| notes \|
	\|---\|---\|---\|---\|---\|
	\| `densenet121-union` \| ONNX \| `(B, T, 3, 64, 64)` \| `(B, T, 1)` \| 1.x frame-wise, DenseNet121 \|
	\| `blinklint-union` \| ONNX \| `(B, T, 3, 64, 64)` \| `(B, T, 1)` \| 1.x sequence, DenseNet121 + LinT \|
	\| `blinklinmult-union` \| ONNX \| crops + `(B, T, 160)` \| `(B, T, 1)` and `(B, 1)` \| 1.x two-stream, the paper's headline model \|
	\| `blinkcnn` \| PyTorch \| `(B, T, 3, 64, 64)` \| `(B, T, 1)` \| v2 frame-wise, ConvNeXt-Femto \|
	\| `blinkcnn-onnx` \| ONNX \| `(B, T, 3, 64, 64)` \| `(B, T, 1)` \| the same weights, frozen — 4.9x faster \|

	Both axes are dynamic: `T=1` is a single frame, any larger `T` is video.

	`blinkcnn` and `blinkcnn-onnx` are the same weights. The graph was exported
	behind a numerical-parity gate — three window lengths, three input
	distributions, worst deviation 2.6e-06, and an end-to-end check through the
	public API at 4.8e-07 — so it is a drop-in replacement, not an
	approximation. Measured on CPU it runs 20.19 → 4.15 ms/frame and loads in
	560 ms rather than 2481. Prefer `blinkcnn-onnx` unless you are fine-tuning.

	The ONNX models need no PyTorch at all: `pip install blinklinmult[onnx]` is a
	~100 MB install against ~600 MB for the training stack.

	## Usage

	```python
	from blinklinmult import BlinkDetector

	model = BlinkDetector.from_pretrained("blinklint-union")
	signal = model.score(crops) # (T,) closure probability per frame
	blinks = model.detect(crops) # [(start, end), ...] frame indices
	```

	Weights download and cache on first use. Crops are `(T, 3, 64, 64)` float32 in
	`[0, 1]`, channel-first — **`BlinkDetector` applies each model's own
	normalisation**, so pass raw `[0, 1]` values and do not pre-normalise.

	Install: `pip install blinklinmult[onnx]` for the ONNX models, plus `torch` for
	`blinkcnn`.

	## Normalisation — the one thing to get right

	The two generations were trained on different input conventions, and applying
	the wrong one costs accuracy without raising an error:

	\| generation \| normalisation \|
	\|---\|---\|
	\| 1.x (`*-union`) \| ImageNet, mean `(0.485, 0.456, 0.406)` / std `(0.229, 0.224, 0.225)` \|
	\| v2 (`blinkcnn`) \| plain `/255` — no shift, no scale \|

	Each file's `.json` sidecar records its own constants. Use `BlinkDetector` and
	this is handled for you.

	## Events, and the operating points

	Blink intervals come from hysteresis extraction over the per-frame signal: a
	run must peak above a high threshold to count as a blink, but extends while it
	stays above a low one. The high threshold decides whether, the low one decides
	where — a single cut has to be low enough to catch a closure's shallow onset
	and high enough to ignore noise, and no one value does both.

	\| model \| threshold \| low ratio \| fitted? \|
	\|---\|---:\|---:\|---\|
	\| `blinkcnn` \| 0.53 \| 0.25 \| yes, on validation \|
	\| the three `-union` models \| 0.50 \| — \| no* — see below \|

	The 1.x thresholds are placeholders, not measurements. Those models were
	released as weights alone, with no operating point, and refitting one would mean
	re-running their validation splits under this codebase. So:

	* frame-level `score()` output is directly comparable across generations;
	* event-level counts from `detect()` are not — `blinkcnn`'s threshold was
	tuned and theirs was not.

	To close that gap, fit on your own validation data:

	```python
	from blinklinmult.fit import fit_operating_point

	point = fit_operating_point(signals, annotations) # validation only
	model.calibrate(point)
	```

	## Provenance

	Each ONNX graph was exported from the original published PyTorch weights and
	verified against them before release at window lengths 1, 15, 30 and 45 —
	including lengths the export never traced, which is what proves the time axis is
	genuinely dynamic rather than unrolled. Maximum observed deviation across all
	three models and all lengths: 3.4e-05, i.e. float32 rounding.

	Every `.json` sidecar carries the source checkpoint's filename and SHA-256, the
	opset, the normalisation, and the parity tolerances.

	## Limitations

	* Eye crops in, not faces. These models score a cropped eye region, not a
	full frame. Crop geometry matters: v2 was trained on a square `2.0 ×` the eye's
	corner-to-corner span. A different crop convention degrades accuracy.
	* `blinklinmult-union` needs the 160-d descriptor stream (iris landmarks,
	eyelid distances, EAR, head pose) alongside the crops. Producing it requires
	the `[preprocess]` extra.
	* Frozen. The 1.x models ship as graphs and cannot be fine-tuned. That is
	deliberate: they are published artefacts.
	* Trained on public blink corpora that skew toward frontal, indoor webcam
	footage. Expect degradation on strong profile views, heavy occlusion, or
	infrared imagery.

	## Citation

	```bibtex
	@article{fodor2023blinklinmult,
	title = {BlinkLinMulT: Transformer-Based Eye Blink Detection},
	author = {Fodor, {\'A}d{\'a}m and Fenech, Kristian and L{\H{o}}rincz, Andr{\'a}s},
	journal = {Journal of Imaging},
	volume = {9},
	number = {10},
	pages = {196},
	year = {2023},
	doi = {10.3390/jimaging9100196}
	}
	```

	## License

	MIT, matching the source repository. The corpora these models were trained on
	each carry their own licence and are not redistributed here.