kiberamos/the-collection-telemetry

Lab 06 · Predictive Maintenance — model card

Generated 2026-10-14T16:35:32Z by labs-training/telemetry/train.py at commit 83b4817c79224403cbcac5344578b017fd1641f6, seed 20261013.

What this is

Two anomaly detectors for a simulated industrial pump, both running in the visitor's browser through ONNX Runtime Web.

The pump and its faults are simulated. The model, thresholds and measurements are real. That sentence is on the lab page too, and it is the honest boundary of what this demonstrates: the detectors were genuinely trained, genuinely exported and genuinely measured on data they had never seen, but the machine they watch is a physics model in src/labs/telemetry/simulator.ts, not a pump.

Training data

Generated by that simulator, which is seeded and deterministic, so these figures are reproducible rather than merely recorded.

Split	Content	Windows	SHA-256 of the windows
Train	6.0 h healthy, 20 seeds	215,020	3f78f4ca2354ea9b…
Validation	1.0 h healthy, 4 seeds	35,804	b98d69c32be2011d…
Test	200 runs of 10 min, one fault each	stride 5	seeded per run

The seed ranges for the three splits are disjoint by construction. Neither detector ever sees a fault during training: both learn what a healthy pump looks like, and a fault is whatever they cannot account for. That is what lets them flag a fault nobody trained them on.

The models

Autoencoder — 1D convolutions over a 50 × 5 window, latent 8, 23,693 parameters, ONNX opset 17. Trained for 24 of a possible 30 epochs (early stopping, patience 5), keeping the best epoch's weights rather than the last. Per-channel normalisation is inside the graph, so the browser cannot apply different statistics than training did.

Isolation Forest — 200 trees over 25 window summary features (mean, standard deviation, min, max and last-minus-first, per channel). The feature extraction is also inside the graph: the browser passes a raw window and reads a score.

The graph's iso_scores output is sklearn's decision_function, NOT score_samples: they differ by the forest's offset_ (-0.5 at the default contamination='auto'). The lab negates it so larger means more anomalous, and the threshold above is on the negated value.

Thresholds

The 99.7th percentile of validation error, plus a 3-window persistence rule — an alarm needs three consecutive windows over the line, not one. Both thresholds come from the validation split only; the test runs below were not involved in choosing them.

• Autoencoder: 0.0299237
• Isolation Forest (on the negated score): 0.0676027

Results

Measured on 200 held-out runs, on onnxruntime — the same runtime the browser uses, so these are not torch numbers for a graph nobody runs.

Autoencoder

Fault	Runs	Recall	Mean time to detect
bearingWear	30	100.0%	7.3 s
coolingFailure	41	100.0%	78.6 s
cavitation	42	100.0%	3.7 s
sensorStuck	37	43.2%	107.8 s
electricalFault	50	100.0%	14.1 s

Overall recall 89.5% over 200 runs.

Isolation Forest

Fault	Runs	Recall	Mean time to detect
bearingWear	30	100.0%	25.0 s
coolingFailure	41	100.0%	127.3 s
cavitation	42	100.0%	18.1 s
sensorStuck	37	70.3%	227.2 s
electricalFault	50	100.0%	89.1 s

Overall recall 94.5% over 200 runs.

What they are bad at

The headline recall figures above average over five faults and hide an uneven one. Against near-perfect recall elsewhere, the autoencoder catches sensorStuck only 43% of the time and the forest manages 70% on sensorStuck.

That is the expected answer rather than a surprise, and it is why the fault is in the set at all. A stuck sensor is a measurement fault: the pump is fine and one channel simply stops moving, at a value that was perfectly normal when it froze. A detector that has learned what "worse" looks like has no reason to complain. The forest does better because three of its features — standard deviation, range and last-minus-first — go to exactly zero for a frozen channel, which no healthy window ever produces. The autoencoder, reconstructing a plausible constant, mostly does not notice.

Shown here rather than smoothed away, because a single averaged recall number would have hidden it and a visitor reading the per-fault table deserves to know why one row is different.

Against the doc 21 §12 targets

These are targets, shown next to what was measured, never in place of it.

	Target	Autoencoder	Isolation Forest
Per-window false-positive rate	< 1.5%	0.213%	0.408%
Per-window inference (p50)	< 15 ms	0.05 ms	1.19 ms
Per-window inference (p95)	—	0.06 ms	1.59 ms
False alarms per healthy hour	—	1.09	3.49

Inference was measured at batch 1, which is how the lab runs it: one window per frame. Time to detect is not inference latency and the two are not comparable — time to detect is bounded by how fast a fault physically develops, which for a cooling failure is minutes.

Export verification

The exported graphs were checked against the models they came from rather than assumed to match:

• autoencoder, largest relative difference from torch: 1.008e-06
• forest, largest relative difference from scikit-learn on the same features: 4.107e-06

A run that failed that check would have written nothing at all.

One honest wrinkle, measured rather than hidden. The forest's window features are computed in float32, and torch and ONNX do not sum fifty numbers in the same order — on shaftRpm.mean, averaging values around 2,950, the two land about 1e-3 apart. A decision forest turns that into a visible difference whenever a window sits near a split threshold: it falls the other way, lands in a different leaf, and its score moves. On the probe set that happened to 1 of 512 windows, and 0 of them would have changed an alarm. The export is rejected unless that last number is zero. The scores this model card reports are the graph's own, measured on the runtime the browser uses, so they describe the model that actually ships.

Reproducing

cd labs-training
uv sync --frozen
uv run python -m telemetry.train

Environment used: Python 3.12.14, torch 2.14.1, onnxruntime 1.31.0, on macOS-27.0.1-arm64-arm-64bit.

Licence

Apache-2.0. Chosen rather than inherited: the training data is generated, not collected, and the weights are trained from scratch, so there is no third-party dataset and no base model whose terms would carry over. Full text at https://www.apache.org/licenses/LICENSE-2.0.

Limitations

• The pump is simulated. A detector that works on this simulator is evidence about the method, not about any particular real machine.
• Both detectors are trained on one pump at one operating point. Nothing here claims they transfer to a different machine without retraining.
• The simulated faults develop smoothly. Real failures are sometimes abrupt, and a detector tuned on smooth onsets may be late on those.
• The thresholds come from one validation split. A deployment would re-derive them on its own healthy data, not inherit these.