File size: 3,314 Bytes
a7c7816 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | ---
license: apache-2.0
library_name: flashback
tags:
- anomaly-detection
- distributed-training
- observability
- debugging
- machine-learning
- flashback
metrics:
- mae
model-index:
- name: flashback-first-bad-step
results:
- task:
type: anomaly-detection
name: First-bad-step localisation
dataset:
name: Flashback Forensics
type: NagaYu/flashback-forensics
split: test
metrics:
- type: mae
name: Mean absolute step error
value: 0.00
- type: accuracy
name: Exact-step hit rate
value: 1.000
---
# Flashback: first-bad-step detector
Given a run's **sketch index** -- a few hundred bytes of summary statistics per
training step -- this model reports the step at which the run first went wrong.
It is small on purpose: 10 logistic weights plus 9 fitted
consensus hyperparameters. The interesting part is not the model, it is that
the *input* is cheap enough to record for every step of every run.
## Results on held-out runs
| detector | mean abs. error (steps) | exact | within 2 | never detected |
|---|---:|---:|---:|---:|
| loss curve alone | 0.50 | 2/6 | 4/6 | 2 |
| **this model** | **0.00** | **6/6** | **6/6** | **0** |
Train split: 12 runs, MAE 0.00.
Splits are by run: no step of a test run appears in training.
## Usage
```python
from flashback.detector import FirstBadStepDetector
from flashback.sketch import StepSketchIndex
from huggingface_hub import snapshot_download
det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step"))
report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r"))
print(report.first_bad_step, report.confidence, report.top_metrics)
```
To produce the sketch index in the first place, add three lines to your
training loop:
```python
from flashback.integrations import FlashbackCallback
trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])
```
## Fitted configuration
```json
{
"win": 32,
"gap": 1,
"mode": "diff",
"k_cal": 8.0,
"k_min": 8.0,
"min_votes": 3,
"vote_window": 2,
"exclude": [
"lr",
"step_wall_ms",
"batch_id_mean",
"batch_uniq_frac"
],
"use_logistic": true,
"logistic_threshold": 0.5,
"version": "1"
}
```
## Feature weights
| feature | weight |
|---|---:|
| `bias` | -9.8951 |
| `log1p_max_z` | +0.1471 |
| `log1p_top5_mean_z` | +0.1950 |
| `frac_metrics_firing` | +18.4084 |
| `hard_signal` | +1.2226 |
| `log1p_z_grad_norm` | +0.3008 |
| `log1p_z_update_max` | +0.1584 |
| `log1p_z_loss` | +0.0378 |
| `log1p_z_hist_tail` | +0.2497 |
| `rel_position` | +15.0875 |
## Limitations
- It needs a **healthy stretch** of the run to calibrate against; a run that is
broken from step 0 has no baseline and will not be localised.
- It reports *regime changes*, so a fault that ramps in smoothly over hundreds
of steps has no single correct answer and the reported step will be early or
late by roughly the ramp length.
- It was fitted on small Transformers (a few hundred thousand to a few million
parameters). The features are scale-free by construction, but transfer to
very large models is **not** demonstrated here.
- No claim of bitwise determinism is made anywhere in Flashback; this is a
statistical detector over recorded statistics.
Generated by Flashback v0.1.0.
|