| --- |
| license: apache-2.0 |
| library_name: flashback |
| tags: |
| - anomaly-detection |
| - distributed-training |
| - observability |
| - debugging |
| - machine-learning |
| - flashback |
| metrics: |
| - mae |
| model-index: |
| - name: flashback-first-bad-step |
| results: |
| - task: |
| type: anomaly-detection |
| name: First-bad-step localisation |
| dataset: |
| name: Flashback Forensics |
| type: NagaYu/flashback-forensics |
| split: test |
| metrics: |
| - type: mae |
| name: Mean absolute step error |
| value: 0.00 |
| - type: accuracy |
| name: Exact-step hit rate |
| value: 1.000 |
| --- |
| |
| # Flashback: first-bad-step detector |
|
|
| Given a run's **sketch index** -- a few hundred bytes of summary statistics per |
| training step -- this model reports the step at which the run first went wrong. |
|
|
| It is small on purpose: 10 logistic weights plus 9 fitted |
| consensus hyperparameters. The interesting part is not the model, it is that |
| the *input* is cheap enough to record for every step of every run. |
|
|
| ## Results on held-out runs |
|
|
| | detector | mean abs. error (steps) | exact | within 2 | never detected | |
| |---|---:|---:|---:|---:| |
| | loss curve alone | 0.50 | 2/6 | 4/6 | 2 | |
| | **this model** | **0.00** | **6/6** | **6/6** | **0** | |
|
|
| Train split: 12 runs, MAE 0.00. |
| Splits are by run: no step of a test run appears in training. |
|
|
| ## Usage |
|
|
| ```python |
| from flashback.detector import FirstBadStepDetector |
| from flashback.sketch import StepSketchIndex |
| from huggingface_hub import snapshot_download |
| |
| det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step")) |
| report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r")) |
| print(report.first_bad_step, report.confidence, report.top_metrics) |
| ``` |
|
|
| To produce the sketch index in the first place, add three lines to your |
| training loop: |
|
|
| ```python |
| from flashback.integrations import FlashbackCallback |
| trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")]) |
| ``` |
|
|
| ## Fitted configuration |
|
|
| ```json |
| { |
| "win": 32, |
| "gap": 1, |
| "mode": "diff", |
| "k_cal": 8.0, |
| "k_min": 8.0, |
| "min_votes": 3, |
| "vote_window": 2, |
| "exclude": [ |
| "lr", |
| "step_wall_ms", |
| "batch_id_mean", |
| "batch_uniq_frac" |
| ], |
| "use_logistic": true, |
| "logistic_threshold": 0.5, |
| "version": "1" |
| } |
| ``` |
|
|
| ## Feature weights |
|
|
| | feature | weight | |
| |---|---:| |
| | `bias` | -9.8951 | |
| | `log1p_max_z` | +0.1471 | |
| | `log1p_top5_mean_z` | +0.1950 | |
| | `frac_metrics_firing` | +18.4084 | |
| | `hard_signal` | +1.2226 | |
| | `log1p_z_grad_norm` | +0.3008 | |
| | `log1p_z_update_max` | +0.1584 | |
| | `log1p_z_loss` | +0.0378 | |
| | `log1p_z_hist_tail` | +0.2497 | |
| | `rel_position` | +15.0875 | |
|
|
| ## Limitations |
|
|
| - It needs a **healthy stretch** of the run to calibrate against; a run that is |
| broken from step 0 has no baseline and will not be localised. |
| - It reports *regime changes*, so a fault that ramps in smoothly over hundreds |
| of steps has no single correct answer and the reported step will be early or |
| late by roughly the ramp length. |
| - It was fitted on small Transformers (a few hundred thousand to a few million |
| parameters). The features are scale-free by construction, but transfer to |
| very large models is **not** demonstrated here. |
| - No claim of bitwise determinism is made anywhere in Flashback; this is a |
| statistical detector over recorded statistics. |
|
|
| Generated by Flashback v0.1.0. |
|
|
