| --- |
| title: Flashback |
| emoji: 🔦 |
| colorFrom: indigo |
| colorTo: gray |
| sdk: static |
| app_file: index.html |
| pinned: false |
| license: apache-2.0 |
| short_description: Find the step your training run broke on, from 452 B/step |
| tags: |
| - machine-learning |
| - distributed-training |
| - checkpointing |
| - observability |
| - debugging |
| - anomaly-detection |
| - pytorch |
| --- |
| |
| # Flashback — find the step your training run broke on |
|
|
| **Finding the step a training run broke on should cost kilobytes, not terabytes.** |
|
|
| A large run fails. Somewhere in the last 40,000 steps a learning rate spiked, a bit |
| flipped, a shard of the dataloader started serving garbage. Today you answer *"when did |
| this start?"* by squinting at a loss curve and reloading whichever checkpoint happens to be |
| nearest — so your answer is quantised to the checkpoint interval, and only exists at all if |
| the loss actually moved. |
|
|
| Flashback keeps a **452-byte sketch of every single step** — 128 internal statistics that |
| usually move long before the loss does — and bisects that index with **zero state |
| reconstructions**. |
|
|
| ## What this Space is |
|
|
| The telemetry here is **real**: recorded from actual training runs of a small Transformer |
| in which a specific fault was injected at a specific step, with the ground truth written |
| down before any detector looked at the data. The page reimplements |
| [`flashback.bisect`](https://github.com/NagaYu/flashback/blob/main/flashback/bisect.py) in |
| JavaScript so it can run for free as a static Space, and **checks itself against the answers |
| Python computed for the same data** — the verification badge at the bottom of the page says |
| whether the port currently agrees. |
|
|
| Pick a fault, then compare the two charts. For the bit-flip scenario the loss curve never |
| crosses its own anomaly threshold at any point in the run; the sketch crosses it on the |
| exact step. |
|
|
| ## Measured results |
|
|
| 12 runs across 6 fault types, ground truth recorded before any detector runs: |
|
|
| | condition | mean error | exact hits | never detected | |
| |---|---:|---:|---:| |
| | (A) full checkpoint every 100 steps + loss curve | 20.0 | 0/12 | 4 | |
| | (B) full checkpoint every step + loss curve (ideal) | 0.5 | 4/12 | 4 | |
| | **(C) Flashback sketch + bisect** | **0.0** | **12/12** | **0** | |
|
|
| ## Add it to your own run |
|
|
| ```python |
| from flashback.integrations import FlashbackCallback |
| |
| trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")]) |
| ``` |
|
|
| ```bash |
| flashback bisect my-run --metric grad_norm |
| ``` |
|
|
| ## Links |
|
|
| - **Code**: <https://github.com/NagaYu/flashback> |
| - **Dataset**: <https://huggingface.co/datasets/NagaYu/flashback-forensics> |
| - **Detector model**: <https://huggingface.co/NagaYu/flashback-first-bad-step> |
|
|
| ## What this demo is honest about |
|
|
| - The detector needs a **healthy stretch** to calibrate against. A run broken from step 0 |
| has no baseline and will not be localised. |
| - It detects **regime changes**, so a fault that ramps in smoothly over hundreds of steps |
| has no single correct answer. The `data poisoning` scenario here is off by ~36 steps at |
| this model size, and the page shows that rather than hiding it. |
| - **No bitwise determinism is claimed.** This is a statistical detector over recorded |
| statistics. |
| - The demo model is tiny (0.13M parameters, 600 steps) so the page stays a few megabytes. |
| The mechanism is size-independent; the repository's benchmark runs the same experiments up |
| to ~30M parameters. |
| - A full Gradio version of this demo — which additionally generates fresh runs on demand — |
| is in the repository as `app.py`. Hugging Face requires a PRO subscription to host Gradio |
| Spaces, so the free public demo is this static one. |
|
|
