| --- |
| license: mit |
| library_name: frm |
| tags: |
| - flow-reasoning-models |
| - reasoning |
| - pytorch |
| --- |
| |
| # Flow Reasoning Models — lightweight release draft |
|
|
| One selected checkpoint per task, with portable benchmark/calibration assets and |
| measured results. This repository stays private during testing. |
|
|
| ## Setup |
|
|
| GitHub installation only; nothing is published to PyPI. |
|
|
| ```bash |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install "git+https://github.com/helblazer811/Flow-Reasoning-Models.git" |
| ``` |
|
|
| ## Download and evaluate |
|
|
| Use standard Hugging Face downloads: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| snapshot_download("helblazer811/FlowReasoningModel", allow_patterns=["sudoku/*"], local_dir="models") |
| ``` |
|
|
| ```bash |
| python -m frm.evaluate --model-dir models/sudoku |
| ``` |
|
|
| Replace `sudoku` with `maze_unique` or `zebra` for the other models. Evaluation |
| checks the downloaded files and uses calibrated sequential restarts. Results go |
| under `outputs/`. Add `--max-samples 8` for a functional check, not a benchmark score. |
| The evaluator computes fixed-depth comparisons too; reported policy NFE excludes |
| that additional comparison work. For manual download, obtain all files in the |
| corresponding model folder from this repository. |
|
|
| ## Selected models and measured results |
|
|
| | Bundle | Training seed | Benchmark | Sequential restart accuracy | Mean NFE | |
| |---|---:|---|---:|---:| |
| | sudoku | 0 | EqR Sudoku-Lite, 2,048 puzzles | 99.7070% (2042/2048) | 899.75 | |
| | maze_unique | 0 | Official EqR Maze-Unique, 1,000 puzzles | 99.9% (999/1000) | 15.708 | |
| | zebra | 0 | 5x5 Zebra, 1,024-puzzle seed0 subset | 99.6094% (1020/1024) | 6.2402 | |
| |
| Sudoku costs 0.8062 mean TFLOPs/puzzle; Maze-Unique costs 0.1781. Zebra FLOPs |
| are unprofiled for this preprocessing, so only NFE is reported. Zebra seed0 scores99.8047% at257NFE with the fixed-depth held-t trajectory; |
| its stopping/restart policy scores99.6094% at6.2402meanNFE. Its earlier SDE |
| result is99.6094%. Every test puzzle accepts on the first attempt. FLOPs for this |
| 5x5 setup are unprofiled; NFE is reported instead. |
| |
| All three hosted checkpoints now use training seed0. Maze-Unique restart seeds0/1/2: |
| 99.9%,98.5%,99.8%. Zebra SDE seeds0/1/2:99.6094%,99.8047%,99.5117%. |
| Only one new Sudoku training seed was run. Zebra seed0 was preferred for a |
| consistent release after its measured default score99.6094% exceeded seed1's |
| 99.4141%. Source checkpoint/step/seed details remain in each manifest. |
| |
| ## Scope and reproducibility |
| |
| - Sudoku is the independently tested fixed-time FPF training variant, evaluated on |
| Sudoku-Lite; this does not establish the paper's full-test headline. Its independent |
| development calibration excludes known training and test puzzles. |
| - Maze-Unique uses the official pinned EqR1k/1k dataset and standard two-stage FPF. |
| Its256-puzzle calibration is sampled from the training split as in the published policy. |
| - Zebra currently trains/tests5x5 with repacked answers. It does **not** reproduce |
| the paper's mixed3x3–6x6 pinned-answer dataset. Calibration uses training puzzles |
| explicitly disjoint from test; no test labels select stopping/restart candidates. |
| |
| Each folder contains `model.pt`, `training.yaml`, `threshold.json`, `test.npz`, |
| `calibration.npz`, `manifest.json`, `provenance.json`, and `expected_results.json`. |
| The manifest records the original checkpoint hash and the compact inference-checkpoint |
| hash. Optimizer state is omitted; raw weights and EMA are retained. These are compact PyTorch checkpoints from the recorded training runs. |
| For training from scratch, download `training.yaml` and run |
| `python -m frm.train --config training.yaml` on the matching data. |
|
|
| Training and evaluation code is available in |
| [Flow-Reasoning-Models](https://github.com/helblazer811/Flow-Reasoning-Models). |
| This lightweight release supports |
| the models and evaluations above, not every paper baseline, ablation, table or figure. |
|
|
