KaryoFlow / reproducibility /protocol.md
procaffenator's picture
Update evaluation results and reproducibility records
da25b15 verified
|
Raw History Blame Contribute Delete
2.64 kB

Evaluation protocol

Dataset 1 contains 340 held-out test images. Dataset 2 contains 1,000 held-out test images. The datasets are trained and evaluated separately.

COCO box AP is averaged over IoU thresholds 0.50–0.95 in increments of 0.05, with maxDets = 100. Predictions from successive refinement calls are pooled before classwise NMS. Each configuration retains its tested resize and padding transforms.

The default KaryoFlow configuration uses 500 proposals, four refinement calls and six cascade stages. H3 uses three stages per call. LQCR combines class probability p and predicted quality q as p × q². Its parent detector is frozen during quality-branch training.

The model list records individual runs rather than representing a three-run mean with a single weight file. RF, RF+LQCR and H3 results use training seeds 42, 123 and 789. Dataset 1 comparators also use three independent training runs. Dataset 2 comparator repeats vary inference seeds 42, 123 and 789 while keeping the checkpoint fixed.

The ranking intervention in Fig. 4 compares p and p × q² within the same quality-augmented checkpoints. These paired values should not be substituted with a separately evaluated parent checkpoint, even when their means round to the same value.

Latency was measured on an RTX A6000 with batch size 1, 20 warm-up iterations and 500 timed iterations per repeat, repeated three times. Timing includes model preprocessing and postprocessing for a preloaded test batch. It excludes disk image loading. Reported standard deviations describe timing repeats, not training runs.

paper_tables.csv retains the manuscript's table cells, including LaTeX notation for uncertainties. The other CSV files provide numeric columns suitable for analysis. Figure-specific measurements are stored in figures/data/; they include validation sweeps, paired AP changes, scale analysis and recorded candidate boxes.

Solver comparison

Euler, Heun and the RF multistep solver are evaluated at one to four steps with three independent training seeds on each dataset. All solvers use the shifted reverse-time grid. At four steps, the maximum difference between solver mean mAP values is below 0.0001 on both datasets. Euler and the multistep rule use four detector calls; Heun uses seven.

LQCR is evaluated with ranking disabled and enabled in each quality-augmented checkpoint. The quality exponent is selected on validation images. The published accuracy, timing and figure records use the corresponding configurations and checkpoints listed in this repository.

Run the solver checks with python -m pytest tests/test_rf_solver_grid.py.