bert_simpson / forgebench /code /ours /batch_equivalence.md
Ronaldo-GOAT's picture
forgebench: ours batched pipeline + equivalence report, Cupid multi-view driver
84bff06 verified
|
Raw History Blame Contribute Delete
10.3 kB

Batch-equivalence check: batch of 2 vs 2 separate single inferences

Method batched: Ours (SS+SLAT) only (the only method with a batched code path). All baselines (ReconViaGen, Pixal3D, Amodal3R, Hunyuan3D-2mv, Cupid) were run UNBATCHED (one object per forward, their official per-object drivers); speed-up for them comes only from co-locating several single-object workers per GPU, which does not change any computation -> no equivalence test needed/applicable.

Runs (same ckpts SS=ssflow_seedcond_v3_20260916/step_0025000, SLAT=slatflow_prod_20260902/step_0020000, seed 42, same 2 objects FaGr_s000001_butternutsquash1 + FaGr_s000002_yellowmustardbottle5):

  • single = ours_combined_cell.sh (ssflow_coords.py, val_daemon single:true), GPU 4
  • single2 = identical repeat of single (GPU 6) -> run-to-run NOISE FLOOR of the unbatched path
  • batched = ours_combined_cell_batched.sh SSB=2 S2B=2 (ssflow_coords_batched.py --batch 2; val_daemon batch_size 2, single:false), GPU 5

1v

object pair SS coords (#) coords sym-diff (#voxels) V/F a V/F b max abs vert diff Chamfer-L1 (50k) Chamfer sampling floor (a vs a)
FaGr_s000001_butternutsquash1 single vs batched 9443/9443 0 206160/412284 206152/412268 n/a (V differ) 0.0033 0.0033
FaGr_s000001_butternutsquash1 single vs single2 (floor) 9443/9443 0 206160/412284 206164/412292 n/a (V differ) 0.0033 0.0033
FaGr_s000002_yellowmustardbottle5 single vs batched 9766/9765 1 187950/375812 188114/376128 n/a (V differ) 0.0033 0.0032
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) 9766/9766 0 187950/375812 187878/375664 n/a (V differ) 0.0032 0.0032
object pair metric a b abs diff
FaGr_s000001_butternutsquash1 single vs batched input_lpips 0.10903 0.10905 2.14e-05
FaGr_s000001_butternutsquash1 single vs batched input_psnr 24.69652 24.70546 8.94e-03
FaGr_s000001_butternutsquash1 single vs batched input_ssim 0.90333 0.90334 8.94e-06
FaGr_s000001_butternutsquash1 single vs batched novel_lpips 0.07460 0.07455 4.61e-05
FaGr_s000001_butternutsquash1 single vs batched novel_psnr 22.07094 22.07439 3.45e-03
FaGr_s000001_butternutsquash1 single vs batched novel_ssim 0.94951 0.94951 7.21e-06
FaGr_s000001_butternutsquash1 single vs batched geome_cd_l1 0.04028 0.04031 3.34e-05
FaGr_s000001_butternutsquash1 single vs batched geome_f01 0.38985 0.38904 8.16e-04
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_lpips 0.10903 0.10933 3.03e-04
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_psnr 24.69652 24.70208 5.56e-03
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_ssim 0.90333 0.90333 7.15e-07
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_lpips 0.07460 0.07454 5.74e-05
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_psnr 22.07094 22.07243 1.49e-03
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_ssim 0.94951 0.94952 1.32e-05
FaGr_s000001_butternutsquash1 single vs single2 (floor) geome_cd_l1 0.04028 0.04028 2.61e-06
FaGr_s000001_butternutsquash1 single vs single2 (floor) geome_f01 0.38985 0.38961 2.44e-04
FaGr_s000002_yellowmustardbottle5 single vs batched input_lpips 0.07058 0.07107 4.95e-04
FaGr_s000002_yellowmustardbottle5 single vs batched input_psnr 18.65362 18.75006 9.64e-02
FaGr_s000002_yellowmustardbottle5 single vs batched input_ssim 0.87413 0.87469 5.63e-04
FaGr_s000002_yellowmustardbottle5 single vs batched novel_lpips 0.05281 0.05350 6.87e-04
FaGr_s000002_yellowmustardbottle5 single vs batched novel_psnr 21.62769 21.60999 1.77e-02
FaGr_s000002_yellowmustardbottle5 single vs batched novel_ssim 0.94570 0.94571 8.44e-06
FaGr_s000002_yellowmustardbottle5 single vs batched geome_cd_l1 0.01265 0.01258 6.85e-05
FaGr_s000002_yellowmustardbottle5 single vs batched geome_f01 0.52419 0.52598 1.79e-03
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) input_lpips 0.07058 0.07036 2.18e-04
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) input_psnr 18.65362 18.65291 7.10e-04
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) input_ssim 0.87413 0.87409 4.23e-05
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) novel_lpips 0.05281 0.05280 1.12e-05
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) novel_psnr 21.62769 21.63569 7.99e-03
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) novel_ssim 0.94570 0.94571 1.22e-05
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) geome_cd_l1 0.01265 0.01267 1.40e-05
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) geome_f01 0.52419 0.52383 3.57e-04

4v

object pair SS coords (#) coords sym-diff (#voxels) V/F a V/F b max abs vert diff Chamfer-L1 (50k) Chamfer sampling floor (a vs a)
FaGr_s000001_butternutsquash1 single vs batched 7778/7777 1 267276/536392 267868/537604 n/a (V differ) 0.0033 0.0033
FaGr_s000001_butternutsquash1 single vs single2 (floor) 7778/7778 0 267276/536392 267108/536024 n/a (V differ) 0.0033 0.0033
FaGr_s000002_yellowmustardbottle5 single vs batched 10256/10256 0 208384/416792 208240/416520 n/a (V differ) 0.0034 0.0034
FaGr_s000002_yellowmustardbottle5 single vs single2 (floor) - - - - - missing mesh (decode OOM in single2 run, GPU oversubscribed)
object pair metric a b abs diff
FaGr_s000001_butternutsquash1 single vs batched input_lpips 0.10835 0.10771 6.42e-04
FaGr_s000001_butternutsquash1 single vs batched input_psnr 24.20930 24.22512 1.58e-02
FaGr_s000001_butternutsquash1 single vs batched input_ssim 0.92051 0.92061 9.86e-05
FaGr_s000001_butternutsquash1 single vs batched novel_lpips 0.03706 0.03715 9.50e-05
FaGr_s000001_butternutsquash1 single vs batched novel_psnr 27.21692 27.24218 2.53e-02
FaGr_s000001_butternutsquash1 single vs batched novel_ssim 0.96570 0.96579 8.91e-05
FaGr_s000001_butternutsquash1 single vs batched geome_cd_l1 0.00443 0.00441 2.08e-05
FaGr_s000001_butternutsquash1 single vs batched geome_f01 0.90865 0.91001 1.37e-03
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_lpips 0.10835 0.10829 5.63e-05
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_psnr 24.20930 24.20380 5.49e-03
FaGr_s000001_butternutsquash1 single vs single2 (floor) input_ssim 0.92051 0.92050 8.91e-06
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_lpips 0.03706 0.03699 6.97e-05
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_psnr 27.21692 27.21410 2.82e-03
FaGr_s000001_butternutsquash1 single vs single2 (floor) novel_ssim 0.96570 0.96572 1.36e-05
FaGr_s000001_butternutsquash1 single vs single2 (floor) geome_cd_l1 0.00443 0.00443 1.96e-06
FaGr_s000001_butternutsquash1 single vs single2 (floor) geome_f01 0.90865 0.90868 3.92e-05
FaGr_s000002_yellowmustardbottle5 single vs batched input_lpips 0.06266 0.06281 1.52e-04
FaGr_s000002_yellowmustardbottle5 single vs batched input_psnr 21.96632 21.96070 5.62e-03
FaGr_s000002_yellowmustardbottle5 single vs batched input_ssim 0.92068 0.92066 2.88e-05
FaGr_s000002_yellowmustardbottle5 single vs batched novel_lpips 0.03690 0.03690 2.28e-06
FaGr_s000002_yellowmustardbottle5 single vs batched novel_psnr 22.77716 22.77316 4.00e-03
FaGr_s000002_yellowmustardbottle5 single vs batched novel_ssim 0.95298 0.95298 2.53e-06
FaGr_s000002_yellowmustardbottle5 single vs batched geome_cd_l1 0.01277 0.01275 1.55e-05
FaGr_s000002_yellowmustardbottle5 single vs batched geome_f01 0.72182 0.72286 1.04e-03

Verdict

Ours (SS+SLAT): batched path is NOT identical beyond float noise -> NOT batched; the full run uses the unbatched ours_combined_cell.sh (paper Table-1 path).

Evidence:

  • Stage-1 SS coords: the unbatched path is bit-deterministic (single vs single2: 0 voxel differences on all 3 comparable pairs), but the batched ODE (B=2) flips occupancy voxels (1 voxel sym-diff in 2 of 4 object/view cells: yellowmustardbottle5 @1v 9766 vs 9765, butternutsquash1 @4v 7778 vs 7777). So batching changes the output where the single path does not.
  • Stage-2 SLAT decode is itself run-to-run non-deterministic (vertex/face counts differ between two identical single runs), so exact mesh identity is unattainable even unbatched; mesh Chamfer-L1 between runs (0.0032-0.0034) equals the 50k-sample Chamfer sampling floor in both the batched and repeat pairs.
  • Metric deltas, batched vs single, exceed the single-vs-single floor: input-view PSNR up to 0.096 dB (floor 0.0056), F@0.01 up to 1.8e-3 (floor 3.6e-4), CD-L1 up to 6.9e-5 (floor 1.4e-5), IV/NV LPIPS up to 6.9e-4 (floor 3.0e-4).
  • These are tiny in absolute terms, but they are systematically above the unbatched noise floor, so under the rule "identical beyond float noise" the batched path fails -> run single.

Baselines (ReconViaGen, Pixal3D SV/MV, Amodal3R, Hunyuan3D-2mv, Cupid): not batched (their drivers run one object per forward pass; no batched path used). Speed-up = several independent single-object worker processes per GPU (sharded object lists, skip-existing, atomic writes), which does not alter any computation -> batch-equivalence test not applicable.

Side note: one single2 4v decode OOMed (GPU 6 was oversubscribed by co-located baseline workers at the time; before the <=2 procs/GPU cap). The full run checks for missing ours meshes and re-runs them.