# FORGE3D: final evaluation protocol, evaluator, inference code, and what still has to be generated

This folder (`forgebench/` of `Ronaldo-GOAT/bert_simpson`) contains everything needed to (1) generate meshes for
FORGE3D (ours) and the baselines on the three benchmarks and (2) score them with the **final** evaluator
`code/eval/eval_final.py`. Every number in section 7 is written by that script (`summary.json`), never by hand.

Contents:

| path | what |
|---|---|
| `code/eval/eval_final.py` | **final evaluator** (FORGE3DBench-150, OmniObject3D-300, Toys4K-300); writes per-object `results.json`, `results.csv` and `summary.json` |
| `code/eval/appeval/` | renderer, metrics and geometry used by the evaluator. `render.py` has the far-clip fix (section 2.5). `geometry.py` adds extent-relative keys and leaves the original keys unchanged |
| `code/eval/heldout_index/` | held-out (novel-view) camera lists: `fb150.json`, `omni300.json`, `omni300_bad_frame_objects.txt` |
| `code/eval/heldout/` | `prepare_heldout.py` (how the held-out views were materialised) and `eval_heldout.py` (reference implementation; `eval_final.masked_view` equals it) |
| `code/eval/tests/` | unit tests (section 8) |
| `code/eval/align_v2.py`, `cammap_pixal3d.py`, `cammap_fb150_recenter.py` | placement of baseline meshes into the GT frame (section 3) |
| `code/ours/` | FORGE3D inference. Final path: `ours_combined_cell_batched.sh` |
| `code/baselines/` | baseline drivers (see `code/SETUP.md` for the official repos, commits and envs) |
| `data/fb150_heldout.tar` (new) | FORGE3DBench-150 held-out views: `fb150_heldout/{inputs,renders}` (2867 views; NV uses the 2798 with visibility >= 0.20) |
| `data/omni3d300_heldout_saved.tar` (new) | OmniObject3D-300 held-out saved views `top, top2, bottom, bottom2` -> `omni3d300_rand/{inputs,renders}`; extract next to `omni3d300_4v_inputs.tar` |
| `todo_final/` | **what must be generated**: `missing_<method>_<dataset>_<setting>.{json,txt}` (+ `.selection.json` subsets) and `SUMMARY.json` |
| `results/` | `summary.json` and `results.csv` of every cell scored with the final protocol |
| `MANIFEST.sha256` | sha256 of every file in this folder |
| `todo/`, `build_rest658.py`, `views_4v.json` | the other 658 FORGE3DBench objects (unchanged from the previous upload, see section 9) |

Tars that were uploaded before and are still needed: `data/fb150_exp_4v.tar` (FB150 inputs, cameras, GT meshes and pointmaps;
the 1v setting uses its `front` view, which is byte-identical to the old `exp_1v`), `data/omni3d300_4v_inputs.tar`,
`data/omni3d300_gt_canon.tar`. The Toys4K-300 exp dir (`toys4k300_rand`: 8 views, cameras, GT) is **not** in this folder.

---------------------------------------------------------------------------------------------------------------------

## 1. Datasets

| benchmark | objects | source of images | GT mesh frame | input views | held-out (NV) views |
|---|---:|---|---|---|---|
| **FORGE3DBench-150 (FB150)** | 150 | Multi-view scene images from the PSL MVScenes benchmark: Factory (78) and Park (72) environments; ground, robot-table and table anchors; 120 scenes, 24 cameras per scene; lit, cluttered, with occlusion. Inputs are crops of the scene frame. The alpha channel is the object's **visible (modal) mask**, i.e. the instance mask with depth > 0 | canonical mesh, centred, **max extent 0.9** | 1v = the front camera (visibility >= 0.40). 4v = front + 3 random other cameras of the same scene with visibility >= 0.40 (`random.Random(f"{obj}/4v/0")`), tags `front, side, back, oside` | the other scene cameras (20 of 24) with visibility >= **0.20** and >= 400 visible px: 2798 views, 13-20 per object (mean 18.7) |
| **OmniObject3D-300** | 300 | official OmniObject3D Blender renders (100 per object) of real scans, no occlusion, alpha = object alpha | scan mesh, canonicalised, max extent 1.0 | 4v = 4 random official renders `front, side, back, oside` (`selection.json -> view_idx`) | 4 saved random renders `top, top2, bottom, bottom2` (1200 views). Optional pool = the remaining 92 official renders per object (`--nv-set all`) |
| **Toys4K-300** | 300 | our Blender renders of textured Toys4K meshes | max extent 1.0, up = +Z | 4v = `front, side, back, oside` (front + 3 random). 1v = `front` | original protocol: 24-view orbit (section 2.3) |

Visibility on FB150 = modal / amodal pixel area of the object in that camera (`views_4v.json -> all_cam_visibility`).
It is copied into `heldout_index/fb150.json` as `vis`.

Camera convention (all datasets): `renders/<obj>_<tag>.npz` stores `fx, fy, cx, cy` (pixels, at `res` x `res`), `c2w_cv` (OpenCV
camera-to-world: x right, y down, z forward, in the GT canonical frame), `bbox = (y0, y1, x0, x1)` (the crop of the full frame
that `inputs/<obj>_<tag>.png` holds) and `res` (FB150 1440, Omni 800, Toys 768).

## 2. Evaluation protocol (final)

### 2.1 FORGE3DBench-150 and OmniObject3D-300: masked GT-image protocol (IV and NV)
One code path (`eval_final.masked_view`, identical to the held-out evaluator `eval_heldout.heldout_view`) for **both** input
views and held-out views:

1. Render the **predicted** mesh (already in the GT canonical frame, section 3) with the unlit nvdiffrast renderer
   (`appeval/render.py`: albedo or texture only, no lighting) at the dataset camera (`K`, `c2w_cv`, `res`), then crop it with `bbox`.
2. Reference = the **dataset's own GT image** of that camera (`inputs/<obj>_<tag>.png`: the real scene-frame crop on FB150, the
   official render on Omni). The mask M = alpha > 0.5. On FB150 M is the **visible** mask, so occluders and occluded object parts
   are never scored. On Omni M is the object alpha.
3. `pred_alpha *= M`, `ref_alpha := M`, both composited over white. Pixels outside M are white in both images and are
   ignored. Pixels in M that the prediction does not cover are compared as white vs GT and are penalised.
   `coverage` = fraction of M covered by the prediction (diagnostic).
4. Metrics per view: LPIPS (AlexNet, primary), SSIM (gaussian, win 11), PSNR, CLIP-ViT-B/32 cosine (`appeval/appearance.py`,
   unchanged). Mean over the views of an object, then mean over objects.

- **IV** = the input cameras: 4v -> `front, side, back, oside`, 1v -> `front`.
- **NV** = held-out cameras never given to any model. The set is **the same for 1v and 4v** (it never contains the 4 input cameras
  of the 4v setting):
  - FB150: `heldout_index/fb150.json`, only views with `vis >= 0.20` (`--min-vis 0.2`, the default). **Views below 20% visibility
    are dropped.**
  - Omni: `top, top2, bottom, bottom2` (`--nv-set saved`, the default). The 92-view pool is optional (`--nv-set all`).
- Omni bad-frame objects: 14 objects whose GT `canon.glb` is not in the render-camera frame
  (`heldout_index/omni300_bad_frame_objects.txt`). Seven are `frame_offset` (2.5-7.5 cm off); seven are `broken_camera`: insole_001-006 and
  ornaments_010, whose stored cameras sit at the origin. `--exclude-bad-frame` (used for the headline) drops them from `means`.
  They are still scored, listed under `excluded_objects`, and `means_including_excluded` is also written.
- The protocol ceiling is NOT perfect. The GT images are lit (scene lighting, shading, tone mapping) and the render is unlit, so the GT mesh
  scored as the prediction does not reach LPIPS 0. The ceilings (GT mesh vs GT image, same code) are in section 7.4. A diagnostic
  `--ref gtrender` replaces the GT image with the GT-mesh render (same camera and mask). It is **not** the protocol and is only
  used for the pipeline-identity test.

### 2.2 Geometry (all datasets, unchanged)
`appeval/geometry.py` on the placed prediction vs the GT canonical mesh, with no alignment inside the evaluator. It samples 100k
surface points per mesh (pytorch3d, seed 0) and reports:
- CD-L1 = 0.5 (mean p->g + mean g->p) of the unsquared Euclidean NN distance;
- F@tau with tau in {0.01, 0.02, 0.05} in GT-frame units (headline **F@0.01**);
- normal consistency;
- volume IoU @128^3.

It also reports `cd_l1_rel` and `f01_rel` (distances divided by the GT max extent), because the FB150 GT has extent 0.9 and Toys/Omni 1.0.

### 2.3 Toys4K-300: ORIGINAL protocol (unchanged)
Toys4K keeps `appeval/evaluate_appforce.eval_object` exactly:
- IV = the prediction rendered at the input cameras vs the input crop, unmasked (object alpha of both);
- NV = 24-view orbit (8 azimuths x elevations {-30, 0, 30}, radius 2.6, fov 40, 512^2), prediction render vs GT-mesh render;
- geometry as in 2.2.

`eval_final.py --dataset toys300` only calls `eval_object` and writes the same summary format. Its numbers equal
`evaluate_appforce.py` (test in section 8). The far-clip fix does not affect Toys, because all its cameras are 2.6 units from the origin.

### 2.4 Mesh placement before scoring (per method)
| method | FB150 | Omni / Toys |
|---|---|---|
| FORGE3D (ours) | none (generated in the GT frame) | none |
| ReconViaGen, Amodal3R, Hunyuan3D-2mv, Cupid | `align_v2.py`: GT-extent scale (0.9 FB150 / 1.0 others), 24 cube + 2000 random rotations scored by 64^3 voxel F1, rigid ICP on the top 8, input-view silhouette tie-break | same |
| Pixal3D 4v | `batch_pixal3d_mv_recenter.py --center origin` + `cammap_fb150_recenter.py` (camera-only, no ICP or scale fit): **X = C'_0 inv(F(d0)) A G + c** | `batch_pixal3d_mv.py` + `cammap_pixal3d.py` (camera-only): X = C0 inv(F(d)) A G, C0 = c2w_cv diag(1,-1,-1,1), d = \|C0 t\| |
| Pixal3D 1v | single-view pipeline + camera-only cammap (`pixal3dsv_1v_cammap`) | Toys 1v: MV pipeline with 1 view + `cammap_pixal3d.py` |

In the formulas, G is the exported glb vertices and A = [[-1,0,0],[0,0,1],[0,1,0]] undoes the glb export axes. F(d) is the official
canonical front camera (Blender c2w, camera at (0,-d,0)). For the re-centred FB150 run, C'_0 is the virtual (re-centred) front camera in the
shifted GT frame and c is the object centre. Both are written by the driver to `<obj>.cams.json`.

### 2.5 Fixes relative to the pre-final code
- **Renderer far-clip** (`appeval/render.py`, `render_input_view`): far was a fixed 20 units. FB150 tiny objects have cameras
  19.2-21.6 units from the origin, so they rendered empty. The far plane is now max(20, |camera centre| + 10). This is identical for every
  camera within 10 units (all Toys/Omni). Example, GT mesh of `PaGr_s000017_smallredcube0`, front camera at 20.7 units: IV LPIPS
  0.746 -> 0.075.
- **Masked IV on FB150**: the old IV compared the unmasked prediction silhouette with the visible-mask crop, which penalised correctly
  predicted occluded parts. IV now uses the visible mask, exactly like NV.
- **Held-out NV** replaces the 24-orbit GT-render NV on FB150 and Omni (the GT image is the reference).
- The summary is produced by the evaluator (`--summarize` merges shards). When duplicate rows exist, the last complete row wins
  and conflicts are listed. Missing, failed and excluded objects are listed explicitly.

## 3. Running the evaluator

Environment: the `mv-sam3d` env (torch, nvdiffrast, pytorch3d, lpips, open_clip, torchmetrics, scikit-image, trimesh). One
process needs about 2-3 GB of GPU memory. Rough speed on an A100: FB150 about 10 s/object (4 IV + about 19 NV views at 1440^2 +
geometry), Omni and Toys about 4-8 s/object.
Set `--exp` / `--heldout-root` to your extracted data (the defaults are the original server's paths).

```bash
cd forgebench/code/eval
# FB150 4v (placed meshes <obj>.glb in $M), 8 shards on 8 GPUs, then merge + summary
for i in 0 1 2 3 4 5 6 7; do
  CUDA_VISIBLE_DEVICES=$i python eval_final.py --dataset fb150 --setting 4v --meshes $M --method ours_fb150_4v \
     --exp $FB --heldout-root $DATA/fb150_heldout --min-vis 0.2 --shard $i --nshards 8 --out $OUT/ours_fb150_4v/s$i &
done; wait
python eval_final.py --dataset fb150 --setting 4v --method ours_fb150_4v --exp $FB --out $OUT/ours_fb150_4v \
     --summarize $OUT/ours_fb150_4v/s*          # -> results.json, results.csv, summary.json
# FB150 1v: same with --setting 1v (IV = front only; NV = same held-out set)
# Omni 4v (headline excludes the 14 bad-frame objects; they are still scored and listed)
python eval_final.py --dataset omni300 --setting 4v --meshes $M --exp $OMNI --nv-set saved --exclude-bad-frame --out ...
# Toys4K (original protocol)
python eval_final.py --dataset toys300 --setting 4v --meshes $M --exp $TOYS --out ...
# diagnostics: GT mesh as the prediction (protocol ceiling) / GT-render reference (identity test)
python eval_final.py --dataset fb150 --setting 4v --pred-is-gt --out ...  [--ref gtrender]
```
`summary.json` keys: `means` (headline, exclusions applied: `iv`, `nv` {lpips, ssim, psnr, clip, coverage, n_views_total,
n_objects}, `geometry` {cd_l1, f01, f02, f05, normal_consistency, vol_iou, cd_l1_rel, f01_rel}), `means_including_excluded`,
`n_scored`, `n_expected`, `missing_objects`, `failed_objects`, `excluded_objects`, `duplicate_conflicts_last_wins`, the protocol
flags and the evaluator's sha256. A cell is final only if `n_scored == n_expected` (minus listed exclusions).

## 4. Reproducing FORGE3D (ours)

Checkpoints on this Hub repo:
- SS-flow: `ckpt/ssflow_seedcond_v3_20260916/step_0080000.pt` (raw weights)
- SLAT-flow: `ckpt/slatflow_mv8_200k_4gpu_nockpt_4567_from21k_20260923/step_0032000.pt`

Seed 42. Code: `code/ours/` + the training repo (`migrator/code/mv-sam3d-for-6d-v2-ssflow/` on this Hub, providing `mvsam3d`,
`tools/val_daemon.py`, `configs/train_slatflow_prod.yaml`) + the SAM3D env (`code/SETUP.md`).

Final path = **batched** (`code/ours/ours_combined_cell_batched.sh`):
1. pass 1 `dump_seed.py` (pointmap seed);
2. pass 2 `ssflow_coords_batched.py --batch $SSB` (SS-flow ODE);
3. pass 3 `finish_cache.py` (stage-1 cache);
4. stage 2 `val_daemon.py --once` with `mkconfig_batched.py` (SLAT-flow, batch `S2B`, decode to glb).

Keep **S2B <= 4**: noisy seeds OOM the batched decoder at 8/16. Meshes land in
`$SCRATCH/runs/step_0032000/<DS>/seed/glb/<obj>.glb` in the GT frame (no placement needed). Use a fresh `$SCRATCH` per cell:
stage 2 decodes every object in the cache.

```bash
export CKPT=<dir with the two ckpt folders> SCRATCH=<work dir> GPUS="0 1 2 3 4 5 6 7"
export R=<data/env root> REPO=<.../mv-sam3d-for-6d-v2> PY=<mv-sam3d env python>     # see the script header
# FB150 4v (1v: NV=1, same exp dir; the 1v inputs are the front view of exp_4v)
OBJFILE=todo_final/missing_ours_fb150_4v.txt SSB=16 S2B=4 bash code/ours/ours_combined_cell_batched.sh "$GPUS" $FB fb150 4 \
  $CKPT/ssflow_seedcond_v3_20260916/step_0080000.pt \
  $CKPT/slatflow_mv8_200k_4gpu_nockpt_4567_from21k_20260923/step_0032000.pt $SCRATCH/ours_fb150_4v ours_fb150_4v
# Toys4K-300: EXP=$TOYS DS=toys4k300_rand NV=4|1 ; Omni-300: EXP=$OMNI DS=omni3d300_rand NV=4
```
Batched vs unbatched: `code/ours/batch_equivalence.md` (differences are at the level of 1 occupancy voxel, 1e-3 F@0.01).

## 5. Baselines
Drivers and exact argument lists: `code/SETUP.md` and `code/baselines/commands_reference.py`. All drivers skip existing
outputs and take `--shard i --nshards n`; seed 42 everywhere. Hunyuan3D-2mv on the random 4v sets needs the slot-permuted inputs
(`code/baselines/omni_reference/hy_slots.py <ds>` -> `<ds>_rand_hyslots`: side->right, back->back, oside->left). None of the
previously uploaded baseline drivers changed; the only differences on disk are path constants. New files:
`baselines/pixal3d_ours/{batch_pixal3d_mv_recenter.py, recenter.py}` and `eval/cammap_fb150_recenter.py`.

## 6. What needs to be generated (state of the original server, scanned 2026-09-26 04:56 KST)

Rule: **baselines generate only the missing objects. FORGE3D (ours) regenerates every object with the final batched code.**
Per cell, `todo_final/missing_<method>_<dataset>_<setting>.json` holds the object list, the count that exists now, the placement
and the exact command. The `.txt` file is the plain object list. RVG and Cupid take `--selection`, so they get
`missing_*.selection.json`. Variables: `$FB` = `exp_4v` of FB150; `$OMNI` = `omni3d300_rand`; `$OMNI_HY` = `omni3d300_rand_hyslots`;
`$TOYS` = `toys4k300_rand`; `$GEN` = output root; `$G` = GPU; `$i`/`$N` = shard index/count.

| method | Toys4K-300 4v | Toys4K-300 1v | Omni3D-300 4v | FB150 4v | FB150 1v |
|---|---|---|---|---|---|
| FORGE3D (ours) | **300** (all; 300 exist) | **300** (all; 288 exist) | **300** (all; 300 exist) | **150** (all; 150 exist) | **150** (all; 150 exist) |
| ReconViaGen | 0 | 0 | **182** (118 exist) | 0 | 0 |
| Amodal3R | 0 | 0 | **177** (123 exist) | 0 | 0 |
| Hunyuan3D-2mv | 0 | 0 | **210** (90 exist) | 0 | 0 |
| Pixal3D | 0 | 0 | **147** (153 exist) | **150** (all, re-centre driver) | 0 |
| Cupid | 0 | 0 | **229** (71 exist) | 0 | 0 |

Generation status on the original server at the time of the scan:
- The Omni 4v baseline schedulers stopped at about 00:50 KST on 09-26 and nothing is running. The 4v baseline meshes that exist are not yet placed or scored.
- FB150 Pixal3D 4v (re-centre, `--center origin`, 3 shards on GPUs 0-2) is queued behind a Toys 1v Pixal3D job that is running,
  with 217/300 done (`after_toys1v.sh`).

After generation:
1. Place the baselines: `align_v2.py` for RVG / Amodal3R / Hy3D-2mv / Cupid, the camera maps for Pixal3D.
2. Score every cell with `eval_final.py` (section 3).
3. Replace the "pending rescore" rows in section 7.

## 7. Results (final protocol only)

Each row lists how it was produced. Rows not produced with the final protocol are marked **pending rescore**. Their old numbers
are not reported here: the old FB150/Omni IV/NV used the unmasked IV, the 24-orbit NV and the far=20 renderer.

### 7.1 Toys4K-300 (original protocol, n = 300 unless noted)
Produced by `appeval/evaluate_appforce.py` (= the Toys path of `eval_final.py`) on the placed meshes; `summary.json` written by
`eval_final.py --summarize` over those result files (`results/toys4k300/<cell>/`).
The FORGE3D rows come from the **unbatched** run (SS 80k + SLAT 32k) and will be replaced by the batched regeneration.

**Toys4K-300 4v**

| method | n | IV LPIPS | IV SSIM | IV PSNR | NV LPIPS | NV SSIM | NV PSNR | CD-L1 | F@0.01 | NC | placement / source |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| FORGE3D (ours) | 300 | 0.0875 | 0.9157 | 22.39 | 0.0512 | 0.9426 | 25.07 | 0.0085 | 0.8502 | 0.8969 | none |
| ReconViaGen | 300 | 0.1508 | 0.8577 | 18.14 | 0.0780 | 0.9246 | 21.12 | 0.0147 | 0.5161 | 0.8448 | align_v2 |
| Amodal3R | 300 | 0.2105 | 0.8255 | 16.01 | 0.1077 | 0.9095 | 18.78 | 0.0221 | 0.4071 | 0.8034 | align_v2 |
| Hunyuan3D-2mv | 300 | 0.1687 | 0.8678 | 17.94 | 0.0851 | 0.9203 | 20.88 | 0.0165 | 0.5139 | 0.8391 | align_v2 |
| Pixal3D | 300 | 0.1141 | 0.8878 | 20.42 | 0.0569 | 0.9395 | 23.82 | 0.0095 | 0.7494 | 0.8761 | cammap_pixal3d |
| Cupid | 300 | 0.1899 | 0.8610 | 17.39 | 0.0990 | 0.9178 | 20.44 | 0.0227 | 0.4808 | 0.7934 | align_v2 |

**Toys4K-300 1v**

| method | n | IV LPIPS | IV SSIM | IV PSNR | NV LPIPS | NV SSIM | NV PSNR | CD-L1 | F@0.01 | NC | placement / source |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| FORGE3D (ours) | 288 | 0.1067 | 0.8954 | 20.60 | 0.0920 | 0.9268 | 20.85 | 0.0253 | 0.5881 | 0.7893 | none **pending: 12 missing (n=288/300)** |
| ReconViaGen | 300 | 0.2003 | 0.8338 | 16.69 | 0.1136 | 0.9136 | 19.01 | 0.0270 | 0.3752 | 0.7718 | align_v2 |
| Amodal3R | 300 | 0.2260 | 0.8169 | 15.98 | 0.1237 | 0.9040 | 17.97 | 0.0289 | 0.3498 | 0.7661 | align_v2 (2 retry rows replaced) |
| Hunyuan3D-2mv | 300 | 0.1898 | 0.8572 | 17.81 | 0.1055 | 0.9155 | 19.87 | 0.0245 | 0.4354 | 0.7971 | align_v2 |
| Pixal3D | 300 | 0.1081 | 0.8959 | 20.83 | 0.1361 | 0.9091 | 18.77 | 0.0444 | 0.3273 | 0.7247 | cammap_pixal3d (MV weights, 1 view) |
| Cupid | 300 | 0.1798 | 0.8633 | 17.73 | 0.1047 | 0.9173 | 19.90 | 0.0237 | 0.4447 | 0.7809 | align_v2 |

### 7.2 FORGE3DBench-150 (masked GT-image protocol, IV + NV with visibility >= 0.20)
All cells are **pending rescore**. FB150 meshes exist for every method except Pixal3D 4v, which is being regenerated. Commands are in section 3.

| method | 4v | 1v |
|---|---|---|
| FORGE3D (ours), RVG, Amodal3R, Hy3D-2mv, Cupid | pending rescore | pending rescore |
| Pixal3D | pending (regeneration + rescore) | pending rescore |

Smoke test of the final evaluator (FORGE3D 4v meshes, first 10 objects of the selection; **not a table row**): n=10: IV LPIPS 0.0733 / SSIM 0.9049 / PSNR 23.26 (views 40); NV LPIPS 0.0818 / SSIM 0.9016 / PSNR 22.73 (views 185); CD-L1 0.0105, F@0.01 0.752, NC 0.8959 (`code/eval/tests/TEST_RESULTS.json`).

### 7.3 OmniObject3D-300 4v (masked GT-image protocol; headline excludes the 14 bad-frame objects)
All cells are **pending rescore**. FORGE3D meshes are complete (300/300, unbatched); every baseline is partial (section 6).

### 7.4 Protocol ceilings (GT mesh scored as the prediction, final code)
- FB150 (all 150, 4v IV / NV vis>=0.2): IV LPIPS 0.1450 / SSIM 0.8604 / PSNR 18.51 (views 600); NV LPIPS 0.1444 / SSIM 0.8615 / PSNR 18.43 (views 2798); 1v IV (front only): LPIPS 0.1477 / SSIM 0.8565 / PSNR 18.35.
- Omni (60 objects: every 5th; bad-frame excluded): IV LPIPS 0.0781 / SSIM 0.9191 / PSNR 23.01 (views 228); NV LPIPS 0.0821 / SSIM 0.9165 / PSNR 22.73 (views 228).
- Toys4K (original protocol, 5 objects): IV LPIPS 0.0484 / PSNR 26.26; NV (orbit, render vs render) exactly perfect.
Methods that bake scene lighting into their texture can score above the GT-mesh ceiling on IV (e.g. FORGE3D on FaGr_s000001_butternutsquash1: IV PSNR 25.0 vs GT-mesh 22.8), because the reference is the lit image.

## 8. Tests (run 2026-09-26 KST, GPU 7)
| test | n obj | IV LPIPS / PSNR | NV LPIPS / PSNR | CD-L1 / F@0.01 / NC |
|---|---:|---|---|---|
| FB150 GT mesh, --ref image (protocol ceiling) | 5 | 0.1554 / 17.66 | 0.1558 / 17.69 | 0.0000 / 1.000 / 1.0000 |
| FB150 GT mesh, --ref gtrender (identity) | 5 | 0.0000 / inf | 1.339e-11 / inf | 0.0000 / 1.000 / 1.0000 |
| Omni GT mesh, --ref image (ceiling, bad-frame excluded) | 4 | 0.0667 / 22.58 | 0.07639 / 22.15 | 0.0000 / 1.000 / 0.9999 |
| Omni GT mesh, --ref gtrender (identity, bad-frame excluded) | 4 | 0.0000 / 186.37 | 1.548e-11 / 186.55 | 0.0000 / 1.000 / 0.9999 |
| Toys GT mesh (original protocol) | 5 | 0.0484 / 26.26 | 2.961e-19 / inf | 0.0000 / 1.000 / 1.0000 |

- `tests/test_equiv_heldout.py` (eval_final.masked_view vs eval_heldout.heldout_view with the original far=20 renderer, FB150 incl. far-camera cubes + Omni): EQUIV (cams < 10 units: 19 views, bit-identical renders 11, max metric diff 2.05e-08, max pix diff 5.96e-08): PASS ; far cams (>=10 units, far-clip fix may change render): 9 views, changed 4
- Toys path vs the unchanged `evaluate_appforce.py` on 2 FORGE3D meshes: max |diff| appearance 3.1e-10, geometry 1.5e-05 (the same order as two runs of the original code: GPU point-sampling noise).
- Real prediction (FORGE3D FB150 4v, first 10 objects): n=10: IV LPIPS 0.0733 / SSIM 0.9049 / PSNR 23.26 (views 40); NV LPIPS 0.0818 / SSIM 0.9016 / PSNR 22.73 (views 185); CD-L1 0.0105, F@0.01 0.752, NC 0.8959 (`code/eval/tests/TEST_RESULTS.json`).
- GT-mesh scored as the prediction: geometry exactly perfect on all datasets; with `--ref gtrender` appearance is exactly perfect (LPIPS 0, SSIM 1, PSNR inf / >180 dB); with the protocol reference (GT image) it gives the ceiling, not 0, because the GT images are lit and the render is unlit.

## 9. Other material in this folder (unchanged from the previous upload)
- `todo/`, `build_rest658.py`, `views_4v.json`: the remaining 658 objects of the full 808-object FORGE3DBench (same builder as
  FB150), meant for generation on other servers. Held-out views for those objects are not built. To score them with the final
  protocol, build them with `code/eval/heldout/prepare_heldout.py`.
- `code/omni_vggt/`: the experimental OmniObject3D real-capture VGGT-Omega seed ablation. It is not part of the main protocol. The
  OpenDataLab keys are read only from the `OPENXLAB_AK` / `OPENXLAB_SK` environment variables and no credential is in this folder.
- `code/baselines/pixal3d_ours/batch_pixal3d_mv_cropK.py` is superseded on FB150 by the re-centre driver.