Diagnosis-Gated Learned Shape Completion
One RGB-D photo in โ a complete 3D shape out โจ
--- A single photo of an object gives a **broken, half-there 3D shape** ๐ซ โ hidden sides are just missing. GapSeg is a little chameleon ๐ฆ๐ that **looks at each object, guesses which ones came out distorted, and re-grows only those** with a learned network. The result beats stitching many camera angles together.
one RGB-D frame โ completed 3D, across 6 GraspNet scenes ๐
## โจ Why it's neat - ๐ **Diagnose first** โ predict per-object distortion from a single frame - ๐งฉ **Complete only what's broken** โ learned shape completion, gated by the diagnosis - ๐ **Beats geometric multi-view fusion** โ with just one view + completion ## ๐ Results (GraspNet, 278 objects, chamfer mm โ lower = better) | method | chamfer | vs single | |---|--:|--:| | ๐ท single view | 6.60 | โ | | ๐งต geometric fusion | 5.64 | 88% โ | | ๐งฉ **single + completion** | **2.75** | 95% โ | | ๐ฆ **diagnosis-gated completion** | **2.84** | 94% โ | > ๐ฏ **One photo + learned completion (2.75 mm) beats fusing many views (5.64 mm).** > Gating it by the diagnoser keeps the gain while only completing the hard objects. ## ๐ Usage ```python from gapseg.pipeline import GapSegPipeline pipe = GapSegPipeline.from_pretrained("haeing/gapseg", device="cuda") # rgb: HxWx3 uint8 ยท depth_m: HxW float (metres) ยท K: 3x3 # instances: [{"inst_id": 1, "mask": HxW bool, "R": 3x3, "t": (3,)}, ...] out = pipe.infer(rgb, depth_m, K, instances) for o in out: print(o["inst_id"], o["dims_cm"], "completed:", o["completed"]) ``` ๐งญ `R`/`t` (object pose) drive completion; objects with no pose or below the diagnosis threshold return their single-view cloud. ## ๐ฆ Contents - ๐ง `sq2_gview/best.pt` โ the G_view diagnoser (RGB-D โ distortion / conditions) - ๐งฉ `sq3_completion/best.pt` โ the PCN shape-completion network - ๐ผ๏ธ `gallery.gif` โ rotating demo across 6 scenesmade with ๐ฆ + ๐ ยท single view, whole shape