| --- |
| license: mit |
| tags: |
| - robotics |
| - 3d-reconstruction |
| - shape-completion |
| - point-cloud |
| - perception-for-manipulation |
| library_name: gapseg |
| --- |
| |
| <p align="center"> |
| <img src="mascot.png" width="240" alt="GapSeg mascot"/> |
| </p> |
|
|
| <h1 align="center">🦎 GapSeg</h1> |
| <p align="center"><b>Diagnosis-Gated Learned Shape Completion</b></p> |
| <p align="center"><i>One RGB-D photo in → a complete 3D shape out ✨</i></p> |
|
|
| --- |
|
|
| A single photo of an object gives a **broken, half-there 3D shape** 🫠 — hidden |
| sides are just missing. GapSeg is a little chameleon 🦎🔍 that **looks at each |
| object, guesses which ones came out distorted, and re-grows only those** with a |
| learned network. The result beats stitching many camera angles together. |
|
|
| <p align="center"> |
| <img src="gallery.gif" width="620" alt="single RGB-D to completed 3D across 6 scenes"/> |
| </p> |
| <p align="center"><sub>one RGB-D frame → completed 3D, across 6 GraspNet scenes 🌀</sub></p> |
|
|
| ## ✨ Why it's neat |
|
|
| - 🔍 **Diagnose first** — predict per-object distortion from a single frame |
| - 🧩 **Complete only what's broken** — learned shape completion, gated by the diagnosis |
| - 🏆 **Beats geometric multi-view fusion** — with just one view + completion |
|
|
| ## 📊 Results (GraspNet, 278 objects, chamfer mm ↓ lower = better) |
|
|
| | method | chamfer | vs single | |
| |---|--:|--:| |
| | 📷 single view | 6.60 | — | |
| | 🧵 geometric fusion | 5.64 | 88% ✅ | |
| | 🧩 **single + completion** | **2.75** | 95% ✅ | |
| | 🦎 **diagnosis-gated completion** | **2.84** | 94% ✅ | |
|
|
| > 🎯 **One photo + learned completion (2.75 mm) beats fusing many views (5.64 mm).** |
| > Gating it by the diagnoser keeps the gain while only completing the hard objects. |
|
|
| ## 🚀 Usage |
|
|
| ```python |
| from gapseg.pipeline import GapSegPipeline |
| pipe = GapSegPipeline.from_pretrained("haeing/gapseg", device="cuda") |
| |
| # rgb: HxWx3 uint8 · depth_m: HxW float (metres) · K: 3x3 |
| # instances: [{"inst_id": 1, "mask": HxW bool, "R": 3x3, "t": (3,)}, ...] |
| out = pipe.infer(rgb, depth_m, K, instances) |
| for o in out: |
| print(o["inst_id"], o["dims_cm"], "completed:", o["completed"]) |
| ``` |
|
|
| 🧭 `R`/`t` (object pose) drive completion; objects with no pose or below the |
| diagnosis threshold return their single-view cloud. |
|
|
| ## 📦 Contents |
| - 🧠 `sq2_gview/best.pt` — the G_view diagnoser (RGB-D → distortion / conditions) |
| - 🧩 `sq3_completion/best.pt` — the PCN shape-completion network |
| - 🖼️ `gallery.gif` — rotating demo across 6 scenes |
|
|
| <p align="center"><sub>made with 🦎 + 🔍 · single view, whole shape</sub></p> |
|
|
