Create README.md

README.md

@@ -0,0 +1,67 @@
+---
+language:
+  - en
+tags:
+  - freeinpaint
+  - 3d-scene-inpainting
+  - gaussian-splatting
+  - multi-view
+  - pose-free
+  - computer-vision
+---
+
+# FreeInpaint
+
+## Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement
+
+[Jingyi Pan](https://rorisis.github.io/)<sup>1</sup>, [Dan Xu](https://www.danxurgb.net/)<sup>2</sup>, [Qiong Luo](https://www.cse.ust.hk/~luo/)<sup>1,2</sup>
+
+<sup>1</sup>The Hong Kong University of Science and Technology (Guangzhou)<br>
+<sup>2</sup>The Hong Kong University of Science and Technology
+
+**NeurIPS 2026**
+
+[Project Page](https://rorisis.github.io/FreeInpaint/) | [arXiv](https://arxiv.org/abs/2610.11857) | [PDF](https://arxiv.org/pdf/2610.11857) | [GitHub](https://github.com/Rorisis/FreeInpaint)
+
+## Abstract
+
+3D scene inpainting aims to recover missing or occluded regions in edited 3D scenes, while ensuring geometric and textural consistency. Existing approaches, however, typically require accurately calibrated camera poses, which restricts their applicability in casual, in-the-wild scenarios and introduces additional preprocessing overhead. To overcome this limitation, we present **FreeInpaint**, a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions. At its core, FreeInpaint extends a 3D foundation model to propagate masked regions from a reference view to other unposed views, bridging 3D reconstruction and scene inpainting while preserving the model's native ability to recover camera poses and scene geometry. Our method addresses two key challenges in adapting feed-forward 3D foundation models to masked inputs. First, masked regions can corrupt cross-view correspondence reasoning, degrading pose estimation and geometry recovery. To address this, we introduce a Learnable Mask Attention mechanism that preserves the spatial anchoring of reliable observations while allowing masked regions to progressively absorb useful context in deeper layers. Second, under severe occlusions, a single forward pass often lacks sufficient appearance evidence for high-fidelity completion. Therefore, we propose a Support Token Refinement strategy, which injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions while preserving the original spatial anchor. Extensive experiments across diverse datasets demonstrate that FreeInpaint achieves superior inpainting quality, eliminating the reliance on pre-computed camera poses while keeping a fast inference speed. The project page is https://rorisis.github.io/FreeInpaint/.
+
+## Model Files
+
+Download the weights from the [checkpoints directory](https://huggingface.co/jyproris/FreeInpaint/tree/main/checkpoints) and arrange them in the cloned FreeInpaint code repository as follows:
+
+```text
+FreeInpaint/
+└── checkpoints/
+    ├── freeinpaint.pth
+    └── ppt-v1/
+        ├── unet/
+        │   └── unet.safetensors
+        └── text_encoder/
+            └── text_encoder.safetensors
+```
+
+- `freeinpaint.pth`: the released FreeInpaint checkpoint.
+- `ppt-v1/`: third-party [PowerPaint](https://github.com/open-mmlab/PowerPaint) weights used for diffusion-based inpainting.
+
+## Usage
+
+Use the inference implementation in the [FreeInpaint GitHub repository](https://github.com/Rorisis/FreeInpaint). Follow its [installation instructions](https://github.com/Rorisis/FreeInpaint#installation), download the checkpoints, and prepare the [demo data](https://github.com/Rorisis/FreeInpaint#demo-data).
+
+## Intended Use and Limitations
+
+FreeInpaint is intended for research in multi-view 3D scene inpainting and object removal from unposed images. Generated content is a plausible completion, not a recovery of the true hidden scene. Results depend on reference-image quality and the diffusion prior; sparse observations and severe occlusions can lead to artifacts or inconsistent geometry.
+
+See the [paper](https://arxiv.org/abs/2610.11857) for training details, evaluations, and limitations. Third-party models and datasets retain their respective licenses and usage conditions.
+
+## Citation
+
+```bibtex
+@inproceedings{pan2026pose,
+  title     = {Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement},
+  author    = {Pan, Jingyi and Xu, Dan and Luo, Qiong},
+  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
+  year      = {2026}
+}
+```