Thinking-In-Boxes / README.md
PradhaanSBhat
Update README.md
429a03e
|
Raw History Blame Contribute Delete
4.38 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Thinking in Boxes
emoji: πŸ“¦
colorFrom: red
colorTo: gray
sdk: gradio
sdk_version: 6.30.0
app_file: demo.py
pinned: false
short_description: 3D Editing in Real Images Made Easy (NeurIPS 2026)

Thinking in Boxes: 3D Editing in Real Images Made Easy

NeurIPS 2026

Pradhaan S Bhat1βˆ— Β· Naveen Chandra R1βˆ— Β· Rishubh Parihar1 Β· Vaibhav Vavilala2 Β· R. Venkatesh Babu1 Β· D. A. Forsyth3 Β· Anand Bhattad4

1 Indian Institute of Science  Β·  2 Apple  Β·  3 UIUC  Β·  4 Johns Hopkins University

βˆ— Equal Contribution

arXiv Project Page Hugging Face Model Hugging Face Dataset GitHub


πŸ“š Abstract

Text and 2D-conditioning interfaces provide weak, ambiguous control over spatial transformations in image editing β€” particularly under large object motions and camera changes. Prior work has used 3D primitives such as boxes, but only as loose conditioning signals indicating approximate object location rather than specifying the transformation.

We instead use 3D boxes as structured specifications: the user provides the input and output boxes of the edit, casting editing as a well-posed geometry problem. This "thinking in boxes" interface, where each box face is color-coded to convey 3D orientation, gives precise control over translation, rotation, scaling, and viewpoint changes in real images while preserving scene and object identity, and recovering previously unseen object regions. To ground transformations in scene appearance, we introduce a depth-aligned planar floor as a global reference frame, shaded with depth-aware cues.

Conditioned on this structure, an image generator produces consistent results under large transformations. Trained in two stages β€” on synthetic multi-object scenes and a small set of real-world videos from Objectron β€” the system generalizes to complex, in-the-wild real images. Our method operates directly on real photographs and substantially outperforms recent state-of-the-art methods on large 3D edits.


Demo Usage

Interactive demo for [NeurIPS'26] Thinking in Boxes: 3D Editing in Real Images Made Easy.

  • Fit 3D boxes to objects, set a source pose, bookmark it on the left rail, edit to a target pose, then generate.
  • Read the tooltips and hover over icons for details on how to use the 3D Editor.
  • Runs on ZeroGPU (free, time-sliced RTX PRO 6000). Generation consumes your daily GPU quota.
  • 3D auto-detection runs in mock mode on the Space (the full WildDet3D detector needs a dedicated GPU/deps). Place and refine 3D boxes by hand. (See the GitHub Repository to use WildDet3D)

Setup notes (for whoever deploys this Space)

  1. Hardware: Settings β†’ Hardware β†’ ZeroGPU.
  2. Secret: add HF_TOKEN from an account that has accepted the FLUX.1-Kontext-dev license β€” the base model is gated and won't download otherwise.
  3. Files in the Space repo: assets/ demo.py, box-editor.html, pipelines/pipeline_flux_kontext.py (the custom pipeline), and requirements.txt.
  4. The default model + LoRA download on first boot (slow once, cached after).