Release v0.1.0: local Image3D Student prototype under Apache-2.0

1738cdf verified 4 days ago

10.5 kB

---
license: apache-2.0
language: en
pipeline_tag: image-to-3d
tags:
- pytorch
- gaussian-splatting
- distillation
- research-prototype
- synthetic-data
model_name: Image3D Student v0.1.0
---
# Image3D Student v0.1.0 — first public prototype

Train a compact image-to-Gaussian-splat student on cached teacher outputs, using Apple Metal or CPU.

**First release: v0.1.0 · Apache-2.0.** See [Windows, Linux and macOS instructions](docs/USAGE.md),
[all publisher model guides](docs/ALL_MODELS.md), [license](LICENSE), [third-party scope](THIRD_PARTY.md),
and [release notes](CHANGELOG.md).

**This is working research code, not a pretrained general-purpose image-to-3D model.**
The included demonstration checkpoint learned only procedural ellipsoids. It has not learned from Idol,
InstantMesh or Hi3D and should not be used as evidence of real-image quality.

## What is implemented

| Component | Status |
|---|---|
| Compact image encoder + coordinate decoder | Implemented, CPU and Metal training tested |
| Training, object-level validation, metrics and checkpoints | Implemented |
| Import/export of degree-zero Gaussian PLY | Implemented; preserves position, anisotropic scale, rotation, colour and opacity |
| Idol supervision | Import its PLY and the exact prepared input image |
| InstantMesh supervision | Import an aligned coarse triangle mesh as an auxiliary surface-occupancy target |
| Hi3D supervision | Import an aligned refined triangle mesh as a second auxiliary surface-occupancy target |
| Running Idol on Mac | Experimental staged adapter; tiny architecture components tested separately; full pretrained pipeline unverified |
| Running InstantMesh or Hi3D on Mac | **Not implemented**; official GPU implementations need porting |
| Binary/1-bit student | **Not implemented**; this small baseline uses FP32 |
| Differentiable rendering / multiview photometric loss | Not implemented in this first baseline |

The three-teacher training interface is present, but a fully local way to **generate all three teachers' data**
is still unresolved. Importing an existing output does not mean the corresponding teacher can run on this Mac.

## Start

On the Mac where this project was built, the dependencies are already installed in the working directory.
From this delivered folder, `bash run-local.sh doctor` reuses that environment. Pass any command described below
to `bash run-local.sh` instead of `image3d`. On a different machine, use the setup steps below.

Use Python 3.11–3.13. From this project directory:

```sh
bash setup.sh
source .venv/bin/activate
image3d doctor
```

`setup.sh` installs the student dependencies only. It does not download teacher weights.
`doctor` checks this process's Metal access. Sandboxed terminals can report Metal unavailable even when a normal
macOS Terminal can use it. Choose `--device mps` to require GPU execution; `auto` may fall back to CPU.

## Run the synthetic pipeline test

```sh
image3d demo-data --out data/demo --count 12 --grid 8
image3d train --manifest data/demo/manifest.jsonl --out runs/demo \
  --device mps --width 64 --steps 100 --lr 0.001 --eval-every 25 --allow-synthetic
image3d generate --checkpoint runs/demo/best.pt \
  --image data/demo/images/fixture-007.png --out runs/demo/student.ply --device mps
image3d preview --ply runs/demo/student.ply --out runs/demo/turntable.png
python -m pytest -q
```

The data split is seeded and groups all samples with the same object ID together. The commands refuse to overwrite
an existing run or exported PLY. Use a new directory for a new experiment. Ctrl-C saves completed optimizer steps
to `last.pt`. This version does not resume optimizer training from a checkpoint.

With validation enabled, `best.pt` selects the lowest held-out distillation loss. `last.pt` always contains the final
completed step. With `--validation-fraction 0`, there is no `best.pt` and no generalization measurement.

## Train with real teacher data

1. Supply an image of one centered object and obtain its Idol PLY.
2. Cache the **exact input framing seen by Idol**, ideally the `*_input.png` saved by Idol.
3. Optionally attach aligned coarse/refined meshes for the same object and image.
4. Repeat for a dataset of different objects. Keep related views in the same `--object-id`.
5. Train and judge unseen objects, not only training loss.

```sh
image3d cache --image teacher/item01/input.png --idol-ply teacher/item01/teacher.ply \
  --id item01-view01 --object-id item01 --out data/teachers --grid 16
image3d train --manifest data/teachers/manifest.jsonl --out runs/teachers-v1 \
  --device mps --steps 1000 --batch-size 1
image3d generate --checkpoint runs/teachers-v1/best.pt \
  --image your-centered-cutout.png --out results/object.ply --device mps
```

The student does not perform automatic background removal. Use already-framed cutouts, with the object scale
and orientation matching the training data. RGBA is composited on black, matching Idol's prepared images.
An arbitrary photo with clutter or different framing is outside the baseline's input contract.

### Attach all three teachers

```sh
image3d cache --image teacher/item02/input.png --idol-ply teacher/item02/teacher.ply \
  --coarse-mesh teacher/item02/instantmesh.obj --coarse-transform transforms/coarse.json \
  --fine-mesh teacher/item02/hi3d.glb --fine-transform transforms/fine.json \
  --id item02-view01 --object-id item02 --out data/teachers --grid 16
```

Each transform file is a JSON 4×4 affine matrix mapping mesh coordinates into the **exported Idol PLY coordinate
frame**. Idol changes its internal z-up frame when exporting PLY; do not apply that rotation twice.
Only use an identity transform if the assets are already aligned. Example identity:

```json
[[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]
```

Targets must fit inside `[-0.5,0.5]^3`. Out-of-bounds targets are rejected, not silently normalized.
Check rotation, scale, translation and camera framing visually before combining teachers. Different teachers can
hallucinate different unseen geometry. This baseline does **not** automatically align shapes or resolve their disagreements.

Geometry losses supervise auxiliary occupancy heads sharing the student features; they do not guarantee that final
Gaussian surfaces exactly follow the refined mesh. Joint rendering-based supervision and confidence filtering are future work.

The manifest stores source labels, transforms, input/target hashes, object IDs and compression counts. A missing teacher
is masked out of the loss. Do not label fabricated outputs as real teacher supervision.

## Experimental local Idol runner

The adapter uses the official Idol architecture at pinned SDK commit
`4702a7f00850e34b6e83a2170cc123c3e9570744`. It loads background removal, conditioning,
denoising, octree decoding and Gaussian decoding in stages, deleting models between stages.
It replaces complex rotary embeddings with the upstream real-valued equivalent and chunks attention along the query
dimension while retaining all keys/values. CUDA compilation and batched classifier-free guidance are disabled.

This reduces simultaneous working memory but **does not establish that full Idol fits in 16 GB**.
The encoders, their operators, loading peaks and full-size attention still need real-weight testing.
The runner has no silent CPU operator fallback, so unsupported operations surface as errors.

Install code dependencies separately:

```sh
uv pip install --python .venv/bin/python -e '.[teacher]'
```

Idol requires access to these gated encoders:

- [DINOv3 ViT-H+](https://huggingface.co/facebook/dinov3-vith16plus-pretrain-lvd1689m)
- [FLUX.2-dev VAE](https://huggingface.co/black-forest-labs/FLUX.2-dev)

Accept their terms using your own Hugging Face account. Set the local cache consistently before logging in:

```sh
export HF_HOME="$PWD/.cache/huggingface"
hf auth login
# Explicitly allows model downloads. Start with a transparent single-object PNG.
image3d teacher-idol --image your-cutout.png --out teacher/item01 \
  --cache "$HF_HOME" --allow-download --device mps --steps 20 --gaussians 32768
```

The download command requires 16 GiB of free disk space as a conservative safety margin, not a measured model minimum.
At build time this Mac had approximately 7.4 GB available before dependency installation, so no model weights were downloaded.
With a complete existing cache, omit `--allow-download`: the runner then uses local files only.
20 steps is a reduced-cost experiment; it is not the teacher's default 50-step quality setting.
Use `--steps 50` once execution and memory have been verified.

## Architecture and losses

The baseline is deliberately small: a four-stage convolutional image encoder produces a shared feature vector.
A coordinate MLP predicts one Gaussian per cell in a normalized object cube, plus coarse/fine geometry logits.
The default 16³ grid has at most 4,096 Gaussians; grid 32 has at most 32,768. Grid 8 is for smoke tests.
An image encoder trained from scratch is not expected to generalize from a handful of examples.

When several Idol Gaussians occupy a cell, the cache retains the highest-opacity one.
This is **lossy compression**, not a fidelity-preserving replacement for the full teacher. It can lose fine geometry,
transparency and colour. Teacher colours outside display RGB are clipped for student supervision.

Training uses balanced cell-occupancy BCE, cell-scaled position error, colour error, log-scale error,
quaternion sign-invariant rotation error, opacity error, and optional geometry occupancy BCE.
It uses direct cached-target regression; no CUDA rasterizer is needed for student training.
The preview renderer is an orthographic CPU diagnostic renderer, not a calibrated quality evaluator.

The model's speed comes from its small one-pass architecture, not from copying InstantMesh's speed through a loss.
Teacher-level quality, real-photo reconstruction, mesh topology and 1-bit compression are not established.

## Reproducibility

`requirements-tested.txt` records the environment used for verification; `pyproject.toml` specifies installable dependencies.
See `VERIFICATION.md` and `examples/synthetic-smoke/` for the delivered test evidence.
No external GPU or cloud training service was used. Source and Python package downloads did use the internet.

Upstream references:
[InstantMesh](https://github.com/TencentARC/InstantMesh),
[Hi3D](https://huggingface.co/hi3d-official/Hi3Dv1.0),
[Idol SDK](https://github.com/speridlabs/idol-sdk).

	---
	license: apache-2.0
	language: en
	pipeline_tag: image-to-3d
	tags:
	- pytorch
	- gaussian-splatting
	- distillation
	- research-prototype
	- synthetic-data
	model_name: Image3D Student v0.1.0
	---
	# Image3D Student v0.1.0 — first public prototype

	Train a compact image-to-Gaussian-splat student on cached teacher outputs, using Apple Metal or CPU.

	First release: v0.1.0 · Apache-2.0. See [Windows, Linux and macOS instructions](docs/USAGE.md),
	[all publisher model guides](docs/ALL_MODELS.md), [license](LICENSE), [third-party scope](THIRD_PARTY.md),
	and [release notes](CHANGELOG.md).

	This is working research code, not a pretrained general-purpose image-to-3D model.
	The included demonstration checkpoint learned only procedural ellipsoids. It has not learned from Idol,
	InstantMesh or Hi3D and should not be used as evidence of real-image quality.

	## What is implemented

	\| Component \| Status \|
	\|---\|---\|
	\| Compact image encoder + coordinate decoder \| Implemented, CPU and Metal training tested \|
	\| Training, object-level validation, metrics and checkpoints \| Implemented \|
	\| Import/export of degree-zero Gaussian PLY \| Implemented; preserves position, anisotropic scale, rotation, colour and opacity \|
	\| Idol supervision \| Import its PLY and the exact prepared input image \|
	\| InstantMesh supervision \| Import an aligned coarse triangle mesh as an auxiliary surface-occupancy target \|
	\| Hi3D supervision \| Import an aligned refined triangle mesh as a second auxiliary surface-occupancy target \|
	\| Running Idol on Mac \| Experimental staged adapter; tiny architecture components tested separately; full pretrained pipeline unverified \|
	\| Running InstantMesh or Hi3D on Mac \| Not implemented; official GPU implementations need porting \|
	\| Binary/1-bit student \| Not implemented; this small baseline uses FP32 \|
	\| Differentiable rendering / multiview photometric loss \| Not implemented in this first baseline \|

	The three-teacher training interface is present, but a fully local way to generate all three teachers' data
	is still unresolved. Importing an existing output does not mean the corresponding teacher can run on this Mac.

	## Start

	On the Mac where this project was built, the dependencies are already installed in the working directory.
	From this delivered folder, `bash run-local.sh doctor` reuses that environment. Pass any command described below
	to `bash run-local.sh` instead of `image3d`. On a different machine, use the setup steps below.

	Use Python 3.11–3.13. From this project directory:

	```sh
	bash setup.sh
	source .venv/bin/activate
	image3d doctor
	```

	`setup.sh` installs the student dependencies only. It does not download teacher weights.
	`doctor` checks this process's Metal access. Sandboxed terminals can report Metal unavailable even when a normal
	macOS Terminal can use it. Choose `--device mps` to require GPU execution; `auto` may fall back to CPU.

	## Run the synthetic pipeline test

	```sh
	image3d demo-data --out data/demo --count 12 --grid 8
	image3d train --manifest data/demo/manifest.jsonl --out runs/demo \
	--device mps --width 64 --steps 100 --lr 0.001 --eval-every 25 --allow-synthetic
	image3d generate --checkpoint runs/demo/best.pt \
	--image data/demo/images/fixture-007.png --out runs/demo/student.ply --device mps
	image3d preview --ply runs/demo/student.ply --out runs/demo/turntable.png
	python -m pytest -q
	```

	The data split is seeded and groups all samples with the same object ID together. The commands refuse to overwrite
	an existing run or exported PLY. Use a new directory for a new experiment. Ctrl-C saves completed optimizer steps
	to `last.pt`. This version does not resume optimizer training from a checkpoint.

	With validation enabled, `best.pt` selects the lowest held-out distillation loss. `last.pt` always contains the final
	completed step. With `--validation-fraction 0`, there is no `best.pt` and no generalization measurement.

	## Train with real teacher data

	1. Supply an image of one centered object and obtain its Idol PLY.
	2. Cache the exact input framing seen by Idol, ideally the `*_input.png` saved by Idol.
	3. Optionally attach aligned coarse/refined meshes for the same object and image.
	4. Repeat for a dataset of different objects. Keep related views in the same `--object-id`.
	5. Train and judge unseen objects, not only training loss.

	```sh
	image3d cache --image teacher/item01/input.png --idol-ply teacher/item01/teacher.ply \
	--id item01-view01 --object-id item01 --out data/teachers --grid 16

	image3d train --manifest data/teachers/manifest.jsonl --out runs/teachers-v1 \
	--device mps --steps 1000 --batch-size 1

	image3d generate --checkpoint runs/teachers-v1/best.pt \
	--image your-centered-cutout.png --out results/object.ply --device mps
	```

	The student does not perform automatic background removal. Use already-framed cutouts, with the object scale
	and orientation matching the training data. RGBA is composited on black, matching Idol's prepared images.
	An arbitrary photo with clutter or different framing is outside the baseline's input contract.

	### Attach all three teachers

	```sh
	image3d cache --image teacher/item02/input.png --idol-ply teacher/item02/teacher.ply \
	--coarse-mesh teacher/item02/instantmesh.obj --coarse-transform transforms/coarse.json \
	--fine-mesh teacher/item02/hi3d.glb --fine-transform transforms/fine.json \
	--id item02-view01 --object-id item02 --out data/teachers --grid 16
	```

	Each transform file is a JSON 4×4 affine matrix mapping mesh coordinates into the **exported Idol PLY coordinate
	frame**. Idol changes its internal z-up frame when exporting PLY; do not apply that rotation twice.
	Only use an identity transform if the assets are already aligned. Example identity:

	```json
	[[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]
	```

	Targets must fit inside `[-0.5,0.5]^3`. Out-of-bounds targets are rejected, not silently normalized.
	Check rotation, scale, translation and camera framing visually before combining teachers. Different teachers can
	hallucinate different unseen geometry. This baseline does not automatically align shapes or resolve their disagreements.

	Geometry losses supervise auxiliary occupancy heads sharing the student features; they do not guarantee that final
	Gaussian surfaces exactly follow the refined mesh. Joint rendering-based supervision and confidence filtering are future work.

	The manifest stores source labels, transforms, input/target hashes, object IDs and compression counts. A missing teacher
	is masked out of the loss. Do not label fabricated outputs as real teacher supervision.

	## Experimental local Idol runner

	The adapter uses the official Idol architecture at pinned SDK commit
	`4702a7f00850e34b6e83a2170cc123c3e9570744`. It loads background removal, conditioning,
	denoising, octree decoding and Gaussian decoding in stages, deleting models between stages.
	It replaces complex rotary embeddings with the upstream real-valued equivalent and chunks attention along the query
	dimension while retaining all keys/values. CUDA compilation and batched classifier-free guidance are disabled.

	This reduces simultaneous working memory but does not establish that full Idol fits in 16 GB.
	The encoders, their operators, loading peaks and full-size attention still need real-weight testing.
	The runner has no silent CPU operator fallback, so unsupported operations surface as errors.

	Install code dependencies separately:

	```sh
	uv pip install --python .venv/bin/python -e '.[teacher]'
	```

	Idol requires access to these gated encoders:

	- [DINOv3 ViT-H+](https://huggingface.co/facebook/dinov3-vith16plus-pretrain-lvd1689m)
	- [FLUX.2-dev VAE](https://huggingface.co/black-forest-labs/FLUX.2-dev)

	Accept their terms using your own Hugging Face account. Set the local cache consistently before logging in:

	```sh
	export HF_HOME="$PWD/.cache/huggingface"
	hf auth login

	# Explicitly allows model downloads. Start with a transparent single-object PNG.
	image3d teacher-idol --image your-cutout.png --out teacher/item01 \
	--cache "$HF_HOME" --allow-download --device mps --steps 20 --gaussians 32768
	```

	The download command requires 16 GiB of free disk space as a conservative safety margin, not a measured model minimum.
	At build time this Mac had approximately 7.4 GB available before dependency installation, so no model weights were downloaded.
	With a complete existing cache, omit `--allow-download`: the runner then uses local files only.
	20 steps is a reduced-cost experiment; it is not the teacher's default 50-step quality setting.
	Use `--steps 50` once execution and memory have been verified.

	## Architecture and losses

	The baseline is deliberately small: a four-stage convolutional image encoder produces a shared feature vector.
	A coordinate MLP predicts one Gaussian per cell in a normalized object cube, plus coarse/fine geometry logits.
	The default 16³ grid has at most 4,096 Gaussians; grid 32 has at most 32,768. Grid 8 is for smoke tests.
	An image encoder trained from scratch is not expected to generalize from a handful of examples.

	When several Idol Gaussians occupy a cell, the cache retains the highest-opacity one.
	This is lossy compression, not a fidelity-preserving replacement for the full teacher. It can lose fine geometry,
	transparency and colour. Teacher colours outside display RGB are clipped for student supervision.

	Training uses balanced cell-occupancy BCE, cell-scaled position error, colour error, log-scale error,
	quaternion sign-invariant rotation error, opacity error, and optional geometry occupancy BCE.
	It uses direct cached-target regression; no CUDA rasterizer is needed for student training.
	The preview renderer is an orthographic CPU diagnostic renderer, not a calibrated quality evaluator.

	The model's speed comes from its small one-pass architecture, not from copying InstantMesh's speed through a loss.
	Teacher-level quality, real-photo reconstruction, mesh topology and 1-bit compression are not established.

	## Reproducibility

	`requirements-tested.txt` records the environment used for verification; `pyproject.toml` specifies installable dependencies.
	See `VERIFICATION.md` and `examples/synthetic-smoke/` for the delivered test evidence.
	No external GPU or cloud training service was used. Source and Python package downloads did use the internet.

	Upstream references:
	[InstantMesh](https://github.com/TencentARC/InstantMesh),
	[Hi3D](https://huggingface.co/hi3d-official/Hi3Dv1.0),
	[Idol SDK](https://github.com/speridlabs/idol-sdk).