Idol on LiteRT: one photo to a 3D Gaussian splat on the phone

Idol (speridlabs/idol with the idol-turbo LoRA) as LiteRT graphs for CompiledModel, converted with litert-torch: on a Galaxy S26, the Android sample in android/ turns one photo into a 3D Gaussian splat of 65,536 Gaussians in 158 s with the default form, or of 262,144 Gaussians in 267 s with the first release's form in 262k/ (the first photo after start on a cool phone, LiteRT 2.3.0; measured on 2026-11-09 and 2026-11-10 JST).

speridlabs/idol turns one photo of an object into a 3D Gaussian splat (262,144 Gaussians at its defaults) that you can turn around, including the sides the picture never showed. This repo holds the author's denoiser with the speridlabs/idol-turbo LoRA (5 steps, no guidance), its octree and Gaussian decoders, the FLUX.2 VAE encoder, and the author's BiRefNet background remover. The DINOv3 image encoder is in a separate, gated repo under its own licence: litert-community/DINOv3-ViT-H16plus-Idol-LiteRT.

Left: the photo by Komet Flicker (Pexels License). Right: the splat the sample app made from it on a Galaxy S26 with the default form (65,536 Gaussians), rendered from the front, the side and the back.

Two forms

The default form (the repo's root) is the fastest of the forms measured on a Galaxy S26: it makes a splat in 158 s against 267 s for the first release's form in 262k/, while its renders agree less closely with the author's default output: 32.3–34.4 dB PSNR against 41.4–46.6 dB (higher is closer). Of the other forms tried, int8 weights did not run on the phone GPUs, a 512 × 512 input made even the author's own PyTorch pipeline miss the 25 dB bar (13.5–22.9 dB against its 1,024 × 1,024 output), the denoiser did not compile for the Galaxy S26's NPU on the phone, and the CPU alone was slower.

	default (the repo's root)	first release (262k/)
Gaussians per splat	65,536 (2,048 anchors)	262,144 (8,192 anchors, the author's default)
GPU precision	FP16	FP16 with fp32 accumulation
Denoiser blocks 1–22 and Gaussian decoder graphs 1–6	one shared graph per chain; the app copies each block's weights in before the call	a graph per block
Galaxy S26: the first photo after start, cool phone (thermal status 0 at the start)	158 s	267 s
Galaxy S26: photos on a phone already warm	329–351 s (4 photos of the armchair, thermal status 3)	339–449 s (5 photos of 3 inputs, thermal status 2)
Pixel 11 Pro: the first photo after start	343 s	not measured
4-view render PSNR against the author's default on the Galaxy S26 (threshold 25 dB)	32.3–34.4 dB	41.4–46.6 dB

Thermal status is Android's: 0 none, 1 light, 2 moderate, 3 severe. Both forms run with the same app and the same Python runner: point them at the repo's root or at 262k/.

The first release's card gives 303 s for the 262k/ form: that photo started on a warmer phone (thermal status 1, skin 38.5 °C, against 34.6 °C here). With this app the 262k/ form makes the same splat as with the first release's, byte for byte.

The rest of this card describes the default form; the first release's card (commit e7ee1226) has the per-stage tables of the 262k/ form.

Speed

On a Galaxy S26, the sample app turns one photo into a splat of 65,536 Gaussians in 158 s, starting from a cool phone (the first photo after start). The phone throttles its GPU during that photo: the GPU clock cap fell from 1,300 to 500 MHz (thermal status 0 → 2). After 12 minutes of back-to-back runs the phone was at thermal status 3; through the next 4 photos the GPU clock cap stayed at 222–282 MHz (one reading of 191 MHz), and they took 329–351 s. At start the app compiles 1 graph before the photo (2 s) and 19 more during it (20 s): the denoiser's 22 middle blocks share one graph and the Gaussian decoder's six middle graphs share another, so each is compiled once per photo. The first release's form in 262k/ (262,144 Gaussians, a graph per block, GPU FP16 with fp32 accumulation) took 267 s the same way, also from thermal status 0 (skin 34.6 °C at the start against 33.5 °C). Background removal (BiRefNet) runs on the CPU: 7.4 s of that time on the Galaxy S26. The other networks run on the GPU at FP16.

Galaxy S26 (SM-S942Q), the sample app, LiteRT 2.3.0; from a cool phone (GPU clock cap 1,300 MHz, thermal status 0 at the start)	first photo after start
photo → .ply written	158 s
background removal (BiRefNet, CPU)	7.4 s
DINOv3 encoder	10.6 s
VAE encoder	5.2 s
denoiser, 5 steps	119.8 s
octree decoder	9.1 s
Gaussian decoder	5.5 s
PLY writer	0.1 s
graphs compiled at app start, before the first photo	1 (2 s)
graphs compiled during the photo	19 (20 s)
peak resident memory (VmHWM)	5.42 GiB
peak GPU memory (kgsl)	5.73 GiB

Galaxy S26 (SM-S942Q), the sample app, LiteRT 2.3.0; right after the cool photo's hold (thermal status 3 at the start)	first photo after start	photo 2	photo 3	photo 4
photo → .ply written	351 s	351 s	341 s	329 s
background removal (BiRefNet, CPU)	14.9 s	18.9 s	15.2 s	15.4 s
DINOv3 encoder	22.5 s	23.6 s	25.1 s	20.6 s
VAE encoder	10.7 s	8.1 s	8.0 s	7.5 s
denoiser, 5 steps	278.1 s	280.3 s	272.6 s	265.4 s
octree decoder	14.3 s	11.7 s	12.1 s	12.2 s
Gaussian decoder	9.2 s	7.7 s	7.6 s	7.4 s
PLY writer	0.1 s	0.0 s	0.0 s	0.0 s
graphs compiled at app start, before the first photo	3 (9 s)	—	—	—
graphs compiled during the photo	18 (29 s)	18 (21 s)	19 (25 s)	18 (20 s)
peak resident memory (VmHWM)	5.30 GiB	4.58 GiB	5.10 GiB	4.63 GiB
peak GPU memory (kgsl)	5.73 GiB	5.12 GiB	5.13 GiB	5.04 GiB

Pixel 11 Pro, the sample app, LiteRT 2.3.0; thermal status 0 at the start	first photo after start
photo → .ply written	343 s
background removal (BiRefNet, CPU)	17.4 s
DINOv3 encoder	8.5 s
VAE encoder	36.1 s
denoiser, 5 steps	242.3 s
octree decoder	12.9 s
Gaussian decoder	23.7 s
PLY writer	0.2 s
graphs compiled at app start, before the first photo	6 (60 s)
graphs compiled during the photo	15 (75 s)
peak resident memory (VmHWM)	5.96 GiB

How close it is to the author's model

Each check runs the author's PyTorch pipeline (float32, CPU) and these graphs on the same input with the same random numbers, against thresholds fixed before the first run. The inputs are two images from the author's gallery (nb-armchair, fig-axolotl) and one photo (a succulent, Pexels). Per part, each graph gets the author's own input for that part and its output is compared with the author's (cosine; the threshold is 0.99); the octree and Gaussian decoders are compared with the author's model run at this form's 65,536 Gaussians (--num-gaussians 65536). The whole pipeline is compared with the author's default (262,144 Gaussians) by rendering both splats from four sides with the same renderer (PSNR; the threshold is 25 dB).

Per-part agreement on the Mac (LiteRT GPU, Metal, fp16): cosine vs the author's model

Part	Compared tensor	armchair	axolotl	succulent
DINOv3 encoder	feature1	0.99989	0.99994	0.99992
VAE encoder	latent tokens	0.99995	0.99993	0.99996
Denoiser (5 steps)	latent velocity, worst step	0.99931	0.99924	0.99931
Octree decoder (8 levels)	child probabilities, worst level	0.99948	0.99937	0.99944
Gaussian decoder	xyz	1.00000	1.00000	1.00000
	colour (f_dc)	0.99980	0.99987	0.99937
	scale	0.99959	0.99925	0.99225
	rotation	0.99953	0.99919	0.99929
	opacity	0.99995	0.99969	0.99969

Per-part agreement on a Galaxy S26 (the app; GPU FP16, BiRefNet on the CPU): cosine vs the author's model

Part	Compared tensor	armchair
DINOv3 encoder	feature1	0.99987
VAE encoder	latent tokens	0.99991
Denoiser (5 steps)	latent velocity, worst step	0.99924
Octree decoder (8 levels)	child probabilities, worst level	0.99943
Gaussian decoder	xyz	1.00000
	colour (f_dc)	0.99973
	scale	0.99939
	rotation	0.99938
	opacity	0.99996

Per-part agreement on a Pixel 11 Pro (the app; GPU FP16, BiRefNet on the CPU): cosine vs the author's model

Part	Compared tensor	armchair
DINOv3 encoder	feature1	0.99989
VAE encoder	latent tokens	0.99997
Denoiser (5 steps)	latent velocity, worst step	0.99930
Octree decoder (8 levels)	child probabilities, worst level	0.99945
Gaussian decoder	xyz	1.00000
	colour (f_dc)	0.99988
	scale	0.99963
	rotation	0.99939
	opacity	0.99998

Background removal

Input	Where	uint8 alpha pixels that differ from the author's	largest difference	crop square (author / here)
armchair	Mac, LiteRT CPU	2,544 of 1,048,576	1 / 255	same
armchair	Galaxy S26, the app	2,542 of 1,048,576	1 / 255	same
armchair	Pixel 11 Pro, the app	2,543 of 1,048,576	1 / 255	same
axolotl	Mac, LiteRT CPU	709 of 1,048,576	1 / 255	same
succulent	Mac, LiteRT CPU	6,115 of 1,048,576	1 / 255	same

Whole pipeline: 4-view render PSNR

Input	Where	PSNR, 4 views pooled (threshold: 25 dB)	per view: 0°, 90°, 180°, 270°
armchair	Mac, LiteRT GPU (Metal, fp16)	35.02 dB	34.23, 36.99, 33.39, 36.48
armchair	Galaxy S26, the app	34.41 dB	34.22, 35.85, 33.69, 34.15
armchair	Pixel 11 Pro, the app	33.58 dB	32.42, 34.81, 31.90, 36.89
axolotl	Mac, LiteRT GPU (Metal, fp16)	31.24 dB	32.84, 28.79, 33.37, 31.52
axolotl	Galaxy S26, the app	32.26 dB	33.27, 30.07, 33.71, 33.04
succulent	Mac, LiteRT GPU (Metal, fp16)	34.85 dB	34.05, 34.89, 35.03, 35.58
succulent	Galaxy S26, the app	34.26 dB	33.47, 34.26, 34.41, 35.05

Minimal usage

Python: the whole pipeline

hf download litert-community/Idol --exclude "262k/*" --local-dir Idol   # the default form; drop --exclude for both
cd Idol
pip install -r python/requirements.txt
python python/run_idol.py photo.jpg --out out --models .   # writes out/gaussian.ply; --models 262k for the other form

run_idol.py downloads the DINOv3 repo (accept its licence on the Hub and hf auth login first) and runs every graph with CompiledModel, the GPU for all but BiRefNet. The steps between graphs are in python/idol_runner.py and python/idol_glue.py. On an M4 Max Mac (Metal GPU) one photo took 53 s and up to 6 GB of memory (the process footprint).

Python: one graph with CompiledModel (background removal)

import sys

import numpy as np
from PIL import Image
from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator
from ai_edge_litert.options import Options

path, photo = sys.argv[1], sys.argv[2]   # bir.tflite from this repo, any photo
model = CompiledModel.from_file(path, options=Options(hardware_accelerators=HardwareAccelerator.CPU))
img = Image.open(photo).convert("RGB")
x = np.asarray(img.resize((1024, 1024), Image.BILINEAR), np.float32) / 255
x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32)
inputs, outputs = model.create_input_buffers(0), model.create_output_buffers(0)
inputs[0].write(x.reshape(-1))                       # [1, 3, 1024, 1024]
model.run_by_index(0, inputs, outputs)
alpha = outputs[0].read(1024 * 1024, np.float32).reshape(1024, 1024)   # 0..1, 1 = the object
mask = Image.fromarray((alpha * 255).astype(np.uint8)).resize(img.size, Image.BILINEAR)
img.putalpha(mask)
img.save("cutout.png")
print(alpha.min(), alpha.max(), (alpha > 0.5).mean())

Kotlin (Android)

val env = Environment.create()
val cpu = CompiledModel.Options(Accelerator.CPU)            // bir.tflite: BiRefNet, the one CPU graph
val bir = CompiledModel.create(File(dir, "bir.tflite").absolutePath, cpu, env)
val inputs = bir.createInputBuffers()
val outputs = bir.createOutputBuffers()
inputs[0].writeFloat(image)        // [1, 3, 1024, 1024]: RGB / 255, ImageNet mean / std
bir.run(inputs, outputs)
val alpha = outputs[0].readFloat() // [1024 * 1024] sigmoid map, 1 = the object
for (b in inputs + outputs) b.close()
bir.close()
val gpu = CompiledModel.Options(Accelerator.GPU).apply {   // every other graph
    gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16)
}

The Android sample app (photo or camera → progress per stage → a splat you can turn with a finger) is in android/. The graphs are too large for the APK; a script copies them into the app's files folder over adb:

hf download litert-community/Idol --exclude "262k/*" --local-dir ~/Downloads/Idol   # the default form; drop --exclude to get 262k/ too
hf download litert-community/DINOv3-ViT-H16plus-Idol-LiteRT --local-dir ~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT
cd ~/Downloads/Idol/android
chmod +x gradlew scripts/install_to_device.sh        # the Hub does not keep the executable bit
./gradlew :app:installDebug
IDOL_DIR=~/Downloads/Idol DINO_DIR=~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT ./scripts/install_to_device.sh
# the first release's form instead (download without --exclude first): IDOL_DIR=~/Downloads/Idol/262k, same DINO_DIR

Background removal (BiRefNet on the CPU) needs about 2,670 MB of free memory for one call. Before compiling it, the app closes GPU graphs until that much is free on top of its 2,500 MB reserve; if other apps hold the rest, it stops with "Not enough memory for background removal (bir) … Close other apps and try again." before starting the call.

What runs where

Stage	Files	Runs on	Calls per photo
Background removal (BiRefNet)	bir.tflite	CPU	1
DINOv3 ViT-H+/16 image encoder	dino_0..3.tflite (DINOv3 repo)	GPU	4
FLUX.2 VAE encoder	vae_0..3.tflite, vae_attn.tflite, vae_4.tflite	GPU	6
Denoiser with the turbo LoRA	den_ctx.tflite, den_in.tflite, den_joint_00.tflite, den_joint_mid.tflite (blocks 1–22, weights from den_joint_w.f32), den_joint_23.tflite	GPU	5 steps × 25, plus den_ctx once per LoRA phase
Octree decoder	oct.tflite (5 signatures by row count)	GPU	8 levels
Gaussian decoder	gs_0.tflite, gs_mid.tflite (graphs 1–6, weights from gs_w.f32), gs_7.tflite	GPU	8
Octree sampling, Gaussian activation, PLY	host code	CPU	—

262k/ holds the first release's form: den_joint_01..22.tflite and gs_1..6.tflite in place of the two shared graphs, the Gaussian decoder at 8,192 anchors, and its own manifest.json (GPU precision FP16 with fp32 accumulation); the files with the same name in both folders are the same bytes, except gs_0.tflite, gs_7.tflite and manifest.json.

manifest.json lists every graph with its signatures, accelerator and precision; the GPU graphs run with FP16 precision. den_joint_mid and gs_mid are one graph each for blocks of the same shape: before each call the app copies that block's weights from the side file into the graph's weight inputs (shared_blocks, block_weights and first_weight_input in manifest.json), so each is compiled once per photo.

first_weight_input is the number of inputs before the weights, counted in numeric order of the input names: 3 for den_joint_mid, whose weights are args_3 to args_32, in that order in each block of den_joint_w.f32. manifest.json lists the inputs in the order the .tflite signature stores them (args_0, args_1, args_10, …, args_19, args_2, …), while LiteRT 2.3.0's CompiledModel lists them in numeric order, so match the inputs by name, as python/idol_runner.py and the app do.

Notes

• Background removal runs on the CPU. BiRefNet's 20 deformable convolutions sample at positions computed from the image. They are rewritten as a gather with bilinear weights and one matmul (same arithmetic; 5.5e-6 from the author's map in PyTorch), and the LiteRT GPU has no kernel for that gather. The gathers run in row tiles one after another: each tile's offsets wait for the previous tile's result. Without that wait the CPU runtime ran every tile's gather before using any of them; the first export ran out of memory on the Galaxy S26, and this one peaks at 3.96 GiB of resident memory (VmHWM) in one call on it.
• The VAE's GroupNorm scales each group by its largest |x − mean| to stay in fp16 range. The graph finds that maximum as a max pool, then a max over the 32 × 32 result (the same value). One REDUCE_MAX over the whole 1024 × 1024 map took 8 s per GPU call on the Galaxy S26; this form takes 0.83 s.
• Attention over 4,096 keys or more is summed in chunks. The product of the attention weights and the values is summed over key chunks of 1,024 after centring the values (exact, since each softmax row sums to 1). One fp16 sum over 8,192 keys lost 18 % in the Gaussian decoder's first block; the chunked form loses 0.6 %. Keep it if you re-export.
• Keep each GPU call short on the Galaxy S26. In the feasibility run the VAE encoder ran as one graph, about 9 s per GPU call; the screen compositor stalled and the process was killed (signal 9) on its fifth call. Every stage here is split into several graphs for that reason (with GPU fp16 with fp32 accumulation, every GPU call of the first form took 0.04–1.83 s on the Galaxy S26 (45 graphs, 49 signatures); this form calls the same blocks, or the Gaussian decoder's at a quarter of the anchors, at FP16).
• The denoiser did not compile for the NPU on the phone. Compiled on the Galaxy S26 for its NPU (Qualcomm HTP; LiteRT 2.3.0, QAIRT 2.50), one denoiser block did not finish in 15 minutes at the default optimization level (O3) and ran out of memory after 28 s at optimization_level=1 (optimize for prepare); the HTP compiler warned it has no tiling rule for the block's concatenations along the 12,294-token axis. In the shipped app every network but BiRefNet (CPU) runs on the GPU.

Licence and provenance

Part	Weights	Licence
Denoiser, octree decoder, Gaussian decoder	speridlabs/idol	Apache-2.0
Turbo LoRA (applied to the denoiser graphs)	speridlabs/idol-turbo	Apache-2.0
FLUX.2 VAE encoder	black-forest-labs/FLUX.2-klein-4B vae/	Apache-2.0
BiRefNet background remover	birefnet.safetensors in speridlabs/idol (BiRefNet)	MIT
DINOv3 ViT-H+/16 (separate repo)	facebook/dinov3-vith16plus-pretrain-lvd1689m, read from the bf16 copy Idol loads (Comfy-Org/Idol clip_vision/dino_v3_vit_h_bf16.safetensors)	DINOv3 License

The graphs here are Apache-2.0 except bir.tflite (MIT). LICENSES.md records where each licence was read.