--- license: apache-2.0 base_model: speridlabs/idol base_model_relation: quantized library_name: litert pipeline_tag: image-to-3d tags: - litert - tflite - android - on-device - gpu - gaussian-splatting - 3d --- # Idol on LiteRT: one photo to a 3D Gaussian splat on the phone Idol (speridlabs/idol with the idol-turbo LoRA) as LiteRT graphs for `CompiledModel`, converted with litert-torch: on a Galaxy S26, the Android sample in `android/` turns one photo into a 3D Gaussian splat of 65,536 Gaussians in 158 s with the default form, or of 262,144 Gaussians in 267 s with the first release's form in `262k/` (the first photo after start on a cool phone, LiteRT 2.3.0; measured on 2026-11-09 and 2026-11-10 JST). [speridlabs/idol](https://huggingface.co/speridlabs/idol) turns one photo of an object into a 3D Gaussian splat (262,144 Gaussians at its defaults) that you can turn around, including the sides the picture never showed. This repo holds the author's denoiser with the [speridlabs/idol-turbo](https://huggingface.co/speridlabs/idol-turbo) LoRA (5 steps, no guidance), its octree and Gaussian decoders, the FLUX.2 VAE encoder, and the author's BiRefNet background remover. The DINOv3 image encoder is in a separate, gated repo under its own licence: [litert-community/DINOv3-ViT-H16plus-Idol-LiteRT](https://huggingface.co/litert-community/DINOv3-ViT-H16plus-Idol-LiteRT). ![The photo, and the splat made from it turned to the front, the side and the back](assets/hero.png) *Left: [the photo](https://www.pexels.com/photo/plant-in-silver-flowerpot-20324601/) by Komet Flicker (Pexels License). Right: the splat the sample app made from it on a Galaxy S26 with the default form (65,536 Gaussians), rendered from the front, the side and the back.* ## Two forms The default form (the repo's root) is the fastest of the forms measured on a Galaxy S26: it makes a splat in 158 s against 267 s for the first release's form in `262k/`, while its renders agree less closely with the author's default output: 32.3–34.4 dB PSNR against 41.4–46.6 dB (higher is closer). Of the other forms tried, int8 weights did not run on the phone GPUs, a 512 × 512 input made even the author's own PyTorch pipeline miss the 25 dB bar (13.5–22.9 dB against its 1,024 × 1,024 output), the denoiser did not compile for the Galaxy S26's NPU on the phone, and the CPU alone was slower. | | default (the repo's root) | first release (`262k/`) | |---|---|---| | Gaussians per splat | 65,536 (2,048 anchors) | 262,144 (8,192 anchors, the author's default) | | GPU precision | FP16 | FP16 with fp32 accumulation | | Denoiser blocks 1–22 and Gaussian decoder graphs 1–6 | one shared graph per chain; the app copies each block's weights in before the call | a graph per block | | Galaxy S26: the first photo after start, cool phone (thermal status 0 at the start) | **158 s** | 267 s | | Galaxy S26: photos on a phone already warm | 329–351 s (4 photos of the armchair, thermal status 3) | 339–449 s (5 photos of 3 inputs, thermal status 2) | | Pixel 11 Pro: the first photo after start | 343 s | not measured | | 4-view render PSNR against the author's default on the Galaxy S26 (threshold 25 dB) | 32.3–34.4 dB | 41.4–46.6 dB | Thermal status is Android's: 0 none, 1 light, 2 moderate, 3 severe. Both forms run with the same app and the same Python runner: point them at the repo's root or at `262k/`. The first release's card gives 303 s for the `262k/` form: that photo started on a warmer phone (thermal status 1, skin 38.5 °C, against 34.6 °C here). With this app the `262k/` form makes the same splat as with the first release's, byte for byte. The rest of this card describes the default form; the first release's card ([commit e7ee1226](https://huggingface.co/litert-community/Idol/blob/e7ee12262a3ed622fa44fb8df8ff4f335dd77064/README.md)) has the per-stage tables of the `262k/` form. ## Speed On a Galaxy S26, the sample app turns one photo into a splat of 65,536 Gaussians in 158 s, starting from a cool phone (the first photo after start). The phone throttles its GPU during that photo: the GPU clock cap fell from 1,300 to 500 MHz (thermal status 0 → 2). After 12 minutes of back-to-back runs the phone was at thermal status 3; through the next 4 photos the GPU clock cap stayed at 222–282 MHz (one reading of 191 MHz), and they took 329–351 s. At start the app compiles 1 graph before the photo (2 s) and 19 more during it (20 s): the denoiser's 22 middle blocks share one graph and the Gaussian decoder's six middle graphs share another, so each is compiled once per photo. The first release's form in `262k/` (262,144 Gaussians, a graph per block, GPU FP16 with fp32 accumulation) took 267 s the same way, also from thermal status 0 (skin 34.6 °C at the start against 33.5 °C). Background removal (BiRefNet) runs on the CPU: 7.4 s of that time on the Galaxy S26. The other networks run on the GPU at FP16. | Galaxy S26 (SM-S942Q), the sample app, LiteRT 2.3.0; from a cool phone (GPU clock cap 1,300 MHz, thermal status 0 at the start) | first photo after start | |---|---| | **photo → .ply written** | **158 s** | | background removal (BiRefNet, CPU) | 7.4 s | | DINOv3 encoder | 10.6 s | | VAE encoder | 5.2 s | | denoiser, 5 steps | 119.8 s | | octree decoder | 9.1 s | | Gaussian decoder | 5.5 s | | PLY writer | 0.1 s | | graphs compiled at app start, before the first photo | 1 (2 s) | | graphs compiled during the photo | 19 (20 s) | | peak resident memory (VmHWM) | 5.42 GiB | | peak GPU memory (kgsl) | 5.73 GiB | | Galaxy S26 (SM-S942Q), the sample app, LiteRT 2.3.0; right after the cool photo's hold (thermal status 3 at the start) | first photo after start | photo 2 | photo 3 | photo 4 | |---|---|---|---|---| | **photo → .ply written** | **351 s** | **351 s** | **341 s** | **329 s** | | background removal (BiRefNet, CPU) | 14.9 s | 18.9 s | 15.2 s | 15.4 s | | DINOv3 encoder | 22.5 s | 23.6 s | 25.1 s | 20.6 s | | VAE encoder | 10.7 s | 8.1 s | 8.0 s | 7.5 s | | denoiser, 5 steps | 278.1 s | 280.3 s | 272.6 s | 265.4 s | | octree decoder | 14.3 s | 11.7 s | 12.1 s | 12.2 s | | Gaussian decoder | 9.2 s | 7.7 s | 7.6 s | 7.4 s | | PLY writer | 0.1 s | 0.0 s | 0.0 s | 0.0 s | | graphs compiled at app start, before the first photo | 3 (9 s) | — | — | — | | graphs compiled during the photo | 18 (29 s) | 18 (21 s) | 19 (25 s) | 18 (20 s) | | peak resident memory (VmHWM) | 5.30 GiB | 4.58 GiB | 5.10 GiB | 4.63 GiB | | peak GPU memory (kgsl) | 5.73 GiB | 5.12 GiB | 5.13 GiB | 5.04 GiB | | Pixel 11 Pro, the sample app, LiteRT 2.3.0; thermal status 0 at the start | first photo after start | |---|---| | **photo → .ply written** | **343 s** | | background removal (BiRefNet, CPU) | 17.4 s | | DINOv3 encoder | 8.5 s | | VAE encoder | 36.1 s | | denoiser, 5 steps | 242.3 s | | octree decoder | 12.9 s | | Gaussian decoder | 23.7 s | | PLY writer | 0.2 s | | graphs compiled at app start, before the first photo | 6 (60 s) | | graphs compiled during the photo | 15 (75 s) | | peak resident memory (VmHWM) | 5.96 GiB | ## How close it is to the author's model Each check runs the author's PyTorch pipeline (float32, CPU) and these graphs on the same input with the same random numbers, against thresholds fixed before the first run. The inputs are two images from the author's gallery (`nb-armchair`, `fig-axolotl`) and one photo (a succulent, Pexels). Per part, each graph gets the author's own input for that part and its output is compared with the author's (cosine; the threshold is 0.99); the octree and Gaussian decoders are compared with the author's model run at this form's 65,536 Gaussians (`--num-gaussians 65536`). The whole pipeline is compared with the author's default (262,144 Gaussians) by rendering both splats from four sides with the same renderer (PSNR; the threshold is 25 dB). ### Per-part agreement on the Mac (LiteRT GPU, Metal, fp16): cosine vs the author's model | Part | Compared tensor | armchair | axolotl | succulent | |---|---|---|---|---| | DINOv3 encoder | feature1 | 0.99989 | 0.99994 | 0.99992 | | VAE encoder | latent tokens | 0.99995 | 0.99993 | 0.99996 | | Denoiser (5 steps) | latent velocity, worst step | 0.99931 | 0.99924 | 0.99931 | | Octree decoder (8 levels) | child probabilities, worst level | 0.99948 | 0.99937 | 0.99944 | | Gaussian decoder | xyz | 1.00000 | 1.00000 | 1.00000 | | | colour (f_dc) | 0.99980 | 0.99987 | 0.99937 | | | scale | 0.99959 | 0.99925 | 0.99225 | | | rotation | 0.99953 | 0.99919 | 0.99929 | | | opacity | 0.99995 | 0.99969 | 0.99969 | ### Per-part agreement on a Galaxy S26 (the app; GPU FP16, BiRefNet on the CPU): cosine vs the author's model | Part | Compared tensor | armchair | |---|---|---| | DINOv3 encoder | feature1 | 0.99987 | | VAE encoder | latent tokens | 0.99991 | | Denoiser (5 steps) | latent velocity, worst step | 0.99924 | | Octree decoder (8 levels) | child probabilities, worst level | 0.99943 | | Gaussian decoder | xyz | 1.00000 | | | colour (f_dc) | 0.99973 | | | scale | 0.99939 | | | rotation | 0.99938 | | | opacity | 0.99996 | ### Per-part agreement on a Pixel 11 Pro (the app; GPU FP16, BiRefNet on the CPU): cosine vs the author's model | Part | Compared tensor | armchair | |---|---|---| | DINOv3 encoder | feature1 | 0.99989 | | VAE encoder | latent tokens | 0.99997 | | Denoiser (5 steps) | latent velocity, worst step | 0.99930 | | Octree decoder (8 levels) | child probabilities, worst level | 0.99945 | | Gaussian decoder | xyz | 1.00000 | | | colour (f_dc) | 0.99988 | | | scale | 0.99963 | | | rotation | 0.99939 | | | opacity | 0.99998 | ### Background removal | Input | Where | uint8 alpha pixels that differ from the author's | largest difference | crop square (author / here) | |---|---|---|---|---| | armchair | Mac, LiteRT CPU | 2,544 of 1,048,576 | 1 / 255 | same | | armchair | Galaxy S26, the app | 2,542 of 1,048,576 | 1 / 255 | same | | armchair | Pixel 11 Pro, the app | 2,543 of 1,048,576 | 1 / 255 | same | | axolotl | Mac, LiteRT CPU | 709 of 1,048,576 | 1 / 255 | same | | succulent | Mac, LiteRT CPU | 6,115 of 1,048,576 | 1 / 255 | same | ### Whole pipeline: 4-view render PSNR | Input | Where | PSNR, 4 views pooled (threshold: 25 dB) | per view: 0°, 90°, 180°, 270° | |---|---|---|---| | armchair | Mac, LiteRT GPU (Metal, fp16) | 35.02 dB | 34.23, 36.99, 33.39, 36.48 | | armchair | Galaxy S26, the app | 34.41 dB | 34.22, 35.85, 33.69, 34.15 | | armchair | Pixel 11 Pro, the app | 33.58 dB | 32.42, 34.81, 31.90, 36.89 | | axolotl | Mac, LiteRT GPU (Metal, fp16) | 31.24 dB | 32.84, 28.79, 33.37, 31.52 | | axolotl | Galaxy S26, the app | 32.26 dB | 33.27, 30.07, 33.71, 33.04 | | succulent | Mac, LiteRT GPU (Metal, fp16) | 34.85 dB | 34.05, 34.89, 35.03, 35.58 | | succulent | Galaxy S26, the app | 34.26 dB | 33.47, 34.26, 34.41, 35.05 | ## Minimal usage ### Python: the whole pipeline ```bash hf download litert-community/Idol --exclude "262k/*" --local-dir Idol # the default form; drop --exclude for both cd Idol pip install -r python/requirements.txt python python/run_idol.py photo.jpg --out out --models . # writes out/gaussian.ply; --models 262k for the other form ``` `run_idol.py` downloads the DINOv3 repo (accept its licence on the Hub and `hf auth login` first) and runs every graph with `CompiledModel`, the GPU for all but BiRefNet. The steps between graphs are in `python/idol_runner.py` and `python/idol_glue.py`. On an M4 Max Mac (Metal GPU) one photo took 53 s and up to 6 GB of memory (the process footprint). ### Python: one graph with `CompiledModel` (background removal) ```python import sys import numpy as np from PIL import Image from ai_edge_litert.compiled_model import CompiledModel from ai_edge_litert.hardware_accelerator import HardwareAccelerator from ai_edge_litert.options import Options path, photo = sys.argv[1], sys.argv[2] # bir.tflite from this repo, any photo model = CompiledModel.from_file(path, options=Options(hardware_accelerators=HardwareAccelerator.CPU)) img = Image.open(photo).convert("RGB") x = np.asarray(img.resize((1024, 1024), Image.BILINEAR), np.float32) / 255 x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32) inputs, outputs = model.create_input_buffers(0), model.create_output_buffers(0) inputs[0].write(x.reshape(-1)) # [1, 3, 1024, 1024] model.run_by_index(0, inputs, outputs) alpha = outputs[0].read(1024 * 1024, np.float32).reshape(1024, 1024) # 0..1, 1 = the object mask = Image.fromarray((alpha * 255).astype(np.uint8)).resize(img.size, Image.BILINEAR) img.putalpha(mask) img.save("cutout.png") print(alpha.min(), alpha.max(), (alpha > 0.5).mean()) ``` ### Kotlin (Android) ```kotlin val env = Environment.create() val cpu = CompiledModel.Options(Accelerator.CPU) // bir.tflite: BiRefNet, the one CPU graph val bir = CompiledModel.create(File(dir, "bir.tflite").absolutePath, cpu, env) val inputs = bir.createInputBuffers() val outputs = bir.createOutputBuffers() inputs[0].writeFloat(image) // [1, 3, 1024, 1024]: RGB / 255, ImageNet mean / std bir.run(inputs, outputs) val alpha = outputs[0].readFloat() // [1024 * 1024] sigmoid map, 1 = the object for (b in inputs + outputs) b.close() bir.close() val gpu = CompiledModel.Options(Accelerator.GPU).apply { // every other graph gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16) } ``` The Android sample app (photo or camera → progress per stage → a splat you can turn with a finger) is in `android/`. The graphs are too large for the APK; a script copies them into the app's files folder over adb: ```bash hf download litert-community/Idol --exclude "262k/*" --local-dir ~/Downloads/Idol # the default form; drop --exclude to get 262k/ too hf download litert-community/DINOv3-ViT-H16plus-Idol-LiteRT --local-dir ~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT cd ~/Downloads/Idol/android chmod +x gradlew scripts/install_to_device.sh # the Hub does not keep the executable bit ./gradlew :app:installDebug IDOL_DIR=~/Downloads/Idol DINO_DIR=~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT ./scripts/install_to_device.sh # the first release's form instead (download without --exclude first): IDOL_DIR=~/Downloads/Idol/262k, same DINO_DIR ``` Background removal (BiRefNet on the CPU) needs about 2,670 MB of free memory for one call. Before compiling it, the app closes GPU graphs until that much is free on top of its 2,500 MB reserve; if other apps hold the rest, it stops with "Not enough memory for background removal (bir) … Close other apps and try again." before starting the call. ## What runs where | Stage | Files | Runs on | Calls per photo | |---|---|---|---| | Background removal (BiRefNet) | `bir.tflite` | CPU | 1 | | DINOv3 ViT-H+/16 image encoder | `dino_0..3.tflite` (DINOv3 repo) | GPU | 4 | | FLUX.2 VAE encoder | `vae_0..3.tflite`, `vae_attn.tflite`, `vae_4.tflite` | GPU | 6 | | Denoiser with the turbo LoRA | `den_ctx.tflite`, `den_in.tflite`, `den_joint_00.tflite`, `den_joint_mid.tflite` (blocks 1–22, weights from `den_joint_w.f32`), `den_joint_23.tflite` | GPU | 5 steps × 25, plus `den_ctx` once per LoRA phase | | Octree decoder | `oct.tflite` (5 signatures by row count) | GPU | 8 levels | | Gaussian decoder | `gs_0.tflite`, `gs_mid.tflite` (graphs 1–6, weights from `gs_w.f32`), `gs_7.tflite` | GPU | 8 | | Octree sampling, Gaussian activation, PLY | host code | CPU | — | `262k/` holds the first release's form: `den_joint_01..22.tflite` and `gs_1..6.tflite` in place of the two shared graphs, the Gaussian decoder at 8,192 anchors, and its own `manifest.json` (GPU precision FP16 with fp32 accumulation); the files with the same name in both folders are the same bytes, except `gs_0.tflite`, `gs_7.tflite` and `manifest.json`. `manifest.json` lists every graph with its signatures, accelerator and precision; the GPU graphs run with `FP16` precision. `den_joint_mid` and `gs_mid` are one graph each for blocks of the same shape: before each call the app copies that block's weights from the side file into the graph's weight inputs (`shared_blocks`, `block_weights` and `first_weight_input` in `manifest.json`), so each is compiled once per photo. `first_weight_input` is the number of inputs before the weights, counted in numeric order of the input names: 3 for `den_joint_mid`, whose weights are `args_3` to `args_32`, in that order in each block of `den_joint_w.f32`. `manifest.json` lists the inputs in the order the `.tflite` signature stores them (`args_0`, `args_1`, `args_10`, …, `args_19`, `args_2`, …), while LiteRT 2.3.0's `CompiledModel` lists them in numeric order, so match the inputs by name, as `python/idol_runner.py` and the app do. ## Notes - **Background removal runs on the CPU.** BiRefNet's 20 deformable convolutions sample at positions computed from the image. They are rewritten as a gather with bilinear weights and one matmul (same arithmetic; 5.5e-6 from the author's map in PyTorch), and the LiteRT GPU has no kernel for that gather. The gathers run in row tiles one after another: each tile's offsets wait for the previous tile's result. Without that wait the CPU runtime ran every tile's gather before using any of them; the first export ran out of memory on the Galaxy S26, and this one peaks at 3.96 GiB of resident memory (VmHWM) in one call on it. - **The VAE's GroupNorm scales each group by its largest |x − mean| to stay in fp16 range.** The graph finds that maximum as a max pool, then a max over the 32 × 32 result (the same value). One REDUCE_MAX over the whole 1024 × 1024 map took 8 s per GPU call on the Galaxy S26; this form takes 0.83 s. - **Attention over 4,096 keys or more is summed in chunks.** The product of the attention weights and the values is summed over key chunks of 1,024 after centring the values (exact, since each softmax row sums to 1). One fp16 sum over 8,192 keys lost 18 % in the Gaussian decoder's first block; the chunked form loses 0.6 %. Keep it if you re-export. - **Keep each GPU call short on the Galaxy S26.** In the feasibility run the VAE encoder ran as one graph, about 9 s per GPU call; the screen compositor stalled and the process was killed (signal 9) on its fifth call. Every stage here is split into several graphs for that reason (with GPU fp16 with fp32 accumulation, every GPU call of the first form took 0.04–1.83 s on the Galaxy S26 (45 graphs, 49 signatures); this form calls the same blocks, or the Gaussian decoder's at a quarter of the anchors, at FP16). - **The denoiser did not compile for the NPU on the phone.** Compiled on the Galaxy S26 for its NPU (Qualcomm HTP; LiteRT 2.3.0, QAIRT 2.50), one denoiser block did not finish in 15 minutes at the default optimization level (O3) and ran out of memory after 28 s at `optimization_level=1` (optimize for prepare); the HTP compiler warned it has no tiling rule for the block's concatenations along the 12,294-token axis. In the shipped app every network but BiRefNet (CPU) runs on the GPU. ## Licence and provenance | Part | Weights | Licence | |---|---|---| | Denoiser, octree decoder, Gaussian decoder | [speridlabs/idol](https://huggingface.co/speridlabs/idol) | Apache-2.0 | | Turbo LoRA (applied to the denoiser graphs) | [speridlabs/idol-turbo](https://huggingface.co/speridlabs/idol-turbo) | Apache-2.0 | | FLUX.2 VAE encoder | [black-forest-labs/FLUX.2-klein-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) `vae/` | Apache-2.0 | | BiRefNet background remover | `birefnet.safetensors` in speridlabs/idol ([BiRefNet](https://github.com/ZhengPeng7/BiRefNet)) | MIT | | DINOv3 ViT-H+/16 (separate repo) | [facebook/dinov3-vith16plus-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vith16plus-pretrain-lvd1689m), read from the bf16 copy Idol loads (Comfy-Org/Idol `clip_vision/dino_v3_vit_h_bf16.safetensors`) | DINOv3 License | The graphs here are Apache-2.0 except `bir.tflite` (MIT). `LICENSES.md` records where each licence was read.