perspectia-models / README.md
Xalcker's picture
Ficha: versión float16 para Android
042a832 verified
|
Raw History Blame Contribute Delete
3.59 kB
metadata
license: apache-2.0
library_name: coreml
tags:
  - inpainting
  - lama
  - coreml
  - tflite
  - litert
  - on-device
pipeline_tag: image-to-image

Big-LaMa for on-device inpainting (Core ML and LiteRT)

Conversions of Big-LaMa (advimman/lama, Apache-2.0) for running on phones, used by the PerspectIA app to fill the background hidden behind objects in its 3D view. The weights are the original Big-LaMa weights (trained on Places2), unchanged; only the format and the way the Fourier transform is computed differ.

Files

File Platform Size Notes
coreml/lama_big_512_p8.mlpackage iOS 17+ 50 MB mlprogram, float16, weights palettized to 8 bits. Run it with computeUnits = .cpuAndGPU (the FFT is not supported on the Neural Engine).
tflite/lama_big_dft_512_fp16.zip Android (LiteRT) 95 MB (104 MB unzipped) Recommended. Same as below with float16 weights (each followed by a DEQUANTIZE); inputs, output and CPU math stay float32. On a float16 NPU the result is identical.
tflite/lama_big_dft_512.zip Android (LiteRT) 190 MB (206 MB unzipped) float32, FFT replaced by matrix products (DFT) so the whole graph runs on GPU/NPU delegates. NHWC inputs/output.

Inputs and output (512×512):

  • Core ML: image (1, 3, 512, 512) RGB in [0, 1]; mask (1, 1, 512, 512), 1 = fill. Output filled, same shape as image: the image with the masked area filled in.
  • TFLite: image (1, 512, 512, 3) and mask (1, 512, 512, 1), same meaning; output (1, 512, 512, 3).

Conversion

  • Source: advimman/lama at commit 7dee0e4a3cf5f73f86a820674bf471454f52b74f; Big-LaMa weights from the smartywu/big-lama copy, revision 05cb2be7f8dbe6ca7c6e78f4fc827a4b2baaa4a9 (big-lama.zip SHA-256 f1b358ca24093b93a106183b98a3dea6e8ed09f3b43ea7251eb2c81e7b4575f6).
  • The checkpoint is loaded with torch.load(weights_only=True); no code from the file is executed.
  • Wrapper as in Qualcomm AI Hub's LaMa: the masked area is blanked out, the mask is passed as a 4th channel, and the prediction is pasted back only inside the mask.
  • Core ML: torch.jit.trace → coremltools 8.3, float16, then k-means palettization to 8 bits.
  • TFLite: each FourierUnit's rfftn/irfftn (norm ortho) is replaced by an equivalent DFT written as matrix products, all in 4-D tensors; the three ConvTranspose2d with output_padding are rewritten as an unpadded transposed convolution plus a crop. Exported with litert-torch 0.10 (ai-edge-torch 0.7.2). The float16 variant casts the weights of CONV_2D, TRANSPOSE_CONV and FULLY_CONNECTED with ai-edge-quantizer (FLOAT_CASTING, weight-only).

Accuracy and speed

Against the original PyTorch model (with FFT), inside the mask:

Mean abs. error Speed
Core ML, 8 bits 0.050 (palettization) ~250 ms on a Mac GPU (Apple Silicon)
TFLite, float32 3e-6 184 ms on a Snapdragon 8 Elite NPU (QNN HTP, float16)
TFLite, float16 weights 2.4e-3 on CPU; on the NPU, identical to float32 166 ms on the same NPU

On the Snapdragon NPU, QNN compiles the graph the first time (2–4 minutes); with QNN's context cache later loads take under a second.

License and attribution

Apache-2.0, like the original repository (see LICENSE). LaMa: Suvorov et al., Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022. The weights were trained by the LaMa authors on the Places2 dataset; check that dataset's terms for your use case.