Instructions to use Xalcker/perspectia-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use Xalcker/perspectia-models with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from Xalcker/perspectia-models: direct link, hf CLI and curl.
- Browser
- Download file 3.59 kB
-
https://huggingface.co/Xalcker/perspectia-models/resolve/main/README.md
- Command line
-
hf download hf://Xalcker/perspectia-models/README.md
-
curl -L -o README.md https://huggingface.co/Xalcker/perspectia-models/resolve/main/README.md
3.59 kB
| license: apache-2.0 | |
| library_name: coreml | |
| tags: | |
| - inpainting | |
| - lama | |
| - coreml | |
| - tflite | |
| - litert | |
| - on-device | |
| pipeline_tag: image-to-image | |
| # Big-LaMa for on-device inpainting (Core ML and LiteRT) | |
| Conversions of **Big-LaMa** ([advimman/lama](https://github.com/advimman/lama), Apache-2.0) for running on phones, | |
| used by the PerspectIA app to fill the background hidden behind objects in its 3D view. The weights are the original | |
| Big-LaMa weights (trained on Places2), unchanged; only the format and the way the Fourier transform is computed | |
| differ. | |
| ## Files | |
| | File | Platform | Size | Notes | | |
| |---|---|---|---| | |
| | `coreml/lama_big_512_p8.mlpackage` | iOS 17+ | 50 MB | `mlprogram`, float16, weights palettized to 8 bits. Run it with `computeUnits = .cpuAndGPU` (the FFT is not supported on the Neural Engine). | | |
| | `tflite/lama_big_dft_512_fp16.zip` | Android (LiteRT) | 95 MB (104 MB unzipped) | **Recommended.** Same as below with float16 weights (each followed by a `DEQUANTIZE`); inputs, output and CPU math stay float32. On a float16 NPU the result is identical. | | |
| | `tflite/lama_big_dft_512.zip` | Android (LiteRT) | 190 MB (206 MB unzipped) | float32, FFT replaced by matrix products (DFT) so the whole graph runs on GPU/NPU delegates. NHWC inputs/output. | | |
| Inputs and output (512×512): | |
| - Core ML: `image` (1, 3, 512, 512) RGB in [0, 1]; `mask` (1, 1, 512, 512), 1 = fill. Output `filled`, same shape as | |
| `image`: the image with the masked area filled in. | |
| - TFLite: `image` (1, 512, 512, 3) and `mask` (1, 512, 512, 1), same meaning; output (1, 512, 512, 3). | |
| ## Conversion | |
| - Source: advimman/lama at commit `7dee0e4a3cf5f73f86a820674bf471454f52b74f`; Big-LaMa weights from the | |
| [smartywu/big-lama](https://huggingface.co/smartywu/big-lama) copy, revision | |
| `05cb2be7f8dbe6ca7c6e78f4fc827a4b2baaa4a9` (`big-lama.zip` SHA-256 | |
| `f1b358ca24093b93a106183b98a3dea6e8ed09f3b43ea7251eb2c81e7b4575f6`). | |
| - The checkpoint is loaded with `torch.load(weights_only=True)`; no code from the file is executed. | |
| - Wrapper as in Qualcomm AI Hub's LaMa: the masked area is blanked out, the mask is passed as a 4th channel, and the | |
| prediction is pasted back only inside the mask. | |
| - Core ML: `torch.jit.trace` → coremltools 8.3, float16, then k-means palettization to 8 bits. | |
| - TFLite: each `FourierUnit`'s `rfftn`/`irfftn` (norm `ortho`) is replaced by an equivalent DFT written as matrix | |
| products, all in 4-D tensors; the three `ConvTranspose2d` with `output_padding` are rewritten as an unpadded | |
| transposed convolution plus a crop. Exported with litert-torch 0.10 (ai-edge-torch 0.7.2). The float16 variant | |
| casts the weights of CONV_2D, TRANSPOSE_CONV and FULLY_CONNECTED with ai-edge-quantizer (FLOAT_CASTING, weight-only). | |
| ## Accuracy and speed | |
| Against the original PyTorch model (with FFT), inside the mask: | |
| | | Mean abs. error | Speed | | |
| |---|---|---| | |
| | Core ML, 8 bits | 0.050 (palettization) | ~250 ms on a Mac GPU (Apple Silicon) | | |
| | TFLite, float32 | 3e-6 | 184 ms on a Snapdragon 8 Elite NPU (QNN HTP, float16) | | |
| | TFLite, float16 weights | 2.4e-3 on CPU; on the NPU, identical to float32 | 166 ms on the same NPU | | |
| On the Snapdragon NPU, QNN compiles the graph the first time (2–4 minutes); with QNN's context cache later loads take | |
| under a second. | |
| ## License and attribution | |
| Apache-2.0, like the original repository (see `LICENSE`). LaMa: Suvorov et al., *Resolution-robust Large Mask | |
| Inpainting with Fourier Convolutions*, WACV 2022. The weights were trained by the LaMa authors on the Places2 dataset; | |
| check that dataset's terms for your use case. | |