Instructions to use litert-community/Idol with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/Idol with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Idol on LiteRT: graphs, manifest, Python runner and Android sample
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +1 -0
- LICENSES.md +37 -0
- android/app/build.gradle.kts +67 -0
- android/app/src/main/AndroidManifest.xml +45 -0
- android/app/src/main/assets/viewer/LICENSE +21 -0
- android/app/src/main/assets/viewer/index.html +281 -0
- android/app/src/main/assets/viewer/main.js +1527 -0
- android/app/src/main/java/com/idol/AutoRun.kt +312 -0
- android/app/src/main/java/com/idol/CardSnippet.kt +75 -0
- android/app/src/main/java/com/idol/DeviceInfo.kt +102 -0
- android/app/src/main/java/com/idol/Fixture.kt +66 -0
- android/app/src/main/java/com/idol/Glue.kt +472 -0
- android/app/src/main/java/com/idol/Graphs.kt +339 -0
- android/app/src/main/java/com/idol/IdolEngine.kt +606 -0
- android/app/src/main/java/com/idol/MainActivity.kt +581 -0
- android/app/src/main/java/com/idol/Manifest.kt +103 -0
- android/app/src/main/java/com/idol/ModelFiles.kt +96 -0
- android/app/src/main/java/com/idol/Photo.kt +89 -0
- android/app/src/main/java/com/idol/Pil.kt +361 -0
- android/app/src/main/java/com/idol/Preprocess.kt +201 -0
- android/app/src/main/java/com/idol/ResultViewer.kt +66 -0
- android/app/src/main/java/com/idol/TorchOps.kt +91 -0
- android/app/src/main/res/layout/activity_main.xml +216 -0
- android/app/src/main/res/values/strings.xml +21 -0
- android/app/src/main/res/values/themes.xml +6 -0
- android/app/src/main/res/xml/file_paths.xml +7 -0
- android/build.gradle.kts +4 -0
- android/gradle.properties +7 -0
- android/gradle/libs.versions.toml +18 -0
- android/gradle/wrapper/gradle-wrapper.jar +0 -0
- android/gradle/wrapper/gradle-wrapper.properties +7 -0
- android/gradlew +251 -0
- android/gradlew.bat +94 -0
- android/licenses/Pillow-MIT-CMU.txt +30 -0
- android/scripts/install_to_device.sh +165 -0
- android/settings.gradle.kts +18 -0
- bir.tflite +3 -0
- den_ctx.tflite +3 -0
- den_in.tflite +3 -0
- den_joint_00.tflite +3 -0
- den_joint_01.tflite +3 -0
- den_joint_02.tflite +3 -0
- den_joint_03.tflite +3 -0
- den_joint_04.tflite +3 -0
- den_joint_05.tflite +3 -0
- den_joint_06.tflite +3 -0
- den_joint_07.tflite +3 -0
- den_joint_08.tflite +3 -0
- den_joint_09.tflite +3 -0
- den_joint_10.tflite +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
den_pos_emb.f32 filter=lfs diff=lfs merge=lfs -text
|
LICENSES.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Licences of the parts of the Idol pipeline (read 2026-11-09)
|
| 2 |
+
|
| 3 |
+
| Part | Weights used here | Licence | Source read |
|
| 4 |
+
|---|---|---|---|
|
| 5 |
+
| Denoiser, octree decoder, Gaussian decoder | speridlabs/idol @2731c229 (`denoiser`, `decoder`, `decoder_gs` .safetensors) | Apache-2.0 | HF card front matter line 2 `license: apache-2.0`; idol-sdk LICENSE @4702a7f0 (Apache License 2.0, lines 1-2) |
|
| 6 |
+
| Turbo LoRA (slow / fast experts) | speridlabs/idol-turbo @45cdd41f (`idol_turbo_lora.safetensors`) | Apache-2.0 | HF card front matter `license: apache-2.0`; card §License "Apache-2.0, inherited from speridlabs/idol" |
|
| 7 |
+
| Background removal (shipped: `bir.tflite`, LiteRT CPU) | speridlabs/idol @2731c229 `birefnet.safetensors` (BiRefNet, Swin-L) | MIT (BiRefNet); the idol repo that packages it is Apache-2.0 | github ZhengPeng7/BiRefNet licence = MIT (GitHub API); idol HF card front matter line 2 |
|
| 8 |
+
| Background removal, alternative in the app (off by default) | litert-community/DIS-ISNet-LiteRT @1b966dbe `dis.tflite` (IS-Net DIS general-use, xuebinqin/DIS) | Apache-2.0 | litert-community/DIS-ISNet-LiteRT card `license: apache-2.0` (Hub API) |
|
| 9 |
+
| FLUX.2 VAE (encoder side used) | black-forest-labs/FLUX.2-klein-4B @e7b7dc27 `vae/` (ungated) | Apache-2.0 | black-forest-labs/flux2 README @50fe5162, line 111: "The autoencoder is released under Apache 2.0"; klein-4B card `license: apache-2.0` |
|
| 10 |
+
| DINOv3 ViT-H+/16 | Comfy-Org/Idol @e907d8dc `clip_vision/dino_v3_vit_h_bf16.safetensors` (bf16 repack of facebook/dinov3-vith16plus-pretrain-lvd1689m, gated) | DINOv3 License (Meta, "Last Updated: August 19, 2025") | facebookresearch/dinov3 LICENSE.md @6876159a |
|
| 11 |
+
| Idol acknowledges TripoSplat | — (no TripoSplat weights used) | MIT | github VAST-AI-Research/TripoSplat licence = MIT (GitHub API) |
|
| 12 |
+
|
| 13 |
+
## DINOv3 License — the clauses that decide redistribution (dinov3_LICENSE.md)
|
| 14 |
+
|
| 15 |
+
- Line 20: "By clicking "I Accept" below or by using or distributing any portion or element of the DINO Materials, you agree to be bound by this Agreement." → using the Comfy-Org repack binds the user of it to the Agreement as if accepted on the gated page.
|
| 16 |
+
- Line 24 (§1.a Grant of Rights): "non-exclusive, worldwide, non-transferable and royalty-free limited license … to use, reproduce, distribute, copy, create derivative works of, and make modifications to the DINO Materials." → a converted .tflite of the encoder is a derivative work and may be distributed.
|
| 17 |
+
- Line 28 (§1.b.i): "If you distribute or make the DINO Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement and you shall provide a copy of this Agreement with any such DINO Materials." → a hosted encoder file must carry the DINOv3 License text and be offered under it (not under Apache-2.0).
|
| 18 |
+
- Line 30 (§1.b.ii): acknowledgement in research publications.
|
| 19 |
+
- Line 34 (§1.b.iv): no use that involves or encourages reverse engineering / decompiling the DINO Materials.
|
| 20 |
+
- Line 36 (§1.b.v): trade-control limits (no ITAR / military / nuclear / espionage / weapons end uses; licensee not a target of trade controls).
|
| 21 |
+
- Line 52 (§5.a): derivative works made by the licensee are owned by the licensee (subject to Meta's ownership of the DINO Materials).
|
| 22 |
+
- No non-commercial clause anywhere in the text (66 lines read in full).
|
| 23 |
+
|
| 24 |
+
## FLUX.2 VAE
|
| 25 |
+
|
| 26 |
+
- flux2 README line 111 (commit 50fe5162): the FLUX.2 autoencoder is released under Apache 2.0. The FLUX.2 [dev] Non-Commercial License of the FLUX.2-dev repo covers the [dev] transformer, not the autoencoder.
|
| 27 |
+
- The klein-4B `vae/` dir is ungated and tagged apache-2.0; the Idol pipeline loads the VAE in bf16 (`conditioning.py` `torch_dtype=torch.bfloat16`) and klein-4B's file is bf16.
|
| 28 |
+
- Verified 2026-11-09 14:04: klein-4B `vae/diffusion_pytorch_model.safetensors` (sha256 ca70d220…, bf16) = the FLUX.2-dev VAE cast to bf16. The FLUX.2-dev VAE was read from Comfy-Org/Idol `vae/flux2-vae.safetensors`, whose LFS sha256 868fe7b3… equals FLUX.2-dev's `ae.safetensors` (Hub API, FLUX.2-dev @26afe3a7): the same file. All 250 float tensors match bit for bit after the bf16 cast (84,046,372 values; 8 attention projections are [512, 512] in klein and [512, 512, 1, 1] in BFL, same values). The one other tensor differs: `bn.num_batches_tracked` (int64 training counter, 0 vs 400000), unused at inference.
|
| 29 |
+
|
| 30 |
+
## Hosting
|
| 31 |
+
|
| 32 |
+
- litert-community/Idol (ungated, Apache-2.0, with the per-part provenance above): the denoiser, octree and Gaussian
|
| 33 |
+
decoder graphs, the VAE encoder graphs, BiRefNet (`bir.tflite`, MIT part named in the card), the side files, the
|
| 34 |
+
manifest, and the Python / Android sources.
|
| 35 |
+
- The DINOv3 encoder graphs and their RoPE tables: a separate litert-community repo, gated (automatic approval), carrying
|
| 36 |
+
the full DINOv3 License text and the provenance (base_model facebook/dinov3-vith16plus-pretrain-lvd1689m), the way
|
| 37 |
+
litert-community/Llama-3.2-1B carries the Llama licence.
|
android/app/build.gradle.kts
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plugins {
|
| 2 |
+
alias(libs.plugins.android.application)
|
| 3 |
+
alias(libs.plugins.kotlin.android)
|
| 4 |
+
}
|
| 5 |
+
|
| 6 |
+
android {
|
| 7 |
+
namespace = "com.idol"
|
| 8 |
+
compileSdk = 35
|
| 9 |
+
buildToolsVersion = "35.0.0"
|
| 10 |
+
|
| 11 |
+
defaultConfig {
|
| 12 |
+
applicationId = "com.idol"
|
| 13 |
+
minSdk = 26
|
| 14 |
+
targetSdk = 35
|
| 15 |
+
versionCode = 1
|
| 16 |
+
versionName = "0.1"
|
| 17 |
+
buildConfigField("String", "LITERT_VERSION", "\"${libs.versions.litert.get()}\"")
|
| 18 |
+
ndk {
|
| 19 |
+
abiFilters += "arm64-v8a"
|
| 20 |
+
}
|
| 21 |
+
}
|
| 22 |
+
|
| 23 |
+
buildTypes {
|
| 24 |
+
release {
|
| 25 |
+
isMinifyEnabled = false
|
| 26 |
+
// sample convenience: installable without a release keystore
|
| 27 |
+
signingConfig = signingConfigs.getByName("debug")
|
| 28 |
+
}
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
androidResources {
|
| 32 |
+
// graphs are never assets here (they are installed in files/models/), but keep any .tflite mmappable
|
| 33 |
+
noCompress += "tflite"
|
| 34 |
+
}
|
| 35 |
+
|
| 36 |
+
packaging {
|
| 37 |
+
// The GPU accelerator library (litert-gpu) is loaded from the extracted native library directory, as in the
|
| 38 |
+
// zoo's other GPU samples on the same phones.
|
| 39 |
+
jniLibs { useLegacyPackaging = true }
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
compileOptions {
|
| 43 |
+
sourceCompatibility = JavaVersion.VERSION_17
|
| 44 |
+
targetCompatibility = JavaVersion.VERSION_17
|
| 45 |
+
}
|
| 46 |
+
buildFeatures {
|
| 47 |
+
buildConfig = true
|
| 48 |
+
}
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
kotlin {
|
| 52 |
+
compilerOptions {
|
| 53 |
+
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
|
| 54 |
+
// LiteRT 2.3.0's Kotlin API carries Kotlin 2.4 metadata; Google's own 2.3.0 samples on Kotlin 2.2.21 pass this
|
| 55 |
+
freeCompilerArgs.add("-Xskip-metadata-version-check")
|
| 56 |
+
}
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
dependencies {
|
| 60 |
+
// LiteRT CompiledModel (CPU / XNNPACK for BiRefNet) and its GPU accelerator (ML Drift, OpenCL) for the rest
|
| 61 |
+
implementation(libs.litert)
|
| 62 |
+
implementation(libs.litert.gpu)
|
| 63 |
+
implementation(libs.androidx.core.ktx)
|
| 64 |
+
implementation(libs.androidx.activity.ktx)
|
| 65 |
+
// WebViewAssetLoader: the splat viewer and the .ply served to the WebView from the app itself
|
| 66 |
+
implementation(libs.androidx.webkit)
|
| 67 |
+
}
|
android/app/src/main/AndroidManifest.xml
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
| 3 |
+
|
| 4 |
+
<!-- "Take a photo" asks for it first; without it the gallery still works. -->
|
| 5 |
+
<uses-permission android:name="android.permission.CAMERA" />
|
| 6 |
+
|
| 7 |
+
<uses-feature
|
| 8 |
+
android:name="android.hardware.camera"
|
| 9 |
+
android:required="false" />
|
| 10 |
+
|
| 11 |
+
<!-- The host side of a run (the 50 MB tensors between graphs, a photo's working copies) is on the Java heap. -->
|
| 12 |
+
<application
|
| 13 |
+
android:allowBackup="false"
|
| 14 |
+
android:label="@string/app_name"
|
| 15 |
+
android:largeHeap="true"
|
| 16 |
+
android:supportsRtl="true"
|
| 17 |
+
android:theme="@style/Theme.Idol">
|
| 18 |
+
|
| 19 |
+
<!-- The GPU accelerator opens the vendor OpenCL library. -->
|
| 20 |
+
<uses-native-library
|
| 21 |
+
android:name="libOpenCL.so"
|
| 22 |
+
android:required="false" />
|
| 23 |
+
|
| 24 |
+
<activity
|
| 25 |
+
android:name=".MainActivity"
|
| 26 |
+
android:configChanges="orientation|screenSize|screenLayout|keyboardHidden"
|
| 27 |
+
android:exported="true"
|
| 28 |
+
android:launchMode="singleTop">
|
| 29 |
+
<intent-filter>
|
| 30 |
+
<action android:name="android.intent.action.MAIN" />
|
| 31 |
+
<category android:name="android.intent.category.LAUNCHER" />
|
| 32 |
+
</intent-filter>
|
| 33 |
+
</activity>
|
| 34 |
+
|
| 35 |
+
<provider
|
| 36 |
+
android:name="androidx.core.content.FileProvider"
|
| 37 |
+
android:authorities="${applicationId}.files"
|
| 38 |
+
android:exported="false"
|
| 39 |
+
android:grantUriPermissions="true">
|
| 40 |
+
<meta-data
|
| 41 |
+
android:name="android.support.FILE_PROVIDER_PATHS"
|
| 42 |
+
android:resource="@xml/file_paths" />
|
| 43 |
+
</provider>
|
| 44 |
+
</application>
|
| 45 |
+
</manifest>
|
android/app/src/main/assets/viewer/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2023 Kevin Kwok
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
android/app/src/main/assets/viewer/index.html
ADDED
|
@@ -0,0 +1,281 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!DOCTYPE html>
|
| 2 |
+
<html lang="en" dir="ltr">
|
| 3 |
+
<head>
|
| 4 |
+
<title>WebGL Gaussian Splat Viewer</title>
|
| 5 |
+
<meta charset="utf-8" />
|
| 6 |
+
<meta
|
| 7 |
+
name="viewport"
|
| 8 |
+
content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=no"
|
| 9 |
+
/>
|
| 10 |
+
<meta name="apple-mobile-web-app-capable" content="yes" />
|
| 11 |
+
<meta
|
| 12 |
+
name="apple-mobile-web-app-status-bar-style"
|
| 13 |
+
content="black-translucent"
|
| 14 |
+
/>
|
| 15 |
+
<style>
|
| 16 |
+
body {
|
| 17 |
+
overflow: hidden;
|
| 18 |
+
margin: 0;
|
| 19 |
+
height: 100vh;
|
| 20 |
+
width: 100vw;
|
| 21 |
+
font-family: sans-serif;
|
| 22 |
+
background: black;
|
| 23 |
+
text-shadow: 0 0 3px black;
|
| 24 |
+
}
|
| 25 |
+
a, body {
|
| 26 |
+
color: white;
|
| 27 |
+
}
|
| 28 |
+
#info {
|
| 29 |
+
z-index: 100;
|
| 30 |
+
position: absolute;
|
| 31 |
+
top: 10px;
|
| 32 |
+
left: 15px;
|
| 33 |
+
}
|
| 34 |
+
h3 {
|
| 35 |
+
margin: 5px 0;
|
| 36 |
+
}
|
| 37 |
+
p {
|
| 38 |
+
margin: 5px 0;
|
| 39 |
+
font-size: small;
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
.cube-wrapper {
|
| 43 |
+
transform-style: preserve-3d;
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
.cube {
|
| 47 |
+
transform-style: preserve-3d;
|
| 48 |
+
transform: rotateX(45deg) rotateZ(45deg);
|
| 49 |
+
animation: rotation 2s infinite;
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
.cube-faces {
|
| 53 |
+
transform-style: preserve-3d;
|
| 54 |
+
height: 80px;
|
| 55 |
+
width: 80px;
|
| 56 |
+
position: relative;
|
| 57 |
+
transform-origin: 0 0;
|
| 58 |
+
transform: translateX(0) translateY(0) translateZ(-40px);
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
.cube-face {
|
| 62 |
+
position: absolute;
|
| 63 |
+
inset: 0;
|
| 64 |
+
background: #0017ff;
|
| 65 |
+
border: solid 1px #ffffff;
|
| 66 |
+
}
|
| 67 |
+
.cube-face.top {
|
| 68 |
+
transform: translateZ(80px);
|
| 69 |
+
}
|
| 70 |
+
.cube-face.front {
|
| 71 |
+
transform-origin: 0 50%;
|
| 72 |
+
transform: rotateY(-90deg);
|
| 73 |
+
}
|
| 74 |
+
.cube-face.back {
|
| 75 |
+
transform-origin: 0 50%;
|
| 76 |
+
transform: rotateY(-90deg) translateZ(-80px);
|
| 77 |
+
}
|
| 78 |
+
.cube-face.right {
|
| 79 |
+
transform-origin: 50% 0;
|
| 80 |
+
transform: rotateX(-90deg) translateY(-80px);
|
| 81 |
+
}
|
| 82 |
+
.cube-face.left {
|
| 83 |
+
transform-origin: 50% 0;
|
| 84 |
+
transform: rotateX(-90deg) translateY(-80px) translateZ(80px);
|
| 85 |
+
}
|
| 86 |
+
|
| 87 |
+
@keyframes rotation {
|
| 88 |
+
0% {
|
| 89 |
+
transform: rotateX(45deg) rotateY(0) rotateZ(45deg);
|
| 90 |
+
animation-timing-function: cubic-bezier(
|
| 91 |
+
0.17,
|
| 92 |
+
0.84,
|
| 93 |
+
0.44,
|
| 94 |
+
1
|
| 95 |
+
);
|
| 96 |
+
}
|
| 97 |
+
50% {
|
| 98 |
+
transform: rotateX(45deg) rotateY(0) rotateZ(225deg);
|
| 99 |
+
animation-timing-function: cubic-bezier(
|
| 100 |
+
0.76,
|
| 101 |
+
0.05,
|
| 102 |
+
0.86,
|
| 103 |
+
0.06
|
| 104 |
+
);
|
| 105 |
+
}
|
| 106 |
+
100% {
|
| 107 |
+
transform: rotateX(45deg) rotateY(0) rotateZ(405deg);
|
| 108 |
+
animation-timing-function: cubic-bezier(
|
| 109 |
+
0.17,
|
| 110 |
+
0.84,
|
| 111 |
+
0.44,
|
| 112 |
+
1
|
| 113 |
+
);
|
| 114 |
+
}
|
| 115 |
+
}
|
| 116 |
+
|
| 117 |
+
.scene,
|
| 118 |
+
#message {
|
| 119 |
+
position: absolute;
|
| 120 |
+
display: flex;
|
| 121 |
+
top: 0;
|
| 122 |
+
right: 0;
|
| 123 |
+
left: 0;
|
| 124 |
+
bottom: 0;
|
| 125 |
+
z-index: 2;
|
| 126 |
+
height: 100%;
|
| 127 |
+
width: 100%;
|
| 128 |
+
align-items: center;
|
| 129 |
+
justify-content: center;
|
| 130 |
+
}
|
| 131 |
+
#message {
|
| 132 |
+
font-weight: bold;
|
| 133 |
+
font-size: large;
|
| 134 |
+
color: red;
|
| 135 |
+
pointer-events: none;
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
details {
|
| 139 |
+
font-size: small;
|
| 140 |
+
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
#progress {
|
| 144 |
+
position: absolute;
|
| 145 |
+
top: 0;
|
| 146 |
+
height: 5px;
|
| 147 |
+
background: blue;
|
| 148 |
+
z-index: 99;
|
| 149 |
+
transition: width 0.1s ease-in-out;
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
#quality {
|
| 153 |
+
position: absolute;
|
| 154 |
+
bottom: 10px;
|
| 155 |
+
z-index: 999;
|
| 156 |
+
right: 10px;
|
| 157 |
+
}
|
| 158 |
+
|
| 159 |
+
#caminfo {
|
| 160 |
+
position: absolute;
|
| 161 |
+
top: 10px;
|
| 162 |
+
z-index: 999;
|
| 163 |
+
right: 10px;
|
| 164 |
+
}
|
| 165 |
+
#canvas {
|
| 166 |
+
display: block;
|
| 167 |
+
position: absolute;
|
| 168 |
+
top: 0;
|
| 169 |
+
left: 0;
|
| 170 |
+
width: 100%;
|
| 171 |
+
height: 100%;
|
| 172 |
+
touch-action: none;
|
| 173 |
+
}
|
| 174 |
+
|
| 175 |
+
#instructions {
|
| 176 |
+
background: rgba(0,0,0,0.6);
|
| 177 |
+
white-space: pre-wrap;
|
| 178 |
+
padding: 10px;
|
| 179 |
+
border-radius: 10px;
|
| 180 |
+
font-size: x-small;
|
| 181 |
+
}
|
| 182 |
+
body.nohf .nohf {
|
| 183 |
+
display: none;
|
| 184 |
+
}
|
| 185 |
+
body.nohf #progress, body.nohf .cube-face {
|
| 186 |
+
background: #ff9d0d;
|
| 187 |
+
}
|
| 188 |
+
/* idol: in the app the keyboard and mouse help, its links and the fps counter are hidden */
|
| 189 |
+
#info, #quality, #caminfo {
|
| 190 |
+
display: none;
|
| 191 |
+
}
|
| 192 |
+
</style>
|
| 193 |
+
</head>
|
| 194 |
+
<body>
|
| 195 |
+
<script>
|
| 196 |
+
if(location.host.includes('hf.space')) document.body.classList.add('nohf');
|
| 197 |
+
</script>
|
| 198 |
+
<div id="info">
|
| 199 |
+
<h3 class="nohf">WebGL 3D Gaussian Splat Viewer</h3>
|
| 200 |
+
<p>
|
| 201 |
+
<small class="nohf">
|
| 202 |
+
By <a href="https://twitter.com/antimatter15">Kevin Kwok</a>.
|
| 203 |
+
Code on
|
| 204 |
+
<a href="https://github.com/antimatter15/splat">Github</a
|
| 205 |
+
>.
|
| 206 |
+
</small>
|
| 207 |
+
</p>
|
| 208 |
+
|
| 209 |
+
<details>
|
| 210 |
+
<summary>Use mouse or arrow keys to navigate.</summary>
|
| 211 |
+
|
| 212 |
+
<div id="instructions">movement (arrow keys)
|
| 213 |
+
- left/right arrow keys to strafe side to side
|
| 214 |
+
- up/down arrow keys to move forward/back
|
| 215 |
+
- space to jump
|
| 216 |
+
|
| 217 |
+
camera angle (wasd)
|
| 218 |
+
- a/d to turn camera left/right
|
| 219 |
+
- w/s to tilt camera up/down
|
| 220 |
+
- q/e to roll camera counterclockwise/clockwise
|
| 221 |
+
- i/k and j/l to orbit
|
| 222 |
+
|
| 223 |
+
trackpad
|
| 224 |
+
- scroll up/down/left/right to orbit
|
| 225 |
+
- pinch to move forward/back
|
| 226 |
+
- ctrl key + scroll to move forward/back
|
| 227 |
+
- shift + scroll to move up/down or strafe
|
| 228 |
+
|
| 229 |
+
mouse
|
| 230 |
+
- click and drag to orbit
|
| 231 |
+
- right click (or ctrl/cmd key) and drag up/down to move
|
| 232 |
+
|
| 233 |
+
touch (mobile)
|
| 234 |
+
- one finger to orbit
|
| 235 |
+
- two finger pinch to move forward/back
|
| 236 |
+
- two finger rotate to rotate camera clockwise/counterclockwise
|
| 237 |
+
- two finger pan to move side-to-side and up-down
|
| 238 |
+
|
| 239 |
+
gamepad
|
| 240 |
+
- if you have a game controller connected it should work
|
| 241 |
+
|
| 242 |
+
other
|
| 243 |
+
- press 0-9 to switch to one of the pre-loaded camera views
|
| 244 |
+
- press '-' or '+'key to cycle loaded cameras
|
| 245 |
+
- press p to resume default animation
|
| 246 |
+
- drag and drop .ply file to convert to .splat
|
| 247 |
+
- drag and drop cameras.json to load cameras
|
| 248 |
+
</div>
|
| 249 |
+
|
| 250 |
+
</details>
|
| 251 |
+
|
| 252 |
+
</div>
|
| 253 |
+
|
| 254 |
+
<div id="progress"></div>
|
| 255 |
+
|
| 256 |
+
<div id="message"></div>
|
| 257 |
+
<div class="scene" id="spinner">
|
| 258 |
+
<div class="cube-wrapper">
|
| 259 |
+
<div class="cube">
|
| 260 |
+
<div class="cube-faces">
|
| 261 |
+
<div class="cube-face bottom"></div>
|
| 262 |
+
<div class="cube-face top"></div>
|
| 263 |
+
<div class="cube-face left"></div>
|
| 264 |
+
<div class="cube-face right"></div>
|
| 265 |
+
<div class="cube-face back"></div>
|
| 266 |
+
<div class="cube-face front"></div>
|
| 267 |
+
</div>
|
| 268 |
+
</div>
|
| 269 |
+
</div>
|
| 270 |
+
</div>
|
| 271 |
+
<canvas id="canvas"></canvas>
|
| 272 |
+
|
| 273 |
+
<div id="quality">
|
| 274 |
+
<span id="fps"></span>
|
| 275 |
+
</div>
|
| 276 |
+
<div id="caminfo">
|
| 277 |
+
<span id="camid"></span>
|
| 278 |
+
</div>
|
| 279 |
+
<script src="main.js"></script>
|
| 280 |
+
</body>
|
| 281 |
+
</html>
|
android/app/src/main/assets/viewer/main.js
ADDED
|
@@ -0,0 +1,1527 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
let cameras = [
|
| 2 |
+
{
|
| 3 |
+
id: 0,
|
| 4 |
+
img_name: "00001",
|
| 5 |
+
width: 1959,
|
| 6 |
+
height: 1090,
|
| 7 |
+
position: [
|
| 8 |
+
-3.0089893469241797, -0.11086489695181866, -3.7527640949141428,
|
| 9 |
+
],
|
| 10 |
+
rotation: [
|
| 11 |
+
[0.876134201218856, 0.06925962026449776, 0.47706599800804744],
|
| 12 |
+
[-0.04747421839895102, 0.9972110940209488, -0.057586739349882114],
|
| 13 |
+
[-0.4797239414934443, 0.027805376500959853, 0.8769787916452908],
|
| 14 |
+
],
|
| 15 |
+
fy: 1164.6601287484507,
|
| 16 |
+
fx: 1159.5880733038064,
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
id: 1,
|
| 20 |
+
img_name: "00009",
|
| 21 |
+
width: 1959,
|
| 22 |
+
height: 1090,
|
| 23 |
+
position: [
|
| 24 |
+
-2.5199776022057296, -0.09704735754873686, -3.6247725540304545,
|
| 25 |
+
],
|
| 26 |
+
rotation: [
|
| 27 |
+
[0.9982731285632193, -0.011928707708098955, -0.05751927260507243],
|
| 28 |
+
[0.0065061360949636325, 0.9955928229282383, -0.09355533724430458],
|
| 29 |
+
[0.058381769258182864, 0.09301955098900708, 0.9939511719154457],
|
| 30 |
+
],
|
| 31 |
+
fy: 1164.6601287484507,
|
| 32 |
+
fx: 1159.5880733038064,
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
id: 2,
|
| 36 |
+
img_name: "00017",
|
| 37 |
+
width: 1959,
|
| 38 |
+
height: 1090,
|
| 39 |
+
position: [
|
| 40 |
+
-0.7737533667465242, -0.3364271945329695, -2.9358969417573753,
|
| 41 |
+
],
|
| 42 |
+
rotation: [
|
| 43 |
+
[0.9998813418672372, 0.013742375651625236, -0.0069605529394208224],
|
| 44 |
+
[-0.014268370388586709, 0.996512943252834, -0.08220929105659476],
|
| 45 |
+
[0.00580653013657589, 0.08229885200307129, 0.9965907801935302],
|
| 46 |
+
],
|
| 47 |
+
fy: 1164.6601287484507,
|
| 48 |
+
fx: 1159.5880733038064,
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
id: 3,
|
| 52 |
+
img_name: "00025",
|
| 53 |
+
width: 1959,
|
| 54 |
+
height: 1090,
|
| 55 |
+
position: [
|
| 56 |
+
1.2198221749590001, -0.2196687861401182, -2.3183162007028453,
|
| 57 |
+
],
|
| 58 |
+
rotation: [
|
| 59 |
+
[0.9208648867765482, 0.0012010625395201253, 0.389880004297208],
|
| 60 |
+
[-0.06298204172269357, 0.987319521752825, 0.14571693239364383],
|
| 61 |
+
[-0.3847611242348369, -0.1587410451475895, 0.9092635249821667],
|
| 62 |
+
],
|
| 63 |
+
fy: 1164.6601287484507,
|
| 64 |
+
fx: 1159.5880733038064,
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
id: 4,
|
| 68 |
+
img_name: "00033",
|
| 69 |
+
width: 1959,
|
| 70 |
+
height: 1090,
|
| 71 |
+
position: [
|
| 72 |
+
1.742387858893817, -0.13848225198886954, -2.0566370113193146,
|
| 73 |
+
],
|
| 74 |
+
rotation: [
|
| 75 |
+
[0.24669889292141334, -0.08370189346592856, -0.9654706879349405],
|
| 76 |
+
[0.11343747891376445, 0.9919082664242816, -0.05700815184573074],
|
| 77 |
+
[0.9624300466054861, -0.09545671285663988, 0.2541976029815521],
|
| 78 |
+
],
|
| 79 |
+
fy: 1164.6601287484507,
|
| 80 |
+
fx: 1159.5880733038064,
|
| 81 |
+
},
|
| 82 |
+
{
|
| 83 |
+
id: 5,
|
| 84 |
+
img_name: "00041",
|
| 85 |
+
width: 1959,
|
| 86 |
+
height: 1090,
|
| 87 |
+
position: [
|
| 88 |
+
3.6567309419223935, -0.16470990600750707, -1.3458085590422042,
|
| 89 |
+
],
|
| 90 |
+
rotation: [
|
| 91 |
+
[0.2341293058324528, -0.02968330457755884, -0.9717522161434825],
|
| 92 |
+
[0.10270823606832301, 0.99469554638321, -0.005638106875665722],
|
| 93 |
+
[0.9667649592295676, -0.09848690996657204, 0.2359360976431732],
|
| 94 |
+
],
|
| 95 |
+
fy: 1164.6601287484507,
|
| 96 |
+
fx: 1159.5880733038064,
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
id: 6,
|
| 100 |
+
img_name: "00049",
|
| 101 |
+
width: 1959,
|
| 102 |
+
height: 1090,
|
| 103 |
+
position: [
|
| 104 |
+
3.9013554243203497, -0.2597500978038105, -0.8106154188297828,
|
| 105 |
+
],
|
| 106 |
+
rotation: [
|
| 107 |
+
[0.6717235545638952, -0.015718162115524837, -0.7406351366386528],
|
| 108 |
+
[0.055627354673906296, 0.9980224478387622, 0.029270992841185218],
|
| 109 |
+
[0.7387104058127439, -0.060861588786650656, 0.6712695459756353],
|
| 110 |
+
],
|
| 111 |
+
fy: 1164.6601287484507,
|
| 112 |
+
fx: 1159.5880733038064,
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
id: 7,
|
| 116 |
+
img_name: "00057",
|
| 117 |
+
width: 1959,
|
| 118 |
+
height: 1090,
|
| 119 |
+
position: [4.742994605467533, -0.05591660945412069, 0.9500365976084458],
|
| 120 |
+
rotation: [
|
| 121 |
+
[-0.17042655709210375, 0.01207080756938, -0.9852964448542146],
|
| 122 |
+
[0.1165090336695526, 0.9931575292530063, -0.00798543433078162],
|
| 123 |
+
[0.9784581921120181, -0.1161568667478904, -0.1706667764862097],
|
| 124 |
+
],
|
| 125 |
+
fy: 1164.6601287484507,
|
| 126 |
+
fx: 1159.5880733038064,
|
| 127 |
+
},
|
| 128 |
+
{
|
| 129 |
+
id: 8,
|
| 130 |
+
img_name: "00065",
|
| 131 |
+
width: 1959,
|
| 132 |
+
height: 1090,
|
| 133 |
+
position: [4.34676307626522, 0.08168160516967145, 1.0876221470355405],
|
| 134 |
+
rotation: [
|
| 135 |
+
[-0.003575447631888379, -0.044792503246552894, -0.9989899137764799],
|
| 136 |
+
[0.10770152645126597, 0.9931680875192705, -0.04491693593046672],
|
| 137 |
+
[0.9941768441149182, -0.10775333677534978, 0.0012732004866391048],
|
| 138 |
+
],
|
| 139 |
+
fy: 1164.6601287484507,
|
| 140 |
+
fx: 1159.5880733038064,
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
id: 9,
|
| 144 |
+
img_name: "00073",
|
| 145 |
+
width: 1959,
|
| 146 |
+
height: 1090,
|
| 147 |
+
position: [3.264984351114202, 0.078974937336732, 1.0117200284114904],
|
| 148 |
+
rotation: [
|
| 149 |
+
[-0.026919994628162257, -0.1565891128261527, -0.9872968974090509],
|
| 150 |
+
[0.08444552208239385, 0.983768234577625, -0.1583319754069128],
|
| 151 |
+
[0.9960643893290491, -0.0876350978794554, -0.013259786205163005],
|
| 152 |
+
],
|
| 153 |
+
fy: 1164.6601287484507,
|
| 154 |
+
fx: 1159.5880733038064,
|
| 155 |
+
},
|
| 156 |
+
];
|
| 157 |
+
|
| 158 |
+
let camera = cameras[0];
|
| 159 |
+
|
| 160 |
+
// idol: the camera of the contract renders (idol/scripts/render_views.py): it looks at the origin from distance 2.2,
|
| 161 |
+
// 20 degrees above the object, image up = -y (idol's PLY is y-down), field of view 40 degrees; the orbit turns it
|
| 162 |
+
// about the vertical axis. Azimuth 90 degrees faces the object's front (scripts/make_hero_app.py).
|
| 163 |
+
const IDOL_DISTANCE = 2.2;
|
| 164 |
+
const IDOL_ELEVATION = (20 * Math.PI) / 180;
|
| 165 |
+
const IDOL_FOV = (40 * Math.PI) / 180;
|
| 166 |
+
const IDOL_START_AZIMUTH = Math.PI / 2;
|
| 167 |
+
const IDOL_TURN_MS = 12000;
|
| 168 |
+
|
| 169 |
+
// idol: world -> camera (rows x, y, z = right, down, forward; t = -R c), column-major like getViewMatrix's result
|
| 170 |
+
function idolViewMatrix(azimuth) {
|
| 171 |
+
const ce = Math.cos(IDOL_ELEVATION);
|
| 172 |
+
const se = Math.sin(IDOL_ELEVATION);
|
| 173 |
+
const c = [
|
| 174 |
+
IDOL_DISTANCE * ce * Math.sin(azimuth),
|
| 175 |
+
-IDOL_DISTANCE * se,
|
| 176 |
+
IDOL_DISTANCE * ce * Math.cos(azimuth),
|
| 177 |
+
];
|
| 178 |
+
const n = Math.hypot(c[0], c[1], c[2]);
|
| 179 |
+
const z = [-c[0] / n, -c[1] / n, -c[2] / n];
|
| 180 |
+
const up = [0, -1, 0];
|
| 181 |
+
let x = [
|
| 182 |
+
z[1] * up[2] - z[2] * up[1],
|
| 183 |
+
z[2] * up[0] - z[0] * up[2],
|
| 184 |
+
z[0] * up[1] - z[1] * up[0],
|
| 185 |
+
];
|
| 186 |
+
const xn = Math.hypot(x[0], x[1], x[2]);
|
| 187 |
+
x = [x[0] / xn, x[1] / xn, x[2] / xn];
|
| 188 |
+
const y = [
|
| 189 |
+
z[1] * x[2] - z[2] * x[1],
|
| 190 |
+
z[2] * x[0] - z[0] * x[2],
|
| 191 |
+
z[0] * x[1] - z[1] * x[0],
|
| 192 |
+
];
|
| 193 |
+
const t = [
|
| 194 |
+
-(x[0] * c[0] + x[1] * c[1] + x[2] * c[2]),
|
| 195 |
+
-(y[0] * c[0] + y[1] * c[1] + y[2] * c[2]),
|
| 196 |
+
-(z[0] * c[0] + z[1] * c[1] + z[2] * c[2]),
|
| 197 |
+
];
|
| 198 |
+
return [x[0], y[0], z[0], 0, x[1], y[1], z[1], 0, x[2], y[2], z[2], 0, t[0], t[1], t[2], 1];
|
| 199 |
+
}
|
| 200 |
+
|
| 201 |
+
function getProjectionMatrix(fx, fy, width, height) {
|
| 202 |
+
const znear = 0.2;
|
| 203 |
+
const zfar = 200;
|
| 204 |
+
return [
|
| 205 |
+
[(2 * fx) / width, 0, 0, 0],
|
| 206 |
+
[0, -(2 * fy) / height, 0, 0],
|
| 207 |
+
[0, 0, zfar / (zfar - znear), 1],
|
| 208 |
+
[0, 0, -(zfar * znear) / (zfar - znear), 0],
|
| 209 |
+
].flat();
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
function getViewMatrix(camera) {
|
| 213 |
+
const R = camera.rotation.flat();
|
| 214 |
+
const t = camera.position;
|
| 215 |
+
const camToWorld = [
|
| 216 |
+
[R[0], R[1], R[2], 0],
|
| 217 |
+
[R[3], R[4], R[5], 0],
|
| 218 |
+
[R[6], R[7], R[8], 0],
|
| 219 |
+
[
|
| 220 |
+
-t[0] * R[0] - t[1] * R[3] - t[2] * R[6],
|
| 221 |
+
-t[0] * R[1] - t[1] * R[4] - t[2] * R[7],
|
| 222 |
+
-t[0] * R[2] - t[1] * R[5] - t[2] * R[8],
|
| 223 |
+
1,
|
| 224 |
+
],
|
| 225 |
+
].flat();
|
| 226 |
+
return camToWorld;
|
| 227 |
+
}
|
| 228 |
+
// function translate4(a, x, y, z) {
|
| 229 |
+
// return [
|
| 230 |
+
// ...a.slice(0, 12),
|
| 231 |
+
// a[0] * x + a[4] * y + a[8] * z + a[12],
|
| 232 |
+
// a[1] * x + a[5] * y + a[9] * z + a[13],
|
| 233 |
+
// a[2] * x + a[6] * y + a[10] * z + a[14],
|
| 234 |
+
// a[3] * x + a[7] * y + a[11] * z + a[15],
|
| 235 |
+
// ];
|
| 236 |
+
// }
|
| 237 |
+
|
| 238 |
+
function multiply4(a, b) {
|
| 239 |
+
return [
|
| 240 |
+
b[0] * a[0] + b[1] * a[4] + b[2] * a[8] + b[3] * a[12],
|
| 241 |
+
b[0] * a[1] + b[1] * a[5] + b[2] * a[9] + b[3] * a[13],
|
| 242 |
+
b[0] * a[2] + b[1] * a[6] + b[2] * a[10] + b[3] * a[14],
|
| 243 |
+
b[0] * a[3] + b[1] * a[7] + b[2] * a[11] + b[3] * a[15],
|
| 244 |
+
b[4] * a[0] + b[5] * a[4] + b[6] * a[8] + b[7] * a[12],
|
| 245 |
+
b[4] * a[1] + b[5] * a[5] + b[6] * a[9] + b[7] * a[13],
|
| 246 |
+
b[4] * a[2] + b[5] * a[6] + b[6] * a[10] + b[7] * a[14],
|
| 247 |
+
b[4] * a[3] + b[5] * a[7] + b[6] * a[11] + b[7] * a[15],
|
| 248 |
+
b[8] * a[0] + b[9] * a[4] + b[10] * a[8] + b[11] * a[12],
|
| 249 |
+
b[8] * a[1] + b[9] * a[5] + b[10] * a[9] + b[11] * a[13],
|
| 250 |
+
b[8] * a[2] + b[9] * a[6] + b[10] * a[10] + b[11] * a[14],
|
| 251 |
+
b[8] * a[3] + b[9] * a[7] + b[10] * a[11] + b[11] * a[15],
|
| 252 |
+
b[12] * a[0] + b[13] * a[4] + b[14] * a[8] + b[15] * a[12],
|
| 253 |
+
b[12] * a[1] + b[13] * a[5] + b[14] * a[9] + b[15] * a[13],
|
| 254 |
+
b[12] * a[2] + b[13] * a[6] + b[14] * a[10] + b[15] * a[14],
|
| 255 |
+
b[12] * a[3] + b[13] * a[7] + b[14] * a[11] + b[15] * a[15],
|
| 256 |
+
];
|
| 257 |
+
}
|
| 258 |
+
|
| 259 |
+
function invert4(a) {
|
| 260 |
+
let b00 = a[0] * a[5] - a[1] * a[4];
|
| 261 |
+
let b01 = a[0] * a[6] - a[2] * a[4];
|
| 262 |
+
let b02 = a[0] * a[7] - a[3] * a[4];
|
| 263 |
+
let b03 = a[1] * a[6] - a[2] * a[5];
|
| 264 |
+
let b04 = a[1] * a[7] - a[3] * a[5];
|
| 265 |
+
let b05 = a[2] * a[7] - a[3] * a[6];
|
| 266 |
+
let b06 = a[8] * a[13] - a[9] * a[12];
|
| 267 |
+
let b07 = a[8] * a[14] - a[10] * a[12];
|
| 268 |
+
let b08 = a[8] * a[15] - a[11] * a[12];
|
| 269 |
+
let b09 = a[9] * a[14] - a[10] * a[13];
|
| 270 |
+
let b10 = a[9] * a[15] - a[11] * a[13];
|
| 271 |
+
let b11 = a[10] * a[15] - a[11] * a[14];
|
| 272 |
+
let det =
|
| 273 |
+
b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;
|
| 274 |
+
if (!det) return null;
|
| 275 |
+
return [
|
| 276 |
+
(a[5] * b11 - a[6] * b10 + a[7] * b09) / det,
|
| 277 |
+
(a[2] * b10 - a[1] * b11 - a[3] * b09) / det,
|
| 278 |
+
(a[13] * b05 - a[14] * b04 + a[15] * b03) / det,
|
| 279 |
+
(a[10] * b04 - a[9] * b05 - a[11] * b03) / det,
|
| 280 |
+
(a[6] * b08 - a[4] * b11 - a[7] * b07) / det,
|
| 281 |
+
(a[0] * b11 - a[2] * b08 + a[3] * b07) / det,
|
| 282 |
+
(a[14] * b02 - a[12] * b05 - a[15] * b01) / det,
|
| 283 |
+
(a[8] * b05 - a[10] * b02 + a[11] * b01) / det,
|
| 284 |
+
(a[4] * b10 - a[5] * b08 + a[7] * b06) / det,
|
| 285 |
+
(a[1] * b08 - a[0] * b10 - a[3] * b06) / det,
|
| 286 |
+
(a[12] * b04 - a[13] * b02 + a[15] * b00) / det,
|
| 287 |
+
(a[9] * b02 - a[8] * b04 - a[11] * b00) / det,
|
| 288 |
+
(a[5] * b07 - a[4] * b09 - a[6] * b06) / det,
|
| 289 |
+
(a[0] * b09 - a[1] * b07 + a[2] * b06) / det,
|
| 290 |
+
(a[13] * b01 - a[12] * b03 - a[14] * b00) / det,
|
| 291 |
+
(a[8] * b03 - a[9] * b01 + a[10] * b00) / det,
|
| 292 |
+
];
|
| 293 |
+
}
|
| 294 |
+
|
| 295 |
+
function rotate4(a, rad, x, y, z) {
|
| 296 |
+
let len = Math.hypot(x, y, z);
|
| 297 |
+
x /= len;
|
| 298 |
+
y /= len;
|
| 299 |
+
z /= len;
|
| 300 |
+
let s = Math.sin(rad);
|
| 301 |
+
let c = Math.cos(rad);
|
| 302 |
+
let t = 1 - c;
|
| 303 |
+
let b00 = x * x * t + c;
|
| 304 |
+
let b01 = y * x * t + z * s;
|
| 305 |
+
let b02 = z * x * t - y * s;
|
| 306 |
+
let b10 = x * y * t - z * s;
|
| 307 |
+
let b11 = y * y * t + c;
|
| 308 |
+
let b12 = z * y * t + x * s;
|
| 309 |
+
let b20 = x * z * t + y * s;
|
| 310 |
+
let b21 = y * z * t - x * s;
|
| 311 |
+
let b22 = z * z * t + c;
|
| 312 |
+
return [
|
| 313 |
+
a[0] * b00 + a[4] * b01 + a[8] * b02,
|
| 314 |
+
a[1] * b00 + a[5] * b01 + a[9] * b02,
|
| 315 |
+
a[2] * b00 + a[6] * b01 + a[10] * b02,
|
| 316 |
+
a[3] * b00 + a[7] * b01 + a[11] * b02,
|
| 317 |
+
a[0] * b10 + a[4] * b11 + a[8] * b12,
|
| 318 |
+
a[1] * b10 + a[5] * b11 + a[9] * b12,
|
| 319 |
+
a[2] * b10 + a[6] * b11 + a[10] * b12,
|
| 320 |
+
a[3] * b10 + a[7] * b11 + a[11] * b12,
|
| 321 |
+
a[0] * b20 + a[4] * b21 + a[8] * b22,
|
| 322 |
+
a[1] * b20 + a[5] * b21 + a[9] * b22,
|
| 323 |
+
a[2] * b20 + a[6] * b21 + a[10] * b22,
|
| 324 |
+
a[3] * b20 + a[7] * b21 + a[11] * b22,
|
| 325 |
+
...a.slice(12, 16),
|
| 326 |
+
];
|
| 327 |
+
}
|
| 328 |
+
|
| 329 |
+
function translate4(a, x, y, z) {
|
| 330 |
+
return [
|
| 331 |
+
...a.slice(0, 12),
|
| 332 |
+
a[0] * x + a[4] * y + a[8] * z + a[12],
|
| 333 |
+
a[1] * x + a[5] * y + a[9] * z + a[13],
|
| 334 |
+
a[2] * x + a[6] * y + a[10] * z + a[14],
|
| 335 |
+
a[3] * x + a[7] * y + a[11] * z + a[15],
|
| 336 |
+
];
|
| 337 |
+
}
|
| 338 |
+
|
| 339 |
+
function createWorker(self) {
|
| 340 |
+
let buffer;
|
| 341 |
+
let vertexCount = 0;
|
| 342 |
+
let viewProj;
|
| 343 |
+
// 6*4 + 4 + 4 = 8*4
|
| 344 |
+
// XYZ - Position (Float32)
|
| 345 |
+
// XYZ - Scale (Float32)
|
| 346 |
+
// RGBA - colors (uint8)
|
| 347 |
+
// IJKL - quaternion/rot (uint8)
|
| 348 |
+
const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
|
| 349 |
+
let lastProj = [];
|
| 350 |
+
let depthIndex = new Uint32Array();
|
| 351 |
+
let lastVertexCount = 0;
|
| 352 |
+
|
| 353 |
+
var _floatView = new Float32Array(1);
|
| 354 |
+
var _int32View = new Int32Array(_floatView.buffer);
|
| 355 |
+
|
| 356 |
+
function floatToHalf(float) {
|
| 357 |
+
_floatView[0] = float;
|
| 358 |
+
var f = _int32View[0];
|
| 359 |
+
|
| 360 |
+
var sign = (f >> 31) & 0x0001;
|
| 361 |
+
var exp = (f >> 23) & 0x00ff;
|
| 362 |
+
var frac = f & 0x007fffff;
|
| 363 |
+
|
| 364 |
+
var newExp;
|
| 365 |
+
if (exp == 0) {
|
| 366 |
+
newExp = 0;
|
| 367 |
+
} else if (exp < 113) {
|
| 368 |
+
newExp = 0;
|
| 369 |
+
frac |= 0x00800000;
|
| 370 |
+
frac = frac >> (113 - exp);
|
| 371 |
+
if (frac & 0x01000000) {
|
| 372 |
+
newExp = 1;
|
| 373 |
+
frac = 0;
|
| 374 |
+
}
|
| 375 |
+
} else if (exp < 142) {
|
| 376 |
+
newExp = exp - 112;
|
| 377 |
+
} else {
|
| 378 |
+
newExp = 31;
|
| 379 |
+
frac = 0;
|
| 380 |
+
}
|
| 381 |
+
|
| 382 |
+
return (sign << 15) | (newExp << 10) | (frac >> 13);
|
| 383 |
+
}
|
| 384 |
+
|
| 385 |
+
function packHalf2x16(x, y) {
|
| 386 |
+
return (floatToHalf(x) | (floatToHalf(y) << 16)) >>> 0;
|
| 387 |
+
}
|
| 388 |
+
|
| 389 |
+
function generateTexture() {
|
| 390 |
+
if (!buffer) return;
|
| 391 |
+
const f_buffer = new Float32Array(buffer);
|
| 392 |
+
const u_buffer = new Uint8Array(buffer);
|
| 393 |
+
|
| 394 |
+
var texwidth = 1024 * 2; // Set to your desired width
|
| 395 |
+
var texheight = Math.ceil((2 * vertexCount) / texwidth); // Set to your desired height
|
| 396 |
+
var texdata = new Uint32Array(texwidth * texheight * 4); // 4 components per pixel (RGBA)
|
| 397 |
+
var texdata_c = new Uint8Array(texdata.buffer);
|
| 398 |
+
var texdata_f = new Float32Array(texdata.buffer);
|
| 399 |
+
|
| 400 |
+
// Here we convert from a .splat file buffer into a texture
|
| 401 |
+
// With a little bit more foresight perhaps this texture file
|
| 402 |
+
// should have been the native format as it'd be very easy to
|
| 403 |
+
// load it into webgl.
|
| 404 |
+
for (let i = 0; i < vertexCount; i++) {
|
| 405 |
+
// x, y, z
|
| 406 |
+
texdata_f[8 * i + 0] = f_buffer[8 * i + 0];
|
| 407 |
+
texdata_f[8 * i + 1] = f_buffer[8 * i + 1];
|
| 408 |
+
texdata_f[8 * i + 2] = f_buffer[8 * i + 2];
|
| 409 |
+
|
| 410 |
+
// r, g, b, a
|
| 411 |
+
texdata_c[4 * (8 * i + 7) + 0] = u_buffer[32 * i + 24 + 0];
|
| 412 |
+
texdata_c[4 * (8 * i + 7) + 1] = u_buffer[32 * i + 24 + 1];
|
| 413 |
+
texdata_c[4 * (8 * i + 7) + 2] = u_buffer[32 * i + 24 + 2];
|
| 414 |
+
texdata_c[4 * (8 * i + 7) + 3] = u_buffer[32 * i + 24 + 3];
|
| 415 |
+
|
| 416 |
+
// quaternions
|
| 417 |
+
let scale = [
|
| 418 |
+
f_buffer[8 * i + 3 + 0],
|
| 419 |
+
f_buffer[8 * i + 3 + 1],
|
| 420 |
+
f_buffer[8 * i + 3 + 2],
|
| 421 |
+
];
|
| 422 |
+
let rot = [
|
| 423 |
+
(u_buffer[32 * i + 28 + 0] - 128) / 128,
|
| 424 |
+
(u_buffer[32 * i + 28 + 1] - 128) / 128,
|
| 425 |
+
(u_buffer[32 * i + 28 + 2] - 128) / 128,
|
| 426 |
+
(u_buffer[32 * i + 28 + 3] - 128) / 128,
|
| 427 |
+
];
|
| 428 |
+
|
| 429 |
+
// Compute the matrix product of S and R (M = S * R)
|
| 430 |
+
const M = [
|
| 431 |
+
1.0 - 2.0 * (rot[2] * rot[2] + rot[3] * rot[3]),
|
| 432 |
+
2.0 * (rot[1] * rot[2] + rot[0] * rot[3]),
|
| 433 |
+
2.0 * (rot[1] * rot[3] - rot[0] * rot[2]),
|
| 434 |
+
|
| 435 |
+
2.0 * (rot[1] * rot[2] - rot[0] * rot[3]),
|
| 436 |
+
1.0 - 2.0 * (rot[1] * rot[1] + rot[3] * rot[3]),
|
| 437 |
+
2.0 * (rot[2] * rot[3] + rot[0] * rot[1]),
|
| 438 |
+
|
| 439 |
+
2.0 * (rot[1] * rot[3] + rot[0] * rot[2]),
|
| 440 |
+
2.0 * (rot[2] * rot[3] - rot[0] * rot[1]),
|
| 441 |
+
1.0 - 2.0 * (rot[1] * rot[1] + rot[2] * rot[2]),
|
| 442 |
+
].map((k, i) => k * scale[Math.floor(i / 3)]);
|
| 443 |
+
|
| 444 |
+
const sigma = [
|
| 445 |
+
M[0] * M[0] + M[3] * M[3] + M[6] * M[6],
|
| 446 |
+
M[0] * M[1] + M[3] * M[4] + M[6] * M[7],
|
| 447 |
+
M[0] * M[2] + M[3] * M[5] + M[6] * M[8],
|
| 448 |
+
M[1] * M[1] + M[4] * M[4] + M[7] * M[7],
|
| 449 |
+
M[1] * M[2] + M[4] * M[5] + M[7] * M[8],
|
| 450 |
+
M[2] * M[2] + M[5] * M[5] + M[8] * M[8],
|
| 451 |
+
];
|
| 452 |
+
|
| 453 |
+
texdata[8 * i + 4] = packHalf2x16(4 * sigma[0], 4 * sigma[1]);
|
| 454 |
+
texdata[8 * i + 5] = packHalf2x16(4 * sigma[2], 4 * sigma[3]);
|
| 455 |
+
texdata[8 * i + 6] = packHalf2x16(4 * sigma[4], 4 * sigma[5]);
|
| 456 |
+
}
|
| 457 |
+
|
| 458 |
+
self.postMessage({ texdata, texwidth, texheight }, [texdata.buffer]);
|
| 459 |
+
}
|
| 460 |
+
|
| 461 |
+
function runSort(viewProj) {
|
| 462 |
+
if (!buffer) return;
|
| 463 |
+
const f_buffer = new Float32Array(buffer);
|
| 464 |
+
if (lastVertexCount == vertexCount) {
|
| 465 |
+
let dot =
|
| 466 |
+
lastProj[2] * viewProj[2] +
|
| 467 |
+
lastProj[6] * viewProj[6] +
|
| 468 |
+
lastProj[10] * viewProj[10];
|
| 469 |
+
if (Math.abs(dot - 1) < 0.01) {
|
| 470 |
+
return;
|
| 471 |
+
}
|
| 472 |
+
} else {
|
| 473 |
+
generateTexture();
|
| 474 |
+
lastVertexCount = vertexCount;
|
| 475 |
+
}
|
| 476 |
+
|
| 477 |
+
console.time("sort");
|
| 478 |
+
let maxDepth = -Infinity;
|
| 479 |
+
let minDepth = Infinity;
|
| 480 |
+
let sizeList = new Int32Array(vertexCount);
|
| 481 |
+
for (let i = 0; i < vertexCount; i++) {
|
| 482 |
+
let depth =
|
| 483 |
+
((viewProj[2] * f_buffer[8 * i + 0] +
|
| 484 |
+
viewProj[6] * f_buffer[8 * i + 1] +
|
| 485 |
+
viewProj[10] * f_buffer[8 * i + 2]) *
|
| 486 |
+
4096) |
|
| 487 |
+
0;
|
| 488 |
+
sizeList[i] = depth;
|
| 489 |
+
if (depth > maxDepth) maxDepth = depth;
|
| 490 |
+
if (depth < minDepth) minDepth = depth;
|
| 491 |
+
}
|
| 492 |
+
|
| 493 |
+
// This is a 16 bit single-pass counting sort
|
| 494 |
+
let depthInv = (256 * 256 - 1) / (maxDepth - minDepth);
|
| 495 |
+
let counts0 = new Uint32Array(256 * 256);
|
| 496 |
+
for (let i = 0; i < vertexCount; i++) {
|
| 497 |
+
sizeList[i] = ((sizeList[i] - minDepth) * depthInv) | 0;
|
| 498 |
+
counts0[sizeList[i]]++;
|
| 499 |
+
}
|
| 500 |
+
let starts0 = new Uint32Array(256 * 256);
|
| 501 |
+
for (let i = 1; i < 256 * 256; i++)
|
| 502 |
+
starts0[i] = starts0[i - 1] + counts0[i - 1];
|
| 503 |
+
depthIndex = new Uint32Array(vertexCount);
|
| 504 |
+
for (let i = 0; i < vertexCount; i++)
|
| 505 |
+
depthIndex[starts0[sizeList[i]]++] = i;
|
| 506 |
+
|
| 507 |
+
console.timeEnd("sort");
|
| 508 |
+
|
| 509 |
+
lastProj = viewProj;
|
| 510 |
+
self.postMessage({ depthIndex, viewProj, vertexCount }, [
|
| 511 |
+
depthIndex.buffer,
|
| 512 |
+
]);
|
| 513 |
+
}
|
| 514 |
+
|
| 515 |
+
function processPlyBuffer(inputBuffer) {
|
| 516 |
+
const ubuf = new Uint8Array(inputBuffer);
|
| 517 |
+
// 10KB ought to be enough for a header...
|
| 518 |
+
const header = new TextDecoder().decode(ubuf.slice(0, 1024 * 10));
|
| 519 |
+
const header_end = "end_header\n";
|
| 520 |
+
const header_end_index = header.indexOf(header_end);
|
| 521 |
+
if (header_end_index < 0)
|
| 522 |
+
throw new Error("Unable to read .ply file header");
|
| 523 |
+
const vertexCount = parseInt(/element vertex (\d+)\n/.exec(header)[1]);
|
| 524 |
+
console.log("Vertex Count", vertexCount);
|
| 525 |
+
let row_offset = 0,
|
| 526 |
+
offsets = {},
|
| 527 |
+
types = {};
|
| 528 |
+
const TYPE_MAP = {
|
| 529 |
+
double: "getFloat64",
|
| 530 |
+
int: "getInt32",
|
| 531 |
+
uint: "getUint32",
|
| 532 |
+
float: "getFloat32",
|
| 533 |
+
short: "getInt16",
|
| 534 |
+
ushort: "getUint16",
|
| 535 |
+
uchar: "getUint8",
|
| 536 |
+
};
|
| 537 |
+
for (let prop of header
|
| 538 |
+
.slice(0, header_end_index)
|
| 539 |
+
.split("\n")
|
| 540 |
+
.filter((k) => k.startsWith("property "))) {
|
| 541 |
+
const [p, type, name] = prop.split(" ");
|
| 542 |
+
const arrayType = TYPE_MAP[type] || "getInt8";
|
| 543 |
+
types[name] = arrayType;
|
| 544 |
+
offsets[name] = row_offset;
|
| 545 |
+
row_offset += parseInt(arrayType.replace(/[^\d]/g, "")) / 8;
|
| 546 |
+
}
|
| 547 |
+
console.log("Bytes per row", row_offset, types, offsets);
|
| 548 |
+
|
| 549 |
+
let dataView = new DataView(
|
| 550 |
+
inputBuffer,
|
| 551 |
+
header_end_index + header_end.length,
|
| 552 |
+
);
|
| 553 |
+
let row = 0;
|
| 554 |
+
const attrs = new Proxy(
|
| 555 |
+
{},
|
| 556 |
+
{
|
| 557 |
+
get(target, prop) {
|
| 558 |
+
if (!types[prop]) throw new Error(prop + " not found");
|
| 559 |
+
return dataView[types[prop]](
|
| 560 |
+
row * row_offset + offsets[prop],
|
| 561 |
+
true,
|
| 562 |
+
);
|
| 563 |
+
},
|
| 564 |
+
},
|
| 565 |
+
);
|
| 566 |
+
|
| 567 |
+
console.time("calculate importance");
|
| 568 |
+
let sizeList = new Float32Array(vertexCount);
|
| 569 |
+
let sizeIndex = new Uint32Array(vertexCount);
|
| 570 |
+
for (row = 0; row < vertexCount; row++) {
|
| 571 |
+
sizeIndex[row] = row;
|
| 572 |
+
if (!types["scale_0"]) continue;
|
| 573 |
+
const size =
|
| 574 |
+
Math.exp(attrs.scale_0) *
|
| 575 |
+
Math.exp(attrs.scale_1) *
|
| 576 |
+
Math.exp(attrs.scale_2);
|
| 577 |
+
const opacity = 1 / (1 + Math.exp(-attrs.opacity));
|
| 578 |
+
sizeList[row] = size * opacity;
|
| 579 |
+
}
|
| 580 |
+
console.timeEnd("calculate importance");
|
| 581 |
+
|
| 582 |
+
console.time("sort");
|
| 583 |
+
sizeIndex.sort((b, a) => sizeList[a] - sizeList[b]);
|
| 584 |
+
console.timeEnd("sort");
|
| 585 |
+
|
| 586 |
+
// 6*4 + 4 + 4 = 8*4
|
| 587 |
+
// XYZ - Position (Float32)
|
| 588 |
+
// XYZ - Scale (Float32)
|
| 589 |
+
// RGBA - colors (uint8)
|
| 590 |
+
// IJKL - quaternion/rot (uint8)
|
| 591 |
+
const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
|
| 592 |
+
const buffer = new ArrayBuffer(rowLength * vertexCount);
|
| 593 |
+
|
| 594 |
+
console.time("build buffer");
|
| 595 |
+
for (let j = 0; j < vertexCount; j++) {
|
| 596 |
+
row = sizeIndex[j];
|
| 597 |
+
|
| 598 |
+
const position = new Float32Array(buffer, j * rowLength, 3);
|
| 599 |
+
const scales = new Float32Array(buffer, j * rowLength + 4 * 3, 3);
|
| 600 |
+
const rgba = new Uint8ClampedArray(
|
| 601 |
+
buffer,
|
| 602 |
+
j * rowLength + 4 * 3 + 4 * 3,
|
| 603 |
+
4,
|
| 604 |
+
);
|
| 605 |
+
const rot = new Uint8ClampedArray(
|
| 606 |
+
buffer,
|
| 607 |
+
j * rowLength + 4 * 3 + 4 * 3 + 4,
|
| 608 |
+
4,
|
| 609 |
+
);
|
| 610 |
+
|
| 611 |
+
if (types["scale_0"]) {
|
| 612 |
+
const qlen = Math.sqrt(
|
| 613 |
+
attrs.rot_0 ** 2 +
|
| 614 |
+
attrs.rot_1 ** 2 +
|
| 615 |
+
attrs.rot_2 ** 2 +
|
| 616 |
+
attrs.rot_3 ** 2,
|
| 617 |
+
);
|
| 618 |
+
|
| 619 |
+
rot[0] = (attrs.rot_0 / qlen) * 128 + 128;
|
| 620 |
+
rot[1] = (attrs.rot_1 / qlen) * 128 + 128;
|
| 621 |
+
rot[2] = (attrs.rot_2 / qlen) * 128 + 128;
|
| 622 |
+
rot[3] = (attrs.rot_3 / qlen) * 128 + 128;
|
| 623 |
+
|
| 624 |
+
scales[0] = Math.exp(attrs.scale_0);
|
| 625 |
+
scales[1] = Math.exp(attrs.scale_1);
|
| 626 |
+
scales[2] = Math.exp(attrs.scale_2);
|
| 627 |
+
} else {
|
| 628 |
+
scales[0] = 0.01;
|
| 629 |
+
scales[1] = 0.01;
|
| 630 |
+
scales[2] = 0.01;
|
| 631 |
+
|
| 632 |
+
rot[0] = 255;
|
| 633 |
+
rot[1] = 0;
|
| 634 |
+
rot[2] = 0;
|
| 635 |
+
rot[3] = 0;
|
| 636 |
+
}
|
| 637 |
+
|
| 638 |
+
position[0] = attrs.x;
|
| 639 |
+
position[1] = attrs.y;
|
| 640 |
+
position[2] = attrs.z;
|
| 641 |
+
|
| 642 |
+
if (types["f_dc_0"]) {
|
| 643 |
+
const SH_C0 = 0.28209479177387814;
|
| 644 |
+
rgba[0] = (0.5 + SH_C0 * attrs.f_dc_0) * 255;
|
| 645 |
+
rgba[1] = (0.5 + SH_C0 * attrs.f_dc_1) * 255;
|
| 646 |
+
rgba[2] = (0.5 + SH_C0 * attrs.f_dc_2) * 255;
|
| 647 |
+
} else {
|
| 648 |
+
rgba[0] = attrs.red;
|
| 649 |
+
rgba[1] = attrs.green;
|
| 650 |
+
rgba[2] = attrs.blue;
|
| 651 |
+
}
|
| 652 |
+
if (types["opacity"]) {
|
| 653 |
+
rgba[3] = (1 / (1 + Math.exp(-attrs.opacity))) * 255;
|
| 654 |
+
} else {
|
| 655 |
+
rgba[3] = 255;
|
| 656 |
+
}
|
| 657 |
+
}
|
| 658 |
+
console.timeEnd("build buffer");
|
| 659 |
+
return buffer;
|
| 660 |
+
}
|
| 661 |
+
|
| 662 |
+
const throttledSort = () => {
|
| 663 |
+
if (!sortRunning) {
|
| 664 |
+
sortRunning = true;
|
| 665 |
+
let lastView = viewProj;
|
| 666 |
+
runSort(lastView);
|
| 667 |
+
setTimeout(() => {
|
| 668 |
+
sortRunning = false;
|
| 669 |
+
if (lastView !== viewProj) {
|
| 670 |
+
throttledSort();
|
| 671 |
+
}
|
| 672 |
+
}, 0);
|
| 673 |
+
}
|
| 674 |
+
};
|
| 675 |
+
|
| 676 |
+
let sortRunning;
|
| 677 |
+
self.onmessage = (e) => {
|
| 678 |
+
if (e.data.ply) {
|
| 679 |
+
vertexCount = 0;
|
| 680 |
+
runSort(viewProj);
|
| 681 |
+
buffer = processPlyBuffer(e.data.ply);
|
| 682 |
+
vertexCount = Math.floor(buffer.byteLength / rowLength);
|
| 683 |
+
postMessage({ buffer: buffer, save: !!e.data.save });
|
| 684 |
+
} else if (e.data.buffer) {
|
| 685 |
+
buffer = e.data.buffer;
|
| 686 |
+
vertexCount = e.data.vertexCount;
|
| 687 |
+
} else if (e.data.vertexCount) {
|
| 688 |
+
vertexCount = e.data.vertexCount;
|
| 689 |
+
} else if (e.data.view) {
|
| 690 |
+
viewProj = e.data.view;
|
| 691 |
+
throttledSort();
|
| 692 |
+
}
|
| 693 |
+
};
|
| 694 |
+
}
|
| 695 |
+
|
| 696 |
+
const vertexShaderSource = `
|
| 697 |
+
#version 300 es
|
| 698 |
+
precision highp float;
|
| 699 |
+
precision highp int;
|
| 700 |
+
|
| 701 |
+
uniform highp usampler2D u_texture;
|
| 702 |
+
uniform mat4 projection, view;
|
| 703 |
+
uniform vec2 focal;
|
| 704 |
+
uniform vec2 viewport;
|
| 705 |
+
|
| 706 |
+
in vec2 position;
|
| 707 |
+
in int index;
|
| 708 |
+
|
| 709 |
+
out vec4 vColor;
|
| 710 |
+
out vec2 vPosition;
|
| 711 |
+
|
| 712 |
+
void main () {
|
| 713 |
+
uvec4 cen = texelFetch(u_texture, ivec2((uint(index) & 0x3ffu) << 1, uint(index) >> 10), 0);
|
| 714 |
+
vec4 cam = view * vec4(uintBitsToFloat(cen.xyz), 1);
|
| 715 |
+
vec4 pos2d = projection * cam;
|
| 716 |
+
|
| 717 |
+
float clip = 1.2 * pos2d.w;
|
| 718 |
+
if (pos2d.z < -clip || pos2d.x < -clip || pos2d.x > clip || pos2d.y < -clip || pos2d.y > clip) {
|
| 719 |
+
gl_Position = vec4(0.0, 0.0, 2.0, 1.0);
|
| 720 |
+
return;
|
| 721 |
+
}
|
| 722 |
+
|
| 723 |
+
uvec4 cov = texelFetch(u_texture, ivec2(((uint(index) & 0x3ffu) << 1) | 1u, uint(index) >> 10), 0);
|
| 724 |
+
vec2 u1 = unpackHalf2x16(cov.x), u2 = unpackHalf2x16(cov.y), u3 = unpackHalf2x16(cov.z);
|
| 725 |
+
mat3 Vrk = mat3(u1.x, u1.y, u2.x, u1.y, u2.y, u3.x, u2.x, u3.x, u3.y);
|
| 726 |
+
|
| 727 |
+
mat3 J = mat3(
|
| 728 |
+
focal.x / cam.z, 0., -(focal.x * cam.x) / (cam.z * cam.z),
|
| 729 |
+
0., -focal.y / cam.z, (focal.y * cam.y) / (cam.z * cam.z),
|
| 730 |
+
0., 0., 0.
|
| 731 |
+
);
|
| 732 |
+
|
| 733 |
+
mat3 T = transpose(mat3(view)) * J;
|
| 734 |
+
mat3 cov2d = transpose(T) * Vrk * T;
|
| 735 |
+
|
| 736 |
+
float mid = (cov2d[0][0] + cov2d[1][1]) / 2.0;
|
| 737 |
+
float radius = length(vec2((cov2d[0][0] - cov2d[1][1]) / 2.0, cov2d[0][1]));
|
| 738 |
+
float lambda1 = mid + radius, lambda2 = mid - radius;
|
| 739 |
+
|
| 740 |
+
if(lambda2 < 0.0) return;
|
| 741 |
+
vec2 diagonalVector = normalize(vec2(cov2d[0][1], lambda1 - cov2d[0][0]));
|
| 742 |
+
vec2 majorAxis = min(sqrt(2.0 * lambda1), 1024.0) * diagonalVector;
|
| 743 |
+
vec2 minorAxis = min(sqrt(2.0 * lambda2), 1024.0) * vec2(diagonalVector.y, -diagonalVector.x);
|
| 744 |
+
|
| 745 |
+
vColor = clamp(pos2d.z/pos2d.w+1.0, 0.0, 1.0) * vec4((cov.w) & 0xffu, (cov.w >> 8) & 0xffu, (cov.w >> 16) & 0xffu, (cov.w >> 24) & 0xffu) / 255.0;
|
| 746 |
+
vPosition = position;
|
| 747 |
+
|
| 748 |
+
vec2 vCenter = vec2(pos2d) / pos2d.w;
|
| 749 |
+
gl_Position = vec4(
|
| 750 |
+
vCenter
|
| 751 |
+
+ position.x * majorAxis / viewport
|
| 752 |
+
+ position.y * minorAxis / viewport, 0.0, 1.0);
|
| 753 |
+
|
| 754 |
+
}
|
| 755 |
+
`.trim();
|
| 756 |
+
|
| 757 |
+
const fragmentShaderSource = `
|
| 758 |
+
#version 300 es
|
| 759 |
+
precision highp float;
|
| 760 |
+
|
| 761 |
+
in vec4 vColor;
|
| 762 |
+
in vec2 vPosition;
|
| 763 |
+
|
| 764 |
+
out vec4 fragColor;
|
| 765 |
+
|
| 766 |
+
void main () {
|
| 767 |
+
float A = -dot(vPosition, vPosition);
|
| 768 |
+
if (A < -4.0) discard;
|
| 769 |
+
float B = exp(A) * vColor.a;
|
| 770 |
+
fragColor = vec4(B * vColor.rgb, B);
|
| 771 |
+
}
|
| 772 |
+
|
| 773 |
+
`.trim();
|
| 774 |
+
|
| 775 |
+
// idol: start at the contract camera facing the object's front (was a fixed matrix for the demo scenes)
|
| 776 |
+
let defaultViewMatrix = idolViewMatrix(IDOL_START_AZIMUTH);
|
| 777 |
+
let viewMatrix = defaultViewMatrix;
|
| 778 |
+
async function main() {
|
| 779 |
+
let carousel = true;
|
| 780 |
+
const params = new URLSearchParams(location.search);
|
| 781 |
+
try {
|
| 782 |
+
viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
|
| 783 |
+
carousel = false;
|
| 784 |
+
} catch (err) {}
|
| 785 |
+
const url = new URL(
|
| 786 |
+
// "nike.splat",
|
| 787 |
+
// location.href,
|
| 788 |
+
params.get("url") || "train.splat",
|
| 789 |
+
"https://huggingface.co/cakewalk/splat-data/resolve/main/",
|
| 790 |
+
);
|
| 791 |
+
const req = await fetch(url, {
|
| 792 |
+
mode: "cors", // no-cors, *cors, same-origin
|
| 793 |
+
credentials: "omit", // include, *same-origin, omit
|
| 794 |
+
});
|
| 795 |
+
console.log(req);
|
| 796 |
+
if (req.status != 200)
|
| 797 |
+
throw new Error(req.status + " Unable to load " + req.url);
|
| 798 |
+
|
| 799 |
+
const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
|
| 800 |
+
const reader = req.body.getReader();
|
| 801 |
+
let splatData = new Uint8Array(req.headers.get("content-length"));
|
| 802 |
+
|
| 803 |
+
const downsample =
|
| 804 |
+
splatData.length / rowLength > 500000 ? 1 : 1 / devicePixelRatio;
|
| 805 |
+
console.log(splatData.length / rowLength, downsample);
|
| 806 |
+
|
| 807 |
+
const worker = new Worker(
|
| 808 |
+
URL.createObjectURL(
|
| 809 |
+
new Blob(["(", createWorker.toString(), ")(self)"], {
|
| 810 |
+
type: "application/javascript",
|
| 811 |
+
}),
|
| 812 |
+
),
|
| 813 |
+
);
|
| 814 |
+
|
| 815 |
+
const canvas = document.getElementById("canvas");
|
| 816 |
+
const fps = document.getElementById("fps");
|
| 817 |
+
const camid = document.getElementById("camid");
|
| 818 |
+
|
| 819 |
+
let projectionMatrix;
|
| 820 |
+
|
| 821 |
+
const gl = canvas.getContext("webgl2", {
|
| 822 |
+
antialias: false,
|
| 823 |
+
});
|
| 824 |
+
|
| 825 |
+
const vertexShader = gl.createShader(gl.VERTEX_SHADER);
|
| 826 |
+
gl.shaderSource(vertexShader, vertexShaderSource);
|
| 827 |
+
gl.compileShader(vertexShader);
|
| 828 |
+
if (!gl.getShaderParameter(vertexShader, gl.COMPILE_STATUS))
|
| 829 |
+
console.error(gl.getShaderInfoLog(vertexShader));
|
| 830 |
+
|
| 831 |
+
const fragmentShader = gl.createShader(gl.FRAGMENT_SHADER);
|
| 832 |
+
gl.shaderSource(fragmentShader, fragmentShaderSource);
|
| 833 |
+
gl.compileShader(fragmentShader);
|
| 834 |
+
if (!gl.getShaderParameter(fragmentShader, gl.COMPILE_STATUS))
|
| 835 |
+
console.error(gl.getShaderInfoLog(fragmentShader));
|
| 836 |
+
|
| 837 |
+
const program = gl.createProgram();
|
| 838 |
+
gl.attachShader(program, vertexShader);
|
| 839 |
+
gl.attachShader(program, fragmentShader);
|
| 840 |
+
gl.linkProgram(program);
|
| 841 |
+
gl.useProgram(program);
|
| 842 |
+
|
| 843 |
+
if (!gl.getProgramParameter(program, gl.LINK_STATUS))
|
| 844 |
+
console.error(gl.getProgramInfoLog(program));
|
| 845 |
+
|
| 846 |
+
gl.disable(gl.DEPTH_TEST); // Disable depth testing
|
| 847 |
+
|
| 848 |
+
// Enable blending
|
| 849 |
+
gl.enable(gl.BLEND);
|
| 850 |
+
gl.blendFuncSeparate(
|
| 851 |
+
gl.ONE_MINUS_DST_ALPHA,
|
| 852 |
+
gl.ONE,
|
| 853 |
+
gl.ONE_MINUS_DST_ALPHA,
|
| 854 |
+
gl.ONE,
|
| 855 |
+
);
|
| 856 |
+
gl.blendEquationSeparate(gl.FUNC_ADD, gl.FUNC_ADD);
|
| 857 |
+
|
| 858 |
+
const u_projection = gl.getUniformLocation(program, "projection");
|
| 859 |
+
const u_viewport = gl.getUniformLocation(program, "viewport");
|
| 860 |
+
const u_focal = gl.getUniformLocation(program, "focal");
|
| 861 |
+
const u_view = gl.getUniformLocation(program, "view");
|
| 862 |
+
|
| 863 |
+
// positions
|
| 864 |
+
const triangleVertices = new Float32Array([-2, -2, 2, -2, 2, 2, -2, 2]);
|
| 865 |
+
const vertexBuffer = gl.createBuffer();
|
| 866 |
+
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
|
| 867 |
+
gl.bufferData(gl.ARRAY_BUFFER, triangleVertices, gl.STATIC_DRAW);
|
| 868 |
+
const a_position = gl.getAttribLocation(program, "position");
|
| 869 |
+
gl.enableVertexAttribArray(a_position);
|
| 870 |
+
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
|
| 871 |
+
gl.vertexAttribPointer(a_position, 2, gl.FLOAT, false, 0, 0);
|
| 872 |
+
|
| 873 |
+
var texture = gl.createTexture();
|
| 874 |
+
gl.bindTexture(gl.TEXTURE_2D, texture);
|
| 875 |
+
|
| 876 |
+
var u_textureLocation = gl.getUniformLocation(program, "u_texture");
|
| 877 |
+
gl.uniform1i(u_textureLocation, 0);
|
| 878 |
+
|
| 879 |
+
const indexBuffer = gl.createBuffer();
|
| 880 |
+
const a_index = gl.getAttribLocation(program, "index");
|
| 881 |
+
gl.enableVertexAttribArray(a_index);
|
| 882 |
+
gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
|
| 883 |
+
gl.vertexAttribIPointer(a_index, 1, gl.INT, false, 0, 0);
|
| 884 |
+
gl.vertexAttribDivisor(a_index, 1);
|
| 885 |
+
|
| 886 |
+
const resize = () => {
|
| 887 |
+
// idol: focal length for a 40 degree field of view across the shorter side of the screen (the square
|
| 888 |
+
// contract render's framing, whole object in view in portrait and landscape)
|
| 889 |
+
camera.fx = camera.fy =
|
| 890 |
+
Math.min(innerWidth, innerHeight) / 2 / Math.tan(IDOL_FOV / 2);
|
| 891 |
+
gl.uniform2fv(u_focal, new Float32Array([camera.fx, camera.fy]));
|
| 892 |
+
|
| 893 |
+
projectionMatrix = getProjectionMatrix(
|
| 894 |
+
camera.fx,
|
| 895 |
+
camera.fy,
|
| 896 |
+
innerWidth,
|
| 897 |
+
innerHeight,
|
| 898 |
+
);
|
| 899 |
+
|
| 900 |
+
gl.uniform2fv(u_viewport, new Float32Array([innerWidth, innerHeight]));
|
| 901 |
+
|
| 902 |
+
gl.canvas.width = Math.round(innerWidth / downsample);
|
| 903 |
+
gl.canvas.height = Math.round(innerHeight / downsample);
|
| 904 |
+
gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
|
| 905 |
+
|
| 906 |
+
gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
|
| 907 |
+
};
|
| 908 |
+
|
| 909 |
+
window.addEventListener("resize", resize);
|
| 910 |
+
resize();
|
| 911 |
+
|
| 912 |
+
worker.onmessage = (e) => {
|
| 913 |
+
if (e.data.buffer) {
|
| 914 |
+
splatData = new Uint8Array(e.data.buffer);
|
| 915 |
+
if (e.data.save) {
|
| 916 |
+
const blob = new Blob([splatData.buffer], {
|
| 917 |
+
type: "application/octet-stream",
|
| 918 |
+
});
|
| 919 |
+
const link = document.createElement("a");
|
| 920 |
+
link.download = "model.splat";
|
| 921 |
+
link.href = URL.createObjectURL(blob);
|
| 922 |
+
document.body.appendChild(link);
|
| 923 |
+
link.click();
|
| 924 |
+
}
|
| 925 |
+
} else if (e.data.texdata) {
|
| 926 |
+
const { texdata, texwidth, texheight } = e.data;
|
| 927 |
+
// console.log(texdata)
|
| 928 |
+
gl.bindTexture(gl.TEXTURE_2D, texture);
|
| 929 |
+
gl.texParameteri(
|
| 930 |
+
gl.TEXTURE_2D,
|
| 931 |
+
gl.TEXTURE_WRAP_S,
|
| 932 |
+
gl.CLAMP_TO_EDGE,
|
| 933 |
+
);
|
| 934 |
+
gl.texParameteri(
|
| 935 |
+
gl.TEXTURE_2D,
|
| 936 |
+
gl.TEXTURE_WRAP_T,
|
| 937 |
+
gl.CLAMP_TO_EDGE,
|
| 938 |
+
);
|
| 939 |
+
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
|
| 940 |
+
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
|
| 941 |
+
|
| 942 |
+
gl.texImage2D(
|
| 943 |
+
gl.TEXTURE_2D,
|
| 944 |
+
0,
|
| 945 |
+
gl.RGBA32UI,
|
| 946 |
+
texwidth,
|
| 947 |
+
texheight,
|
| 948 |
+
0,
|
| 949 |
+
gl.RGBA_INTEGER,
|
| 950 |
+
gl.UNSIGNED_INT,
|
| 951 |
+
texdata,
|
| 952 |
+
);
|
| 953 |
+
gl.activeTexture(gl.TEXTURE0);
|
| 954 |
+
gl.bindTexture(gl.TEXTURE_2D, texture);
|
| 955 |
+
} else if (e.data.depthIndex) {
|
| 956 |
+
const { depthIndex, viewProj } = e.data;
|
| 957 |
+
gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
|
| 958 |
+
gl.bufferData(gl.ARRAY_BUFFER, depthIndex, gl.DYNAMIC_DRAW);
|
| 959 |
+
vertexCount = e.data.vertexCount;
|
| 960 |
+
}
|
| 961 |
+
};
|
| 962 |
+
|
| 963 |
+
let activeKeys = [];
|
| 964 |
+
let currentCameraIndex = 0;
|
| 965 |
+
|
| 966 |
+
window.addEventListener("keydown", (e) => {
|
| 967 |
+
// if (document.activeElement != document.body) return;
|
| 968 |
+
carousel = false;
|
| 969 |
+
if (!activeKeys.includes(e.code)) activeKeys.push(e.code);
|
| 970 |
+
if (/\d/.test(e.key)) {
|
| 971 |
+
currentCameraIndex = parseInt(e.key);
|
| 972 |
+
camera = cameras[currentCameraIndex];
|
| 973 |
+
viewMatrix = getViewMatrix(camera);
|
| 974 |
+
}
|
| 975 |
+
if (["-", "_"].includes(e.key)) {
|
| 976 |
+
currentCameraIndex =
|
| 977 |
+
(currentCameraIndex + cameras.length - 1) % cameras.length;
|
| 978 |
+
viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
|
| 979 |
+
}
|
| 980 |
+
if (["+", "="].includes(e.key)) {
|
| 981 |
+
currentCameraIndex = (currentCameraIndex + 1) % cameras.length;
|
| 982 |
+
viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
|
| 983 |
+
}
|
| 984 |
+
camid.innerText = "cam " + currentCameraIndex;
|
| 985 |
+
if (e.code == "KeyV") {
|
| 986 |
+
location.hash =
|
| 987 |
+
"#" +
|
| 988 |
+
JSON.stringify(
|
| 989 |
+
viewMatrix.map((k) => Math.round(k * 100) / 100),
|
| 990 |
+
);
|
| 991 |
+
camid.innerText = "";
|
| 992 |
+
} else if (e.code === "KeyP") {
|
| 993 |
+
carousel = true;
|
| 994 |
+
camid.innerText = "";
|
| 995 |
+
}
|
| 996 |
+
});
|
| 997 |
+
window.addEventListener("keyup", (e) => {
|
| 998 |
+
activeKeys = activeKeys.filter((k) => k !== e.code);
|
| 999 |
+
});
|
| 1000 |
+
window.addEventListener("blur", () => {
|
| 1001 |
+
activeKeys = [];
|
| 1002 |
+
});
|
| 1003 |
+
|
| 1004 |
+
window.addEventListener(
|
| 1005 |
+
"wheel",
|
| 1006 |
+
(e) => {
|
| 1007 |
+
carousel = false;
|
| 1008 |
+
e.preventDefault();
|
| 1009 |
+
const lineHeight = 10;
|
| 1010 |
+
const scale =
|
| 1011 |
+
e.deltaMode == 1
|
| 1012 |
+
? lineHeight
|
| 1013 |
+
: e.deltaMode == 2
|
| 1014 |
+
? innerHeight
|
| 1015 |
+
: 1;
|
| 1016 |
+
let inv = invert4(viewMatrix);
|
| 1017 |
+
if (e.shiftKey) {
|
| 1018 |
+
inv = translate4(
|
| 1019 |
+
inv,
|
| 1020 |
+
(e.deltaX * scale) / innerWidth,
|
| 1021 |
+
(e.deltaY * scale) / innerHeight,
|
| 1022 |
+
0,
|
| 1023 |
+
);
|
| 1024 |
+
} else if (e.ctrlKey || e.metaKey) {
|
| 1025 |
+
// inv = rotate4(inv, (e.deltaX * scale) / innerWidth, 0, 0, 1);
|
| 1026 |
+
// inv = translate4(inv, 0, (e.deltaY * scale) / innerHeight, 0);
|
| 1027 |
+
// let preY = inv[13];
|
| 1028 |
+
inv = translate4(
|
| 1029 |
+
inv,
|
| 1030 |
+
0,
|
| 1031 |
+
0,
|
| 1032 |
+
(-10 * (e.deltaY * scale)) / innerHeight,
|
| 1033 |
+
);
|
| 1034 |
+
// inv[13] = preY;
|
| 1035 |
+
} else {
|
| 1036 |
+
let d = 4;
|
| 1037 |
+
inv = translate4(inv, 0, 0, d);
|
| 1038 |
+
inv = rotate4(inv, -(e.deltaX * scale) / innerWidth, 0, 1, 0);
|
| 1039 |
+
inv = rotate4(inv, (e.deltaY * scale) / innerHeight, 1, 0, 0);
|
| 1040 |
+
inv = translate4(inv, 0, 0, -d);
|
| 1041 |
+
}
|
| 1042 |
+
|
| 1043 |
+
viewMatrix = invert4(inv);
|
| 1044 |
+
},
|
| 1045 |
+
{ passive: false },
|
| 1046 |
+
);
|
| 1047 |
+
|
| 1048 |
+
let startX, startY, down;
|
| 1049 |
+
canvas.addEventListener("mousedown", (e) => {
|
| 1050 |
+
carousel = false;
|
| 1051 |
+
e.preventDefault();
|
| 1052 |
+
startX = e.clientX;
|
| 1053 |
+
startY = e.clientY;
|
| 1054 |
+
down = e.ctrlKey || e.metaKey ? 2 : 1;
|
| 1055 |
+
});
|
| 1056 |
+
canvas.addEventListener("contextmenu", (e) => {
|
| 1057 |
+
carousel = false;
|
| 1058 |
+
e.preventDefault();
|
| 1059 |
+
startX = e.clientX;
|
| 1060 |
+
startY = e.clientY;
|
| 1061 |
+
down = 2;
|
| 1062 |
+
});
|
| 1063 |
+
|
| 1064 |
+
canvas.addEventListener("mousemove", (e) => {
|
| 1065 |
+
e.preventDefault();
|
| 1066 |
+
if (down == 1) {
|
| 1067 |
+
let inv = invert4(viewMatrix);
|
| 1068 |
+
let dx = (5 * (e.clientX - startX)) / innerWidth;
|
| 1069 |
+
let dy = (5 * (e.clientY - startY)) / innerHeight;
|
| 1070 |
+
let d = 4;
|
| 1071 |
+
|
| 1072 |
+
inv = translate4(inv, 0, 0, d);
|
| 1073 |
+
inv = rotate4(inv, dx, 0, 1, 0);
|
| 1074 |
+
inv = rotate4(inv, -dy, 1, 0, 0);
|
| 1075 |
+
inv = translate4(inv, 0, 0, -d);
|
| 1076 |
+
// let postAngle = Math.atan2(inv[0], inv[10])
|
| 1077 |
+
// inv = rotate4(inv, postAngle - preAngle, 0, 0, 1)
|
| 1078 |
+
// console.log(postAngle)
|
| 1079 |
+
viewMatrix = invert4(inv);
|
| 1080 |
+
|
| 1081 |
+
startX = e.clientX;
|
| 1082 |
+
startY = e.clientY;
|
| 1083 |
+
} else if (down == 2) {
|
| 1084 |
+
let inv = invert4(viewMatrix);
|
| 1085 |
+
// inv = rotateY(inv, );
|
| 1086 |
+
// let preY = inv[13];
|
| 1087 |
+
inv = translate4(
|
| 1088 |
+
inv,
|
| 1089 |
+
(-10 * (e.clientX - startX)) / innerWidth,
|
| 1090 |
+
0,
|
| 1091 |
+
(10 * (e.clientY - startY)) / innerHeight,
|
| 1092 |
+
);
|
| 1093 |
+
// inv[13] = preY;
|
| 1094 |
+
viewMatrix = invert4(inv);
|
| 1095 |
+
|
| 1096 |
+
startX = e.clientX;
|
| 1097 |
+
startY = e.clientY;
|
| 1098 |
+
}
|
| 1099 |
+
});
|
| 1100 |
+
canvas.addEventListener("mouseup", (e) => {
|
| 1101 |
+
e.preventDefault();
|
| 1102 |
+
down = false;
|
| 1103 |
+
startX = 0;
|
| 1104 |
+
startY = 0;
|
| 1105 |
+
});
|
| 1106 |
+
|
| 1107 |
+
let altX = 0,
|
| 1108 |
+
altY = 0;
|
| 1109 |
+
canvas.addEventListener(
|
| 1110 |
+
"touchstart",
|
| 1111 |
+
(e) => {
|
| 1112 |
+
e.preventDefault();
|
| 1113 |
+
if (e.touches.length === 1) {
|
| 1114 |
+
carousel = false;
|
| 1115 |
+
startX = e.touches[0].clientX;
|
| 1116 |
+
startY = e.touches[0].clientY;
|
| 1117 |
+
down = 1;
|
| 1118 |
+
} else if (e.touches.length === 2) {
|
| 1119 |
+
// console.log('beep')
|
| 1120 |
+
carousel = false;
|
| 1121 |
+
startX = e.touches[0].clientX;
|
| 1122 |
+
altX = e.touches[1].clientX;
|
| 1123 |
+
startY = e.touches[0].clientY;
|
| 1124 |
+
altY = e.touches[1].clientY;
|
| 1125 |
+
down = 1;
|
| 1126 |
+
}
|
| 1127 |
+
},
|
| 1128 |
+
{ passive: false },
|
| 1129 |
+
);
|
| 1130 |
+
canvas.addEventListener(
|
| 1131 |
+
"touchmove",
|
| 1132 |
+
(e) => {
|
| 1133 |
+
e.preventDefault();
|
| 1134 |
+
if (e.touches.length === 1 && down) {
|
| 1135 |
+
let inv = invert4(viewMatrix);
|
| 1136 |
+
let dx = (4 * (e.touches[0].clientX - startX)) / innerWidth;
|
| 1137 |
+
let dy = (4 * (e.touches[0].clientY - startY)) / innerHeight;
|
| 1138 |
+
|
| 1139 |
+
// idol: orbit about the object at the origin (was a point 4 units ahead)
|
| 1140 |
+
let d = Math.hypot(inv[12], inv[13], inv[14]);
|
| 1141 |
+
inv = translate4(inv, 0, 0, d);
|
| 1142 |
+
// inv = translate4(inv, -x, -y, -z);
|
| 1143 |
+
// inv = translate4(inv, x, y, z);
|
| 1144 |
+
inv = rotate4(inv, dx, 0, 1, 0);
|
| 1145 |
+
inv = rotate4(inv, -dy, 1, 0, 0);
|
| 1146 |
+
inv = translate4(inv, 0, 0, -d);
|
| 1147 |
+
|
| 1148 |
+
viewMatrix = invert4(inv);
|
| 1149 |
+
|
| 1150 |
+
startX = e.touches[0].clientX;
|
| 1151 |
+
startY = e.touches[0].clientY;
|
| 1152 |
+
} else if (e.touches.length === 2) {
|
| 1153 |
+
// alert('beep')
|
| 1154 |
+
const dtheta =
|
| 1155 |
+
Math.atan2(startY - altY, startX - altX) -
|
| 1156 |
+
Math.atan2(
|
| 1157 |
+
e.touches[0].clientY - e.touches[1].clientY,
|
| 1158 |
+
e.touches[0].clientX - e.touches[1].clientX,
|
| 1159 |
+
);
|
| 1160 |
+
const dscale =
|
| 1161 |
+
Math.hypot(startX - altX, startY - altY) /
|
| 1162 |
+
Math.hypot(
|
| 1163 |
+
e.touches[0].clientX - e.touches[1].clientX,
|
| 1164 |
+
e.touches[0].clientY - e.touches[1].clientY,
|
| 1165 |
+
);
|
| 1166 |
+
const dx =
|
| 1167 |
+
(e.touches[0].clientX +
|
| 1168 |
+
e.touches[1].clientX -
|
| 1169 |
+
(startX + altX)) /
|
| 1170 |
+
2;
|
| 1171 |
+
const dy =
|
| 1172 |
+
(e.touches[0].clientY +
|
| 1173 |
+
e.touches[1].clientY -
|
| 1174 |
+
(startY + altY)) /
|
| 1175 |
+
2;
|
| 1176 |
+
let inv = invert4(viewMatrix);
|
| 1177 |
+
// inv = translate4(inv, 0, 0, d);
|
| 1178 |
+
inv = rotate4(inv, dtheta, 0, 0, 1);
|
| 1179 |
+
|
| 1180 |
+
inv = translate4(inv, -dx / innerWidth, -dy / innerHeight, 0);
|
| 1181 |
+
|
| 1182 |
+
// let preY = inv[13];
|
| 1183 |
+
inv = translate4(inv, 0, 0, 3 * (1 - dscale));
|
| 1184 |
+
// inv[13] = preY;
|
| 1185 |
+
|
| 1186 |
+
viewMatrix = invert4(inv);
|
| 1187 |
+
|
| 1188 |
+
startX = e.touches[0].clientX;
|
| 1189 |
+
altX = e.touches[1].clientX;
|
| 1190 |
+
startY = e.touches[0].clientY;
|
| 1191 |
+
altY = e.touches[1].clientY;
|
| 1192 |
+
}
|
| 1193 |
+
},
|
| 1194 |
+
{ passive: false },
|
| 1195 |
+
);
|
| 1196 |
+
canvas.addEventListener(
|
| 1197 |
+
"touchend",
|
| 1198 |
+
(e) => {
|
| 1199 |
+
e.preventDefault();
|
| 1200 |
+
down = false;
|
| 1201 |
+
startX = 0;
|
| 1202 |
+
startY = 0;
|
| 1203 |
+
},
|
| 1204 |
+
{ passive: false },
|
| 1205 |
+
);
|
| 1206 |
+
|
| 1207 |
+
let jumpDelta = 0;
|
| 1208 |
+
let vertexCount = 0;
|
| 1209 |
+
|
| 1210 |
+
let lastFrame = 0;
|
| 1211 |
+
let avgFps = 0;
|
| 1212 |
+
let start = 0;
|
| 1213 |
+
|
| 1214 |
+
window.addEventListener("gamepadconnected", (e) => {
|
| 1215 |
+
const gp = navigator.getGamepads()[e.gamepad.index];
|
| 1216 |
+
console.log(
|
| 1217 |
+
`Gamepad connected at index ${gp.index}: ${gp.id}. It has ${gp.buttons.length} buttons and ${gp.axes.length} axes.`,
|
| 1218 |
+
);
|
| 1219 |
+
});
|
| 1220 |
+
window.addEventListener("gamepaddisconnected", (e) => {
|
| 1221 |
+
console.log("Gamepad disconnected");
|
| 1222 |
+
});
|
| 1223 |
+
|
| 1224 |
+
let leftGamepadTrigger, rightGamepadTrigger;
|
| 1225 |
+
|
| 1226 |
+
const frame = (now) => {
|
| 1227 |
+
let inv = invert4(viewMatrix);
|
| 1228 |
+
let shiftKey =
|
| 1229 |
+
activeKeys.includes("Shift") ||
|
| 1230 |
+
activeKeys.includes("ShiftLeft") ||
|
| 1231 |
+
activeKeys.includes("ShiftRight");
|
| 1232 |
+
|
| 1233 |
+
if (activeKeys.includes("ArrowUp")) {
|
| 1234 |
+
if (shiftKey) {
|
| 1235 |
+
inv = translate4(inv, 0, -0.03, 0);
|
| 1236 |
+
} else {
|
| 1237 |
+
inv = translate4(inv, 0, 0, 0.1);
|
| 1238 |
+
}
|
| 1239 |
+
}
|
| 1240 |
+
if (activeKeys.includes("ArrowDown")) {
|
| 1241 |
+
if (shiftKey) {
|
| 1242 |
+
inv = translate4(inv, 0, 0.03, 0);
|
| 1243 |
+
} else {
|
| 1244 |
+
inv = translate4(inv, 0, 0, -0.1);
|
| 1245 |
+
}
|
| 1246 |
+
}
|
| 1247 |
+
if (activeKeys.includes("ArrowLeft"))
|
| 1248 |
+
inv = translate4(inv, -0.03, 0, 0);
|
| 1249 |
+
//
|
| 1250 |
+
if (activeKeys.includes("ArrowRight"))
|
| 1251 |
+
inv = translate4(inv, 0.03, 0, 0);
|
| 1252 |
+
// inv = rotate4(inv, 0.01, 0, 1, 0);
|
| 1253 |
+
if (activeKeys.includes("KeyA")) inv = rotate4(inv, -0.01, 0, 1, 0);
|
| 1254 |
+
if (activeKeys.includes("KeyD")) inv = rotate4(inv, 0.01, 0, 1, 0);
|
| 1255 |
+
if (activeKeys.includes("KeyQ")) inv = rotate4(inv, 0.01, 0, 0, 1);
|
| 1256 |
+
if (activeKeys.includes("KeyE")) inv = rotate4(inv, -0.01, 0, 0, 1);
|
| 1257 |
+
if (activeKeys.includes("KeyW")) inv = rotate4(inv, 0.005, 1, 0, 0);
|
| 1258 |
+
if (activeKeys.includes("KeyS")) inv = rotate4(inv, -0.005, 1, 0, 0);
|
| 1259 |
+
|
| 1260 |
+
const gamepads = navigator.getGamepads ? navigator.getGamepads() : [];
|
| 1261 |
+
let isJumping = activeKeys.includes("Space");
|
| 1262 |
+
for (let gamepad of gamepads) {
|
| 1263 |
+
if (!gamepad) continue;
|
| 1264 |
+
|
| 1265 |
+
const axisThreshold = 0.1; // Threshold to detect when the axis is intentionally moved
|
| 1266 |
+
const moveSpeed = 0.06;
|
| 1267 |
+
const rotateSpeed = 0.02;
|
| 1268 |
+
|
| 1269 |
+
// Assuming the left stick controls translation (axes 0 and 1)
|
| 1270 |
+
if (Math.abs(gamepad.axes[0]) > axisThreshold) {
|
| 1271 |
+
inv = translate4(inv, moveSpeed * gamepad.axes[0], 0, 0);
|
| 1272 |
+
carousel = false;
|
| 1273 |
+
}
|
| 1274 |
+
if (Math.abs(gamepad.axes[1]) > axisThreshold) {
|
| 1275 |
+
inv = translate4(inv, 0, 0, -moveSpeed * gamepad.axes[1]);
|
| 1276 |
+
carousel = false;
|
| 1277 |
+
}
|
| 1278 |
+
if (gamepad.buttons[12].pressed || gamepad.buttons[13].pressed) {
|
| 1279 |
+
inv = translate4(
|
| 1280 |
+
inv,
|
| 1281 |
+
0,
|
| 1282 |
+
-moveSpeed *
|
| 1283 |
+
(gamepad.buttons[12].pressed -
|
| 1284 |
+
gamepad.buttons[13].pressed),
|
| 1285 |
+
0,
|
| 1286 |
+
);
|
| 1287 |
+
carousel = false;
|
| 1288 |
+
}
|
| 1289 |
+
|
| 1290 |
+
if (gamepad.buttons[14].pressed || gamepad.buttons[15].pressed) {
|
| 1291 |
+
inv = translate4(
|
| 1292 |
+
inv,
|
| 1293 |
+
-moveSpeed *
|
| 1294 |
+
(gamepad.buttons[14].pressed -
|
| 1295 |
+
gamepad.buttons[15].pressed),
|
| 1296 |
+
0,
|
| 1297 |
+
0,
|
| 1298 |
+
);
|
| 1299 |
+
carousel = false;
|
| 1300 |
+
}
|
| 1301 |
+
|
| 1302 |
+
// Assuming the right stick controls rotation (axes 2 and 3)
|
| 1303 |
+
if (Math.abs(gamepad.axes[2]) > axisThreshold) {
|
| 1304 |
+
inv = rotate4(inv, rotateSpeed * gamepad.axes[2], 0, 1, 0);
|
| 1305 |
+
carousel = false;
|
| 1306 |
+
}
|
| 1307 |
+
if (Math.abs(gamepad.axes[3]) > axisThreshold) {
|
| 1308 |
+
inv = rotate4(inv, -rotateSpeed * gamepad.axes[3], 1, 0, 0);
|
| 1309 |
+
carousel = false;
|
| 1310 |
+
}
|
| 1311 |
+
|
| 1312 |
+
let tiltAxis = gamepad.buttons[6].value - gamepad.buttons[7].value;
|
| 1313 |
+
if (Math.abs(tiltAxis) > axisThreshold) {
|
| 1314 |
+
inv = rotate4(inv, rotateSpeed * tiltAxis, 0, 0, 1);
|
| 1315 |
+
carousel = false;
|
| 1316 |
+
}
|
| 1317 |
+
if (gamepad.buttons[4].pressed && !leftGamepadTrigger) {
|
| 1318 |
+
camera =
|
| 1319 |
+
cameras[(cameras.indexOf(camera) + 1) % cameras.length];
|
| 1320 |
+
inv = invert4(getViewMatrix(camera));
|
| 1321 |
+
carousel = false;
|
| 1322 |
+
}
|
| 1323 |
+
if (gamepad.buttons[5].pressed && !rightGamepadTrigger) {
|
| 1324 |
+
camera =
|
| 1325 |
+
cameras[
|
| 1326 |
+
(cameras.indexOf(camera) + cameras.length - 1) %
|
| 1327 |
+
cameras.length
|
| 1328 |
+
];
|
| 1329 |
+
inv = invert4(getViewMatrix(camera));
|
| 1330 |
+
carousel = false;
|
| 1331 |
+
}
|
| 1332 |
+
leftGamepadTrigger = gamepad.buttons[4].pressed;
|
| 1333 |
+
rightGamepadTrigger = gamepad.buttons[5].pressed;
|
| 1334 |
+
if (gamepad.buttons[0].pressed) {
|
| 1335 |
+
isJumping = true;
|
| 1336 |
+
carousel = false;
|
| 1337 |
+
}
|
| 1338 |
+
if (gamepad.buttons[3].pressed) {
|
| 1339 |
+
carousel = true;
|
| 1340 |
+
}
|
| 1341 |
+
}
|
| 1342 |
+
|
| 1343 |
+
if (
|
| 1344 |
+
["KeyJ", "KeyK", "KeyL", "KeyI"].some((k) => activeKeys.includes(k))
|
| 1345 |
+
) {
|
| 1346 |
+
let d = 4;
|
| 1347 |
+
inv = translate4(inv, 0, 0, d);
|
| 1348 |
+
inv = rotate4(
|
| 1349 |
+
inv,
|
| 1350 |
+
activeKeys.includes("KeyJ")
|
| 1351 |
+
? -0.05
|
| 1352 |
+
: activeKeys.includes("KeyL")
|
| 1353 |
+
? 0.05
|
| 1354 |
+
: 0,
|
| 1355 |
+
0,
|
| 1356 |
+
1,
|
| 1357 |
+
0,
|
| 1358 |
+
);
|
| 1359 |
+
inv = rotate4(
|
| 1360 |
+
inv,
|
| 1361 |
+
activeKeys.includes("KeyI")
|
| 1362 |
+
? 0.05
|
| 1363 |
+
: activeKeys.includes("KeyK")
|
| 1364 |
+
? -0.05
|
| 1365 |
+
: 0,
|
| 1366 |
+
1,
|
| 1367 |
+
0,
|
| 1368 |
+
0,
|
| 1369 |
+
);
|
| 1370 |
+
inv = translate4(inv, 0, 0, -d);
|
| 1371 |
+
}
|
| 1372 |
+
|
| 1373 |
+
viewMatrix = invert4(inv);
|
| 1374 |
+
|
| 1375 |
+
if (carousel) {
|
| 1376 |
+
// idol: a full turn about the vertical axis, about 12 s per turn, until the first touch (was a swing
|
| 1377 |
+
// left and right for the demo scenes)
|
| 1378 |
+
const azimuth =
|
| 1379 |
+
IDOL_START_AZIMUTH +
|
| 1380 |
+
(2 * Math.PI * Math.max(0, Date.now() - start)) / IDOL_TURN_MS;
|
| 1381 |
+
viewMatrix = idolViewMatrix(azimuth);
|
| 1382 |
+
}
|
| 1383 |
+
|
| 1384 |
+
if (isJumping) {
|
| 1385 |
+
jumpDelta = Math.min(1, jumpDelta + 0.05);
|
| 1386 |
+
} else {
|
| 1387 |
+
jumpDelta = Math.max(0, jumpDelta - 0.05);
|
| 1388 |
+
}
|
| 1389 |
+
|
| 1390 |
+
let inv2 = invert4(viewMatrix);
|
| 1391 |
+
inv2 = translate4(inv2, 0, -jumpDelta, 0);
|
| 1392 |
+
inv2 = rotate4(inv2, -0.1 * jumpDelta, 1, 0, 0);
|
| 1393 |
+
let actualViewMatrix = invert4(inv2);
|
| 1394 |
+
|
| 1395 |
+
const viewProj = multiply4(projectionMatrix, actualViewMatrix);
|
| 1396 |
+
worker.postMessage({ view: viewProj });
|
| 1397 |
+
|
| 1398 |
+
const currentFps = 1000 / (now - lastFrame) || 0;
|
| 1399 |
+
avgFps = avgFps * 0.9 + currentFps * 0.1;
|
| 1400 |
+
|
| 1401 |
+
if (vertexCount > 0) {
|
| 1402 |
+
document.getElementById("spinner").style.display = "none";
|
| 1403 |
+
gl.uniformMatrix4fv(u_view, false, actualViewMatrix);
|
| 1404 |
+
gl.clear(gl.COLOR_BUFFER_BIT);
|
| 1405 |
+
gl.drawArraysInstanced(gl.TRIANGLE_FAN, 0, 4, vertexCount);
|
| 1406 |
+
} else {
|
| 1407 |
+
gl.clear(gl.COLOR_BUFFER_BIT);
|
| 1408 |
+
document.getElementById("spinner").style.display = "";
|
| 1409 |
+
start = Date.now() + 2000;
|
| 1410 |
+
}
|
| 1411 |
+
const progress = (100 * vertexCount) / (splatData.length / rowLength);
|
| 1412 |
+
if (progress < 100) {
|
| 1413 |
+
document.getElementById("progress").style.width = progress + "%";
|
| 1414 |
+
} else {
|
| 1415 |
+
document.getElementById("progress").style.display = "none";
|
| 1416 |
+
}
|
| 1417 |
+
fps.innerText = Math.round(avgFps) + " fps";
|
| 1418 |
+
if (isNaN(currentCameraIndex)) {
|
| 1419 |
+
camid.innerText = "";
|
| 1420 |
+
}
|
| 1421 |
+
lastFrame = now;
|
| 1422 |
+
requestAnimationFrame(frame);
|
| 1423 |
+
};
|
| 1424 |
+
|
| 1425 |
+
frame();
|
| 1426 |
+
|
| 1427 |
+
const isPly = (splatData) =>
|
| 1428 |
+
splatData[0] == 112 &&
|
| 1429 |
+
splatData[1] == 108 &&
|
| 1430 |
+
splatData[2] == 121 &&
|
| 1431 |
+
splatData[3] == 10;
|
| 1432 |
+
|
| 1433 |
+
const selectFile = (file) => {
|
| 1434 |
+
const fr = new FileReader();
|
| 1435 |
+
if (/\.json$/i.test(file.name)) {
|
| 1436 |
+
fr.onload = () => {
|
| 1437 |
+
cameras = JSON.parse(fr.result);
|
| 1438 |
+
viewMatrix = getViewMatrix(cameras[0]);
|
| 1439 |
+
projectionMatrix = getProjectionMatrix(
|
| 1440 |
+
camera.fx / downsample,
|
| 1441 |
+
camera.fy / downsample,
|
| 1442 |
+
canvas.width,
|
| 1443 |
+
canvas.height,
|
| 1444 |
+
);
|
| 1445 |
+
gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
|
| 1446 |
+
|
| 1447 |
+
console.log("Loaded Cameras");
|
| 1448 |
+
};
|
| 1449 |
+
fr.readAsText(file);
|
| 1450 |
+
} else {
|
| 1451 |
+
stopLoading = true;
|
| 1452 |
+
fr.onload = () => {
|
| 1453 |
+
splatData = new Uint8Array(fr.result);
|
| 1454 |
+
console.log("Loaded", Math.floor(splatData.length / rowLength));
|
| 1455 |
+
|
| 1456 |
+
if (isPly(splatData)) {
|
| 1457 |
+
// ply file magic header means it should be handled differently
|
| 1458 |
+
worker.postMessage({ ply: splatData.buffer, save: true });
|
| 1459 |
+
} else {
|
| 1460 |
+
worker.postMessage({
|
| 1461 |
+
buffer: splatData.buffer,
|
| 1462 |
+
vertexCount: Math.floor(splatData.length / rowLength),
|
| 1463 |
+
});
|
| 1464 |
+
}
|
| 1465 |
+
};
|
| 1466 |
+
fr.readAsArrayBuffer(file);
|
| 1467 |
+
}
|
| 1468 |
+
};
|
| 1469 |
+
|
| 1470 |
+
window.addEventListener("hashchange", (e) => {
|
| 1471 |
+
try {
|
| 1472 |
+
viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
|
| 1473 |
+
carousel = false;
|
| 1474 |
+
} catch (err) {}
|
| 1475 |
+
});
|
| 1476 |
+
|
| 1477 |
+
const preventDefault = (e) => {
|
| 1478 |
+
e.preventDefault();
|
| 1479 |
+
e.stopPropagation();
|
| 1480 |
+
};
|
| 1481 |
+
document.addEventListener("dragenter", preventDefault);
|
| 1482 |
+
document.addEventListener("dragover", preventDefault);
|
| 1483 |
+
document.addEventListener("dragleave", preventDefault);
|
| 1484 |
+
document.addEventListener("drop", (e) => {
|
| 1485 |
+
e.preventDefault();
|
| 1486 |
+
e.stopPropagation();
|
| 1487 |
+
selectFile(e.dataTransfer.files[0]);
|
| 1488 |
+
});
|
| 1489 |
+
|
| 1490 |
+
let bytesRead = 0;
|
| 1491 |
+
let lastVertexCount = -1;
|
| 1492 |
+
let stopLoading = false;
|
| 1493 |
+
|
| 1494 |
+
while (true) {
|
| 1495 |
+
const { done, value } = await reader.read();
|
| 1496 |
+
if (done || stopLoading) break;
|
| 1497 |
+
|
| 1498 |
+
splatData.set(value, bytesRead);
|
| 1499 |
+
bytesRead += value.length;
|
| 1500 |
+
|
| 1501 |
+
if (vertexCount > lastVertexCount) {
|
| 1502 |
+
if (!isPly(splatData)) {
|
| 1503 |
+
worker.postMessage({
|
| 1504 |
+
buffer: splatData.buffer,
|
| 1505 |
+
vertexCount: Math.floor(bytesRead / rowLength),
|
| 1506 |
+
});
|
| 1507 |
+
}
|
| 1508 |
+
lastVertexCount = vertexCount;
|
| 1509 |
+
}
|
| 1510 |
+
}
|
| 1511 |
+
if (!stopLoading) {
|
| 1512 |
+
if (isPly(splatData)) {
|
| 1513 |
+
// ply file magic header means it should be handled differently
|
| 1514 |
+
worker.postMessage({ ply: splatData.buffer, save: false });
|
| 1515 |
+
} else {
|
| 1516 |
+
worker.postMessage({
|
| 1517 |
+
buffer: splatData.buffer,
|
| 1518 |
+
vertexCount: Math.floor(bytesRead / rowLength),
|
| 1519 |
+
});
|
| 1520 |
+
}
|
| 1521 |
+
}
|
| 1522 |
+
}
|
| 1523 |
+
|
| 1524 |
+
main().catch((err) => {
|
| 1525 |
+
document.getElementById("spinner").style.display = "none";
|
| 1526 |
+
document.getElementById("message").innerText = err.toString();
|
| 1527 |
+
});
|
android/app/src/main/java/com/idol/AutoRun.kt
ADDED
|
@@ -0,0 +1,312 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.os.Build
|
| 4 |
+
import android.os.Process
|
| 5 |
+
import android.os.SystemClock
|
| 6 |
+
import android.util.Log
|
| 7 |
+
import com.google.ai.edge.litert.CompiledModel
|
| 8 |
+
import org.json.JSONArray
|
| 9 |
+
import org.json.JSONObject
|
| 10 |
+
import java.io.File
|
| 11 |
+
|
| 12 |
+
/**
|
| 13 |
+
* Debug builds only: runs a plan of rows through the app and writes what each produced, for the device lines of
|
| 14 |
+
* CONTRACT.md ((a) parity, (b) the .ply, (c) speed, (d) failure paths, (e) memory).
|
| 15 |
+
*
|
| 16 |
+
* adb shell am start -n com.idol/.MainActivity --es autorun plans/<plan>.json --es out runs/<name>
|
| 17 |
+
*
|
| 18 |
+
* Paths are under the app's files dir. Plan: {"preload": true, "reserve_mb": 2500, "mem_check": true,
|
| 19 |
+
* "handoff": "host" | "direct", "matting": "bir" | "dis", "cpu_threads": 6, "gpu_priority": "default", "rows": [...]}
|
| 20 |
+
* (every key optional; a row may override "preload"). Rows:
|
| 21 |
+
* {"id", "kind": "e2e", "fixture": "fx/<tag>", "runs": 4, "preload": true}: the app's whole pipeline from the
|
| 22 |
+
* fixture's raw image and mode with its noise arrays; <id>.<run>.ply + <id>.<run>.json per run (run 0 is the
|
| 23 |
+
* warm-up), <id>.alpha.u8 once (raw uint8 matting alpha), <id>.json for the row.
|
| 24 |
+
* {"id", "kind": "parity", "fixture": "fx/<tag>"}: each part on the fixture's reference input, <id>.<name>.f32.
|
| 25 |
+
* {"id", "kind": "image", "image": "<path>", "mode": "real"}: a normal run of a picture (failure paths);
|
| 26 |
+
* <id>.json with the outcome and the message on the screen.
|
| 27 |
+
* {"id", "kind": "snippet", "fixture": "fx/<tag>"}: the model card's Kotlin block (CardSnippet.kt) on the
|
| 28 |
+
* fixture's picture; its line goes to <id>.json, DONE.json and logcat.
|
| 29 |
+
* Output: <out>/header.json (device, options, the load and compile seconds of each preload), the row files,
|
| 30 |
+
* <out>/DONE.json; logcat tag IdolAutoRun: START / ROW / DONE lines. A file <out>/STOP ends the plan after the
|
| 31 |
+
* current run.
|
| 32 |
+
*/
|
| 33 |
+
class AutoRun(private val activity: MainActivity, private val planFile: File, private val outDir: File) {
|
| 34 |
+
private val header = JSONObject()
|
| 35 |
+
private val loads = JSONArray()
|
| 36 |
+
|
| 37 |
+
fun start() {
|
| 38 |
+
Thread({
|
| 39 |
+
try {
|
| 40 |
+
runAll()
|
| 41 |
+
} catch (e: Throwable) {
|
| 42 |
+
Log.e(TAG, "IDOL_AUTORUN FAILED", e)
|
| 43 |
+
runCatching {
|
| 44 |
+
outDir.mkdirs()
|
| 45 |
+
File(outDir, "DONE.json").writeText(
|
| 46 |
+
JSONObject().put("status", "FAILED").put("error", e.toString()).toString(1),
|
| 47 |
+
)
|
| 48 |
+
}
|
| 49 |
+
Log.i(TAG, "IDOL_AUTORUN DONE status=FAILED pid=${Process.myPid()}")
|
| 50 |
+
} finally {
|
| 51 |
+
activity.autorunFinished()
|
| 52 |
+
}
|
| 53 |
+
}, "IdolAutoRun").start()
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
private fun stopped() = File(outDir, "STOP").exists()
|
| 57 |
+
|
| 58 |
+
private fun writeHeader() = File(outDir, "header.json").writeText(header.toString(1))
|
| 59 |
+
|
| 60 |
+
private fun runAll() {
|
| 61 |
+
val plan = JSONObject(planFile.readText())
|
| 62 |
+
outDir.mkdirs()
|
| 63 |
+
File(outDir, "DONE.json").delete()
|
| 64 |
+
header.put("plan", planFile.name).put("plan_json", plan)
|
| 65 |
+
.put("device", DeviceInfo.displayName()).put("model", Build.MODEL).put("android_sdk", Build.VERSION.SDK_INT)
|
| 66 |
+
.put("fingerprint", Build.FINGERPRINT).put("litert", BuildConfig.LITERT_VERSION)
|
| 67 |
+
.put("app_version", BuildConfig.VERSION_NAME).put("pid", Process.myPid())
|
| 68 |
+
.put("mem_total_kb", DeviceInfo.memTotalKb()).put("mem_available_kb", DeviceInfo.memAvailableKb())
|
| 69 |
+
.put("thermal", DeviceInfo.thermalStatus(activity)).put("cpuset", DeviceInfo.cpuset())
|
| 70 |
+
.put("start_ms", System.currentTimeMillis()).put("load", loads)
|
| 71 |
+
writeHeader()
|
| 72 |
+
Log.i(TAG, "IDOL_AUTORUN START plan=${planFile.name} out=${outDir.name} pid=${Process.myPid()}")
|
| 73 |
+
val rows = plan.getJSONArray("rows")
|
| 74 |
+
val summary = JSONArray()
|
| 75 |
+
val done = JSONObject().put("status", "DONE")
|
| 76 |
+
for (r in 0 until rows.length()) {
|
| 77 |
+
if (stopped()) {
|
| 78 |
+
done.put("stopped", true)
|
| 79 |
+
break
|
| 80 |
+
}
|
| 81 |
+
val row = rows.getJSONObject(r)
|
| 82 |
+
val id = row.getString("id")
|
| 83 |
+
val kind = row.optString("kind", "e2e")
|
| 84 |
+
Log.i(TAG, "IDOL_AUTORUN ROW_START id=$id kind=$kind")
|
| 85 |
+
val rec = try {
|
| 86 |
+
when (kind) {
|
| 87 |
+
"e2e" -> e2e(plan, row)
|
| 88 |
+
"parity" -> parity(plan, row)
|
| 89 |
+
"image" -> image(plan, row)
|
| 90 |
+
"snippet" -> snippet(row)
|
| 91 |
+
else -> JSONObject().put("id", id).put("kind", kind).put("status", "unknown_kind")
|
| 92 |
+
}
|
| 93 |
+
} catch (e: Throwable) {
|
| 94 |
+
Log.e(TAG, "row $id failed", e)
|
| 95 |
+
JSONObject().put("id", id).put("kind", kind).put("status", "failed").put("error", e.toString())
|
| 96 |
+
}
|
| 97 |
+
summary.put(rec)
|
| 98 |
+
if (kind == "snippet") done.put("snippet", rec.optString("line"))
|
| 99 |
+
Log.i(TAG, "IDOL_AUTORUN ROW id=$id kind=$kind status=${rec.optString("status")} ${rec.optString("summary")}")
|
| 100 |
+
}
|
| 101 |
+
done.put("rows", summary).put("pid", Process.myPid()).put("end_ms", System.currentTimeMillis())
|
| 102 |
+
.put("self_status_kb", JSONObject(DeviceInfo.selfStatusKb() as Map<*, *>))
|
| 103 |
+
File(outDir, "DONE.json").writeText(done.toString(1))
|
| 104 |
+
Log.i(TAG, "IDOL_AUTORUN DONE rows=${summary.length()} pid=${Process.myPid()}")
|
| 105 |
+
}
|
| 106 |
+
|
| 107 |
+
/** The engine options of [row] in [plan] (the row's keys first). */
|
| 108 |
+
private fun options(plan: JSONObject, row: JSONObject): EngineOptions {
|
| 109 |
+
fun has(k: String) = row.has(k) || plan.has(k)
|
| 110 |
+
fun src(k: String) = if (row.has(k)) row else plan
|
| 111 |
+
val defaults = EngineOptions()
|
| 112 |
+
val manifestMatting = ModelFiles.manifest(ModelFiles.dir(activity.filesDir))?.defaultMatting ?: "bir"
|
| 113 |
+
val priority = when (if (has("gpu_priority")) src("gpu_priority").getString("gpu_priority") else "default") {
|
| 114 |
+
"low" -> CompiledModel.GpuOptions.Priority.LOW
|
| 115 |
+
"normal" -> CompiledModel.GpuOptions.Priority.NORMAL
|
| 116 |
+
"high" -> CompiledModel.GpuOptions.Priority.HIGH
|
| 117 |
+
else -> null
|
| 118 |
+
}
|
| 119 |
+
return EngineOptions(
|
| 120 |
+
preload = if (has("preload")) src("preload").getBoolean("preload") else defaults.preload,
|
| 121 |
+
reserveMb = if (has("reserve_mb")) src("reserve_mb").getLong("reserve_mb") else defaults.reserveMb,
|
| 122 |
+
memoryCheck = if (has("mem_check")) src("mem_check").getBoolean("mem_check") else defaults.memoryCheck,
|
| 123 |
+
directHandoff = has("handoff") && src("handoff").getString("handoff") == "direct",
|
| 124 |
+
matting = if (has("matting")) src("matting").getString("matting") else manifestMatting,
|
| 125 |
+
cpuThreads = if (has("cpu_threads")) src("cpu_threads").getInt("cpu_threads") else defaults.cpuThreads,
|
| 126 |
+
gpuPriority = priority,
|
| 127 |
+
disFloor = if (has("dis_floor")) src("dis_floor").getInt("dis_floor") else defaults.disFloor,
|
| 128 |
+
)
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
private fun json(o: EngineOptions) = JSONObject().put("preload", o.preload).put("reserve_mb", o.reserveMb)
|
| 132 |
+
.put("mem_check", o.memoryCheck).put("handoff", if (o.directHandoff) "direct" else "host")
|
| 133 |
+
.put("matting", o.matting).put("cpu_threads", o.cpuThreads)
|
| 134 |
+
.put("gpu_priority", o.gpuPriority?.name ?: "default").put("dis_floor", o.disFloor)
|
| 135 |
+
|
| 136 |
+
private fun outcomeFields(rec: JSONObject, outcome: Outcome) {
|
| 137 |
+
when (outcome) {
|
| 138 |
+
is Outcome.Done -> rec.put("status", "done")
|
| 139 |
+
is Outcome.NoObject -> rec.put("status", "no_object").put("reason", outcome.reason)
|
| 140 |
+
is Outcome.Missing -> rec.put("status", "missing_models").put("message", ModelFiles.message(outcome.problems))
|
| 141 |
+
is Outcome.Failed -> rec.put("status", outcome.kind).put("message", outcome.message)
|
| 142 |
+
}
|
| 143 |
+
}
|
| 144 |
+
|
| 145 |
+
private fun e2e(plan: JSONObject, row: JSONObject): JSONObject {
|
| 146 |
+
val id = row.getString("id")
|
| 147 |
+
val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
|
| 148 |
+
val runs = row.optInt("runs", 4)
|
| 149 |
+
val opts = options(plan, row)
|
| 150 |
+
val rowRec = JSONObject().put("id", id).put("kind", "e2e").put("fixture", row.getString("fixture"))
|
| 151 |
+
.put("mode", fx.mode).put("runs", runs).put("options", json(opts))
|
| 152 |
+
val problems = activity.ensureEngine(opts)
|
| 153 |
+
if (problems.isNotEmpty()) {
|
| 154 |
+
return rowRec.put("status", "missing_models").put("message", ModelFiles.message(problems))
|
| 155 |
+
}
|
| 156 |
+
if (opts.preload) {
|
| 157 |
+
val load = activity.onModel { requireNotNull(Idol.engine).preload(activity.progress) }
|
| 158 |
+
loads.put(JSONObject().put("row", id).put("record", load))
|
| 159 |
+
writeHeader()
|
| 160 |
+
rowRec.put("load_total_s", load.optDouble("total_s")).put("load_loaded", load.optInt("loaded"))
|
| 161 |
+
.put("load_of", load.optInt("of"))
|
| 162 |
+
}
|
| 163 |
+
val photo = Photo.fromFile(fx.image)
|
| 164 |
+
val noise = fx.noise()
|
| 165 |
+
val statuses = JSONArray()
|
| 166 |
+
val measured = ArrayList<Double>()
|
| 167 |
+
var alphaWritten = false
|
| 168 |
+
for (k in 0 until runs) {
|
| 169 |
+
if (stopped()) {
|
| 170 |
+
rowRec.put("stopped_after", k)
|
| 171 |
+
break
|
| 172 |
+
}
|
| 173 |
+
val ply = File(outDir, "$id.$k.ply")
|
| 174 |
+
val rec = JSONObject().put("id", id).put("run", k).put("warmup", k == 0)
|
| 175 |
+
.put("fixture", row.getString("fixture")).put("mode", fx.mode).put("noise", noise.source)
|
| 176 |
+
.put("options", json(opts))
|
| 177 |
+
.put("thermal_start", DeviceInfo.thermalStatus(activity))
|
| 178 |
+
.put("mem_available_kb_start", DeviceInfo.memAvailableKb())
|
| 179 |
+
.put("vmhwm_reset", DeviceInfo.resetPeakRss())
|
| 180 |
+
.put("cpuset", DeviceInfo.cpuset())
|
| 181 |
+
val sampler = GpuPageSampler().also { it.start() }
|
| 182 |
+
val t0 = SystemClock.elapsedRealtimeNanos()
|
| 183 |
+
val outcome = activity.runAndWait(photo, fx.mode, noise, ply, showResult = false)
|
| 184 |
+
val wall = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
|
| 185 |
+
val gpuMax = sampler.stop()
|
| 186 |
+
val st = DeviceInfo.selfStatusKb()
|
| 187 |
+
rec.put("wall_s", wall).put("thermal_end", DeviceInfo.thermalStatus(activity))
|
| 188 |
+
.put("vmhwm_kb", st["VmHWM"] ?: -1L).put("vmrss_kb", st["VmRSS"] ?: -1L)
|
| 189 |
+
.put("vmswap_kb", st["VmSwap"] ?: -1L).put("gpu_page_alloc_max_bytes", gpuMax)
|
| 190 |
+
.put("mem_available_kb_end", DeviceInfo.memAvailableKb())
|
| 191 |
+
outcomeFields(rec, outcome)
|
| 192 |
+
if (outcome is Outcome.Done) {
|
| 193 |
+
val r = outcome.result
|
| 194 |
+
val stages = JSONObject()
|
| 195 |
+
for ((key, v) in r.times) if (key != "generate" && key != "all") stages.put(key, v)
|
| 196 |
+
rec.put("stages", stages).put("generate", r.times["generate"]).put("total", r.times["all"])
|
| 197 |
+
.put("ply", ply.name).put("ply_bytes", ply.length()).put("gaussians", r.gaussians)
|
| 198 |
+
.put("box", JSONArray(r.pre.box.toList()))
|
| 199 |
+
.put("alpha_size", JSONArray(listOf(r.pre.alphaWidth, r.pre.alphaHeight)))
|
| 200 |
+
.put("own_alpha", r.pre.ownAlpha).put("handoff", r.handoff)
|
| 201 |
+
.put("compiles_in_run", JSONObject().put("count", r.compiles.size)
|
| 202 |
+
.put("seconds", r.compiles.sumOf { it.seconds })
|
| 203 |
+
.put("graphs", JSONArray(r.compiles.map { it.json() })))
|
| 204 |
+
.put("memory_events", JSONArray(r.events))
|
| 205 |
+
if (!alphaWritten) {
|
| 206 |
+
File(outDir, "$id.alpha.u8").writeBytes(r.pre.alphaRaw)
|
| 207 |
+
rowRec.put("alpha", JSONObject().put("file", "$id.alpha.u8").put("width", r.pre.alphaWidth)
|
| 208 |
+
.put("height", r.pre.alphaHeight).put("box", JSONArray(r.pre.box.toList())))
|
| 209 |
+
alphaWritten = true
|
| 210 |
+
}
|
| 211 |
+
if (k > 0) r.times["generate"]?.let { measured.add(it) }
|
| 212 |
+
}
|
| 213 |
+
File(outDir, "$id.$k.json").writeText(rec.toString(1))
|
| 214 |
+
statuses.put(rec.optString("status"))
|
| 215 |
+
Log.i(TAG, "IDOL_AUTORUN RUN id=$id run=$k warmup=${k == 0} status=${rec.optString("status")} " +
|
| 216 |
+
"generate_s=${rec.optDouble("generate", -1.0)} vmhwm_kb=${rec.optLong("vmhwm_kb")}")
|
| 217 |
+
}
|
| 218 |
+
val allDone = (0 until statuses.length()).all { statuses.getString(it) == "done" }
|
| 219 |
+
rowRec.put("statuses", statuses).put("status", if (allDone && statuses.length() > 0) "done" else "incomplete")
|
| 220 |
+
if (measured.isNotEmpty()) {
|
| 221 |
+
val sorted = measured.sorted()
|
| 222 |
+
val median = if (sorted.size % 2 == 1) sorted[sorted.size / 2]
|
| 223 |
+
else (sorted[sorted.size / 2 - 1] + sorted[sorted.size / 2]) / 2
|
| 224 |
+
rowRec.put("generate_measured_s", JSONArray(measured)).put("generate_median_s", median)
|
| 225 |
+
.put("summary", "generate median %.2f s over %d measured runs".format(median, measured.size))
|
| 226 |
+
}
|
| 227 |
+
File(outDir, "$id.json").writeText(rowRec.toString(1))
|
| 228 |
+
return rowRec
|
| 229 |
+
}
|
| 230 |
+
|
| 231 |
+
private fun parity(plan: JSONObject, row: JSONObject): JSONObject {
|
| 232 |
+
val id = row.getString("id")
|
| 233 |
+
val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
|
| 234 |
+
val opts = options(plan, row).copy(preload = false)
|
| 235 |
+
val problems = activity.ensureEngine(opts)
|
| 236 |
+
if (problems.isNotEmpty()) {
|
| 237 |
+
return JSONObject().put("id", id).put("kind", "parity").put("status", "missing_models")
|
| 238 |
+
.put("message", ModelFiles.message(problems))
|
| 239 |
+
}
|
| 240 |
+
activity.showRunScreen("Part agreement: ${fx.dir.name}")
|
| 241 |
+
val t0 = SystemClock.elapsedRealtimeNanos()
|
| 242 |
+
val rec = activity.onModel { requireNotNull(Idol.engine).parity(fx, outDir, id, activity.progress) }
|
| 243 |
+
rec.put("kind", "parity").put("status", "done").put("options", json(opts))
|
| 244 |
+
.put("wall_s", (SystemClock.elapsedRealtimeNanos() - t0) / 1e9)
|
| 245 |
+
.put("summary", "${rec.getJSONObject("arrays").length()} arrays")
|
| 246 |
+
File(outDir, "$id.json").writeText(rec.toString(1))
|
| 247 |
+
return rec
|
| 248 |
+
}
|
| 249 |
+
|
| 250 |
+
private fun image(plan: JSONObject, row: JSONObject): JSONObject {
|
| 251 |
+
val id = row.getString("id")
|
| 252 |
+
val path = row.getString("image")
|
| 253 |
+
val mode = row.optString("mode", "real")
|
| 254 |
+
val rec = JSONObject().put("id", id).put("kind", "image").put("image", path).put("mode", mode)
|
| 255 |
+
activity.ensureEngine(options(plan, row))
|
| 256 |
+
val photo = try {
|
| 257 |
+
Photo.fromFile(File(activity.filesDir, path))
|
| 258 |
+
} catch (e: Throwable) {
|
| 259 |
+
return rec.put("status", "decode_failed").put("error", e.toString())
|
| 260 |
+
}
|
| 261 |
+
val outcome = activity.runAndWait(
|
| 262 |
+
photo, mode, Noise.random(MainActivity.DEFAULT_SEED), File(outDir, "$id.ply"),
|
| 263 |
+
showResult = row.optBoolean("show_result", false),
|
| 264 |
+
)
|
| 265 |
+
outcomeFields(rec, outcome)
|
| 266 |
+
if (outcome is Outcome.Done) rec.put("generate", outcome.result.times["generate"])
|
| 267 |
+
// the outcome is on the screen once the UI thread has run the posts before this one
|
| 268 |
+
rec.put("message_on_screen", activity.homeMessageText()).put("pid", Process.myPid())
|
| 269 |
+
rec.put("summary", rec.optString("message_on_screen").lineSequence().firstOrNull() ?: "")
|
| 270 |
+
File(outDir, "$id.json").writeText(rec.toString(1))
|
| 271 |
+
return rec
|
| 272 |
+
}
|
| 273 |
+
|
| 274 |
+
private fun snippet(row: JSONObject): JSONObject {
|
| 275 |
+
val id = row.getString("id")
|
| 276 |
+
val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
|
| 277 |
+
val photo = Photo.fromFile(fx.image)
|
| 278 |
+
val rgb = Preprocess.resized(photo.image).rgb()
|
| 279 |
+
val image = Matting.birefnetInput(rgb)
|
| 280 |
+
val dir = ModelFiles.dir(activity.filesDir)
|
| 281 |
+
val t0 = SystemClock.elapsedRealtimeNanos()
|
| 282 |
+
val line = activity.onModel {
|
| 283 |
+
// the card's block builds its own environment and graph; free the engine's graphs first
|
| 284 |
+
Idol.engine?.pool?.releaseAll()
|
| 285 |
+
val bir = cardSnippet(dir, image)
|
| 286 |
+
if (!fx.has("pixels") || !fx.has("feature1")) return@onModel bir
|
| 287 |
+
// the DINOv3 card's block on the fixture's preprocessed picture, against the reference's feature1
|
| 288 |
+
val got = dinoCardSnippet(dir, fx.floats("pixels"))
|
| 289 |
+
val want = fx.floats("feature1")
|
| 290 |
+
var dot = 0.0
|
| 291 |
+
var ng = 0.0
|
| 292 |
+
var nw = 0.0
|
| 293 |
+
for (i in want.indices) {
|
| 294 |
+
dot += got[i].toDouble() * want[i]
|
| 295 |
+
ng += got[i].toDouble() * got[i]
|
| 296 |
+
nw += want[i].toDouble() * want[i]
|
| 297 |
+
}
|
| 298 |
+
"$bir | dino card snippet: ${got.size} values, cos vs the reference feature1 %.6f"
|
| 299 |
+
.format(dot / Math.sqrt(ng * nw))
|
| 300 |
+
}
|
| 301 |
+
Log.i(TAG, "IDOL_AUTORUN SNIPPET $line")
|
| 302 |
+
val rec = JSONObject().put("id", id).put("kind", "snippet").put("status", "done").put("line", line)
|
| 303 |
+
.put("picture", JSONArray(listOf(rgb.width, rgb.height)))
|
| 304 |
+
.put("seconds", (SystemClock.elapsedRealtimeNanos() - t0) / 1e9).put("summary", line)
|
| 305 |
+
File(outDir, "$id.json").writeText(rec.toString(1))
|
| 306 |
+
return rec
|
| 307 |
+
}
|
| 308 |
+
|
| 309 |
+
companion object {
|
| 310 |
+
const val TAG = "IdolAutoRun"
|
| 311 |
+
}
|
| 312 |
+
}
|
android/app/src/main/java/com/idol/CardSnippet.kt
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import com.google.ai.edge.litert.Accelerator
|
| 4 |
+
import com.google.ai.edge.litert.CompiledModel
|
| 5 |
+
import com.google.ai.edge.litert.Environment
|
| 6 |
+
import java.io.File
|
| 7 |
+
|
| 8 |
+
/**
|
| 9 |
+
* The model card's Kotlin block, run on the phone as written (autorun plan plans/snippet.json). [dir] is the app's
|
| 10 |
+
* model folder; [image] is the card's `image`: Matting.birefnetInput(the picture resized to a short side of 1024),
|
| 11 |
+
* [1, 3, 1024, 1024]. Returns one line about the alpha it produced.
|
| 12 |
+
*/
|
| 13 |
+
fun cardSnippet(dir: File, image: FloatArray): String {
|
| 14 |
+
// ---- the card's block, verbatim
|
| 15 |
+
val env = Environment.create()
|
| 16 |
+
val cpu = CompiledModel.Options(Accelerator.CPU) // bir.tflite: BiRefNet, the one CPU graph
|
| 17 |
+
val bir = CompiledModel.create(File(dir, "bir.tflite").absolutePath, cpu, env)
|
| 18 |
+
val inputs = bir.createInputBuffers()
|
| 19 |
+
val outputs = bir.createOutputBuffers()
|
| 20 |
+
inputs[0].writeFloat(image) // [1, 3, 1024, 1024]: RGB / 255, ImageNet mean / std
|
| 21 |
+
bir.run(inputs, outputs)
|
| 22 |
+
val alpha = outputs[0].readFloat() // [1024 * 1024] sigmoid map, 1 = the object
|
| 23 |
+
bir.close()
|
| 24 |
+
val gpu = CompiledModel.Options(Accelerator.GPU).apply { // every other graph
|
| 25 |
+
gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM)
|
| 26 |
+
}
|
| 27 |
+
// ---- end of the card's block
|
| 28 |
+
for (b in inputs + outputs) runCatching { b.close() }
|
| 29 |
+
runCatching { env.close() }
|
| 30 |
+
var mn = Float.POSITIVE_INFINITY
|
| 31 |
+
var mx = Float.NEGATIVE_INFINITY
|
| 32 |
+
var sum = 0.0
|
| 33 |
+
var above = 0
|
| 34 |
+
for (v in alpha) {
|
| 35 |
+
if (v < mn) mn = v
|
| 36 |
+
if (v > mx) mx = v
|
| 37 |
+
sum += v
|
| 38 |
+
if (v > 0.5f) above++
|
| 39 |
+
}
|
| 40 |
+
return "card snippet: alpha ${alpha.size} values, min %.4f max %.4f mean %.4f, fraction > 0.5 %.4f; gpu options built (%s)"
|
| 41 |
+
.format(mn, mx, sum / alpha.size, above.toDouble() / alpha.size, gpu.gpuOptions?.precision)
|
| 42 |
+
}
|
| 43 |
+
|
| 44 |
+
/**
|
| 45 |
+
* The DINOv3 repo card's Kotlin block, run on the phone as written (autorun plan plans/snippet.json, after
|
| 46 |
+
* [cardSnippet]). [dir] holds dino_0..3.tflite and the RoPE tables; [pixels] is the card's `pixels`: the picture Idol
|
| 47 |
+
* feeds, [1, 3, 1024, 1024] RGB in [0, 1]. Returns the feature (for the check against the fixture's feature1).
|
| 48 |
+
*/
|
| 49 |
+
fun dinoCardSnippet(dir: File, pixels: FloatArray): FloatArray {
|
| 50 |
+
// ---- the DINOv3 card's block, verbatim
|
| 51 |
+
fun floats(f: File): FloatArray {
|
| 52 |
+
val b = java.nio.ByteBuffer.wrap(f.readBytes()).order(java.nio.ByteOrder.LITTLE_ENDIAN).asFloatBuffer()
|
| 53 |
+
return FloatArray(b.remaining()).also { b.get(it) }
|
| 54 |
+
}
|
| 55 |
+
val env = Environment.create()
|
| 56 |
+
val gpu = CompiledModel.Options(Accelerator.GPU).apply {
|
| 57 |
+
gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM)
|
| 58 |
+
}
|
| 59 |
+
val rope = mapOf("args_1" to floats(File(dir, "dino_rope_cos.f32")), "args_2" to floats(File(dir, "dino_rope_sin.f32")))
|
| 60 |
+
var x = pixels // [1, 3, 1024, 1024] RGB in [0, 1]
|
| 61 |
+
for (i in 0..3) { // 8 layers per graph
|
| 62 |
+
val model = CompiledModel.create(File(dir, "dino_$i.tflite").absolutePath, gpu, env)
|
| 63 |
+
val inputs = listOf("args_0", "args_1", "args_2").associateWith { model.createInputBuffer(it, "serving_default") }
|
| 64 |
+
val output = model.createOutputBuffer("output_0", "serving_default")
|
| 65 |
+
for ((name, buffer) in inputs) buffer.writeFloat(if (name == "args_0") x else rope.getValue(name))
|
| 66 |
+
model.run(inputs, mapOf("output_0" to output), "serving_default")
|
| 67 |
+
x = output.readFloat() // [1, 4101, 1280]
|
| 68 |
+
for (b in inputs.values + output) b.close()
|
| 69 |
+
model.close()
|
| 70 |
+
}
|
| 71 |
+
env.close()
|
| 72 |
+
// x: the image feature Idol's denoiser reads (1 class, 4 register and 64 x 64 patch tokens)
|
| 73 |
+
// ---- end of the DINOv3 card's block
|
| 74 |
+
return x
|
| 75 |
+
}
|
android/app/src/main/java/com/idol/DeviceInfo.kt
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.content.Context
|
| 4 |
+
import android.os.Build
|
| 5 |
+
import android.os.PowerManager
|
| 6 |
+
import java.io.File
|
| 7 |
+
|
| 8 |
+
/** Device facts for the status line and the run JSON. */
|
| 9 |
+
object DeviceInfo {
|
| 10 |
+
/** `MemAvailable` of `/proc/meminfo` in kB (what the memory guard compares with), or -1. */
|
| 11 |
+
fun memAvailableKb(): Long = meminfo("MemAvailable:")
|
| 12 |
+
|
| 13 |
+
fun memTotalKb(): Long = meminfo("MemTotal:")
|
| 14 |
+
|
| 15 |
+
private fun meminfo(key: String): Long = runCatching {
|
| 16 |
+
File("/proc/meminfo").useLines { lines ->
|
| 17 |
+
lines.first { it.startsWith(key) }.trim().split(Regex("\\s+"))[1].toLong()
|
| 18 |
+
}
|
| 19 |
+
}.getOrDefault(-1L)
|
| 20 |
+
|
| 21 |
+
/** VmHWM / VmRSS / VmSwap of this process in kB (`/proc/self/status`). */
|
| 22 |
+
fun selfStatusKb(): Map<String, Long> = runCatching {
|
| 23 |
+
File("/proc/self/status").readLines().mapNotNull { line ->
|
| 24 |
+
val key = line.substringBefore(':')
|
| 25 |
+
if (key in setOf("VmHWM", "VmRSS", "VmSwap")) key to line.substringAfter(':').trim().split(' ')[0].toLong()
|
| 26 |
+
else null
|
| 27 |
+
}.toMap()
|
| 28 |
+
}.getOrDefault(emptyMap())
|
| 29 |
+
|
| 30 |
+
/**
|
| 31 |
+
* Resets this process's VmHWM to its current RSS (`echo 5 > /proc/self/clear_refs`), so that VmHWM read after a run
|
| 32 |
+
* is that run's peak. Returns false when the system does not allow it (the VmHWM is then the process's peak).
|
| 33 |
+
*/
|
| 34 |
+
fun resetPeakRss(): Boolean = runCatching {
|
| 35 |
+
File("/proc/self/clear_refs").writeText("5")
|
| 36 |
+
true
|
| 37 |
+
}.getOrDefault(false)
|
| 38 |
+
|
| 39 |
+
/** The system's GPU pages (`/sys/class/kgsl/kgsl/page_alloc`, bytes), or -1 when the file cannot be read. */
|
| 40 |
+
fun gpuPageAllocBytes(): Long = runCatching {
|
| 41 |
+
File("/sys/class/kgsl/kgsl/page_alloc").readText().trim().split(Regex("\\s+"))[0].toLong()
|
| 42 |
+
}.getOrDefault(-1L)
|
| 43 |
+
|
| 44 |
+
/** "Galaxy S26 (SM-S942Q)" for the phones this sample was measured on, else `Build.MODEL`. */
|
| 45 |
+
fun displayName(): String {
|
| 46 |
+
val model = Build.MODEL
|
| 47 |
+
val market = MARKET_NAMES.entries.firstOrNull { model.startsWith(it.key) }?.value
|
| 48 |
+
return if (market == null || market == model) model else "$market ($model)"
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
fun thermalStatus(context: Context): Int =
|
| 52 |
+
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
|
| 53 |
+
context.getSystemService(PowerManager::class.java).currentThermalStatus
|
| 54 |
+
} else {
|
| 55 |
+
-1
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
/** The cpuset of this process (`/top-app` while the app is in front). */
|
| 59 |
+
fun cpuset(): String = runCatching {
|
| 60 |
+
File("/proc/self/cgroup").readLines().firstOrNull { ":cpuset:" in it } ?: "none"
|
| 61 |
+
}.getOrDefault("unreadable")
|
| 62 |
+
|
| 63 |
+
private val MARKET_NAMES = mapOf("SM-S942" to "Galaxy S26")
|
| 64 |
+
}
|
| 65 |
+
|
| 66 |
+
/**
|
| 67 |
+
* Samples the system's GPU pages every [periodMs] while a run is in progress and keeps the maximum (CONTRACT.md
|
| 68 |
+
* line (e)). Best effort: when the app cannot read the file, [maxBytes] stays -1.
|
| 69 |
+
*/
|
| 70 |
+
class GpuPageSampler(private val periodMs: Long = 100) {
|
| 71 |
+
@Volatile var maxBytes = -1L
|
| 72 |
+
private set
|
| 73 |
+
@Volatile private var running = false
|
| 74 |
+
private var thread: Thread? = null
|
| 75 |
+
|
| 76 |
+
fun start() {
|
| 77 |
+
maxBytes = DeviceInfo.gpuPageAllocBytes()
|
| 78 |
+
if (maxBytes < 0) return
|
| 79 |
+
running = true
|
| 80 |
+
thread = Thread({
|
| 81 |
+
while (running) {
|
| 82 |
+
val v = DeviceInfo.gpuPageAllocBytes()
|
| 83 |
+
if (v > maxBytes) maxBytes = v
|
| 84 |
+
try {
|
| 85 |
+
Thread.sleep(periodMs)
|
| 86 |
+
} catch (_: InterruptedException) {
|
| 87 |
+
break
|
| 88 |
+
}
|
| 89 |
+
}
|
| 90 |
+
}, "IdolGpuPages").apply { isDaemon = true; start() }
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
fun stop(): Long {
|
| 94 |
+
running = false
|
| 95 |
+
thread?.interrupt()
|
| 96 |
+
thread?.join(500)
|
| 97 |
+
thread = null
|
| 98 |
+
val v = DeviceInfo.gpuPageAllocBytes()
|
| 99 |
+
if (v > maxBytes) maxBytes = v
|
| 100 |
+
return maxBytes
|
| 101 |
+
}
|
| 102 |
+
}
|
android/app/src/main/java/com/idol/Fixture.kt
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import org.json.JSONObject
|
| 4 |
+
import java.io.File
|
| 5 |
+
import java.io.FileOutputStream
|
| 6 |
+
import java.io.RandomAccessFile
|
| 7 |
+
import java.nio.ByteBuffer
|
| 8 |
+
import java.nio.ByteOrder
|
| 9 |
+
import java.nio.channels.FileChannel
|
| 10 |
+
|
| 11 |
+
/**
|
| 12 |
+
* A fixture directory `files/fx/<tag>/` as scripts/make_fixtures.py writes it: fixture.json
|
| 13 |
+
* {"tag", "mode", "seed", "image", "arrays": {name: {"file", "shape"}}}, the raw input image and little-endian
|
| 14 |
+
* float32 arrays (the reference run's shared randomness and each part's own reference input).
|
| 15 |
+
*/
|
| 16 |
+
class Fixture(val dir: File) {
|
| 17 |
+
private val json = JSONObject(File(dir, "fixture.json").readText())
|
| 18 |
+
val tag: String = json.optString("tag", dir.name)
|
| 19 |
+
val mode: String = json.getString("mode")
|
| 20 |
+
val seed: Long = json.optLong("seed", 0L)
|
| 21 |
+
val image: File = File(dir, json.getString("image"))
|
| 22 |
+
private val arrays = json.getJSONObject("arrays")
|
| 23 |
+
|
| 24 |
+
fun has(name: String) = arrays.has(name)
|
| 25 |
+
|
| 26 |
+
fun shape(name: String): IntArray {
|
| 27 |
+
val a = arrays.getJSONObject(name).getJSONArray("shape")
|
| 28 |
+
return IntArray(a.length()) { a.getInt(it) }
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
fun floats(name: String): FloatArray {
|
| 32 |
+
val e = arrays.getJSONObject(name)
|
| 33 |
+
val shape = shape(name)
|
| 34 |
+
val n = shape.fold(1) { a, b -> a * b }
|
| 35 |
+
return readF32(File(dir, e.getString("file")), n)
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
/** The five noise arrays, in place of the app's random draws. */
|
| 39 |
+
fun noise(): Noise = Noise(
|
| 40 |
+
vae = floats("vae_noise"),
|
| 41 |
+
latent = floats("latent_noise"),
|
| 42 |
+
camera = floats("camera_noise"),
|
| 43 |
+
pool = floats("octree_pool"),
|
| 44 |
+
leaf = floats("leaf_pool"),
|
| 45 |
+
source = "fixture ${dir.name}",
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
companion object {
|
| 49 |
+
/** [n] little-endian float32 values from [file] (its size must be n x 4 bytes). */
|
| 50 |
+
fun readF32(file: File, n: Int): FloatArray {
|
| 51 |
+
require(file.length() == n * 4L) { "${file.name} is ${file.length()} bytes, expected ${n * 4L}" }
|
| 52 |
+
val out = FloatArray(n)
|
| 53 |
+
RandomAccessFile(file, "r").use { raf ->
|
| 54 |
+
val map = raf.channel.map(FileChannel.MapMode.READ_ONLY, 0, n * 4L)
|
| 55 |
+
map.order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().get(out)
|
| 56 |
+
}
|
| 57 |
+
return out
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
fun writeF32(file: File, data: FloatArray) {
|
| 61 |
+
val buf = ByteBuffer.allocate(data.size * 4).order(ByteOrder.LITTLE_ENDIAN)
|
| 62 |
+
buf.asFloatBuffer().put(data)
|
| 63 |
+
FileOutputStream(file).use { it.write(buf.array()) }
|
| 64 |
+
}
|
| 65 |
+
}
|
| 66 |
+
}
|
android/app/src/main/java/com/idol/Glue.kt
ADDED
|
@@ -0,0 +1,472 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import org.json.JSONObject
|
| 4 |
+
import java.io.BufferedOutputStream
|
| 5 |
+
import java.io.File
|
| 6 |
+
import java.io.FileOutputStream
|
| 7 |
+
import java.nio.ByteBuffer
|
| 8 |
+
import java.nio.ByteOrder
|
| 9 |
+
import kotlin.math.PI
|
| 10 |
+
import kotlin.math.cos
|
| 11 |
+
import kotlin.math.exp
|
| 12 |
+
import kotlin.math.expm1
|
| 13 |
+
import kotlin.math.ln
|
| 14 |
+
import kotlin.math.ln1p
|
| 15 |
+
import kotlin.math.pow
|
| 16 |
+
import kotlin.math.sin
|
| 17 |
+
import kotlin.math.sqrt
|
| 18 |
+
import kotlin.math.tanh
|
| 19 |
+
|
| 20 |
+
/** The five arrays of shared randomness (CONTRACT.md "Shared randomness"), flat in C order. */
|
| 21 |
+
class Noise(
|
| 22 |
+
/** [1, 32, 128, 128] normal: the VAE posterior noise. */
|
| 23 |
+
val vae: FloatArray,
|
| 24 |
+
/** [1, 8192, 16] normal: x at t = 1. */
|
| 25 |
+
val latent: FloatArray,
|
| 26 |
+
/** [1, 1, 5] normal: the camera at t = 1. */
|
| 27 |
+
val camera: FloatArray,
|
| 28 |
+
/** [8, 8192] uniform: the octree sampler's per-row offsets. */
|
| 29 |
+
val pool: FloatArray,
|
| 30 |
+
/** [1, 8192, 3] uniform: the leaf positions. */
|
| 31 |
+
val leaf: FloatArray,
|
| 32 |
+
/** "seed <n>" or the fixture it came from. */
|
| 33 |
+
val source: String,
|
| 34 |
+
) {
|
| 35 |
+
init {
|
| 36 |
+
require(vae.size == 32 * 128 * 128 && latent.size == Glue.NUM_POINTS * Glue.IN_CH && camera.size == Glue.CAM_CH)
|
| 37 |
+
require(pool.size == Glue.MAX_LEVEL * Glue.NUM_POINTS && leaf.size == Glue.NUM_POINTS * 3)
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
companion object {
|
| 41 |
+
/**
|
| 42 |
+
* The app's own draws for a run from a picture: same distributions as the reference (normal, uniform [0, 1)),
|
| 43 |
+
* from java.util.Random, not torch's generator; the fixtures carry the reference's own numbers.
|
| 44 |
+
*/
|
| 45 |
+
fun random(seed: Long): Noise {
|
| 46 |
+
val r = java.util.Random(seed)
|
| 47 |
+
fun normal(n: Int) = FloatArray(n) { r.nextGaussian().toFloat() }
|
| 48 |
+
fun uniform(n: Int) = FloatArray(n) { r.nextFloat() }
|
| 49 |
+
val vae = normal(32 * 128 * 128)
|
| 50 |
+
val latent = normal(Glue.NUM_POINTS * Glue.IN_CH)
|
| 51 |
+
val camera = normal(Glue.CAM_CH)
|
| 52 |
+
val pool = uniform(Glue.MAX_LEVEL * Glue.NUM_POINTS)
|
| 53 |
+
val leaf = uniform(Glue.NUM_POINTS * 3)
|
| 54 |
+
return Noise(vae, latent, camera, pool, leaf, "seed $seed")
|
| 55 |
+
}
|
| 56 |
+
}
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
/** Activated Gaussians (z-up, as GaussianFixedlenDecoder.to_representation returns them), N = 8192 x 32. */
|
| 60 |
+
class Gaussians(
|
| 61 |
+
val count: Int,
|
| 62 |
+
val xyz: FloatArray,
|
| 63 |
+
val featuresDc: FloatArray,
|
| 64 |
+
val scaling: FloatArray,
|
| 65 |
+
/** (w, x, y, z) */
|
| 66 |
+
val rotation: FloatArray,
|
| 67 |
+
val opacity: FloatArray,
|
| 68 |
+
)
|
| 69 |
+
|
| 70 |
+
/** gs_rep.json: the Gaussian decoder's representation constants. */
|
| 71 |
+
class GaussianRep(
|
| 72 |
+
val baseOffsetScale: Float,
|
| 73 |
+
val lrXyz: Float,
|
| 74 |
+
val lrFeaturesDc: Float,
|
| 75 |
+
val lrOpacity: Float,
|
| 76 |
+
val lrScaling: Float,
|
| 77 |
+
val lrRotation: Float,
|
| 78 |
+
val perturbeSize: Float,
|
| 79 |
+
val filterKernelSize: Double,
|
| 80 |
+
val scalingBias: Float,
|
| 81 |
+
val opacityBias: Float,
|
| 82 |
+
) {
|
| 83 |
+
companion object {
|
| 84 |
+
fun parse(o: JSONObject): GaussianRep {
|
| 85 |
+
val rep = o.getJSONObject("rep")
|
| 86 |
+
val lr = rep.getJSONObject("lr")
|
| 87 |
+
return GaussianRep(
|
| 88 |
+
baseOffsetScale = o.getDouble("base_offset_scale").toFloat(),
|
| 89 |
+
lrXyz = lr.getDouble("_xyz").toFloat(),
|
| 90 |
+
lrFeaturesDc = lr.getDouble("_features_dc").toFloat(),
|
| 91 |
+
lrOpacity = lr.getDouble("_opacity").toFloat(),
|
| 92 |
+
lrScaling = lr.getDouble("_scaling").toFloat(),
|
| 93 |
+
lrRotation = lr.getDouble("_rotation").toFloat(),
|
| 94 |
+
perturbeSize = rep.getDouble("perturbe_size").toFloat(),
|
| 95 |
+
filterKernelSize = rep.getDouble("3d_filter_kernel_size"),
|
| 96 |
+
scalingBias = rep.getDouble("scaling_bias").toFloat(),
|
| 97 |
+
opacityBias = rep.getDouble("opacity_bias").toFloat(),
|
| 98 |
+
)
|
| 99 |
+
}
|
| 100 |
+
}
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
/**
|
| 104 |
+
* The host glue between the graphs, ported line by line from scripts/idol_glue.py, scripts/idol_graphs.py
|
| 105 |
+
* (sinusoid, point_pe, lora_gate) and scripts/idol_runner.py (vae_tokens): the same order of float32 operations as
|
| 106 |
+
* the torch / numpy code (each transcendental is computed in double and rounded to float32).
|
| 107 |
+
*/
|
| 108 |
+
object Glue {
|
| 109 |
+
const val MAX_LEVEL = 8
|
| 110 |
+
const val PER_ANCHOR = 32
|
| 111 |
+
const val PHASE_THRESHOLD = 0.5
|
| 112 |
+
const val NUM_POINTS = 8192
|
| 113 |
+
const val IN_CH = 16
|
| 114 |
+
const val CAM_CH = 5
|
| 115 |
+
const val COND_TOKENS = 4101
|
| 116 |
+
const val LORA_SCALE = 128.0f / 32.0f
|
| 117 |
+
val OCT_BUCKETS = intArrayOf(64, 256, 1024, 4096, 8192)
|
| 118 |
+
const val PLY_FLOATS = 17
|
| 119 |
+
val PLY_NAMES = listOf(
|
| 120 |
+
"x", "y", "z", "nx", "ny", "nz", "f_dc_0", "f_dc_1", "f_dc_2", "opacity",
|
| 121 |
+
"scale_0", "scale_1", "scale_2", "rot_0", "rot_1", "rot_2", "rot_3",
|
| 122 |
+
)
|
| 123 |
+
|
| 124 |
+
private val PI_F = PI.toFloat()
|
| 125 |
+
|
| 126 |
+
private fun expf(x: Float) = exp(x.toDouble()).toFloat()
|
| 127 |
+
private fun logf(x: Float) = ln(x.toDouble()).toFloat()
|
| 128 |
+
|
| 129 |
+
/** time_grid: t = r s / (1 + (r - 1) s), s = numpy.linspace(1, 0, steps + 1), float64. */
|
| 130 |
+
fun timeGrid(steps: Int = 5, rescale: Double = 3.0): DoubleArray {
|
| 131 |
+
val num = steps + 1
|
| 132 |
+
val step = (0.0 - 1.0) / (num - 1)
|
| 133 |
+
val s = DoubleArray(num) { i -> i * step + 1.0 }
|
| 134 |
+
s[num - 1] = 0.0
|
| 135 |
+
return DoubleArray(num) { rescale * s[it] / (1 + (rescale - 1) * s[it]) }
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
/** lora_gate: [1, 64], the slow expert's half (t >= 0.5) or the fast one's at LORA_SCALE. */
|
| 139 |
+
fun loraGate(slow: Boolean): FloatArray = FloatArray(64) { i ->
|
| 140 |
+
if ((i < 32) == slow) LORA_SCALE else 0f
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
/** sinusoid: SinusoidalEmbedder's frequency features of one scalar -> [1, dim]. */
|
| 144 |
+
fun sinusoid(value: Double, dim: Int = 256, maxPeriod: Double = 10000.0, twoPi: Boolean = false): FloatArray {
|
| 145 |
+
val half = dim / 2
|
| 146 |
+
val c = (-ln(maxPeriod)).toFloat()
|
| 147 |
+
val v = value.toFloat()
|
| 148 |
+
val out = FloatArray(dim)
|
| 149 |
+
for (k in 0 until half) {
|
| 150 |
+
val f = expf((c * k.toFloat()) / half.toFloat())
|
| 151 |
+
var a = v * f
|
| 152 |
+
if (twoPi) {
|
| 153 |
+
a *= 2f
|
| 154 |
+
a *= PI_F
|
| 155 |
+
}
|
| 156 |
+
out[k] = cos(a.toDouble()).toFloat()
|
| 157 |
+
out[half + k] = sin(a.toDouble()).toFloat()
|
| 158 |
+
}
|
| 159 |
+
return out
|
| 160 |
+
}
|
| 161 |
+
|
| 162 |
+
/**
|
| 163 |
+
* torch.linspace(start, end, steps) in float32 as the reference runs' torch computes it: start + step * i below
|
| 164 |
+
* halfway, end - step * (steps - 1 - i) from there, each one fused multiply-add (one rounding). Only called with
|
| 165 |
+
* (0, 10, 170): there the product and the sum are exact in double, so the double expression rounds once too.
|
| 166 |
+
*/
|
| 167 |
+
private fun linspaceF(start: Float, end: Float, steps: Int): FloatArray {
|
| 168 |
+
val step = (end - start) / (steps - 1).toFloat()
|
| 169 |
+
val halfway = steps / 2
|
| 170 |
+
return FloatArray(steps) { i ->
|
| 171 |
+
if (i < halfway) (step.toDouble() * i + start).toFloat() else (end - step.toDouble() * (steps - i - 1)).toFloat()
|
| 172 |
+
}
|
| 173 |
+
}
|
| 174 |
+
|
| 175 |
+
/**
|
| 176 |
+
* The frequencies where torch's float32 pow(2, x) is one ulp off the correctly rounded value, read from the Mac
|
| 177 |
+
* reference run's point embedding (device/fx/g1_gs_0): with them, all 170 frequency columns of that embedding agree
|
| 178 |
+
* to one ulp of sin / cos (without them six columns are off by up to 1.1e-3).
|
| 179 |
+
*/
|
| 180 |
+
private val TORCH_POW_ULP = mapOf(43 to -1, 72 to -1, 95 to -1, 113 to 1, 138 to -1, 159 to 1)
|
| 181 |
+
|
| 182 |
+
/** torch.pow(2.0, torch.linspace(0.0, 10.0, 170)), float32. */
|
| 183 |
+
private val PE_FREQS: FloatArray by lazy {
|
| 184 |
+
val lin = linspaceF(0f, 10f, 1024 / 3 / 2)
|
| 185 |
+
FloatArray(lin.size) {
|
| 186 |
+
val f = 2.0.pow(lin[it].toDouble()).toFloat()
|
| 187 |
+
Float.fromBits(f.toRawBits() + (TORCH_POW_ULP[it] ?: 0))
|
| 188 |
+
}
|
| 189 |
+
}
|
| 190 |
+
|
| 191 |
+
/** point_pe: PcdAbsolutePositionEmbedderv2 of [n] points (x, y, z) -> [n, 1024]. */
|
| 192 |
+
fun pointPe(points: FloatArray, n: Int, channels: Int = 1024): FloatArray {
|
| 193 |
+
val freqs = PE_FREQS
|
| 194 |
+
val fd = freqs.size
|
| 195 |
+
val out = FloatArray(n * channels)
|
| 196 |
+
for (p in 0 until n) {
|
| 197 |
+
for (c in 0 until 3) {
|
| 198 |
+
val v = points[p * 3 + c]
|
| 199 |
+
val base = p * channels + c * 2 * fd
|
| 200 |
+
for (j in 0 until fd) {
|
| 201 |
+
val ang = (v * freqs[j]) * PI_F
|
| 202 |
+
out[base + j] = sin(ang.toDouble()).toFloat()
|
| 203 |
+
out[base + fd + j] = cos(ang.toDouble()).toFloat()
|
| 204 |
+
}
|
| 205 |
+
}
|
| 206 |
+
}
|
| 207 |
+
return out
|
| 208 |
+
}
|
| 209 |
+
|
| 210 |
+
/**
|
| 211 |
+
* vae_tokens: posterior sample with the shared noise (logvar clipped to [-30, 20]), 2 x 2 patchify, bn ->
|
| 212 |
+
* [1, 4096, 128]. moments [1, 64, 128, 128] (mean, logvar), noise [1, 32, 128, 128].
|
| 213 |
+
*/
|
| 214 |
+
fun vaeTokens(moments: FloatArray, noise: FloatArray, bnMean: FloatArray, bnStd: FloatArray): FloatArray {
|
| 215 |
+
val hw = 128
|
| 216 |
+
val plane = hw * hw
|
| 217 |
+
val out = FloatArray(4096 * 128)
|
| 218 |
+
for (c in 0 until 32) for (y in 0 until hw) for (x in 0 until hw) {
|
| 219 |
+
val i = c * plane + y * hw + x
|
| 220 |
+
var lv = moments[32 * plane + i]
|
| 221 |
+
lv = if (lv < -30f) -30f else if (lv > 20f) 20f else lv
|
| 222 |
+
val lat = moments[i] + expf(0.5f * lv) * noise[i]
|
| 223 |
+
val k = c * 4 + (y % 2) * 2 + (x % 2)
|
| 224 |
+
val t = (y / 2) * 64 + (x / 2)
|
| 225 |
+
out[t * 128 + k] = (lat - bnMean[k]) / bnStd[k]
|
| 226 |
+
}
|
| 227 |
+
return out
|
| 228 |
+
}
|
| 229 |
+
|
| 230 |
+
/** torch.softmax over the last dimension (8) of [rows, 8] logits: exp(x - max) times 1 / sum. */
|
| 231 |
+
fun softmax8(logits: FloatArray, rows: Int): FloatArray {
|
| 232 |
+
val out = FloatArray(rows * 8)
|
| 233 |
+
for (r in 0 until rows) {
|
| 234 |
+
val b = r * 8
|
| 235 |
+
var mx = logits[b]
|
| 236 |
+
for (k in 1 until 8) if (logits[b + k] > mx) mx = logits[b + k]
|
| 237 |
+
var sum = 0f
|
| 238 |
+
for (k in 0 until 8) {
|
| 239 |
+
val e = expf(logits[b + k] - mx)
|
| 240 |
+
out[b + k] = e
|
| 241 |
+
sum += e
|
| 242 |
+
}
|
| 243 |
+
val inv = 1f / sum
|
| 244 |
+
for (k in 0 until 8) out[b + k] = out[b + k] * inv
|
| 245 |
+
}
|
| 246 |
+
return out
|
| 247 |
+
}
|
| 248 |
+
|
| 249 |
+
/**
|
| 250 |
+
* systematic_rows: idol's systematic_sample for one batch, row i drawing counts[i] samples from its 8 child
|
| 251 |
+
* probabilities with the offset u[i] / counts[i] -> [rows, 8] sample counts. The cumulative sum accumulates in
|
| 252 |
+
* double as torch's CPU cumsum does for float32.
|
| 253 |
+
*/
|
| 254 |
+
fun systematicRows(probs: FloatArray, counts: IntArray, u: FloatArray, rows: Int): IntArray {
|
| 255 |
+
val P = 8
|
| 256 |
+
val out = IntArray(rows * P)
|
| 257 |
+
val p = FloatArray(P)
|
| 258 |
+
val cdf = FloatArray(P)
|
| 259 |
+
for (r in 0 until rows) {
|
| 260 |
+
val n = counts[r]
|
| 261 |
+
if (n == 0) continue
|
| 262 |
+
var s = 0f
|
| 263 |
+
for (k in 0 until P) {
|
| 264 |
+
val v = probs[r * P + k]
|
| 265 |
+
p[k] = if (v < 0f) 0f else v
|
| 266 |
+
s += p[k]
|
| 267 |
+
}
|
| 268 |
+
if (s == 0f) {
|
| 269 |
+
for (k in 0 until P) p[k] = 1.0f / P
|
| 270 |
+
} else {
|
| 271 |
+
val d = if (s < 1f) 1f else s
|
| 272 |
+
for (k in 0 until P) p[k] = p[k] / d
|
| 273 |
+
}
|
| 274 |
+
var acc = 0.0
|
| 275 |
+
for (k in 0 until P) {
|
| 276 |
+
acc += p[k]
|
| 277 |
+
val c = acc.toFloat()
|
| 278 |
+
cdf[k] = if (c > 1f) 1f else c
|
| 279 |
+
}
|
| 280 |
+
val nf = n.toFloat()
|
| 281 |
+
val u0 = u[r] / nf
|
| 282 |
+
for (j in 0 until n) {
|
| 283 |
+
var us = u0 + j.toFloat() / nf
|
| 284 |
+
if (us > 1f) us = 1f
|
| 285 |
+
var idx = 0
|
| 286 |
+
while (idx < P && cdf[idx] < us) idx++
|
| 287 |
+
if (idx > P - 1) idx = P - 1
|
| 288 |
+
out[r * P + idx]++
|
| 289 |
+
}
|
| 290 |
+
}
|
| 291 |
+
return out
|
| 292 |
+
}
|
| 293 |
+
|
| 294 |
+
/**
|
| 295 |
+
* octree_descend: idol OctreeProbabilityFixedlenDecoder.sample for one batch. [probsFn] (cells [rows, 3], rows,
|
| 296 |
+
* level) -> child probabilities [rows, 8]. pool [8, 8192], leaf [8192, 3] -> anchors [8192, 3] in [0, 1]^3.
|
| 297 |
+
* [cellsOut], when given, receives each level's cells (the parity record).
|
| 298 |
+
*/
|
| 299 |
+
fun octreeDescend(
|
| 300 |
+
probsFn: (FloatArray, Int, Int) -> FloatArray,
|
| 301 |
+
pool: FloatArray,
|
| 302 |
+
leaf: FloatArray,
|
| 303 |
+
numPoints: Int = NUM_POINTS,
|
| 304 |
+
levels: Int = MAX_LEVEL,
|
| 305 |
+
): FloatArray {
|
| 306 |
+
var prev = IntArray(3)
|
| 307 |
+
var rows = 1
|
| 308 |
+
var counts = intArrayOf(numPoints)
|
| 309 |
+
for (l in 1..levels) {
|
| 310 |
+
val res = (1 shl (l - 1)).toFloat()
|
| 311 |
+
val cells = FloatArray(rows * 3) { (prev[it].toFloat() + 0.5f) / res }
|
| 312 |
+
val probs = probsFn(cells, rows, l)
|
| 313 |
+
val u = FloatArray(rows) { pool[(l - 1) * numPoints + it] }
|
| 314 |
+
val sampled = systematicRows(probs, counts, u, rows)
|
| 315 |
+
var m = 0
|
| 316 |
+
for (v in sampled) if (v > 0) m++
|
| 317 |
+
val next = IntArray(m * 3)
|
| 318 |
+
val nextCounts = IntArray(m)
|
| 319 |
+
var o = 0
|
| 320 |
+
for (r in 0 until rows) for (c in 0 until 8) {
|
| 321 |
+
val s = sampled[r * 8 + c]
|
| 322 |
+
if (s <= 0) continue
|
| 323 |
+
// CHILD_OFFSET[c] = (i, j, k) with c = k * 4 + j * 2 + i
|
| 324 |
+
next[o * 3] = prev[r * 3] * 2 + (c and 1)
|
| 325 |
+
next[o * 3 + 1] = prev[r * 3 + 1] * 2 + ((c shr 1) and 1)
|
| 326 |
+
next[o * 3 + 2] = prev[r * 3 + 2] * 2 + ((c shr 2) and 1)
|
| 327 |
+
nextCounts[o] = s
|
| 328 |
+
o++
|
| 329 |
+
}
|
| 330 |
+
prev = next
|
| 331 |
+
counts = nextCounts
|
| 332 |
+
rows = m
|
| 333 |
+
}
|
| 334 |
+
val out = FloatArray(numPoints * 3)
|
| 335 |
+
val scale = (1 shl levels).toFloat()
|
| 336 |
+
var p = 0
|
| 337 |
+
for (r in 0 until rows) for (k in 0 until counts[r]) {
|
| 338 |
+
check(p < numPoints) { "octree sampling produced more than $numPoints points" }
|
| 339 |
+
for (d in 0 until 3) out[p * 3 + d] = (prev[r * 3 + d].toFloat() + leaf[p * 3 + d]) / scale
|
| 340 |
+
p++
|
| 341 |
+
}
|
| 342 |
+
check(p == numPoints) { "octree sampling produced $p points, not $numPoints" }
|
| 343 |
+
return out
|
| 344 |
+
}
|
| 345 |
+
|
| 346 |
+
private fun softplus(x: Float): Float = if (x > 20f) x else ln1p(expf(x).toDouble()).toFloat()
|
| 347 |
+
|
| 348 |
+
private fun sigmoid(x: Float): Float = 1f / (1f + expf(-x))
|
| 349 |
+
|
| 350 |
+
/**
|
| 351 |
+
* to_gaussians: GaussianFixedlenDecoder.to_representation + Gaussian.__init__ for one batch. anchors [L, 3],
|
| 352 |
+
* features [L, 480] (per anchor: xyz 32x3, f_dc 32x3, scaling 32x3, rotation 32x4, opacity 32, offset scale 32),
|
| 353 |
+
* perturbation [32, 3].
|
| 354 |
+
*/
|
| 355 |
+
fun toGaussians(anchors: FloatArray, features: FloatArray, rep: GaussianRep, perturbation: FloatArray): Gaussians {
|
| 356 |
+
val L = anchors.size / 3
|
| 357 |
+
val n = L * PER_ANCHOR
|
| 358 |
+
val width = PER_ANCHOR * (3 + 3 + 3 + 4 + 1 + 1)
|
| 359 |
+
require(features.size == L * width) { "features hold ${features.size} values, not $L x $width" }
|
| 360 |
+
val xyz = FloatArray(n * 3)
|
| 361 |
+
val fdc = FloatArray(n * 3)
|
| 362 |
+
val scaling = FloatArray(n * 3)
|
| 363 |
+
val rotation = FloatArray(n * 4)
|
| 364 |
+
val opacity = FloatArray(n)
|
| 365 |
+
// inverse_softplus(scaling_bias), inverse_sigmoid(opacity_bias): float32 tensors in the reference
|
| 366 |
+
val sb = rep.scalingBias
|
| 367 |
+
val scalingOffset = sb + logf(-expm1(-sb.toDouble()).toFloat())
|
| 368 |
+
val ob = rep.opacityBias
|
| 369 |
+
val opacityOffset = logf(ob / (1f - ob))
|
| 370 |
+
val kernel2 = (rep.filterKernelSize * rep.filterKernelSize).toFloat()
|
| 371 |
+
val xyzAt = 0
|
| 372 |
+
val fdcAt = PER_ANCHOR * 3
|
| 373 |
+
val scaleAt = PER_ANCHOR * 6
|
| 374 |
+
val rotAt = PER_ANCHOR * 9
|
| 375 |
+
val opaAt = PER_ANCHOR * 13
|
| 376 |
+
val oscAt = PER_ANCHOR * 14
|
| 377 |
+
for (a in 0 until L) {
|
| 378 |
+
val f = a * width
|
| 379 |
+
for (j in 0 until PER_ANCHOR) {
|
| 380 |
+
val g = a * PER_ANCHOR + j
|
| 381 |
+
val offsetScale = softplus(features[f + oscAt + j] + rep.baseOffsetScale)
|
| 382 |
+
for (c in 0 until 3) {
|
| 383 |
+
var off = features[f + xyzAt + j * 3 + c] * rep.lrXyz + perturbation[j * 3 + c]
|
| 384 |
+
off = tanh(off.toDouble()).toFloat() * 0.5f * rep.perturbeSize * offsetScale
|
| 385 |
+
val raw = off + anchors[a * 3 + c]
|
| 386 |
+
xyz[g * 3 + c] = raw * 1.0f - 0.5f
|
| 387 |
+
fdc[g * 3 + c] = features[f + fdcAt + j * 3 + c] * rep.lrFeaturesDc
|
| 388 |
+
val s = softplus(features[f + scaleAt + j * 3 + c] * rep.lrScaling + scalingOffset)
|
| 389 |
+
scaling[g * 3 + c] = sqrt((s * s + kernel2).toDouble()).toFloat()
|
| 390 |
+
}
|
| 391 |
+
val r0 = features[f + rotAt + j * 4] * rep.lrRotation + 1.0f
|
| 392 |
+
val r1 = features[f + rotAt + j * 4 + 1] * rep.lrRotation + 0.0f
|
| 393 |
+
val r2 = features[f + rotAt + j * 4 + 2] * rep.lrRotation + 0.0f
|
| 394 |
+
val r3 = features[f + rotAt + j * 4 + 3] * rep.lrRotation + 0.0f
|
| 395 |
+
var ss = r0 * r0
|
| 396 |
+
ss += r1 * r1
|
| 397 |
+
ss += r2 * r2
|
| 398 |
+
ss += r3 * r3
|
| 399 |
+
val norm = maxOf(sqrt(ss.toDouble()).toFloat(), 1e-12f)
|
| 400 |
+
rotation[g * 4] = r0 / norm
|
| 401 |
+
rotation[g * 4 + 1] = r1 / norm
|
| 402 |
+
rotation[g * 4 + 2] = r2 / norm
|
| 403 |
+
rotation[g * 4 + 3] = r3 / norm
|
| 404 |
+
opacity[g] = sigmoid(features[f + opaAt + j] * rep.lrOpacity + opacityOffset)
|
| 405 |
+
}
|
| 406 |
+
}
|
| 407 |
+
return Gaussians(n, xyz, fdc, scaling, rotation, opacity)
|
| 408 |
+
}
|
| 409 |
+
|
| 410 |
+
/** ply_rows: idol Gaussian.save_ply's 17 attributes per Gaussian (z-up turned to y-up), float32. */
|
| 411 |
+
fun plyRows(g: Gaussians): FloatArray {
|
| 412 |
+
val n = g.count
|
| 413 |
+
val rows = FloatArray(n * PLY_FLOATS)
|
| 414 |
+
val eps = 1e-6f
|
| 415 |
+
val hi = (1.0 - 1e-6).toFloat()
|
| 416 |
+
// quaternion_multiply([sqrt(.5), sqrt(.5), 0, 0], rotation) in float64 (numpy promotes), rounded at the end
|
| 417 |
+
val aw = sqrt(0.5)
|
| 418 |
+
val ax = sqrt(0.5)
|
| 419 |
+
val ay = 0.0
|
| 420 |
+
val az = 0.0
|
| 421 |
+
for (i in 0 until n) {
|
| 422 |
+
val o = i * PLY_FLOATS
|
| 423 |
+
rows[o] = g.xyz[i * 3]
|
| 424 |
+
rows[o + 1] = -g.xyz[i * 3 + 2]
|
| 425 |
+
rows[o + 2] = g.xyz[i * 3 + 1]
|
| 426 |
+
rows[o + 6] = g.featuresDc[i * 3]
|
| 427 |
+
rows[o + 7] = g.featuresDc[i * 3 + 1]
|
| 428 |
+
rows[o + 8] = g.featuresDc[i * 3 + 2]
|
| 429 |
+
var op = g.opacity[i]
|
| 430 |
+
op = if (op < eps) eps else if (op > hi) hi else op
|
| 431 |
+
rows[o + 9] = logf(op / (1f - op))
|
| 432 |
+
for (c in 0 until 3) rows[o + 10 + c] = logf(maxOf(g.scaling[i * 3 + c], eps))
|
| 433 |
+
val bw = g.rotation[i * 4].toDouble()
|
| 434 |
+
val bx = g.rotation[i * 4 + 1].toDouble()
|
| 435 |
+
val by = g.rotation[i * 4 + 2].toDouble()
|
| 436 |
+
val bz = g.rotation[i * 4 + 3].toDouble()
|
| 437 |
+
rows[o + 13] = (aw * bw - ax * bx - ay * by - az * bz).toFloat()
|
| 438 |
+
rows[o + 14] = (aw * bx + ax * bw + ay * bz - az * by).toFloat()
|
| 439 |
+
rows[o + 15] = (aw * by - ax * bz + ay * bw + az * bx).toFloat()
|
| 440 |
+
rows[o + 16] = (aw * bz + ax * by - ay * bx + az * bw).toFloat()
|
| 441 |
+
}
|
| 442 |
+
return rows
|
| 443 |
+
}
|
| 444 |
+
|
| 445 |
+
/** write_ply: binary little-endian PLY with the 17 float properties (the bytes plyfile writes). */
|
| 446 |
+
fun writePly(rows: FloatArray, file: File) {
|
| 447 |
+
val n = rows.size / PLY_FLOATS
|
| 448 |
+
val header = buildString {
|
| 449 |
+
append("ply\nformat binary_little_endian 1.0\nelement vertex ").append(n).append('\n')
|
| 450 |
+
for (name in PLY_NAMES) append("property float ").append(name).append('\n')
|
| 451 |
+
append("end_header\n")
|
| 452 |
+
}
|
| 453 |
+
val tmp = File(file.parentFile, file.name + ".tmp")
|
| 454 |
+
BufferedOutputStream(FileOutputStream(tmp), 1 shl 20).use { out ->
|
| 455 |
+
out.write(header.toByteArray(Charsets.US_ASCII))
|
| 456 |
+
val chunk = 4096 * PLY_FLOATS
|
| 457 |
+
val buf = ByteBuffer.allocate(chunk * 4).order(ByteOrder.LITTLE_ENDIAN)
|
| 458 |
+
var i = 0
|
| 459 |
+
while (i < rows.size) {
|
| 460 |
+
val m = minOf(chunk, rows.size - i)
|
| 461 |
+
buf.clear()
|
| 462 |
+
buf.asFloatBuffer().put(rows, i, m)
|
| 463 |
+
out.write(buf.array(), 0, m * 4)
|
| 464 |
+
i += m
|
| 465 |
+
}
|
| 466 |
+
}
|
| 467 |
+
if (!tmp.renameTo(file)) {
|
| 468 |
+
tmp.copyTo(file, overwrite = true)
|
| 469 |
+
tmp.delete()
|
| 470 |
+
}
|
| 471 |
+
}
|
| 472 |
+
}
|
android/app/src/main/java/com/idol/Graphs.kt
ADDED
|
@@ -0,0 +1,339 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.os.SystemClock
|
| 4 |
+
import android.util.Log
|
| 5 |
+
import com.google.ai.edge.litert.Accelerator
|
| 6 |
+
import com.google.ai.edge.litert.CompiledModel
|
| 7 |
+
import com.google.ai.edge.litert.Environment
|
| 8 |
+
import com.google.ai.edge.litert.TensorBuffer
|
| 9 |
+
import org.json.JSONObject
|
| 10 |
+
import java.io.File
|
| 11 |
+
|
| 12 |
+
/**
|
| 13 |
+
* One compiled graph with its I/O buffers, made once per signature and reused. Inputs written with a tag are not
|
| 14 |
+
* written again while the buffer still holds that tag (the RoPE tables, the position embedding, the LoRA gate, the
|
| 15 |
+
* conditioning of the current phase...).
|
| 16 |
+
*/
|
| 17 |
+
class Graph(val spec: GraphSpec, private val model: CompiledModel) : AutoCloseable {
|
| 18 |
+
val name: String get() = spec.name
|
| 19 |
+
private val inputs = HashMap<String, List<TensorBuffer>>()
|
| 20 |
+
private val outputs = HashMap<String, List<TensorBuffer>>()
|
| 21 |
+
private val tags = HashMap<String, Array<Any?>>()
|
| 22 |
+
var runs = 0
|
| 23 |
+
private set
|
| 24 |
+
|
| 25 |
+
init {
|
| 26 |
+
try {
|
| 27 |
+
for (sig in spec.signatures.values) {
|
| 28 |
+
inputs[sig.key] = sig.inputs.map { model.createInputBuffer(it.name, sig.key) }
|
| 29 |
+
outputs[sig.key] = sig.outputs.map { model.createOutputBuffer(it.name, sig.key) }
|
| 30 |
+
tags[sig.key] = arrayOfNulls(sig.inputs.size)
|
| 31 |
+
}
|
| 32 |
+
} catch (e: Throwable) {
|
| 33 |
+
close()
|
| 34 |
+
throw e
|
| 35 |
+
}
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
private fun sig(key: String?): SignatureSpec = spec.signature(key)
|
| 39 |
+
|
| 40 |
+
/** Writes [data] to input [index] of signature [key], unless the buffer already holds [tag]. */
|
| 41 |
+
fun write(index: Int, data: FloatArray, tag: Any? = null, key: String? = null) {
|
| 42 |
+
val s = sig(key)
|
| 43 |
+
val t = tags.getValue(s.key)
|
| 44 |
+
if (tag != null && t[index] == tag) return
|
| 45 |
+
val want = s.inputs[index].size
|
| 46 |
+
require(data.size == want) { "$name ${s.key} input $index: ${data.size} values, expected $want" }
|
| 47 |
+
inputs.getValue(s.key)[index].writeFloat(data)
|
| 48 |
+
t[index] = tag
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
/**
|
| 52 |
+
* Runs signature [key]. [overrides] replace input buffers by index (another graph's output buffer: the direct
|
| 53 |
+
* handoff); the graph's own buffer at that index keeps its content and loses its tag.
|
| 54 |
+
*/
|
| 55 |
+
fun run(key: String? = null, overrides: Map<Int, TensorBuffer> = emptyMap()) {
|
| 56 |
+
val s = sig(key)
|
| 57 |
+
val ins = inputs.getValue(s.key)
|
| 58 |
+
val outs = outputs.getValue(s.key)
|
| 59 |
+
val inMap = HashMap<String, TensorBuffer>()
|
| 60 |
+
for ((i, t) in s.inputs.withIndex()) inMap[t.name] = overrides[i] ?: ins[i]
|
| 61 |
+
val outMap = HashMap<String, TensorBuffer>()
|
| 62 |
+
for ((i, t) in s.outputs.withIndex()) outMap[t.name] = outs[i]
|
| 63 |
+
model.run(inMap, outMap, s.key)
|
| 64 |
+
runs++
|
| 65 |
+
}
|
| 66 |
+
|
| 67 |
+
/** Output [index] of signature [key] as float32 (waits for the GPU). */
|
| 68 |
+
fun read(index: Int, key: String? = null): FloatArray {
|
| 69 |
+
val s = sig(key)
|
| 70 |
+
val want = s.outputs[index].size
|
| 71 |
+
val v = outputs.getValue(s.key)[index].readFloat()
|
| 72 |
+
check(v.size >= want) { "$name ${s.key} output $index: ${v.size} values, expected $want" }
|
| 73 |
+
return if (v.size == want) v else v.copyOf(want)
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
fun outputBuffer(index: Int, key: String? = null): TensorBuffer = outputs.getValue(sig(key).key)[index]
|
| 77 |
+
|
| 78 |
+
/** Forget the tag of input [index] (its buffer was not written by us). */
|
| 79 |
+
fun untag(index: Int, key: String? = null) {
|
| 80 |
+
tags.getValue(sig(key).key)[index] = null
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
override fun close() {
|
| 84 |
+
for (l in inputs.values) for (b in l) runCatching { b.close() }
|
| 85 |
+
for (l in outputs.values) for (b in l) runCatching { b.close() }
|
| 86 |
+
inputs.clear()
|
| 87 |
+
outputs.clear()
|
| 88 |
+
runCatching { model.close() }
|
| 89 |
+
}
|
| 90 |
+
}
|
| 91 |
+
|
| 92 |
+
/** The memory guard found no graph left to close and the next one does not fit above the reserve. */
|
| 93 |
+
class NotEnoughMemoryException(message: String) : Exception(message)
|
| 94 |
+
|
| 95 |
+
/**
|
| 96 |
+
* The compiled graphs the app keeps, all on the model thread. All 46 graphs do not fit in a phone's memory at once
|
| 97 |
+
* (one denoiser block alone holds about 0.7 GB of GPU memory), so the pool keeps as many as fit above a reserve of
|
| 98 |
+
* `MemAvailable` and, when the next graph does not fit, closes the resident graph whose next use in the run's
|
| 99 |
+
* schedule is farthest away (Belady's rule over the fixed graph order of a run, read cyclically for the next run).
|
| 100 |
+
* Every compile, close and memory shortfall is recorded.
|
| 101 |
+
*/
|
| 102 |
+
class GraphPool(
|
| 103 |
+
private val env: Environment,
|
| 104 |
+
private val dir: File,
|
| 105 |
+
private val manifest: Manifest,
|
| 106 |
+
var reserveKb: Long,
|
| 107 |
+
var memoryCheck: Boolean,
|
| 108 |
+
var cpuThreads: Int,
|
| 109 |
+
var gpuPriority: CompiledModel.GpuOptions.Priority?,
|
| 110 |
+
) : AutoCloseable {
|
| 111 |
+
class CompileRecord(
|
| 112 |
+
val graph: String,
|
| 113 |
+
val seconds: Double,
|
| 114 |
+
val memAvailableBeforeKb: Long,
|
| 115 |
+
val memAvailableAfterKb: Long,
|
| 116 |
+
val phase: String,
|
| 117 |
+
) {
|
| 118 |
+
fun json(): JSONObject = JSONObject().put("graph", graph).put("seconds", seconds)
|
| 119 |
+
.put("mem_available_before_kb", memAvailableBeforeKb).put("mem_available_after_kb", memAvailableAfterKb)
|
| 120 |
+
.put("phase", phase)
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
private val resident = LinkedHashMap<String, Graph>()
|
| 124 |
+
private val lastUse = HashMap<String, Long>()
|
| 125 |
+
private val pinned = HashSet<String>()
|
| 126 |
+
private val footprintKb = HashMap<String, Long>()
|
| 127 |
+
private var schedule: List<String> = emptyList()
|
| 128 |
+
private var cursor = -1
|
| 129 |
+
private var clock = 0L
|
| 130 |
+
|
| 131 |
+
/** What the compile records are filed under ("preload", "run", "parity", ...). */
|
| 132 |
+
var phase = "run"
|
| 133 |
+
val compiles = ArrayList<CompileRecord>()
|
| 134 |
+
val events = ArrayList<JSONObject>()
|
| 135 |
+
|
| 136 |
+
val residentNames: List<String> get() = resident.keys.toList()
|
| 137 |
+
|
| 138 |
+
fun isResident(name: String) = name in resident
|
| 139 |
+
|
| 140 |
+
/** The order the next run will ask for graphs in (resets the position in it). */
|
| 141 |
+
fun setSchedule(order: List<String>) {
|
| 142 |
+
schedule = order
|
| 143 |
+
cursor = -1
|
| 144 |
+
}
|
| 145 |
+
|
| 146 |
+
fun pin(name: String) {
|
| 147 |
+
pinned.add(name)
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
fun unpin(name: String) {
|
| 151 |
+
pinned.remove(name)
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
/** The graph [name], compiled now if it is not resident (closing others first when memory is short). */
|
| 155 |
+
fun get(name: String): Graph {
|
| 156 |
+
advance(name)
|
| 157 |
+
lastUse[name] = clock++
|
| 158 |
+
resident[name]?.let { return it }
|
| 159 |
+
makeRoom(name, evict = true)
|
| 160 |
+
return compile(name)
|
| 161 |
+
}
|
| 162 |
+
|
| 163 |
+
/** Compiles [name] if it fits without closing another graph; false when it does not fit. */
|
| 164 |
+
fun preload(name: String): Boolean {
|
| 165 |
+
if (name in resident) return true
|
| 166 |
+
val need = estimateKb(name) + reserveKb
|
| 167 |
+
val avail = DeviceInfo.memAvailableKb()
|
| 168 |
+
if (avail in 0 until need) {
|
| 169 |
+
event("preload_stop", name, avail, need)
|
| 170 |
+
return false
|
| 171 |
+
}
|
| 172 |
+
lastUse[name] = clock++
|
| 173 |
+
compile(name)
|
| 174 |
+
return true
|
| 175 |
+
}
|
| 176 |
+
|
| 177 |
+
fun release(name: String) {
|
| 178 |
+
resident.remove(name)?.let {
|
| 179 |
+
it.close()
|
| 180 |
+
event("close", name, DeviceInfo.memAvailableKb(), null)
|
| 181 |
+
}
|
| 182 |
+
}
|
| 183 |
+
|
| 184 |
+
fun releaseIf(predicate: (String) -> Boolean) {
|
| 185 |
+
for (n in resident.keys.filter(predicate)) release(n)
|
| 186 |
+
}
|
| 187 |
+
|
| 188 |
+
fun releaseAll() = releaseIf { true }
|
| 189 |
+
|
| 190 |
+
override fun close() = releaseAll()
|
| 191 |
+
|
| 192 |
+
private fun advance(name: String) {
|
| 193 |
+
if (schedule.isEmpty()) return
|
| 194 |
+
for (i in cursor + 1 until schedule.size) if (schedule[i] == name) {
|
| 195 |
+
cursor = i
|
| 196 |
+
return
|
| 197 |
+
}
|
| 198 |
+
for (i in 0..minOf(cursor, schedule.size - 1)) if (schedule[i] == name) {
|
| 199 |
+
cursor = i
|
| 200 |
+
return
|
| 201 |
+
}
|
| 202 |
+
}
|
| 203 |
+
|
| 204 |
+
/** Distance in the schedule to the next use of [name] after the current position (cyclic); MAX when never. */
|
| 205 |
+
private fun nextUse(name: String): Int {
|
| 206 |
+
if (schedule.isEmpty()) return Int.MAX_VALUE
|
| 207 |
+
for (i in cursor + 1 until schedule.size) if (schedule[i] == name) return i - cursor
|
| 208 |
+
for (i in 0..minOf(cursor, schedule.size - 1)) if (schedule[i] == name) return schedule.size - cursor + i
|
| 209 |
+
return Int.MAX_VALUE
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
private fun victim(exclude: String): String? {
|
| 213 |
+
val candidates = resident.keys.filter { it != exclude && it !in pinned }
|
| 214 |
+
if (candidates.isEmpty()) return null
|
| 215 |
+
if (schedule.isEmpty()) return candidates.minByOrNull { lastUse[it] ?: 0L }
|
| 216 |
+
return candidates.maxWithOrNull(compareBy<String>({ nextUse(it) }, { -(lastUse[it] ?: 0L) }))
|
| 217 |
+
}
|
| 218 |
+
|
| 219 |
+
/**
|
| 220 |
+
* Closes graphs until `MemAvailable` covers the estimate for [name] plus the reserve. With [memoryCheck], fails
|
| 221 |
+
* when nothing is left to close and even the reserve is not there (or the measured size of [name] is not).
|
| 222 |
+
*/
|
| 223 |
+
private fun makeRoom(name: String, evict: Boolean) {
|
| 224 |
+
val est = estimateKb(name)
|
| 225 |
+
while (true) {
|
| 226 |
+
val avail = DeviceInfo.memAvailableKb()
|
| 227 |
+
if (avail < 0 || avail >= est + reserveKb) return
|
| 228 |
+
val v = if (evict) victim(name) else null
|
| 229 |
+
if (v == null) {
|
| 230 |
+
event("short", name, avail, est + reserveKb)
|
| 231 |
+
val measured = footprintKb[name]
|
| 232 |
+
val floor = reserveKb + (measured ?: 0L)
|
| 233 |
+
if (memoryCheck && avail < floor) {
|
| 234 |
+
val now = awaitAvailable(floor, MAX_WAIT_SHORT_MS)
|
| 235 |
+
event("waited", name, now, floor)
|
| 236 |
+
if (now in 0 until floor) {
|
| 237 |
+
throw NotEnoughMemoryException(
|
| 238 |
+
"Not enough memory for $name: ${now / 1024} MB available, ${floor / 1024} MB needed " +
|
| 239 |
+
"(reserve ${reserveKb / 1024} MB). Close other apps and try again.",
|
| 240 |
+
)
|
| 241 |
+
}
|
| 242 |
+
}
|
| 243 |
+
return
|
| 244 |
+
}
|
| 245 |
+
event("evict", v, avail, est + reserveKb, "for" to name)
|
| 246 |
+
val freed = footprintKb[v]
|
| 247 |
+
release(v)
|
| 248 |
+
// A closed graph can return its GPU memory a few seconds late (Pixel 11 Pro, PowerVR: MemAvailable rose
|
| 249 |
+
// from 1.6 to 8.9 GB within 8 s of the closes); without this wait the loop closed every resident graph.
|
| 250 |
+
if (freed != null && freed > 0) awaitAvailable(avail + freed / 2, MAX_WAIT_EVICT_MS)
|
| 251 |
+
}
|
| 252 |
+
}
|
| 253 |
+
|
| 254 |
+
/** Polls `MemAvailable` until it reaches [targetKb] or [maxMs] pass; returns the last value read. */
|
| 255 |
+
private fun awaitAvailable(targetKb: Long, maxMs: Long): Long {
|
| 256 |
+
val t0 = SystemClock.elapsedRealtime()
|
| 257 |
+
var now = DeviceInfo.memAvailableKb()
|
| 258 |
+
while (now in 0 until targetKb && SystemClock.elapsedRealtime() - t0 < maxMs) {
|
| 259 |
+
Thread.sleep(POLL_MS)
|
| 260 |
+
now = DeviceInfo.memAvailableKb()
|
| 261 |
+
}
|
| 262 |
+
return now
|
| 263 |
+
}
|
| 264 |
+
|
| 265 |
+
private fun compile(name: String): Graph {
|
| 266 |
+
val spec = manifest.graph(name)
|
| 267 |
+
val file = File(dir, spec.file)
|
| 268 |
+
val options = if (spec.onCpu) {
|
| 269 |
+
CompiledModel.Options(Accelerator.CPU).apply {
|
| 270 |
+
cpuOptions = CompiledModel.CpuOptions(numThreads = cpuThreads)
|
| 271 |
+
}
|
| 272 |
+
} else {
|
| 273 |
+
CompiledModel.Options(Accelerator.GPU).apply {
|
| 274 |
+
gpuOptions = CompiledModel.GpuOptions(
|
| 275 |
+
// one copy of the weights for a graph with several signatures (the octree decoder's buckets)
|
| 276 |
+
constantTensorSharing = if (spec.signatures.size > 1) true else null,
|
| 277 |
+
precision = precisionOf(spec.precision),
|
| 278 |
+
priority = gpuPriority,
|
| 279 |
+
)
|
| 280 |
+
}
|
| 281 |
+
}
|
| 282 |
+
val before = DeviceInfo.memAvailableKb()
|
| 283 |
+
val t0 = SystemClock.elapsedRealtimeNanos()
|
| 284 |
+
val model = CompiledModel.create(file.absolutePath, options, env)
|
| 285 |
+
val graph = try {
|
| 286 |
+
Graph(spec, model)
|
| 287 |
+
} catch (e: Throwable) {
|
| 288 |
+
runCatching { model.close() }
|
| 289 |
+
throw e
|
| 290 |
+
}
|
| 291 |
+
val seconds = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
|
| 292 |
+
val after = DeviceInfo.memAvailableKb()
|
| 293 |
+
if (before >= 0 && after >= 0) footprintKb[name] = maxOf(footprintKb[name] ?: 0L, before - after)
|
| 294 |
+
resident[name] = graph
|
| 295 |
+
compiles.add(CompileRecord(name, seconds, before, after, phase))
|
| 296 |
+
event("compile", name, after, null, "seconds" to seconds)
|
| 297 |
+
Log.i(TAG, "compiled $name in ${"%.2f".format(seconds)} s (${spec.accelerator}, ${spec.precision}), " +
|
| 298 |
+
"MemAvailable ${before / 1024} -> ${after / 1024} MB, resident ${resident.size}")
|
| 299 |
+
return graph
|
| 300 |
+
}
|
| 301 |
+
|
| 302 |
+
/**
|
| 303 |
+
* The memory [name] is expected to take: what it took when it was last compiled, else an estimate from the
|
| 304 |
+
* file (weights at the run precision) and its largest tensor (a transformer block keeps a few dozen of those).
|
| 305 |
+
*/
|
| 306 |
+
fun estimateKb(name: String): Long {
|
| 307 |
+
footprintKb[name]?.let { if (it > 0) return it }
|
| 308 |
+
val spec = manifest.graph(name)
|
| 309 |
+
val elementBytes = if (spec.precision == "fp32" || spec.onCpu) 4L else 2L
|
| 310 |
+
val weights = if (elementBytes == 4L) spec.bytes else spec.bytes / 2
|
| 311 |
+
val transformer = name.startsWith("dino_") || name.startsWith("den_") || name == "oct" ||
|
| 312 |
+
name.startsWith("gs_") || name == "bir"
|
| 313 |
+
val activations = spec.maxIoElements * elementBytes * (if (transformer) 26 else 8)
|
| 314 |
+
return (weights + activations) / 1024
|
| 315 |
+
}
|
| 316 |
+
|
| 317 |
+
private fun event(kind: String, graph: String, availKb: Long, needKb: Long?, vararg extra: Pair<String, Any>) {
|
| 318 |
+
val e = JSONObject().put("t_ms", SystemClock.elapsedRealtime()).put("event", kind).put("graph", graph)
|
| 319 |
+
.put("mem_available_kb", availKb).put("resident", resident.size)
|
| 320 |
+
if (needKb != null) e.put("need_kb", needKb)
|
| 321 |
+
for ((k, v) in extra) e.put(k, v)
|
| 322 |
+
events.add(e)
|
| 323 |
+
if (kind != "compile") Log.i(TAG, "memory: $e")
|
| 324 |
+
}
|
| 325 |
+
|
| 326 |
+
companion object {
|
| 327 |
+
const val TAG = "IdolGraphs"
|
| 328 |
+
private const val POLL_MS = 100L
|
| 329 |
+
private const val MAX_WAIT_EVICT_MS = 1_500L
|
| 330 |
+
private const val MAX_WAIT_SHORT_MS = 8_000L
|
| 331 |
+
|
| 332 |
+
fun precisionOf(p: String): CompiledModel.GpuOptions.Precision = when (p) {
|
| 333 |
+
"fp32" -> CompiledModel.GpuOptions.Precision.FP32
|
| 334 |
+
"fp16" -> CompiledModel.GpuOptions.Precision.FP16
|
| 335 |
+
"fp16_with_fp32_accum" -> CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM
|
| 336 |
+
else -> CompiledModel.GpuOptions.Precision.DEFAULT
|
| 337 |
+
}
|
| 338 |
+
}
|
| 339 |
+
}
|
android/app/src/main/java/com/idol/IdolEngine.kt
ADDED
|
@@ -0,0 +1,606 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.os.SystemClock
|
| 4 |
+
import android.util.Log
|
| 5 |
+
import com.google.ai.edge.litert.CompiledModel
|
| 6 |
+
import com.google.ai.edge.litert.Environment
|
| 7 |
+
import com.google.ai.edge.litert.LiteRtException
|
| 8 |
+
import org.json.JSONArray
|
| 9 |
+
import org.json.JSONObject
|
| 10 |
+
import java.io.File
|
| 11 |
+
import java.util.concurrent.atomic.AtomicLong
|
| 12 |
+
import kotlin.math.sqrt
|
| 13 |
+
|
| 14 |
+
/** The stages the Run screen lists, in order. */
|
| 15 |
+
enum class Stage(val label: String) {
|
| 16 |
+
MATTING("Removing background"),
|
| 17 |
+
DINO("Image features (DINOv3)"),
|
| 18 |
+
VAE("Image latent (VAE)"),
|
| 19 |
+
DENOISER("Generating the 3D shape"),
|
| 20 |
+
OCTREE("Placing points"),
|
| 21 |
+
GAUSSIANS("Decoding Gaussians"),
|
| 22 |
+
PLY("Saving the .ply"),
|
| 23 |
+
}
|
| 24 |
+
|
| 25 |
+
/** What the engine tells the screen while it works (called on the model thread). */
|
| 26 |
+
interface Progress {
|
| 27 |
+
fun stage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {}
|
| 28 |
+
fun status(text: String) {}
|
| 29 |
+
fun cutout(composite: Image8) {}
|
| 30 |
+
}
|
| 31 |
+
|
| 32 |
+
class CancelledException : Exception("cancelled")
|
| 33 |
+
|
| 34 |
+
data class EngineOptions(
|
| 35 |
+
/** Compile the graphs before the first run and keep them (true), or per stage, closed after it (false). */
|
| 36 |
+
val preload: Boolean = true,
|
| 37 |
+
/**
|
| 38 |
+
* MemAvailable the memory guard keeps free. MemAvailable counts the page cache of the mapped graph files, and with
|
| 39 |
+
* 1 GB kept a compile burst outran its reclaim: lmkd killed the app on the Galaxy S26 at about 1.9 GB available.
|
| 40 |
+
*/
|
| 41 |
+
val reserveMb: Long = 2500,
|
| 42 |
+
/** Fail a run with a message when a graph does not fit even after closing all others (instead of trying). */
|
| 43 |
+
val memoryCheck: Boolean = true,
|
| 44 |
+
/** Pass a graph's output buffer straight to the next graph instead of copying it through the host. */
|
| 45 |
+
val directHandoff: Boolean = false,
|
| 46 |
+
/** "bir" (BiRefNet on the CPU, the author's matting) or "dis" (IS-Net DIS on the GPU). */
|
| 47 |
+
val matting: String = "bir",
|
| 48 |
+
val cpuThreads: Int = 6,
|
| 49 |
+
val gpuPriority: CompiledModel.GpuOptions.Priority? = null,
|
| 50 |
+
val disFloor: Int = Matting.DIS_ALPHA_FLOOR,
|
| 51 |
+
)
|
| 52 |
+
|
| 53 |
+
class RunResult(
|
| 54 |
+
val ply: File,
|
| 55 |
+
val gaussians: Int,
|
| 56 |
+
/** Seconds: matting, preprocess (matting included, as idol_runner times it), dinov3, vae, denoiser, octree,
|
| 57 |
+
* gaussian_decoder, ply, generate (preprocessed pixels ready -> .ply written), all. */
|
| 58 |
+
val times: LinkedHashMap<String, Double>,
|
| 59 |
+
val pre: Preprocessed,
|
| 60 |
+
val compiles: List<GraphPool.CompileRecord>,
|
| 61 |
+
val events: List<JSONObject>,
|
| 62 |
+
val noiseSource: String,
|
| 63 |
+
val handoff: String,
|
| 64 |
+
)
|
| 65 |
+
|
| 66 |
+
/**
|
| 67 |
+
* The phone pipeline, step for step scripts/idol_runner.py: matting -> preprocessing -> DINOv3 (4 graphs) + VAE
|
| 68 |
+
* (6 graphs) -> 5 Euler steps (den_ctx once per LoRA phase, den_in, 24 joint graphs) -> octree (8 levels, row
|
| 69 |
+
* buckets) -> Gaussian decoder (8 graphs) -> .ply. Every graph runs on its own GPU call (the 2 s rule of F0.md
|
| 70 |
+
* finding 2), every call on the one model thread. Create, use and close it on that thread.
|
| 71 |
+
*/
|
| 72 |
+
class IdolEngine(val modelDir: File, val manifest: Manifest, val options: EngineOptions) : AutoCloseable {
|
| 73 |
+
private val env: Environment = Environment.create()
|
| 74 |
+
val pool = GraphPool(
|
| 75 |
+
env, modelDir, manifest, options.reserveMb * 1024, options.memoryCheck, options.cpuThreads, options.gpuPriority,
|
| 76 |
+
)
|
| 77 |
+
|
| 78 |
+
private val posEmb = side("den_pos_emb.f32")
|
| 79 |
+
private val ropeCos = side("dino_rope_cos.f32")
|
| 80 |
+
private val ropeSin = side("dino_rope_sin.f32")
|
| 81 |
+
private val perturbation = side("gs_perturbation.f32")
|
| 82 |
+
private val bnMean: FloatArray
|
| 83 |
+
private val bnStd: FloatArray
|
| 84 |
+
private val rep: GaussianRep
|
| 85 |
+
private val gateSlow = Glue.loraGate(true)
|
| 86 |
+
private val gateFast = Glue.loraGate(false)
|
| 87 |
+
private val timeGrid = Glue.timeGrid()
|
| 88 |
+
|
| 89 |
+
/** Set from the screen thread; the model thread stops at the next graph. */
|
| 90 |
+
@Volatile var cancel = false
|
| 91 |
+
|
| 92 |
+
/** Direct handoff, until the first failure (then host copies, and the reason is kept). */
|
| 93 |
+
var direct = options.directHandoff
|
| 94 |
+
private set
|
| 95 |
+
var directFailure: String? = null
|
| 96 |
+
private set
|
| 97 |
+
|
| 98 |
+
private var progress: Progress = object : Progress {}
|
| 99 |
+
|
| 100 |
+
init {
|
| 101 |
+
val bn = JSONObject(File(modelDir, "vae_bn.json").readText())
|
| 102 |
+
val mean = bn.getJSONArray("bn_mean")
|
| 103 |
+
val variance = bn.getJSONArray("bn_var")
|
| 104 |
+
val eps = bn.getDouble("bn_eps").toFloat()
|
| 105 |
+
bnMean = FloatArray(mean.length()) { mean.getDouble(it).toFloat() }
|
| 106 |
+
bnStd = FloatArray(variance.length()) { sqrt((variance.getDouble(it).toFloat() + eps).toDouble()).toFloat() }
|
| 107 |
+
rep = GaussianRep.parse(JSONObject(File(modelDir, "gs_rep.json").readText()))
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
private fun side(name: String): FloatArray {
|
| 111 |
+
val s = requireNotNull(manifest.sideFiles[name]) { "manifest.json has no side file $name" }
|
| 112 |
+
return Fixture.readF32(File(modelDir, name), s.elements.toInt())
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
override fun close() {
|
| 116 |
+
pool.close()
|
| 117 |
+
runCatching { env.close() }
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
// ------------------------------------------------------------------------------------------------ schedule
|
| 121 |
+
// Each chain's graphs in manifest order (manifest.json lists them in pipeline order: dino_0..3, vae_0..3, vae_attn,
|
| 122 |
+
// vae_4, den_joint_00..23, gs_0..7), so a stage split into more or fewer graphs needs a new manifest, not new code.
|
| 123 |
+
private fun chain(prefix: String): List<String> = manifest.graphs.keys.filter { it.startsWith(prefix) }.also {
|
| 124 |
+
require(it.isNotEmpty()) { "manifest.json has no $prefix graphs" }
|
| 125 |
+
}
|
| 126 |
+
private val dinoNames = chain("dino_")
|
| 127 |
+
private val vaeNames = chain("vae_")
|
| 128 |
+
private val jointNames = chain("den_joint_")
|
| 129 |
+
private val gsNames = chain("gs_")
|
| 130 |
+
|
| 131 |
+
/** The order a run asks for graphs in. */
|
| 132 |
+
fun runSchedule(): List<String> {
|
| 133 |
+
val s = ArrayList<String>()
|
| 134 |
+
s.add(options.matting)
|
| 135 |
+
s.addAll(dinoNames)
|
| 136 |
+
s.addAll(vaeNames)
|
| 137 |
+
var phase: Boolean? = null
|
| 138 |
+
for (i in 0 until timeGrid.size - 1) {
|
| 139 |
+
val slow = timeGrid[i] >= Glue.PHASE_THRESHOLD
|
| 140 |
+
if (slow != phase) s.add("den_ctx")
|
| 141 |
+
phase = slow
|
| 142 |
+
s.add("den_in")
|
| 143 |
+
s.addAll(jointNames)
|
| 144 |
+
}
|
| 145 |
+
repeat(Glue.MAX_LEVEL) { s.add("oct") }
|
| 146 |
+
s.addAll(gsNames)
|
| 147 |
+
return s
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
/** The GPU graphs of a run in first-use order (the CPU matting graph is built for each matting and closed). */
|
| 151 |
+
fun preloadOrder(): List<String> = runSchedule().distinct().filter { !manifest.graph(it).onCpu }
|
| 152 |
+
|
| 153 |
+
/**
|
| 154 |
+
* preload = true: compiles the graphs before the first run, in first-use order, as many as fit above the reserve.
|
| 155 |
+
* Returns the load record (seconds per graph and the total).
|
| 156 |
+
*/
|
| 157 |
+
fun preload(progress: Progress): JSONObject {
|
| 158 |
+
cancel = false
|
| 159 |
+
val names = preloadOrder()
|
| 160 |
+
pool.phase = "preload"
|
| 161 |
+
pool.setSchedule(runSchedule())
|
| 162 |
+
val first = pool.compiles.size
|
| 163 |
+
val t0 = SystemClock.elapsedRealtimeNanos()
|
| 164 |
+
var loaded = 0
|
| 165 |
+
var stoppedAt: String? = null
|
| 166 |
+
try {
|
| 167 |
+
for (n in names) {
|
| 168 |
+
if (cancel) throw CancelledException()
|
| 169 |
+
progress.status("Loading models ${loaded + 1}/${names.size}")
|
| 170 |
+
if (!pool.preload(n)) {
|
| 171 |
+
stoppedAt = n
|
| 172 |
+
break
|
| 173 |
+
}
|
| 174 |
+
loaded++
|
| 175 |
+
}
|
| 176 |
+
} finally {
|
| 177 |
+
pool.phase = "run"
|
| 178 |
+
}
|
| 179 |
+
val total = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
|
| 180 |
+
progress.status(
|
| 181 |
+
if (stoppedAt == null) "Models loaded ($loaded/${names.size})"
|
| 182 |
+
else "Loaded $loaded/${names.size} models; the rest load during the run (memory)",
|
| 183 |
+
)
|
| 184 |
+
val graphs = JSONArray()
|
| 185 |
+
for (r in pool.compiles.subList(first, pool.compiles.size)) graphs.put(r.json())
|
| 186 |
+
return JSONObject().put("graphs", graphs).put("loaded", loaded).put("of", names.size)
|
| 187 |
+
.put("stopped_at", stoppedAt ?: JSONObject.NULL).put("total_s", total)
|
| 188 |
+
.put("compile_s", pool.compiles.subList(first, pool.compiles.size).sumOf { it.seconds })
|
| 189 |
+
.put("resident", JSONArray(pool.residentNames))
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
// ------------------------------------------------------------------------------------------------ the run
|
| 193 |
+
/** A picture (or a fixture's raw image) -> .ply at [outPly]. */
|
| 194 |
+
fun generate(image: Image8, mode: String, noise: Noise, outPly: File, progress: Progress): RunResult {
|
| 195 |
+
cancel = false
|
| 196 |
+
this.progress = progress
|
| 197 |
+
val firstCompile = pool.compiles.size
|
| 198 |
+
val firstEvent = pool.events.size
|
| 199 |
+
pool.phase = "run"
|
| 200 |
+
pool.setSchedule(runSchedule())
|
| 201 |
+
val times = LinkedHashMap<String, Double>()
|
| 202 |
+
val tAll = now()
|
| 203 |
+
|
| 204 |
+
progress.stage(Stage.MATTING, null, null, false)
|
| 205 |
+
var mattingS = 0.0
|
| 206 |
+
val tPre = now()
|
| 207 |
+
val pre = Preprocess.run(image, mode) { rgb ->
|
| 208 |
+
val t = now()
|
| 209 |
+
val a = alpha(rgb)
|
| 210 |
+
mattingS = since(t)
|
| 211 |
+
a
|
| 212 |
+
}
|
| 213 |
+
times["matting"] = mattingS
|
| 214 |
+
times["preprocess"] = since(tPre)
|
| 215 |
+
pool.release(options.matting)
|
| 216 |
+
progress.cutout(pre.composite)
|
| 217 |
+
progress.stage(Stage.MATTING, null, times["preprocess"], true)
|
| 218 |
+
val pixels = Preprocess.pixels(pre.composite)
|
| 219 |
+
|
| 220 |
+
val tGen = now()
|
| 221 |
+
progress.stage(Stage.DINO, null, null, false)
|
| 222 |
+
var t = now()
|
| 223 |
+
val f1 = dinov3(pixels)
|
| 224 |
+
times["dinov3"] = since(t)
|
| 225 |
+
progress.stage(Stage.DINO, null, times["dinov3"], true)
|
| 226 |
+
if (!options.preload) pool.releaseIf { it.startsWith("dino_") }
|
| 227 |
+
|
| 228 |
+
progress.stage(Stage.VAE, null, null, false)
|
| 229 |
+
t = now()
|
| 230 |
+
val moments = vaeMoments(pixels)
|
| 231 |
+
times["vae"] = since(t)
|
| 232 |
+
val f2 = cond2(Glue.vaeTokens(moments, noise.vae, bnMean, bnStd))
|
| 233 |
+
progress.stage(Stage.VAE, null, times["vae"], true)
|
| 234 |
+
if (!options.preload) pool.releaseIf { it.startsWith("vae_") }
|
| 235 |
+
|
| 236 |
+
val tDen = now()
|
| 237 |
+
val latent = sample(noise, f1, f2, times)
|
| 238 |
+
progress.stage(Stage.DENOISER, null, since(tDen), true)
|
| 239 |
+
if (!options.preload) pool.releaseIf { it.startsWith("den_") }
|
| 240 |
+
|
| 241 |
+
progress.stage(Stage.OCTREE, null, null, false)
|
| 242 |
+
t = now()
|
| 243 |
+
val latentTag = nextTag()
|
| 244 |
+
val anchors = Glue.octreeDescend({ cells, rows, level ->
|
| 245 |
+
progress.stage(Stage.OCTREE, "level $level of ${Glue.MAX_LEVEL}", null, false)
|
| 246 |
+
Glue.softmax8(octreeLogits(cells, rows, level, latent, latentTag), rows)
|
| 247 |
+
}, noise.pool, noise.leaf)
|
| 248 |
+
times["octree"] = since(t)
|
| 249 |
+
progress.stage(Stage.OCTREE, null, times["octree"], true)
|
| 250 |
+
if (!options.preload) pool.release("oct")
|
| 251 |
+
|
| 252 |
+
progress.stage(Stage.GAUSSIANS, null, null, false)
|
| 253 |
+
t = now()
|
| 254 |
+
val features = gaussianFeatures(anchors, latent, latentTag)
|
| 255 |
+
times["gaussian_decoder"] = since(t)
|
| 256 |
+
if (!options.preload) pool.releaseIf { it.startsWith("gs_") }
|
| 257 |
+
val g = Glue.toGaussians(anchors, features, rep, perturbation)
|
| 258 |
+
progress.stage(Stage.GAUSSIANS, null, since(t), true)
|
| 259 |
+
|
| 260 |
+
progress.stage(Stage.PLY, null, null, false)
|
| 261 |
+
t = now()
|
| 262 |
+
Glue.writePly(Glue.plyRows(g), outPly)
|
| 263 |
+
times["ply"] = since(t)
|
| 264 |
+
progress.stage(Stage.PLY, null, times["ply"], true)
|
| 265 |
+
times["generate"] = since(tGen)
|
| 266 |
+
times["all"] = since(tAll)
|
| 267 |
+
Log.i(TAG, "run done: " + times.entries.joinToString { "${it.key} ${"%.2f".format(it.value)}" })
|
| 268 |
+
return RunResult(
|
| 269 |
+
outPly, g.count, times, pre,
|
| 270 |
+
pool.compiles.subList(firstCompile, pool.compiles.size).toList(),
|
| 271 |
+
pool.events.subList(firstEvent, pool.events.size).toList(),
|
| 272 |
+
noise.source, handoffName(),
|
| 273 |
+
)
|
| 274 |
+
}
|
| 275 |
+
|
| 276 |
+
fun handoffName(): String = when {
|
| 277 |
+
direct -> "direct"
|
| 278 |
+
options.directHandoff -> "host (direct failed: $directFailure)"
|
| 279 |
+
else -> "host"
|
| 280 |
+
}
|
| 281 |
+
|
| 282 |
+
/** feature2 = 5 zero tokens + the 4,096 VAE tokens: [1, 4101, 128]. */
|
| 283 |
+
private fun cond2(tokens: FloatArray): FloatArray {
|
| 284 |
+
val f2 = FloatArray(Glue.COND_TOKENS * 128)
|
| 285 |
+
System.arraycopy(tokens, 0, f2, 5 * 128, tokens.size)
|
| 286 |
+
return f2
|
| 287 |
+
}
|
| 288 |
+
|
| 289 |
+
// ------------------------------------------------------------------------------------------------ stages
|
| 290 |
+
private fun graph(name: String): Graph {
|
| 291 |
+
if (cancel) throw CancelledException()
|
| 292 |
+
if (!pool.isResident(name)) progress.status("Loading $name")
|
| 293 |
+
val g = pool.get(name)
|
| 294 |
+
progress.status("")
|
| 295 |
+
return g
|
| 296 |
+
}
|
| 297 |
+
|
| 298 |
+
/** The matting graph's uint8 alpha for the resized RGB picture; the graph is closed right after. */
|
| 299 |
+
private fun alpha(rgb: Image8): ByteArray {
|
| 300 |
+
val m = options.matting
|
| 301 |
+
val input = if (m == "bir") Matting.birefnetInput(rgb) else Matting.disInput(rgb)
|
| 302 |
+
try {
|
| 303 |
+
val g = graph(m)
|
| 304 |
+
g.write(0, input)
|
| 305 |
+
g.run()
|
| 306 |
+
val map = g.read(0)
|
| 307 |
+
pool.release(m)
|
| 308 |
+
return if (m == "bir") Matting.birefnetAlpha(map, rgb.width, rgb.height)
|
| 309 |
+
else Matting.disAlpha(map, rgb.width, rgb.height, options.disFloor)
|
| 310 |
+
} finally {
|
| 311 |
+
pool.release(m)
|
| 312 |
+
}
|
| 313 |
+
}
|
| 314 |
+
|
| 315 |
+
private class Link(val from: Graph, val fromOut: Int, val to: Graph, val toIn: Int, val direct: Boolean)
|
| 316 |
+
|
| 317 |
+
/**
|
| 318 |
+
* The next graph [toName] of a chain, fed [from]'s output [fromOut] at input [toIn]: copied through the host now
|
| 319 |
+
* (the read waits for [from]'s GPU work), or handed over as the buffer itself at [run] with direct handoff.
|
| 320 |
+
*/
|
| 321 |
+
private fun link(from: Graph, fromOut: Int, toName: String, toIn: Int = 0): Link {
|
| 322 |
+
if (direct) {
|
| 323 |
+
pool.pin(from.name)
|
| 324 |
+
try {
|
| 325 |
+
return Link(from, fromOut, graph(toName), toIn, true)
|
| 326 |
+
} catch (e: Throwable) {
|
| 327 |
+
pool.unpin(from.name)
|
| 328 |
+
throw e
|
| 329 |
+
}
|
| 330 |
+
}
|
| 331 |
+
val data = from.read(fromOut)
|
| 332 |
+
val to = graph(toName)
|
| 333 |
+
to.write(toIn, data)
|
| 334 |
+
return Link(from, fromOut, to, toIn, false)
|
| 335 |
+
}
|
| 336 |
+
|
| 337 |
+
private fun run(l: Link) {
|
| 338 |
+
if (!l.direct) {
|
| 339 |
+
l.to.run()
|
| 340 |
+
return
|
| 341 |
+
}
|
| 342 |
+
try {
|
| 343 |
+
l.to.run(overrides = mapOf(l.toIn to l.from.outputBuffer(l.fromOut)))
|
| 344 |
+
l.to.untag(l.toIn)
|
| 345 |
+
} catch (e: LiteRtException) {
|
| 346 |
+
Log.w(TAG, "direct handoff ${l.from.name} -> ${l.to.name} failed, host copies from now on", e)
|
| 347 |
+
direct = false
|
| 348 |
+
directFailure = "${l.from.name} -> ${l.to.name}: ${e.message}"
|
| 349 |
+
l.to.write(l.toIn, l.from.read(l.fromOut))
|
| 350 |
+
l.to.run()
|
| 351 |
+
} finally {
|
| 352 |
+
pool.unpin(l.from.name)
|
| 353 |
+
}
|
| 354 |
+
}
|
| 355 |
+
|
| 356 |
+
private fun dinov3(pixels: FloatArray): FloatArray {
|
| 357 |
+
var g = graph(dinoNames[0])
|
| 358 |
+
g.write(0, pixels)
|
| 359 |
+
g.write(1, ropeCos, "rope_cos")
|
| 360 |
+
g.write(2, ropeSin, "rope_sin")
|
| 361 |
+
g.run()
|
| 362 |
+
for (name in dinoNames.drop(1)) {
|
| 363 |
+
val l = link(g, 0, name)
|
| 364 |
+
l.to.write(1, ropeCos, "rope_cos")
|
| 365 |
+
l.to.write(2, ropeSin, "rope_sin")
|
| 366 |
+
run(l)
|
| 367 |
+
g = l.to
|
| 368 |
+
}
|
| 369 |
+
return g.read(0)
|
| 370 |
+
}
|
| 371 |
+
|
| 372 |
+
private fun vaeMoments(pixels: FloatArray): FloatArray {
|
| 373 |
+
var g = graph(vaeNames[0])
|
| 374 |
+
g.write(0, pixels)
|
| 375 |
+
g.run()
|
| 376 |
+
for (name in vaeNames.drop(1)) {
|
| 377 |
+
val l = link(g, 0, name)
|
| 378 |
+
run(l)
|
| 379 |
+
g = l.to
|
| 380 |
+
}
|
| 381 |
+
return g.read(0)
|
| 382 |
+
}
|
| 383 |
+
|
| 384 |
+
/** The conditioning of one LoRA phase (den_ctx), kept for the steps of that phase. */
|
| 385 |
+
private class Ctx(val h: FloatArray, val tag: Long)
|
| 386 |
+
|
| 387 |
+
/** sample: 5 Euler steps from the shared noise -> the final latent [1, 8192, 16]. */
|
| 388 |
+
private fun sample(noise: Noise, f1: FloatArray, f2: FloatArray, times: MutableMap<String, Double>): FloatArray {
|
| 389 |
+
val xLat = noise.latent.copyOf()
|
| 390 |
+
val xCam = noise.camera.copyOf()
|
| 391 |
+
val ctx = HashMap<Boolean, Ctx>()
|
| 392 |
+
val condTag = nextTag()
|
| 393 |
+
var den = 0.0
|
| 394 |
+
val steps = timeGrid.size - 1
|
| 395 |
+
for (i in 0 until steps) {
|
| 396 |
+
progress.stage(Stage.DENOISER, "step ${i + 1} of $steps", null, false)
|
| 397 |
+
val t0 = now()
|
| 398 |
+
val (v, vc) = velocity(xLat, xCam, timeGrid[i], f1, f2, condTag, ctx)
|
| 399 |
+
den += since(t0)
|
| 400 |
+
val dt = (timeGrid[i] - timeGrid[i + 1]).toFloat()
|
| 401 |
+
for (k in xLat.indices) xLat[k] = xLat[k] - v[k] * dt
|
| 402 |
+
for (k in xCam.indices) xCam[k] = xCam[k] - vc[k] * dt
|
| 403 |
+
}
|
| 404 |
+
times["denoiser"] = den
|
| 405 |
+
return xLat
|
| 406 |
+
}
|
| 407 |
+
|
| 408 |
+
/** velocity: one denoiser call at time t -> (latent velocity [1, 8192, 16], camera velocity [1, 1, 5]). */
|
| 409 |
+
private fun velocity(
|
| 410 |
+
xLat: FloatArray,
|
| 411 |
+
xCam: FloatArray,
|
| 412 |
+
t: Double,
|
| 413 |
+
f1: FloatArray,
|
| 414 |
+
f2: FloatArray,
|
| 415 |
+
condTag: Long,
|
| 416 |
+
ctx: HashMap<Boolean, Ctx>,
|
| 417 |
+
): Pair<FloatArray, FloatArray> {
|
| 418 |
+
val slow = t >= Glue.PHASE_THRESHOLD
|
| 419 |
+
val gate = if (slow) gateSlow else gateFast
|
| 420 |
+
val gateTag = if (slow) "gate_slow" else "gate_fast"
|
| 421 |
+
val c = ctx.getOrPut(slow) {
|
| 422 |
+
val g = graph("den_ctx")
|
| 423 |
+
g.write(0, f1, "f1_$condTag")
|
| 424 |
+
g.write(1, f2, "f2_$condTag")
|
| 425 |
+
g.write(2, gate, gateTag)
|
| 426 |
+
g.run()
|
| 427 |
+
Ctx(g.read(0), nextTag())
|
| 428 |
+
}
|
| 429 |
+
val din = graph("den_in")
|
| 430 |
+
din.write(0, xLat)
|
| 431 |
+
din.write(1, xCam)
|
| 432 |
+
din.write(2, Glue.sinusoid(1000 * t))
|
| 433 |
+
din.write(3, posEmb, "pos_emb")
|
| 434 |
+
din.write(4, gate, gateTag)
|
| 435 |
+
din.run()
|
| 436 |
+
val hCam = din.read(1)
|
| 437 |
+
val tMod = din.read(2)
|
| 438 |
+
val tEmb = din.read(3)
|
| 439 |
+
val tModTag = nextTag()
|
| 440 |
+
var prev = din
|
| 441 |
+
for ((k, name) in jointNames.withIndex()) {
|
| 442 |
+
val l = link(prev, 0, name)
|
| 443 |
+
val j = l.to
|
| 444 |
+
when (k) {
|
| 445 |
+
0 -> {
|
| 446 |
+
j.write(1, c.h, c.tag)
|
| 447 |
+
j.write(2, hCam)
|
| 448 |
+
j.write(3, tMod, tModTag)
|
| 449 |
+
j.write(4, gate, gateTag)
|
| 450 |
+
}
|
| 451 |
+
jointNames.size - 1 -> {
|
| 452 |
+
j.write(1, tMod, tModTag)
|
| 453 |
+
j.write(2, tEmb)
|
| 454 |
+
j.write(3, gate, gateTag)
|
| 455 |
+
}
|
| 456 |
+
else -> {
|
| 457 |
+
j.write(1, tMod, tModTag)
|
| 458 |
+
j.write(2, gate, gateTag)
|
| 459 |
+
}
|
| 460 |
+
}
|
| 461 |
+
run(l)
|
| 462 |
+
prev = j
|
| 463 |
+
}
|
| 464 |
+
return prev.read(0) to prev.read(1)
|
| 465 |
+
}
|
| 466 |
+
|
| 467 |
+
/** octree_logits: the decoder's child logits for [n] cells at [level], rows padded to the bucket -> [n, 8]. */
|
| 468 |
+
private fun octreeLogits(cells: FloatArray, n: Int, level: Int, latent: FloatArray, latentTag: Any?): FloatArray {
|
| 469 |
+
val b = Glue.OCT_BUCKETS.first { n <= it }
|
| 470 |
+
val key = "L$b"
|
| 471 |
+
val c = FloatArray(b * 3)
|
| 472 |
+
System.arraycopy(cells, 0, c, 0, n * 3)
|
| 473 |
+
val pe = Glue.pointPe(c, b)
|
| 474 |
+
val lf = Glue.sinusoid((1 shl level).toDouble(), maxPeriod = 1024.0, twoPi = true)
|
| 475 |
+
val g = graph("oct")
|
| 476 |
+
g.write(0, c, key = key)
|
| 477 |
+
g.write(1, pe, key = key)
|
| 478 |
+
g.write(2, lf, key = key)
|
| 479 |
+
g.write(3, latent, latentTag, key)
|
| 480 |
+
g.run(key)
|
| 481 |
+
return g.read(0, key).copyOf(n * 8)
|
| 482 |
+
}
|
| 483 |
+
|
| 484 |
+
/** The Gaussian decoder's 8 graphs: anchors [8192, 3] + latent -> features [8192, 480]. */
|
| 485 |
+
private fun gaussianFeatures(anchors: FloatArray, latent: FloatArray, latentTag: Any?): FloatArray {
|
| 486 |
+
val pe = Glue.pointPe(anchors, Glue.NUM_POINTS)
|
| 487 |
+
var g = graph(gsNames[0])
|
| 488 |
+
g.write(0, anchors)
|
| 489 |
+
g.write(1, pe)
|
| 490 |
+
g.write(2, latent, latentTag)
|
| 491 |
+
g.run()
|
| 492 |
+
for (name in gsNames.drop(1)) {
|
| 493 |
+
val l = link(g, 0, name)
|
| 494 |
+
l.to.write(1, latent, latentTag)
|
| 495 |
+
run(l)
|
| 496 |
+
g = l.to
|
| 497 |
+
}
|
| 498 |
+
return g.read(0)
|
| 499 |
+
}
|
| 500 |
+
|
| 501 |
+
// ------------------------------------------------------------------------------------------------ parity
|
| 502 |
+
/**
|
| 503 |
+
* CONTRACT.md line (a) on the phone: each part gets the fixture's reference input for it (no chaining) and its
|
| 504 |
+
* output is written as `<id>.<name>.f32` (little-endian float32) in [out]; the matting alpha as `<id>.alpha.u8`.
|
| 505 |
+
*/
|
| 506 |
+
fun parity(fx: Fixture, out: File, id: String, progress: Progress): JSONObject {
|
| 507 |
+
cancel = false
|
| 508 |
+
this.progress = progress
|
| 509 |
+
pool.phase = "parity"
|
| 510 |
+
pool.setSchedule(emptyList())
|
| 511 |
+
val arrays = JSONObject()
|
| 512 |
+
val seconds = JSONObject()
|
| 513 |
+
fun dump(name: String, data: FloatArray, shape: IntArray) {
|
| 514 |
+
Fixture.writeF32(File(out, "$id.$name.f32"), data)
|
| 515 |
+
arrays.put(name, JSONObject().put("file", "$id.$name.f32").put("shape", JSONArray(shape.toList())))
|
| 516 |
+
}
|
| 517 |
+
val result = JSONObject().put("id", id).put("fixture", fx.dir.name).put("matting", options.matting)
|
| 518 |
+
|
| 519 |
+
progress.stage(Stage.MATTING, null, null, false)
|
| 520 |
+
var t = now()
|
| 521 |
+
val photo = Photo.fromFile(fx.image)
|
| 522 |
+
val pre = Preprocess.run(photo.image, fx.mode) { alpha(it) }
|
| 523 |
+
File(out, "$id.alpha.u8").writeBytes(pre.alphaRaw)
|
| 524 |
+
result.put("alpha", JSONObject().put("file", "$id.alpha.u8").put("width", pre.alphaWidth)
|
| 525 |
+
.put("height", pre.alphaHeight).put("box", JSONArray(pre.box.toList())))
|
| 526 |
+
seconds.put("matting", since(t))
|
| 527 |
+
progress.stage(Stage.MATTING, null, since(t), true)
|
| 528 |
+
|
| 529 |
+
val pixels = fx.floats("pixels")
|
| 530 |
+
progress.stage(Stage.DINO, null, null, false)
|
| 531 |
+
t = now()
|
| 532 |
+
dump("feature1", dinov3(pixels), intArrayOf(1, Glue.COND_TOKENS, 1280))
|
| 533 |
+
seconds.put("dinov3", since(t))
|
| 534 |
+
pool.releaseIf { it.startsWith("dino_") }
|
| 535 |
+
progress.stage(Stage.DINO, null, since(t), true)
|
| 536 |
+
|
| 537 |
+
progress.stage(Stage.VAE, null, null, false)
|
| 538 |
+
t = now()
|
| 539 |
+
val moments = vaeMoments(pixels)
|
| 540 |
+
dump("vae_moments", moments, intArrayOf(1, 64, 128, 128))
|
| 541 |
+
dump("vae_tokens", Glue.vaeTokens(moments, fx.floats("vae_noise"), bnMean, bnStd), intArrayOf(1, 4096, 128))
|
| 542 |
+
seconds.put("vae", since(t))
|
| 543 |
+
pool.releaseIf { it.startsWith("vae_") }
|
| 544 |
+
progress.stage(Stage.VAE, null, since(t), true)
|
| 545 |
+
|
| 546 |
+
t = now()
|
| 547 |
+
val f1 = fx.floats("feature1")
|
| 548 |
+
val f2 = fx.floats("feature2")
|
| 549 |
+
// t comes from the engine's float64 grid, as the reference passes float(ts[i]) (1000 t rounds differently from
|
| 550 |
+
// the fixture's float32 copy); the fixture's grid only confirms it is the same schedule
|
| 551 |
+
val tg = fx.floats("t_grid")
|
| 552 |
+
result.put("t_grid_matches_fixture", tg.size == timeGrid.size && tg.indices.all { tg[it] == timeGrid[it].toFloat() })
|
| 553 |
+
val ctx = HashMap<Boolean, Ctx>()
|
| 554 |
+
val condTag = nextTag()
|
| 555 |
+
for (i in 0 until timeGrid.size - 1) {
|
| 556 |
+
progress.stage(Stage.DENOISER, "step ${i + 1} of ${timeGrid.size - 1}", null, false)
|
| 557 |
+
val (v, vc) = velocity(fx.floats("x_latent_$i"), fx.floats("x_camera_$i"), timeGrid[i], f1, f2, condTag, ctx)
|
| 558 |
+
dump("v_latent_$i", v, intArrayOf(1, Glue.NUM_POINTS, Glue.IN_CH))
|
| 559 |
+
dump("v_camera_$i", vc, intArrayOf(1, 1, Glue.CAM_CH))
|
| 560 |
+
}
|
| 561 |
+
seconds.put("denoiser", since(t))
|
| 562 |
+
pool.releaseIf { it.startsWith("den_") }
|
| 563 |
+
progress.stage(Stage.DENOISER, null, since(t), true)
|
| 564 |
+
|
| 565 |
+
t = now()
|
| 566 |
+
val latent = fx.floats("latent")
|
| 567 |
+
val latentTag = nextTag()
|
| 568 |
+
for (level in 1..Glue.MAX_LEVEL) {
|
| 569 |
+
progress.stage(Stage.OCTREE, "level $level of ${Glue.MAX_LEVEL}", null, false)
|
| 570 |
+
val cells = fx.floats("oct_cells_$level")
|
| 571 |
+
val n = cells.size / 3
|
| 572 |
+
dump("oct_logits_$level", octreeLogits(cells, n, level, latent, latentTag), intArrayOf(1, n, 8))
|
| 573 |
+
}
|
| 574 |
+
seconds.put("octree", since(t))
|
| 575 |
+
pool.release("oct")
|
| 576 |
+
progress.stage(Stage.OCTREE, null, since(t), true)
|
| 577 |
+
|
| 578 |
+
progress.stage(Stage.GAUSSIANS, null, null, false)
|
| 579 |
+
t = now()
|
| 580 |
+
val anchors = fx.floats("anchors")
|
| 581 |
+
val feats = gaussianFeatures(anchors, latent, latentTag)
|
| 582 |
+
seconds.put("gaussian_decoder", since(t))
|
| 583 |
+
pool.releaseIf { it.startsWith("gs_") }
|
| 584 |
+
dump("gs_features", feats, intArrayOf(1, Glue.NUM_POINTS, 480))
|
| 585 |
+
val g = Glue.toGaussians(anchors, feats, rep, perturbation)
|
| 586 |
+
dump("g_xyz", g.xyz, intArrayOf(g.count, 3))
|
| 587 |
+
dump("g_features_dc", g.featuresDc, intArrayOf(g.count, 1, 3))
|
| 588 |
+
dump("g_scaling", g.scaling, intArrayOf(g.count, 3))
|
| 589 |
+
dump("g_rotation", g.rotation, intArrayOf(g.count, 4))
|
| 590 |
+
dump("g_opacity", g.opacity, intArrayOf(g.count, 1))
|
| 591 |
+
progress.stage(Stage.GAUSSIANS, null, since(t), true)
|
| 592 |
+
pool.phase = "run"
|
| 593 |
+
return result.put("arrays", arrays).put("seconds", seconds).put("handoff", handoffName())
|
| 594 |
+
}
|
| 595 |
+
|
| 596 |
+
companion object {
|
| 597 |
+
const val TAG = "IdolEngine"
|
| 598 |
+
private val tags = AtomicLong(1)
|
| 599 |
+
|
| 600 |
+
fun nextTag(): Long = tags.getAndIncrement()
|
| 601 |
+
|
| 602 |
+
private fun now() = SystemClock.elapsedRealtimeNanos()
|
| 603 |
+
|
| 604 |
+
private fun since(t0: Long) = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
|
| 605 |
+
}
|
| 606 |
+
}
|
android/app/src/main/java/com/idol/MainActivity.kt
ADDED
|
@@ -0,0 +1,581 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.Manifest
|
| 4 |
+
import android.content.ActivityNotFoundException
|
| 5 |
+
import android.content.ComponentCallbacks2
|
| 6 |
+
import android.content.Intent
|
| 7 |
+
import android.content.pm.ApplicationInfo
|
| 8 |
+
import android.content.pm.PackageManager
|
| 9 |
+
import android.net.Uri
|
| 10 |
+
import android.os.Build
|
| 11 |
+
import android.os.Bundle
|
| 12 |
+
import android.os.Handler
|
| 13 |
+
import android.os.Looper
|
| 14 |
+
import android.os.SystemClock
|
| 15 |
+
import android.provider.OpenableColumns
|
| 16 |
+
import android.util.Log
|
| 17 |
+
import android.util.TypedValue
|
| 18 |
+
import android.view.View
|
| 19 |
+
import android.view.WindowManager
|
| 20 |
+
import android.webkit.WebView
|
| 21 |
+
import android.widget.Button
|
| 22 |
+
import android.widget.CheckBox
|
| 23 |
+
import android.widget.FrameLayout
|
| 24 |
+
import android.widget.ImageView
|
| 25 |
+
import android.widget.LinearLayout
|
| 26 |
+
import android.widget.RadioGroup
|
| 27 |
+
import android.widget.TextView
|
| 28 |
+
import androidx.activity.ComponentActivity
|
| 29 |
+
import androidx.activity.OnBackPressedCallback
|
| 30 |
+
import androidx.activity.result.PickVisualMediaRequest
|
| 31 |
+
import androidx.activity.result.contract.ActivityResultContracts
|
| 32 |
+
import androidx.core.content.FileProvider
|
| 33 |
+
import androidx.core.view.ViewCompat
|
| 34 |
+
import androidx.core.view.WindowInsetsCompat
|
| 35 |
+
import org.json.JSONObject
|
| 36 |
+
import java.io.File
|
| 37 |
+
import java.text.SimpleDateFormat
|
| 38 |
+
import java.util.Date
|
| 39 |
+
import java.util.Locale
|
| 40 |
+
import java.util.concurrent.Callable
|
| 41 |
+
import java.util.concurrent.CountDownLatch
|
| 42 |
+
import java.util.concurrent.ExecutorService
|
| 43 |
+
import java.util.concurrent.Executors
|
| 44 |
+
|
| 45 |
+
/** How a run ended, for the screen and the autorun. */
|
| 46 |
+
sealed class Outcome {
|
| 47 |
+
class Done(val result: RunResult) : Outcome()
|
| 48 |
+
class NoObject(val reason: String) : Outcome()
|
| 49 |
+
class Missing(val problems: List<ModelFiles.Problem>) : Outcome()
|
| 50 |
+
class Failed(val kind: String, val message: String) : Outcome()
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
/** The model state of the process: one model thread and one engine, kept across activity instances. */
|
| 54 |
+
object Idol {
|
| 55 |
+
/** Every LiteRT call (compile, run, close) happens on this one thread. */
|
| 56 |
+
val model: ExecutorService = Executors.newSingleThreadExecutor { r -> Thread(r, "IdolModel") }
|
| 57 |
+
@Volatile var engine: IdolEngine? = null
|
| 58 |
+
@Volatile var problems: List<ModelFiles.Problem>? = null
|
| 59 |
+
@Volatile var started = false
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
/**
|
| 63 |
+
* Home (pick or take a photo) -> Run (the stages with their seconds) -> Result (the splat viewer). Every run, from a
|
| 64 |
+
* button or from the debug autorun ([AutoRun]), goes through [startRun]; the work is on the model thread, never on
|
| 65 |
+
* the UI thread.
|
| 66 |
+
*/
|
| 67 |
+
class MainActivity : ComponentActivity() {
|
| 68 |
+
private lateinit var home: View
|
| 69 |
+
private lateinit var runScreen: View
|
| 70 |
+
private lateinit var resultScreen: View
|
| 71 |
+
private lateinit var modeGroup: RadioGroup
|
| 72 |
+
private lateinit var disCheck: CheckBox
|
| 73 |
+
private lateinit var homeMessage: TextView
|
| 74 |
+
private lateinit var homeStatus: TextView
|
| 75 |
+
private lateinit var picture: ImageView
|
| 76 |
+
private lateinit var cutout: ImageView
|
| 77 |
+
private lateinit var stageList: LinearLayout
|
| 78 |
+
private lateinit var totalText: TextView
|
| 79 |
+
private lateinit var runStatus: TextView
|
| 80 |
+
private lateinit var viewerBox: FrameLayout
|
| 81 |
+
private var web: WebView? = null
|
| 82 |
+
|
| 83 |
+
private val ui = Handler(Looper.getMainLooper())
|
| 84 |
+
private val decoder = Executors.newSingleThreadExecutor { r -> Thread(r, "IdolDecode") }
|
| 85 |
+
private val stageRows = ArrayList<TextView>()
|
| 86 |
+
|
| 87 |
+
private class StageState {
|
| 88 |
+
var detail: String? = null
|
| 89 |
+
var seconds: Double? = null
|
| 90 |
+
var startedAt = 0L
|
| 91 |
+
var running = false
|
| 92 |
+
var done = false
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
private val stages = Array(Stage.entries.size) { StageState() }
|
| 96 |
+
@Volatile private var running = false
|
| 97 |
+
@Volatile var autorunActive = false
|
| 98 |
+
private set
|
| 99 |
+
private var lastPly: File? = null
|
| 100 |
+
private var deviceLine = ""
|
| 101 |
+
|
| 102 |
+
/** Engine progress, posted to the screen. */
|
| 103 |
+
val progress = object : Progress {
|
| 104 |
+
override fun stage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {
|
| 105 |
+
ui.post { onStage(stage, detail, seconds, done) }
|
| 106 |
+
}
|
| 107 |
+
|
| 108 |
+
override fun status(text: String) {
|
| 109 |
+
ui.post {
|
| 110 |
+
runStatus.text = text
|
| 111 |
+
homeStatus.text = if (text.isEmpty()) deviceLine else "$text\n$deviceLine"
|
| 112 |
+
}
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
override fun cutout(composite: Image8) {
|
| 116 |
+
val bmp = Photo.displayBitmap(composite, 512)
|
| 117 |
+
ui.post {
|
| 118 |
+
cutout.setImageBitmap(bmp)
|
| 119 |
+
cutout.visibility = View.VISIBLE
|
| 120 |
+
}
|
| 121 |
+
}
|
| 122 |
+
}
|
| 123 |
+
|
| 124 |
+
private val pickPhoto = registerForActivityResult(ActivityResultContracts.PickVisualMedia()) { uri ->
|
| 125 |
+
if (uri != null) startFromPicture { Photo.fromUri(contentResolver, uri, displayName(uri)) }
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
private val takePhoto = registerForActivityResult(ActivityResultContracts.TakePicture()) { ok ->
|
| 129 |
+
// the camera app can take enough memory for Android to stop this process while it is open; the activity is
|
| 130 |
+
// then created again for the result, so the photo's path is a fixed one, not a field
|
| 131 |
+
val file = cameraFile()
|
| 132 |
+
if (ok && file.isFile) startFromPicture { Photo.fromFile(file) }
|
| 133 |
+
}
|
| 134 |
+
|
| 135 |
+
private val cameraPermission = registerForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
|
| 136 |
+
if (granted) launchCamera() else showHome(getString(R.string.camera_denied))
|
| 137 |
+
}
|
| 138 |
+
|
| 139 |
+
override fun onCreate(savedInstanceState: Bundle?) {
|
| 140 |
+
super.onCreate(savedInstanceState)
|
| 141 |
+
if (isDebuggable()) {
|
| 142 |
+
// the autorun drives a phone nobody unlocked: stay in front (cpuset top-app) with the screen on
|
| 143 |
+
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
|
| 144 |
+
if (Build.VERSION.SDK_INT >= 27) {
|
| 145 |
+
setShowWhenLocked(true)
|
| 146 |
+
setTurnScreenOn(true)
|
| 147 |
+
}
|
| 148 |
+
}
|
| 149 |
+
setContentView(R.layout.activity_main)
|
| 150 |
+
// targetSdk 35 draws edge to edge: keep the screens clear of the status and navigation bars
|
| 151 |
+
val root = findViewById<View>(R.id.root)
|
| 152 |
+
ViewCompat.setOnApplyWindowInsetsListener(root) { v, insets ->
|
| 153 |
+
val bars = insets.getInsets(WindowInsetsCompat.Type.systemBars())
|
| 154 |
+
v.setPadding(bars.left, bars.top, bars.right, bars.bottom)
|
| 155 |
+
insets
|
| 156 |
+
}
|
| 157 |
+
home = findViewById(R.id.home)
|
| 158 |
+
runScreen = findViewById(R.id.run)
|
| 159 |
+
resultScreen = findViewById(R.id.result)
|
| 160 |
+
modeGroup = findViewById(R.id.mode)
|
| 161 |
+
disCheck = findViewById(R.id.dis)
|
| 162 |
+
homeMessage = findViewById(R.id.home_message)
|
| 163 |
+
homeStatus = findViewById(R.id.home_status)
|
| 164 |
+
picture = findViewById(R.id.picture)
|
| 165 |
+
cutout = findViewById(R.id.cutout)
|
| 166 |
+
stageList = findViewById(R.id.stages)
|
| 167 |
+
totalText = findViewById(R.id.total)
|
| 168 |
+
runStatus = findViewById(R.id.run_status)
|
| 169 |
+
viewerBox = findViewById(R.id.viewer)
|
| 170 |
+
|
| 171 |
+
for (s in Stage.entries) {
|
| 172 |
+
val tv = TextView(this)
|
| 173 |
+
tv.setTextSize(TypedValue.COMPLEX_UNIT_SP, 16f)
|
| 174 |
+
tv.setPadding(0, dp(6), 0, dp(6))
|
| 175 |
+
stageList.addView(tv)
|
| 176 |
+
stageRows.add(tv)
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
findViewById<Button>(R.id.pick).setOnClickListener {
|
| 180 |
+
pickPhoto.launch(PickVisualMediaRequest(ActivityResultContracts.PickVisualMedia.ImageOnly))
|
| 181 |
+
}
|
| 182 |
+
findViewById<Button>(R.id.camera).setOnClickListener {
|
| 183 |
+
if (checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
|
| 184 |
+
launchCamera()
|
| 185 |
+
} else {
|
| 186 |
+
cameraPermission.launch(Manifest.permission.CAMERA)
|
| 187 |
+
}
|
| 188 |
+
}
|
| 189 |
+
findViewById<Button>(R.id.share).setOnClickListener { share() }
|
| 190 |
+
findViewById<Button>(R.id.new_run).setOnClickListener { showHome(null) }
|
| 191 |
+
|
| 192 |
+
val prefs = getSharedPreferences("settings", MODE_PRIVATE)
|
| 193 |
+
disCheck.isChecked = prefs.getBoolean("dis", false)
|
| 194 |
+
disCheck.setOnCheckedChangeListener { _, checked ->
|
| 195 |
+
prefs.edit().putBoolean("dis", checked).apply()
|
| 196 |
+
if (!running && !autorunActive) Idol.model.execute { setup(uiOptions(), preload = true) }
|
| 197 |
+
}
|
| 198 |
+
|
| 199 |
+
onBackPressedDispatcher.addCallback(this, object : OnBackPressedCallback(true) {
|
| 200 |
+
override fun handleOnBackPressed() = onBack()
|
| 201 |
+
})
|
| 202 |
+
|
| 203 |
+
deviceLine = "${DeviceInfo.displayName()} · LiteRT ${BuildConfig.LITERT_VERSION}"
|
| 204 |
+
homeStatus.text = deviceLine
|
| 205 |
+
showHome(null)
|
| 206 |
+
val autorun = isDebuggable() && intent?.getStringExtra("autorun") != null
|
| 207 |
+
if (!Idol.started) {
|
| 208 |
+
Idol.started = true
|
| 209 |
+
// an autorun builds its own engine with its plan's options: no preload here then
|
| 210 |
+
val opts = uiOptions()
|
| 211 |
+
Idol.model.execute { setup(opts, preload = !autorun) }
|
| 212 |
+
} else {
|
| 213 |
+
Idol.problems?.takeIf { it.isNotEmpty() }?.let { showMissing(it) }
|
| 214 |
+
}
|
| 215 |
+
handleIntent(intent)
|
| 216 |
+
}
|
| 217 |
+
|
| 218 |
+
override fun onNewIntent(intent: Intent) {
|
| 219 |
+
super.onNewIntent(intent)
|
| 220 |
+
handleIntent(intent)
|
| 221 |
+
}
|
| 222 |
+
|
| 223 |
+
private fun handleIntent(intent: Intent?) {
|
| 224 |
+
val plan = intent?.getStringExtra("autorun") ?: return
|
| 225 |
+
if (!isDebuggable() || autorunActive) return
|
| 226 |
+
autorunActive = true
|
| 227 |
+
AutoRun(this, File(filesDir, plan), File(filesDir, intent.getStringExtra("out") ?: "runs/autorun")).start()
|
| 228 |
+
}
|
| 229 |
+
|
| 230 |
+
fun autorunFinished() {
|
| 231 |
+
ui.post { autorunActive = false }
|
| 232 |
+
}
|
| 233 |
+
|
| 234 |
+
override fun onTrimMemory(level: Int) {
|
| 235 |
+
super.onTrimMemory(level)
|
| 236 |
+
// In the background (the camera app in front, for one) the resident graphs make this process the first one
|
| 237 |
+
// Android stops: give their memory back while nothing runs; the next run compiles them again.
|
| 238 |
+
if (level >= ComponentCallbacks2.TRIM_MEMORY_BACKGROUND && !running && !autorunActive) {
|
| 239 |
+
Idol.model.execute { if (!running) Idol.engine?.pool?.releaseAll() }
|
| 240 |
+
}
|
| 241 |
+
}
|
| 242 |
+
|
| 243 |
+
override fun onDestroy() {
|
| 244 |
+
ResultViewer.destroy(viewerBox, web)
|
| 245 |
+
web = null
|
| 246 |
+
if (isFinishing && !running && !autorunActive) {
|
| 247 |
+
Idol.model.execute {
|
| 248 |
+
Idol.engine?.close()
|
| 249 |
+
Idol.engine = null
|
| 250 |
+
Idol.started = false
|
| 251 |
+
}
|
| 252 |
+
}
|
| 253 |
+
super.onDestroy()
|
| 254 |
+
}
|
| 255 |
+
|
| 256 |
+
private fun isDebuggable() = applicationInfo.flags and ApplicationInfo.FLAG_DEBUGGABLE != 0
|
| 257 |
+
|
| 258 |
+
private fun dp(v: Int) = (v * resources.displayMetrics.density + 0.5f).toInt()
|
| 259 |
+
|
| 260 |
+
// ------------------------------------------------------------------------------------------------ model thread
|
| 261 |
+
/** The options of a run from the buttons: preload on, the matting the setting picks (else the manifest's). */
|
| 262 |
+
private fun uiOptions(): EngineOptions {
|
| 263 |
+
val matting = if (getSharedPreferences("settings", MODE_PRIVATE).getBoolean("dis", false)) "dis" else ""
|
| 264 |
+
return EngineOptions(preload = true, matting = matting)
|
| 265 |
+
}
|
| 266 |
+
|
| 267 |
+
/**
|
| 268 |
+
* On the model thread: checks the model files (sizes; SHA-256 once per install), then builds the engine with
|
| 269 |
+
* [options] (an empty matting = the manifest's default) and, with [preload], compiles the graphs.
|
| 270 |
+
*/
|
| 271 |
+
private fun setup(options: EngineOptions, preload: Boolean) {
|
| 272 |
+
Idol.engine?.close()
|
| 273 |
+
Idol.engine = null
|
| 274 |
+
val dir = ModelFiles.dir(filesDir)
|
| 275 |
+
val manifest = ModelFiles.manifest(dir)
|
| 276 |
+
val matting = options.matting.ifEmpty { manifest?.defaultMatting ?: "bir" }
|
| 277 |
+
val problems = ModelFiles.check(this, dir, matting, hash = true) { k, n ->
|
| 278 |
+
progress.status("Checking model files ${k + 1}/$n")
|
| 279 |
+
}
|
| 280 |
+
Idol.problems = problems
|
| 281 |
+
if (problems.isNotEmpty() || manifest == null) {
|
| 282 |
+
progress.status("")
|
| 283 |
+
ui.post { if (!running) showMissing(problems) }
|
| 284 |
+
return
|
| 285 |
+
}
|
| 286 |
+
ui.post { if (homeMessage.text.startsWith("Model files")) homeMessage.text = "" }
|
| 287 |
+
val engine = try {
|
| 288 |
+
IdolEngine(dir, manifest, options.copy(matting = matting))
|
| 289 |
+
} catch (e: Throwable) {
|
| 290 |
+
Log.e(TAG, "engine", e)
|
| 291 |
+
progress.status("")
|
| 292 |
+
ui.post { showHome("The model files could not be read: ${e.message}") }
|
| 293 |
+
return
|
| 294 |
+
}
|
| 295 |
+
Idol.engine = engine
|
| 296 |
+
if (preload && options.preload) {
|
| 297 |
+
try {
|
| 298 |
+
engine.preload(progress)
|
| 299 |
+
} catch (e: Throwable) {
|
| 300 |
+
Log.e(TAG, "preload", e)
|
| 301 |
+
progress.status("Loading stopped: ${e.message}")
|
| 302 |
+
}
|
| 303 |
+
} else {
|
| 304 |
+
progress.status("")
|
| 305 |
+
}
|
| 306 |
+
}
|
| 307 |
+
|
| 308 |
+
/** Runs [block] on the model thread and waits for it (the autorun's engine calls). */
|
| 309 |
+
fun <T> onModel(block: () -> T): T = Idol.model.submit(Callable { block() }).get()
|
| 310 |
+
|
| 311 |
+
/** The autorun's engine: rebuilt (without preloading) unless the current one has [options]. */
|
| 312 |
+
fun ensureEngine(options: EngineOptions): List<ModelFiles.Problem> = onModel {
|
| 313 |
+
val e = Idol.engine
|
| 314 |
+
if (e != null && e.options == options) {
|
| 315 |
+
emptyList()
|
| 316 |
+
} else {
|
| 317 |
+
setup(options, preload = false)
|
| 318 |
+
Idol.problems ?: emptyList()
|
| 319 |
+
}
|
| 320 |
+
}
|
| 321 |
+
|
| 322 |
+
private fun generate(photo: Photo, mode: String, noise: Noise, ply: File): Outcome {
|
| 323 |
+
val e = Idol.engine ?: return Idol.problems?.takeIf { it.isNotEmpty() }?.let { Outcome.Missing(it) }
|
| 324 |
+
?: Outcome.Failed("not_ready", "The models are not loaded.")
|
| 325 |
+
return try {
|
| 326 |
+
ply.parentFile?.mkdirs()
|
| 327 |
+
Outcome.Done(e.generate(photo.image, mode, noise, ply, progress))
|
| 328 |
+
} catch (x: NoObjectException) {
|
| 329 |
+
Outcome.NoObject(x.reason)
|
| 330 |
+
} catch (x: NotEnoughMemoryException) {
|
| 331 |
+
Outcome.Failed("memory", x.message ?: "Not enough memory.")
|
| 332 |
+
} catch (x: CancelledException) {
|
| 333 |
+
Outcome.Failed("cancelled", "Stopped.")
|
| 334 |
+
} catch (x: OutOfMemoryError) {
|
| 335 |
+
Log.e(TAG, "run", x)
|
| 336 |
+
Outcome.Failed("memory", "Not enough memory for this picture.")
|
| 337 |
+
} catch (x: Throwable) {
|
| 338 |
+
Log.e(TAG, "run", x)
|
| 339 |
+
Outcome.Failed("error", "Something went wrong: ${x.message ?: x.javaClass.simpleName}")
|
| 340 |
+
} finally {
|
| 341 |
+
progress.status("")
|
| 342 |
+
}
|
| 343 |
+
}
|
| 344 |
+
|
| 345 |
+
// ------------------------------------------------------------------------------------------------ runs
|
| 346 |
+
private fun startFromPicture(decode: () -> Photo) {
|
| 347 |
+
if (running) return
|
| 348 |
+
val mode = currentMode()
|
| 349 |
+
showRun(null)
|
| 350 |
+
decoder.execute {
|
| 351 |
+
val photo = try {
|
| 352 |
+
decode()
|
| 353 |
+
} catch (e: Throwable) {
|
| 354 |
+
Log.e(TAG, "decode", e)
|
| 355 |
+
ui.post { showHome("This picture could not be opened: ${e.message ?: e.javaClass.simpleName}") }
|
| 356 |
+
return@execute
|
| 357 |
+
}
|
| 358 |
+
ui.post { startRun(photo, mode, Noise.random(DEFAULT_SEED), newResultFile(), showResult = true) {} }
|
| 359 |
+
}
|
| 360 |
+
}
|
| 361 |
+
|
| 362 |
+
/**
|
| 363 |
+
* Starts a run of [photo] on the model thread; [done] gets the outcome on the UI thread. With [showResult] a
|
| 364 |
+
* finished run opens the viewer; without (the autorun) the Run screen stays with its times.
|
| 365 |
+
*/
|
| 366 |
+
fun startRun(photo: Photo, mode: String, noise: Noise, ply: File, showResult: Boolean, done: (Outcome) -> Unit) {
|
| 367 |
+
running = true
|
| 368 |
+
showRun(photo)
|
| 369 |
+
Idol.model.execute {
|
| 370 |
+
val outcome = generate(photo, mode, noise, ply)
|
| 371 |
+
ui.post {
|
| 372 |
+
running = false
|
| 373 |
+
when (outcome) {
|
| 374 |
+
is Outcome.Done -> {
|
| 375 |
+
val t = outcome.result.times
|
| 376 |
+
totalText.text = "Total %.1f s (3D generation %.1f s)".format(t["all"] ?: 0.0, t["generate"] ?: 0.0)
|
| 377 |
+
if (showResult) showResult(outcome.result.ply)
|
| 378 |
+
}
|
| 379 |
+
is Outcome.NoObject -> showHome(getString(R.string.no_object))
|
| 380 |
+
is Outcome.Missing -> showMissing(outcome.problems)
|
| 381 |
+
is Outcome.Failed -> showHome(if (outcome.kind == "cancelled") null else outcome.message)
|
| 382 |
+
}
|
| 383 |
+
done(outcome)
|
| 384 |
+
}
|
| 385 |
+
}
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
/** The autorun's run: [startRun] from another thread, waiting for the outcome. */
|
| 389 |
+
fun runAndWait(photo: Photo, mode: String, noise: Noise, ply: File, showResult: Boolean): Outcome {
|
| 390 |
+
val latch = CountDownLatch(1)
|
| 391 |
+
var result: Outcome? = null
|
| 392 |
+
ui.post {
|
| 393 |
+
if (running) {
|
| 394 |
+
result = Outcome.Failed("busy", "A run is already in progress.")
|
| 395 |
+
latch.countDown()
|
| 396 |
+
} else {
|
| 397 |
+
startRun(photo, mode, noise, ply, showResult) {
|
| 398 |
+
result = it
|
| 399 |
+
latch.countDown()
|
| 400 |
+
}
|
| 401 |
+
}
|
| 402 |
+
}
|
| 403 |
+
latch.await()
|
| 404 |
+
return result!!
|
| 405 |
+
}
|
| 406 |
+
|
| 407 |
+
/** The Run screen for a run the autorun drives without a picture (the parity row). */
|
| 408 |
+
fun showRunScreen(title: String) {
|
| 409 |
+
ui.post {
|
| 410 |
+
showRun(null)
|
| 411 |
+
runStatus.text = title
|
| 412 |
+
}
|
| 413 |
+
}
|
| 414 |
+
|
| 415 |
+
/** The message the Home screen shows now (the autorun records it). */
|
| 416 |
+
fun homeMessageText(): String {
|
| 417 |
+
val latch = CountDownLatch(1)
|
| 418 |
+
var text = ""
|
| 419 |
+
ui.post {
|
| 420 |
+
text = if (home.visibility == View.VISIBLE) homeMessage.text.toString() else ""
|
| 421 |
+
latch.countDown()
|
| 422 |
+
}
|
| 423 |
+
latch.await()
|
| 424 |
+
return text
|
| 425 |
+
}
|
| 426 |
+
|
| 427 |
+
private fun currentMode() = if (modeGroup.checkedRadioButtonId == R.id.mode_render) "synthetic" else "real"
|
| 428 |
+
|
| 429 |
+
private fun cameraFile() = File(File(cacheDir, "camera"), "photo.jpg")
|
| 430 |
+
|
| 431 |
+
private fun launchCamera() {
|
| 432 |
+
val file = cameraFile().apply { parentFile?.mkdirs() }
|
| 433 |
+
try {
|
| 434 |
+
takePhoto.launch(FileProvider.getUriForFile(this, "$packageName.files", file))
|
| 435 |
+
} catch (e: ActivityNotFoundException) {
|
| 436 |
+
showHome(getString(R.string.no_camera_app))
|
| 437 |
+
}
|
| 438 |
+
}
|
| 439 |
+
|
| 440 |
+
private fun resultsDir() = File(filesDir, "results").apply { mkdirs() }
|
| 441 |
+
|
| 442 |
+
/** A new file in files/results/ (the last few results are kept). */
|
| 443 |
+
private fun newResultFile(): File {
|
| 444 |
+
val dir = resultsDir()
|
| 445 |
+
dir.listFiles { f -> f.name.endsWith(".ply") }?.sortedByDescending { it.lastModified() }?.drop(KEEP_RESULTS - 1)
|
| 446 |
+
?.forEach { it.delete() }
|
| 447 |
+
val stamp = SimpleDateFormat("yyyyMMdd_HHmmss", Locale.US).format(Date())
|
| 448 |
+
return File(dir, "idol_$stamp.ply")
|
| 449 |
+
}
|
| 450 |
+
|
| 451 |
+
private fun displayName(uri: Uri): String =
|
| 452 |
+
runCatching {
|
| 453 |
+
contentResolver.query(uri, arrayOf(OpenableColumns.DISPLAY_NAME), null, null, null)?.use { c ->
|
| 454 |
+
if (c.moveToFirst()) c.getString(0) else null
|
| 455 |
+
}
|
| 456 |
+
}.getOrNull() ?: uri.lastPathSegment ?: "photo"
|
| 457 |
+
|
| 458 |
+
private fun share() {
|
| 459 |
+
val ply = lastPly ?: return
|
| 460 |
+
val uri = FileProvider.getUriForFile(this, "$packageName.files", ply)
|
| 461 |
+
val send = Intent(Intent.ACTION_SEND).apply {
|
| 462 |
+
type = "application/octet-stream"
|
| 463 |
+
putExtra(Intent.EXTRA_STREAM, uri)
|
| 464 |
+
addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION)
|
| 465 |
+
}
|
| 466 |
+
try {
|
| 467 |
+
startActivity(Intent.createChooser(send, getString(R.string.share)))
|
| 468 |
+
} catch (e: ActivityNotFoundException) {
|
| 469 |
+
runStatus.text = getString(R.string.no_share_app)
|
| 470 |
+
}
|
| 471 |
+
}
|
| 472 |
+
|
| 473 |
+
// ------------------------------------------------------------------------------------------------ screens
|
| 474 |
+
private fun onBack() {
|
| 475 |
+
when {
|
| 476 |
+
resultScreen.visibility == View.VISIBLE -> showHome(null)
|
| 477 |
+
runScreen.visibility == View.VISIBLE && running -> {
|
| 478 |
+
Idol.engine?.cancel = true
|
| 479 |
+
runStatus.text = getString(R.string.stopping)
|
| 480 |
+
}
|
| 481 |
+
runScreen.visibility == View.VISIBLE -> showHome(null)
|
| 482 |
+
else -> finish()
|
| 483 |
+
}
|
| 484 |
+
}
|
| 485 |
+
|
| 486 |
+
private fun showHome(message: String?) {
|
| 487 |
+
ResultViewer.destroy(viewerBox, web)
|
| 488 |
+
web = null
|
| 489 |
+
stopTicker()
|
| 490 |
+
home.visibility = View.VISIBLE
|
| 491 |
+
runScreen.visibility = View.GONE
|
| 492 |
+
resultScreen.visibility = View.GONE
|
| 493 |
+
homeMessage.text = message ?: ""
|
| 494 |
+
homeMessage.visibility = if (message.isNullOrEmpty()) View.GONE else View.VISIBLE
|
| 495 |
+
}
|
| 496 |
+
|
| 497 |
+
private fun showMissing(problems: List<ModelFiles.Problem>) {
|
| 498 |
+
showHome(ModelFiles.message(problems))
|
| 499 |
+
}
|
| 500 |
+
|
| 501 |
+
private fun showRun(photo: Photo?) {
|
| 502 |
+
ResultViewer.destroy(viewerBox, web)
|
| 503 |
+
web = null
|
| 504 |
+
home.visibility = View.GONE
|
| 505 |
+
resultScreen.visibility = View.GONE
|
| 506 |
+
runScreen.visibility = View.VISIBLE
|
| 507 |
+
if (photo != null) picture.setImageBitmap(photo.displayBitmap()) else picture.setImageDrawable(null)
|
| 508 |
+
cutout.setImageDrawable(null)
|
| 509 |
+
cutout.visibility = View.INVISIBLE
|
| 510 |
+
for (s in stages) {
|
| 511 |
+
s.detail = null
|
| 512 |
+
s.seconds = null
|
| 513 |
+
s.running = false
|
| 514 |
+
s.done = false
|
| 515 |
+
}
|
| 516 |
+
totalText.text = ""
|
| 517 |
+
runStatus.text = ""
|
| 518 |
+
renderStages()
|
| 519 |
+
startTicker()
|
| 520 |
+
}
|
| 521 |
+
|
| 522 |
+
private fun showResult(ply: File) {
|
| 523 |
+
stopTicker()
|
| 524 |
+
lastPly = ply
|
| 525 |
+
home.visibility = View.GONE
|
| 526 |
+
runScreen.visibility = View.GONE
|
| 527 |
+
resultScreen.visibility = View.VISIBLE
|
| 528 |
+
web = ResultViewer.show(this, viewerBox, resultsDir(), ply)
|
| 529 |
+
}
|
| 530 |
+
|
| 531 |
+
private fun onStage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {
|
| 532 |
+
val s = stages[stage.ordinal]
|
| 533 |
+
s.detail = detail
|
| 534 |
+
if (done) {
|
| 535 |
+
s.running = false
|
| 536 |
+
s.done = true
|
| 537 |
+
s.seconds = seconds
|
| 538 |
+
} else if (!s.running) {
|
| 539 |
+
s.running = true
|
| 540 |
+
s.startedAt = SystemClock.elapsedRealtime()
|
| 541 |
+
}
|
| 542 |
+
renderStages()
|
| 543 |
+
}
|
| 544 |
+
|
| 545 |
+
private fun renderStages() {
|
| 546 |
+
val now = SystemClock.elapsedRealtime()
|
| 547 |
+
for ((i, st) in Stage.entries.withIndex()) {
|
| 548 |
+
val s = stages[i]
|
| 549 |
+
val label = if (s.detail != null) "${st.label}, ${s.detail}" else st.label
|
| 550 |
+
stageRows[i].text = when {
|
| 551 |
+
s.done -> "✓ $label — %.1f s".format(s.seconds ?: 0.0)
|
| 552 |
+
s.running -> "● $label — %.1f s".format((now - s.startedAt) / 1000.0)
|
| 553 |
+
else -> "○ $label"
|
| 554 |
+
}
|
| 555 |
+
stageRows[i].alpha = if (s.done || s.running) 1f else 0.5f
|
| 556 |
+
}
|
| 557 |
+
}
|
| 558 |
+
|
| 559 |
+
private val ticker = object : Runnable {
|
| 560 |
+
override fun run() {
|
| 561 |
+
renderStages()
|
| 562 |
+
ui.postDelayed(this, 250)
|
| 563 |
+
}
|
| 564 |
+
}
|
| 565 |
+
|
| 566 |
+
private fun startTicker() {
|
| 567 |
+
ui.removeCallbacks(ticker)
|
| 568 |
+
ui.postDelayed(ticker, 250)
|
| 569 |
+
}
|
| 570 |
+
|
| 571 |
+
private fun stopTicker() {
|
| 572 |
+
ui.removeCallbacks(ticker)
|
| 573 |
+
}
|
| 574 |
+
|
| 575 |
+
companion object {
|
| 576 |
+
const val TAG = "Idol"
|
| 577 |
+
/** The seed of a run from a picture (scripts/run_idol.py's default; the app draws with java.util.Random). */
|
| 578 |
+
const val DEFAULT_SEED = 42L
|
| 579 |
+
const val KEEP_RESULTS = 5
|
| 580 |
+
}
|
| 581 |
+
}
|
android/app/src/main/java/com/idol/Manifest.kt
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import org.json.JSONArray
|
| 4 |
+
import org.json.JSONObject
|
| 5 |
+
import java.io.File
|
| 6 |
+
|
| 7 |
+
/** One input or output tensor of a graph signature: its name in the flatbuffer and its shape. */
|
| 8 |
+
class TensorSpec(val name: String, val shape: IntArray) {
|
| 9 |
+
val size: Int = shape.fold(1) { a, b -> a * b }
|
| 10 |
+
}
|
| 11 |
+
|
| 12 |
+
/** A signature of a graph: inputs in signature order, outputs sorted by their trailing index (output_0, output_1, ...). */
|
| 13 |
+
class SignatureSpec(val key: String, val inputs: List<TensorSpec>, val outputs: List<TensorSpec>)
|
| 14 |
+
|
| 15 |
+
class GraphSpec(
|
| 16 |
+
val name: String,
|
| 17 |
+
val file: String,
|
| 18 |
+
val bytes: Long,
|
| 19 |
+
val sha256: String?,
|
| 20 |
+
/** "gpu" (ML Drift) or "cpu" (XNNPACK: BiRefNet, whose deformable convolutions have no GPU path). */
|
| 21 |
+
val accelerator: String,
|
| 22 |
+
/** "fp16_with_fp32_accum", "fp16" or "fp32". */
|
| 23 |
+
val precision: String,
|
| 24 |
+
val signatures: Map<String, SignatureSpec>,
|
| 25 |
+
) {
|
| 26 |
+
val onCpu: Boolean get() = accelerator == "cpu"
|
| 27 |
+
|
| 28 |
+
/** The largest input or output tensor, in elements. */
|
| 29 |
+
val maxIoElements: Long =
|
| 30 |
+
signatures.values.flatMap { it.inputs + it.outputs }.maxOfOrNull { it.size.toLong() } ?: 0L
|
| 31 |
+
|
| 32 |
+
fun signature(key: String? = null): SignatureSpec =
|
| 33 |
+
if (key == null) signatures.values.first() else requireNotNull(signatures[key]) { "$name has no signature $key" }
|
| 34 |
+
}
|
| 35 |
+
|
| 36 |
+
class SideFile(val name: String, val shape: IntArray) {
|
| 37 |
+
val elements: Long = shape.fold(1L) { a, b -> a * b }
|
| 38 |
+
val bytes: Long = elements * 4
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
/**
|
| 42 |
+
* `models/manifest.json` as `scripts/make_manifest.py` writes it: every graph's file, size, SHA-256, accelerator,
|
| 43 |
+
* precision and signatures, the raw side files (little-endian float32, with their shapes) and the JSON files. The
|
| 44 |
+
* app builds its file checks, its buffers and its run calls from it.
|
| 45 |
+
*/
|
| 46 |
+
class Manifest(
|
| 47 |
+
val graphs: Map<String, GraphSpec>,
|
| 48 |
+
val sideFiles: Map<String, SideFile>,
|
| 49 |
+
val jsonFiles: List<String>,
|
| 50 |
+
/** The matting graph the app uses unless the setting picks the alternative ("bir" or "dis"). */
|
| 51 |
+
val defaultMatting: String,
|
| 52 |
+
) {
|
| 53 |
+
fun graph(name: String): GraphSpec = requireNotNull(graphs[name]) { "manifest.json has no graph $name" }
|
| 54 |
+
|
| 55 |
+
companion object {
|
| 56 |
+
const val FILE = "manifest.json"
|
| 57 |
+
|
| 58 |
+
fun read(file: File): Manifest = parse(JSONObject(file.readText()))
|
| 59 |
+
|
| 60 |
+
fun parse(o: JSONObject): Manifest {
|
| 61 |
+
val graphs = LinkedHashMap<String, GraphSpec>()
|
| 62 |
+
val g = o.getJSONObject("graphs")
|
| 63 |
+
for (name in g.keys()) {
|
| 64 |
+
val e = g.getJSONObject(name)
|
| 65 |
+
val sigs = LinkedHashMap<String, SignatureSpec>()
|
| 66 |
+
val s = e.getJSONObject("signatures")
|
| 67 |
+
for (key in s.keys()) {
|
| 68 |
+
val sig = s.getJSONObject(key)
|
| 69 |
+
val inputs = tensors(sig.getJSONArray("inputs"))
|
| 70 |
+
val outputs = tensors(sig.getJSONArray("outputs")).sortedBy { trailingIndex(it.name) }
|
| 71 |
+
sigs[key] = SignatureSpec(key, inputs, outputs)
|
| 72 |
+
}
|
| 73 |
+
graphs[name] = GraphSpec(
|
| 74 |
+
name = name,
|
| 75 |
+
file = e.getString("file"),
|
| 76 |
+
bytes = e.getLong("bytes"),
|
| 77 |
+
sha256 = e.optString("sha256", "").ifEmpty { null },
|
| 78 |
+
accelerator = e.optString("accelerator", "gpu"),
|
| 79 |
+
precision = e.optString("precision", "fp16_with_fp32_accum"),
|
| 80 |
+
signatures = sigs,
|
| 81 |
+
)
|
| 82 |
+
}
|
| 83 |
+
val side = LinkedHashMap<String, SideFile>()
|
| 84 |
+
o.optJSONObject("side_files")?.let { sf ->
|
| 85 |
+
for (name in sf.keys()) side[name] = SideFile(name, ints(sf.getJSONObject(name).getJSONArray("shape")))
|
| 86 |
+
}
|
| 87 |
+
val json = ArrayList<String>()
|
| 88 |
+
o.optJSONArray("json_files")?.let { a -> for (i in 0 until a.length()) json.add(a.getString(i)) }
|
| 89 |
+
val matting = o.optJSONObject("matting")?.optString("default", "")?.ifEmpty { null }
|
| 90 |
+
?: if ("bir" in graphs) "bir" else "dis"
|
| 91 |
+
return Manifest(graphs, side, json, matting)
|
| 92 |
+
}
|
| 93 |
+
|
| 94 |
+
private fun tensors(a: JSONArray): List<TensorSpec> = (0 until a.length()).map { i ->
|
| 95 |
+
val t = a.getJSONArray(i)
|
| 96 |
+
TensorSpec(t.getString(0), ints(t.getJSONArray(1)))
|
| 97 |
+
}
|
| 98 |
+
|
| 99 |
+
private fun ints(a: JSONArray) = IntArray(a.length()) { a.getInt(it) }
|
| 100 |
+
|
| 101 |
+
private fun trailingIndex(name: String): Int = name.substringAfterLast('_').toIntOrNull() ?: 0
|
| 102 |
+
}
|
| 103 |
+
}
|
android/app/src/main/java/com/idol/ModelFiles.kt
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.content.Context
|
| 4 |
+
import java.io.File
|
| 5 |
+
import java.io.FileInputStream
|
| 6 |
+
import java.security.MessageDigest
|
| 7 |
+
|
| 8 |
+
/**
|
| 9 |
+
* The files `scripts/install_to_device.sh` puts in the app's `files/models/`: `manifest.json`, every graph it lists
|
| 10 |
+
* (sizes and SHA-256 from the manifest), the side files (float32, size from their shapes) and the JSON files. The
|
| 11 |
+
* app checks them before it builds a graph.
|
| 12 |
+
*/
|
| 13 |
+
object ModelFiles {
|
| 14 |
+
/** The two matting graphs; a run needs only the one the setting picks. */
|
| 15 |
+
val MATTING_GRAPHS = setOf("bir", "dis")
|
| 16 |
+
|
| 17 |
+
fun dir(filesDir: File): File = File(filesDir, "models")
|
| 18 |
+
|
| 19 |
+
/** A file problem the app reports instead of building a graph. */
|
| 20 |
+
class Problem(val file: String, val message: String)
|
| 21 |
+
|
| 22 |
+
/** The graphs a run with [matting] needs, in the manifest's order. */
|
| 23 |
+
fun neededGraphs(manifest: Manifest, matting: String): List<String> =
|
| 24 |
+
manifest.graphs.keys.filter { it == matting || it !in MATTING_GRAPHS }
|
| 25 |
+
|
| 26 |
+
/** Reads the manifest, or returns null when it is missing or unreadable (reported by [check]). */
|
| 27 |
+
fun manifest(dir: File): Manifest? = runCatching { Manifest.read(File(dir, Manifest.FILE)) }.getOrNull()
|
| 28 |
+
|
| 29 |
+
/**
|
| 30 |
+
* Missing files and wrong sizes (cheap); with [hash] also the SHA-256 of every graph the manifest gives one for,
|
| 31 |
+
* once per install: a match is remembered by size and modification time in the app's preferences.
|
| 32 |
+
* [progress] gets (files checked, files to check) while hashing.
|
| 33 |
+
*/
|
| 34 |
+
fun check(
|
| 35 |
+
context: Context,
|
| 36 |
+
dir: File,
|
| 37 |
+
matting: String,
|
| 38 |
+
hash: Boolean,
|
| 39 |
+
progress: (Int, Int) -> Unit = { _, _ -> },
|
| 40 |
+
): List<Problem> {
|
| 41 |
+
val manifestFile = File(dir, Manifest.FILE)
|
| 42 |
+
if (!manifestFile.isFile) return listOf(Problem(Manifest.FILE, "${Manifest.FILE} (missing)"))
|
| 43 |
+
val manifest = manifest(dir) ?: return listOf(Problem(Manifest.FILE, "${Manifest.FILE} (unreadable)"))
|
| 44 |
+
val out = ArrayList<Problem>()
|
| 45 |
+
if (matting !in manifest.graphs) out.add(Problem(matting, "the manifest has no matting graph \"$matting\""))
|
| 46 |
+
val graphs = neededGraphs(manifest, matting).map { manifest.graph(it) }
|
| 47 |
+
for (g in graphs) sizeProblem(File(dir, g.file), g.bytes)?.let { out.add(it) }
|
| 48 |
+
for (s in manifest.sideFiles.values) sizeProblem(File(dir, s.name), s.bytes)?.let { out.add(it) }
|
| 49 |
+
for (j in manifest.jsonFiles) {
|
| 50 |
+
val f = File(dir, j)
|
| 51 |
+
if (!f.isFile || f.length() == 0L) out.add(Problem(j, "$j (missing)"))
|
| 52 |
+
}
|
| 53 |
+
if (!hash || out.isNotEmpty()) return out
|
| 54 |
+
val prefs = context.getSharedPreferences("verified", Context.MODE_PRIVATE)
|
| 55 |
+
val toHash = graphs.filter { it.sha256 != null }
|
| 56 |
+
for ((k, g) in toHash.withIndex()) {
|
| 57 |
+
progress(k, toHash.size)
|
| 58 |
+
val f = File(dir, g.file)
|
| 59 |
+
val key = "${f.length()}:${f.lastModified()}:${g.sha256}"
|
| 60 |
+
if (prefs.getString(g.file, null) == key) continue
|
| 61 |
+
val digest = sha256(f)
|
| 62 |
+
if (digest != g.sha256) {
|
| 63 |
+
out.add(Problem(g.file, "${g.file} (SHA-256 ${digest.take(12)}..., expected ${g.sha256!!.take(12)}...)"))
|
| 64 |
+
} else {
|
| 65 |
+
prefs.edit().putString(g.file, key).apply()
|
| 66 |
+
}
|
| 67 |
+
}
|
| 68 |
+
progress(toHash.size, toHash.size)
|
| 69 |
+
return out
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
private fun sizeProblem(f: File, bytes: Long): Problem? = when {
|
| 73 |
+
!f.isFile -> Problem(f.name, "${f.name} (missing)")
|
| 74 |
+
f.length() != bytes -> Problem(f.name, "${f.name} (${f.length()} bytes, expected $bytes)")
|
| 75 |
+
else -> null
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
fun sha256(file: File): String {
|
| 79 |
+
val md = MessageDigest.getInstance("SHA-256")
|
| 80 |
+
FileInputStream(file).use { input ->
|
| 81 |
+
val buf = ByteArray(1 shl 20)
|
| 82 |
+
while (true) {
|
| 83 |
+
val n = input.read(buf)
|
| 84 |
+
if (n < 0) break
|
| 85 |
+
md.update(buf, 0, n)
|
| 86 |
+
}
|
| 87 |
+
}
|
| 88 |
+
return md.digest().joinToString("") { "%02x".format(it) }
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
/** The message the Home screen shows for [problems]. */
|
| 92 |
+
fun message(problems: List<Problem>): String =
|
| 93 |
+
"Model files are missing. Run scripts/install_to_device.sh (see README).\n" +
|
| 94 |
+
problems.take(12).joinToString("\n") { "• ${it.message}" } +
|
| 95 |
+
if (problems.size > 12) "\n• ... and ${problems.size - 12} more" else ""
|
| 96 |
+
}
|
android/app/src/main/java/com/idol/Photo.kt
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.content.ContentResolver
|
| 4 |
+
import android.graphics.Bitmap
|
| 5 |
+
import android.graphics.BitmapFactory
|
| 6 |
+
import android.net.Uri
|
| 7 |
+
import java.io.File
|
| 8 |
+
import java.nio.ByteBuffer
|
| 9 |
+
|
| 10 |
+
/**
|
| 11 |
+
* A picture as the pipeline reads it: the decoded bytes at full size, unpremultiplied, without colour conversion and
|
| 12 |
+
* without the EXIF rotation (what PIL's Image.open gives the author's pipeline: idol/pipeline.py opens the file and
|
| 13 |
+
* does not transpose it). RGBA when the file has an alpha channel, else RGB.
|
| 14 |
+
*/
|
| 15 |
+
class Photo(val name: String, val image: Image8) {
|
| 16 |
+
val width: Int get() = image.width
|
| 17 |
+
val height: Int get() = image.height
|
| 18 |
+
|
| 19 |
+
/** A screen copy of at most [side] pixels a side (nearest sampling), opaque. */
|
| 20 |
+
fun displayBitmap(side: Int = 1024): Bitmap = Photo.displayBitmap(image, side)
|
| 21 |
+
|
| 22 |
+
companion object {
|
| 23 |
+
/** Pictures above this many pixels are decoded at half size per side (and again) to bound the memory. */
|
| 24 |
+
const val MAX_PIXELS = 40_000_000L
|
| 25 |
+
|
| 26 |
+
fun fromUri(resolver: ContentResolver, uri: Uri, name: String): Photo {
|
| 27 |
+
val bytes = requireNotNull(resolver.openInputStream(uri)) { "cannot open $uri" }.use { it.readBytes() }
|
| 28 |
+
return Photo(name, decode(bytes))
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
fun fromFile(file: File): Photo = Photo(file.name, decode(file.readBytes()))
|
| 32 |
+
|
| 33 |
+
private fun decode(bytes: ByteArray): Image8 {
|
| 34 |
+
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
|
| 35 |
+
BitmapFactory.decodeByteArray(bytes, 0, bytes.size, bounds)
|
| 36 |
+
require(bounds.outWidth > 0 && bounds.outHeight > 0) { "not an image the phone can decode" }
|
| 37 |
+
var sample = 1
|
| 38 |
+
while (bounds.outWidth.toLong() * bounds.outHeight / (sample.toLong() * sample) > MAX_PIXELS) sample *= 2
|
| 39 |
+
val options = BitmapFactory.Options().apply {
|
| 40 |
+
inPremultiplied = false
|
| 41 |
+
inPreferredConfig = Bitmap.Config.ARGB_8888
|
| 42 |
+
inSampleSize = sample
|
| 43 |
+
}
|
| 44 |
+
val decoded = requireNotNull(BitmapFactory.decodeByteArray(bytes, 0, bytes.size, options)) {
|
| 45 |
+
"not an image the phone can decode"
|
| 46 |
+
}
|
| 47 |
+
val bitmap = if (decoded.config == Bitmap.Config.ARGB_8888) decoded else decoded.copy(Bitmap.Config.ARGB_8888, false)
|
| 48 |
+
val w = bitmap.width
|
| 49 |
+
val h = bitmap.height
|
| 50 |
+
// copyPixelsToBuffer keeps the stored values (getPixels would convert colour spaces); ARGB_8888 is R, G, B, A
|
| 51 |
+
val rgba = ByteBuffer.allocateDirect(w * h * 4)
|
| 52 |
+
bitmap.copyPixelsToBuffer(rgba)
|
| 53 |
+
rgba.rewind()
|
| 54 |
+
val hasAlpha = bitmap.hasAlpha()
|
| 55 |
+
val ch = if (hasAlpha) 4 else 3
|
| 56 |
+
val out = ByteArray(w * h * ch)
|
| 57 |
+
val row = ByteArray(w * 4)
|
| 58 |
+
for (y in 0 until h) {
|
| 59 |
+
rgba.get(row)
|
| 60 |
+
var o = y * w * ch
|
| 61 |
+
for (x in 0 until w) {
|
| 62 |
+
out[o] = row[x * 4]
|
| 63 |
+
out[o + 1] = row[x * 4 + 1]
|
| 64 |
+
out[o + 2] = row[x * 4 + 2]
|
| 65 |
+
if (hasAlpha) out[o + 3] = row[x * 4 + 3]
|
| 66 |
+
o += ch
|
| 67 |
+
}
|
| 68 |
+
}
|
| 69 |
+
if (bitmap !== decoded) bitmap.recycle()
|
| 70 |
+
decoded.recycle()
|
| 71 |
+
return Image8(w, h, ch, out)
|
| 72 |
+
}
|
| 73 |
+
|
| 74 |
+
fun displayBitmap(image: Image8, side: Int): Bitmap {
|
| 75 |
+
val step = maxOf(1, (maxOf(image.width, image.height) + side - 1) / side)
|
| 76 |
+
val dw = maxOf(1, image.width / step)
|
| 77 |
+
val dh = maxOf(1, image.height / step)
|
| 78 |
+
val ch = image.channels
|
| 79 |
+
val s = image.data
|
| 80 |
+
val px = IntArray(dw * dh)
|
| 81 |
+
for (y in 0 until dh) for (x in 0 until dw) {
|
| 82 |
+
val i = ((y * step) * image.width + x * step) * ch
|
| 83 |
+
px[y * dw + x] = (0xFF shl 24) or ((s[i].toInt() and 0xFF) shl 16) or
|
| 84 |
+
((s[i + 1].toInt() and 0xFF) shl 8) or (s[i + 2].toInt() and 0xFF)
|
| 85 |
+
}
|
| 86 |
+
return Bitmap.createBitmap(px, dw, dh, Bitmap.Config.ARGB_8888)
|
| 87 |
+
}
|
| 88 |
+
}
|
| 89 |
+
}
|
android/app/src/main/java/com/idol/Pil.kt
ADDED
|
@@ -0,0 +1,361 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import kotlin.math.PI
|
| 4 |
+
import kotlin.math.ceil
|
| 5 |
+
import kotlin.math.sin
|
| 6 |
+
|
| 7 |
+
/** An 8-bit image, channels interleaved, rows top to bottom: what a PIL image of mode RGB (3) or RGBA (4) holds. */
|
| 8 |
+
class Image8(val width: Int, val height: Int, val channels: Int, val data: ByteArray) {
|
| 9 |
+
init {
|
| 10 |
+
require(data.size == width * height * channels) { "${data.size} bytes for ${width}x${height}x$channels" }
|
| 11 |
+
}
|
| 12 |
+
|
| 13 |
+
fun copy() = Image8(width, height, channels, data.copyOf())
|
| 14 |
+
|
| 15 |
+
/** One channel as a width x height plane. */
|
| 16 |
+
fun channel(c: Int): ByteArray {
|
| 17 |
+
val out = ByteArray(width * height)
|
| 18 |
+
var p = c
|
| 19 |
+
for (i in out.indices) {
|
| 20 |
+
out[i] = data[p]
|
| 21 |
+
p += channels
|
| 22 |
+
}
|
| 23 |
+
return out
|
| 24 |
+
}
|
| 25 |
+
|
| 26 |
+
/** PIL's convert("RGB"): the first three channels. */
|
| 27 |
+
fun rgb(): Image8 {
|
| 28 |
+
if (channels == 3) return this
|
| 29 |
+
val out = ByteArray(width * height * 3)
|
| 30 |
+
var p = 0
|
| 31 |
+
var o = 0
|
| 32 |
+
for (i in 0 until width * height) {
|
| 33 |
+
out[o] = data[p]
|
| 34 |
+
out[o + 1] = data[p + 1]
|
| 35 |
+
out[o + 2] = data[p + 2]
|
| 36 |
+
p += channels
|
| 37 |
+
o += 3
|
| 38 |
+
}
|
| 39 |
+
return Image8(width, height, 3, out)
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
/** PIL's putalpha(L): an RGBA image with this image's colours and [alpha] as its alpha channel. */
|
| 43 |
+
fun withAlpha(alpha: ByteArray): Image8 {
|
| 44 |
+
require(alpha.size == width * height)
|
| 45 |
+
val out = ByteArray(width * height * 4)
|
| 46 |
+
var p = 0
|
| 47 |
+
var o = 0
|
| 48 |
+
for (i in 0 until width * height) {
|
| 49 |
+
out[o] = data[p]
|
| 50 |
+
out[o + 1] = data[p + 1]
|
| 51 |
+
out[o + 2] = data[p + 2]
|
| 52 |
+
out[o + 3] = alpha[i]
|
| 53 |
+
p += channels
|
| 54 |
+
o += 4
|
| 55 |
+
}
|
| 56 |
+
return Image8(width, height, 4, out)
|
| 57 |
+
}
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
/**
|
| 61 |
+
* The parts of Pillow 13.0.0 the author's preprocessing uses (idol/preprocess.py, scripts/idol_glue.py preprocess),
|
| 62 |
+
* pixel for pixel, from Pillow's src/libImaging: Resample.c (Lanczos, 22-bit fixed-point coefficients, the pass order
|
| 63 |
+
* of ImagingResampleInner), Convert.c (RGBA <-> RGBa around a resize), RankFilter.c + Filter.c (MinFilter(3) with edge
|
| 64 |
+
* replication), Crop.c (zero fill outside the image) and Paste.c (paste with an L mask).
|
| 65 |
+
* Pillow: Copyright (c) 1997-2011 by Secret Labs AB, Copyright (c) 1995-2011 by Fredrik Lundh and contributors,
|
| 66 |
+
* Copyright (c) 2010 by Jeffrey 'Alex' Clark and contributors, under the MIT-CMU License (licenses/Pillow-MIT-CMU.txt).
|
| 67 |
+
*/
|
| 68 |
+
object Pil {
|
| 69 |
+
private const val PRECISION_BITS = 32 - 8 - 2
|
| 70 |
+
|
| 71 |
+
private class Coeffs(val ksize: Int, val bounds: IntArray, val kk: IntArray)
|
| 72 |
+
|
| 73 |
+
private fun sinc(x0: Double): Double {
|
| 74 |
+
if (x0 == 0.0) return 1.0
|
| 75 |
+
val x = x0 * PI
|
| 76 |
+
return sin(x) / x
|
| 77 |
+
}
|
| 78 |
+
|
| 79 |
+
/** lanczos_filter: truncated sinc, support 3. */
|
| 80 |
+
private fun lanczos(x: Double): Double = if (-3.0 <= x && x < 3.0) sinc(x) * sinc(x / 3) else 0.0
|
| 81 |
+
|
| 82 |
+
/** precompute_coeffs for the box [0, inSize) -> outSize, then normalize_coeffs_8bpc. */
|
| 83 |
+
private fun coeffs(inSize: Int, outSize: Int): Coeffs {
|
| 84 |
+
val in0 = 0.0f
|
| 85 |
+
val in1 = inSize.toFloat()
|
| 86 |
+
val scale = (in1 - in0).toDouble() / outSize
|
| 87 |
+
val filterscale = if (scale < 1.0) 1.0 else scale
|
| 88 |
+
val support = 3.0 * filterscale
|
| 89 |
+
val ksize = ceil(support).toInt() * 2 + 1
|
| 90 |
+
val pre = DoubleArray(outSize * ksize)
|
| 91 |
+
val bounds = IntArray(outSize * 2)
|
| 92 |
+
val invFilterscale = 1.0 / filterscale
|
| 93 |
+
for (xx in 0 until outSize) {
|
| 94 |
+
val center = in0 + (xx + 0.5) * scale
|
| 95 |
+
var ww = 0.0
|
| 96 |
+
var xmin = (center - support + 0.5).toInt()
|
| 97 |
+
if (xmin < 0) xmin = 0
|
| 98 |
+
var xmax = (center + support + 0.5).toInt()
|
| 99 |
+
if (xmax > inSize) xmax = inSize
|
| 100 |
+
xmax -= xmin
|
| 101 |
+
val base = xx * ksize
|
| 102 |
+
for (x in 0 until xmax) {
|
| 103 |
+
val w = lanczos((x + xmin - center + 0.5) * invFilterscale)
|
| 104 |
+
pre[base + x] = w
|
| 105 |
+
ww += w
|
| 106 |
+
}
|
| 107 |
+
if (ww != 0.0) for (x in 0 until xmax) pre[base + x] /= ww
|
| 108 |
+
bounds[xx * 2] = xmin
|
| 109 |
+
bounds[xx * 2 + 1] = xmax
|
| 110 |
+
}
|
| 111 |
+
val one = (1 shl PRECISION_BITS).toDouble()
|
| 112 |
+
val kk = IntArray(pre.size) { i ->
|
| 113 |
+
val v = pre[i]
|
| 114 |
+
if (v < 0) (-0.5 + v * one).toInt() else (0.5 + v * one).toInt()
|
| 115 |
+
}
|
| 116 |
+
return Coeffs(ksize, bounds, kk)
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
private fun clip8(ss: Int): Byte {
|
| 120 |
+
val v = ss shr PRECISION_BITS
|
| 121 |
+
return (if (v < 0) 0 else if (v > 255) 255 else v).toByte()
|
| 122 |
+
}
|
| 123 |
+
|
| 124 |
+
/** _ImagingResampleHorizontal_8bpc: [rows] output rows from input rows offset.. */
|
| 125 |
+
private fun horizontal(src: Image8, outWidth: Int, rows: Int, offset: Int, c: Coeffs): Image8 {
|
| 126 |
+
val ch = src.channels
|
| 127 |
+
val inW = src.width
|
| 128 |
+
val s = src.data
|
| 129 |
+
val out = ByteArray(outWidth * rows * ch)
|
| 130 |
+
val half = 1 shl (PRECISION_BITS - 1)
|
| 131 |
+
val acc = IntArray(ch)
|
| 132 |
+
for (yy in 0 until rows) {
|
| 133 |
+
val inRow = (yy + offset) * inW * ch
|
| 134 |
+
val outRow = yy * outWidth * ch
|
| 135 |
+
for (xx in 0 until outWidth) {
|
| 136 |
+
val xmin = c.bounds[xx * 2]
|
| 137 |
+
val xmax = c.bounds[xx * 2 + 1]
|
| 138 |
+
val k = xx * c.ksize
|
| 139 |
+
val o = outRow + xx * ch
|
| 140 |
+
when (ch) {
|
| 141 |
+
3 -> {
|
| 142 |
+
var s0 = half
|
| 143 |
+
var s1 = half
|
| 144 |
+
var s2 = half
|
| 145 |
+
var p = inRow + xmin * 3
|
| 146 |
+
for (x in 0 until xmax) {
|
| 147 |
+
val w = c.kk[k + x]
|
| 148 |
+
s0 += (s[p].toInt() and 0xFF) * w
|
| 149 |
+
s1 += (s[p + 1].toInt() and 0xFF) * w
|
| 150 |
+
s2 += (s[p + 2].toInt() and 0xFF) * w
|
| 151 |
+
p += 3
|
| 152 |
+
}
|
| 153 |
+
out[o] = clip8(s0)
|
| 154 |
+
out[o + 1] = clip8(s1)
|
| 155 |
+
out[o + 2] = clip8(s2)
|
| 156 |
+
}
|
| 157 |
+
4 -> {
|
| 158 |
+
var s0 = half
|
| 159 |
+
var s1 = half
|
| 160 |
+
var s2 = half
|
| 161 |
+
var s3 = half
|
| 162 |
+
var p = inRow + xmin * 4
|
| 163 |
+
for (x in 0 until xmax) {
|
| 164 |
+
val w = c.kk[k + x]
|
| 165 |
+
s0 += (s[p].toInt() and 0xFF) * w
|
| 166 |
+
s1 += (s[p + 1].toInt() and 0xFF) * w
|
| 167 |
+
s2 += (s[p + 2].toInt() and 0xFF) * w
|
| 168 |
+
s3 += (s[p + 3].toInt() and 0xFF) * w
|
| 169 |
+
p += 4
|
| 170 |
+
}
|
| 171 |
+
out[o] = clip8(s0)
|
| 172 |
+
out[o + 1] = clip8(s1)
|
| 173 |
+
out[o + 2] = clip8(s2)
|
| 174 |
+
out[o + 3] = clip8(s3)
|
| 175 |
+
}
|
| 176 |
+
else -> {
|
| 177 |
+
acc.fill(half)
|
| 178 |
+
var p = inRow + xmin * ch
|
| 179 |
+
for (x in 0 until xmax) {
|
| 180 |
+
val w = c.kk[k + x]
|
| 181 |
+
for (b in 0 until ch) acc[b] += (s[p + b].toInt() and 0xFF) * w
|
| 182 |
+
p += ch
|
| 183 |
+
}
|
| 184 |
+
for (b in 0 until ch) out[o + b] = clip8(acc[b])
|
| 185 |
+
}
|
| 186 |
+
}
|
| 187 |
+
}
|
| 188 |
+
}
|
| 189 |
+
return Image8(outWidth, rows, ch, out)
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
/** _ImagingResampleVertical_8bpc over every column of [src]. */
|
| 193 |
+
private fun vertical(src: Image8, outHeight: Int, c: Coeffs): Image8 {
|
| 194 |
+
val ch = src.channels
|
| 195 |
+
val w = src.width
|
| 196 |
+
val stride = w * ch
|
| 197 |
+
val s = src.data
|
| 198 |
+
val out = ByteArray(w * outHeight * ch)
|
| 199 |
+
val half = 1 shl (PRECISION_BITS - 1)
|
| 200 |
+
val acc = IntArray(stride)
|
| 201 |
+
for (yy in 0 until outHeight) {
|
| 202 |
+
val ymin = c.bounds[yy * 2]
|
| 203 |
+
val ymax = c.bounds[yy * 2 + 1]
|
| 204 |
+
val k = yy * c.ksize
|
| 205 |
+
// the same sums as Pillow's per-pixel loop (integer adds, so the order does not change them), row by row
|
| 206 |
+
acc.fill(half)
|
| 207 |
+
for (y in 0 until ymax) {
|
| 208 |
+
val wgt = c.kk[k + y]
|
| 209 |
+
val row = (y + ymin) * stride
|
| 210 |
+
for (i in 0 until stride) acc[i] += (s[row + i].toInt() and 0xFF) * wgt
|
| 211 |
+
}
|
| 212 |
+
val o = yy * stride
|
| 213 |
+
for (i in 0 until stride) out[o + i] = clip8(acc[i])
|
| 214 |
+
}
|
| 215 |
+
return Image8(w, outHeight, ch, out)
|
| 216 |
+
}
|
| 217 |
+
|
| 218 |
+
/** ImagingResample with LANCZOS on an 8-bit image (any channel count): no premultiplication here. */
|
| 219 |
+
fun resizeLanczos(img: Image8, outWidth: Int, outHeight: Int): Image8 {
|
| 220 |
+
val inW = img.width
|
| 221 |
+
val inH = img.height
|
| 222 |
+
val needHorizontal = outWidth != inW
|
| 223 |
+
val needVertical = outHeight != inH
|
| 224 |
+
if (!needHorizontal && !needVertical) return img.copy()
|
| 225 |
+
// If height is being scaled down more than twice the amount the width is, run the vertical pass first.
|
| 226 |
+
val horizontalFirst = !((inH - outHeight) > 0 && (inH - outHeight) > (inW - outWidth) * 2)
|
| 227 |
+
var vert: Coeffs? = null
|
| 228 |
+
if ((needHorizontal && horizontalFirst) || needVertical) vert = coeffs(inH, outHeight)
|
| 229 |
+
var yFirst = 0
|
| 230 |
+
var yLast = 0
|
| 231 |
+
var horiz: Coeffs? = null
|
| 232 |
+
if (needHorizontal) {
|
| 233 |
+
if (horizontalFirst) {
|
| 234 |
+
val v = vert!!
|
| 235 |
+
yFirst = v.bounds[0]
|
| 236 |
+
yLast = v.bounds[outHeight * 2 - 2] + v.bounds[outHeight * 2 - 1]
|
| 237 |
+
if (yFirst != 0) for (i in 0 until outHeight) v.bounds[i * 2] -= yFirst
|
| 238 |
+
}
|
| 239 |
+
horiz = coeffs(inW, outWidth)
|
| 240 |
+
}
|
| 241 |
+
var cur = img
|
| 242 |
+
if (horizontalFirst) {
|
| 243 |
+
if (needHorizontal) cur = horizontal(cur, outWidth, yLast - yFirst, yFirst, horiz!!)
|
| 244 |
+
if (needVertical) cur = vertical(cur, outHeight, vert!!)
|
| 245 |
+
} else {
|
| 246 |
+
if (needVertical) cur = vertical(cur, outHeight, vert!!)
|
| 247 |
+
if (needHorizontal) cur = horizontal(cur, outWidth, outHeight, 0, horiz!!)
|
| 248 |
+
}
|
| 249 |
+
return cur
|
| 250 |
+
}
|
| 251 |
+
|
| 252 |
+
/**
|
| 253 |
+
* Image.resize(size, LANCZOS): an RGBA image goes through RGBa (premultiplied) and back, as Pillow does; RGB and
|
| 254 |
+
* single-channel images are resized directly; the same size returns a copy.
|
| 255 |
+
*/
|
| 256 |
+
fun resize(img: Image8, outWidth: Int, outHeight: Int): Image8 {
|
| 257 |
+
if (outWidth == img.width && outHeight == img.height) return img.copy()
|
| 258 |
+
if (img.channels != 4) return resizeLanczos(img, outWidth, outHeight)
|
| 259 |
+
return unpremultiply(resizeLanczos(premultiply(img), outWidth, outHeight))
|
| 260 |
+
}
|
| 261 |
+
|
| 262 |
+
/** Convert.c rgbA2rgba (RGBA -> RGBa): MULDIV255(c, a). */
|
| 263 |
+
fun premultiply(img: Image8): Image8 {
|
| 264 |
+
require(img.channels == 4)
|
| 265 |
+
val s = img.data
|
| 266 |
+
val out = ByteArray(s.size)
|
| 267 |
+
var p = 0
|
| 268 |
+
while (p < s.size) {
|
| 269 |
+
val a = s[p + 3].toInt() and 0xFF
|
| 270 |
+
for (b in 0 until 3) {
|
| 271 |
+
val t = (s[p + b].toInt() and 0xFF) * a + 128
|
| 272 |
+
out[p + b] = (((t shr 8) + t) shr 8).toByte()
|
| 273 |
+
}
|
| 274 |
+
out[p + 3] = s[p + 3]
|
| 275 |
+
p += 4
|
| 276 |
+
}
|
| 277 |
+
return Image8(img.width, img.height, 4, out)
|
| 278 |
+
}
|
| 279 |
+
|
| 280 |
+
/** Convert.c rgba2rgbA (RGBa -> RGBA): CLIP8(255 * c / a), alpha 0 and 255 unchanged. */
|
| 281 |
+
fun unpremultiply(img: Image8): Image8 {
|
| 282 |
+
require(img.channels == 4)
|
| 283 |
+
val s = img.data
|
| 284 |
+
val out = ByteArray(s.size)
|
| 285 |
+
var p = 0
|
| 286 |
+
while (p < s.size) {
|
| 287 |
+
val a = s[p + 3].toInt() and 0xFF
|
| 288 |
+
if (a == 255 || a == 0) {
|
| 289 |
+
out[p] = s[p]
|
| 290 |
+
out[p + 1] = s[p + 1]
|
| 291 |
+
out[p + 2] = s[p + 2]
|
| 292 |
+
} else {
|
| 293 |
+
for (b in 0 until 3) {
|
| 294 |
+
val v = (255 * (s[p + b].toInt() and 0xFF)) / a
|
| 295 |
+
out[p + b] = (if (v <= 0) 0 else if (v < 256) v else 255).toByte()
|
| 296 |
+
}
|
| 297 |
+
}
|
| 298 |
+
out[p + 3] = s[p + 3]
|
| 299 |
+
p += 4
|
| 300 |
+
}
|
| 301 |
+
return Image8(img.width, img.height, 4, out)
|
| 302 |
+
}
|
| 303 |
+
|
| 304 |
+
/** ImageFilter.MinFilter(3) on an L plane: the 3 x 3 minimum, edges replicated (ImagingExpand + RankFilter). */
|
| 305 |
+
fun minFilter3(src: ByteArray, w: Int, h: Int): ByteArray {
|
| 306 |
+
val out = ByteArray(w * h)
|
| 307 |
+
for (y in 0 until h) {
|
| 308 |
+
val y0 = if (y > 0) y - 1 else 0
|
| 309 |
+
val y1 = if (y < h - 1) y + 1 else h - 1
|
| 310 |
+
for (x in 0 until w) {
|
| 311 |
+
val x0 = if (x > 0) x - 1 else 0
|
| 312 |
+
val x1 = if (x < w - 1) x + 1 else w - 1
|
| 313 |
+
var m = 255
|
| 314 |
+
for (yy in y0..y1) {
|
| 315 |
+
val row = yy * w
|
| 316 |
+
for (xx in x0..x1) {
|
| 317 |
+
val v = src[row + xx].toInt() and 0xFF
|
| 318 |
+
if (v < m) m = v
|
| 319 |
+
}
|
| 320 |
+
}
|
| 321 |
+
out[y * w + x] = m.toByte()
|
| 322 |
+
}
|
| 323 |
+
}
|
| 324 |
+
return out
|
| 325 |
+
}
|
| 326 |
+
|
| 327 |
+
/** Image.crop((x0, y0, x1, y1)): the region, zeros where it leaves the image. */
|
| 328 |
+
fun crop(img: Image8, x0: Int, y0: Int, x1: Int, y1: Int): Image8 {
|
| 329 |
+
val w = maxOf(0, x1 - x0)
|
| 330 |
+
val h = maxOf(0, y1 - y0)
|
| 331 |
+
val ch = img.channels
|
| 332 |
+
val out = ByteArray(w * h * ch)
|
| 333 |
+
val xs = maxOf(0, x0)
|
| 334 |
+
val xe = minOf(img.width, x1)
|
| 335 |
+
if (xe > xs) {
|
| 336 |
+
for (y in maxOf(0, y0) until minOf(img.height, y1)) {
|
| 337 |
+
System.arraycopy(img.data, (y * img.width + xs) * ch, out, ((y - y0) * w + (xs - x0)) * ch, (xe - xs) * ch)
|
| 338 |
+
}
|
| 339 |
+
}
|
| 340 |
+
return Image8(w, h, ch, out)
|
| 341 |
+
}
|
| 342 |
+
|
| 343 |
+
/**
|
| 344 |
+
* Image.new("RGB", size, (0, 0, 0)).paste(rgba, mask=rgba.split()[3]): Paste.c paste_mask_L's BLEND on black,
|
| 345 |
+
* DIV255(c * a).
|
| 346 |
+
*/
|
| 347 |
+
fun compositeOnBlack(rgba: Image8): Image8 {
|
| 348 |
+
require(rgba.channels == 4)
|
| 349 |
+
val n = rgba.width * rgba.height
|
| 350 |
+
val s = rgba.data
|
| 351 |
+
val out = ByteArray(n * 3)
|
| 352 |
+
for (i in 0 until n) {
|
| 353 |
+
val a = s[i * 4 + 3].toInt() and 0xFF
|
| 354 |
+
for (b in 0 until 3) {
|
| 355 |
+
val t = (s[i * 4 + b].toInt() and 0xFF) * a + 128
|
| 356 |
+
out[i * 3 + b] = (((t shr 8) + t) shr 8).toByte()
|
| 357 |
+
}
|
| 358 |
+
}
|
| 359 |
+
return Image8(rgba.width, rgba.height, 3, out)
|
| 360 |
+
}
|
| 361 |
+
}
|
android/app/src/main/java/com/idol/Preprocess.kt
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import kotlin.math.max
|
| 4 |
+
import kotlin.math.min
|
| 5 |
+
import kotlin.math.round
|
| 6 |
+
|
| 7 |
+
/** The picture has nothing to cut out (the app's "No object found in this picture."). */
|
| 8 |
+
class NoObjectException(val reason: String) : Exception("no object found: $reason")
|
| 9 |
+
|
| 10 |
+
/** What the encoders see, and how it was cut out (scripts/idol_glue.py preprocess). */
|
| 11 |
+
class Preprocessed(
|
| 12 |
+
/** 1024 x 1024 RGB: the object on black. */
|
| 13 |
+
val composite: Image8,
|
| 14 |
+
/** The uint8 alpha at the resized size, before the minimum filter (the matting graph's, or the picture's own). */
|
| 15 |
+
val alphaRaw: ByteArray,
|
| 16 |
+
val alphaWidth: Int,
|
| 17 |
+
val alphaHeight: Int,
|
| 18 |
+
/** The crop square (left, top, right, bottom) in the resized picture. */
|
| 19 |
+
val box: IntArray,
|
| 20 |
+
/** True when the picture brought its own alpha (RGBA with a pixel below 255) and no matting ran. */
|
| 21 |
+
val ownAlpha: Boolean,
|
| 22 |
+
)
|
| 23 |
+
|
| 24 |
+
/**
|
| 25 |
+
* The host steps around the two matting graphs, ported line by line from scripts/idol_glue.py: birefnet_alpha (the
|
| 26 |
+
* author's BiRefNet.remove_background) and dis_alpha (IS-Net DIS as its authors run it, plus the alpha floor).
|
| 27 |
+
*/
|
| 28 |
+
object Matting {
|
| 29 |
+
const val SIZE = 1024
|
| 30 |
+
private val BIREFNET_MEAN = floatArrayOf(0.485f, 0.456f, 0.406f)
|
| 31 |
+
private val BIREFNET_STD = floatArrayOf(0.229f, 0.224f, 0.225f)
|
| 32 |
+
|
| 33 |
+
/** uint8 DIS alpha values below this are set to 0 (idol_glue.DIS_ALPHA_FLOOR, DIS only). */
|
| 34 |
+
const val DIS_ALPHA_FLOOR = 8
|
| 35 |
+
|
| 36 |
+
/** The RGB picture as float32 planes [3, h, w], each value c (or c / 255 with [divide]). */
|
| 37 |
+
private fun planes(rgb: Image8, divide: Boolean): FloatArray {
|
| 38 |
+
val n = rgb.width * rgb.height
|
| 39 |
+
val out = FloatArray(3 * n)
|
| 40 |
+
val s = rgb.data
|
| 41 |
+
for (i in 0 until n) {
|
| 42 |
+
for (c in 0 until 3) {
|
| 43 |
+
val v = (s[i * 3 + c].toInt() and 0xFF).toFloat()
|
| 44 |
+
out[c * n + i] = if (divide) v / 255.0f else v
|
| 45 |
+
}
|
| 46 |
+
}
|
| 47 |
+
return out
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
/** BiRefNet's input [1, 3, 1024, 1024]: RGB / 255, bilinear (align_corners) to 1024 x 1024, ImageNet mean / std. */
|
| 51 |
+
fun birefnetInput(rgb: Image8): FloatArray {
|
| 52 |
+
val t = TorchOps.bilinear(planes(rgb, true), 3, rgb.height, rgb.width, SIZE, SIZE, alignCorners = true)
|
| 53 |
+
val n = SIZE * SIZE
|
| 54 |
+
for (c in 0 until 3) {
|
| 55 |
+
val m = BIREFNET_MEAN[c]
|
| 56 |
+
val s = BIREFNET_STD[c]
|
| 57 |
+
for (i in c * n until (c + 1) * n) t[i] = (t[i] - m) / s
|
| 58 |
+
}
|
| 59 |
+
return t
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
/** BiRefNet's sigmoid map [1, 1, 1024, 1024] -> uint8 alpha at w x h: bilinear (align_corners), x 255, truncated. */
|
| 63 |
+
fun birefnetAlpha(map: FloatArray, w: Int, h: Int): ByteArray {
|
| 64 |
+
val a = TorchOps.bilinear(map, 1, SIZE, SIZE, h, w, alignCorners = true)
|
| 65 |
+
return ByteArray(a.size) { i -> toUint8(a[i]) }
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
/** DIS's input [1, 3, 1024, 1024]: bilinear to 1024 x 1024 on 0..255, then x / 255 - 0.5. */
|
| 69 |
+
fun disInput(rgb: Image8): FloatArray {
|
| 70 |
+
val t = TorchOps.bilinear(planes(rgb, false), 3, rgb.height, rgb.width, SIZE, SIZE, alignCorners = false)
|
| 71 |
+
for (i in t.indices) t[i] = t[i] / 255.0f - 0.5f
|
| 72 |
+
return t
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
/**
|
| 76 |
+
* DIS's sigmoid map -> uint8 alpha at w x h: bilinear back, min-max normalised (the division guarded by 1e-12 as in
|
| 77 |
+
* the authors' code), x 255 truncated, values below [floor] set to 0.
|
| 78 |
+
*/
|
| 79 |
+
fun disAlpha(map: FloatArray, w: Int, h: Int, floor: Int = DIS_ALPHA_FLOOR): ByteArray {
|
| 80 |
+
val a = TorchOps.bilinear(map, 1, SIZE, SIZE, h, w, alignCorners = false)
|
| 81 |
+
var mi = Float.POSITIVE_INFINITY
|
| 82 |
+
var ma = Float.NEGATIVE_INFINITY
|
| 83 |
+
for (v in a) {
|
| 84 |
+
if (v < mi) mi = v
|
| 85 |
+
if (v > ma) ma = v
|
| 86 |
+
}
|
| 87 |
+
val d = max(ma - mi, 1e-12f)
|
| 88 |
+
val out = ByteArray(a.size)
|
| 89 |
+
for (i in a.indices) {
|
| 90 |
+
val u = toUint8((a[i] - mi) / d).toInt() and 0xFF
|
| 91 |
+
out[i] = (if (u >= floor) u else 0).toByte()
|
| 92 |
+
}
|
| 93 |
+
return out
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
/** torch: (a.clamp(0, 1) * 255).to(torch.uint8), the cast truncating. */
|
| 97 |
+
private fun toUint8(v: Float): Byte {
|
| 98 |
+
val c = if (v < 0f) 0f else if (v > 1f) 1f else v
|
| 99 |
+
return (c * 255f).toInt().toByte()
|
| 100 |
+
}
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
object Preprocess {
|
| 104 |
+
const val IMAGE_SIZE = 1024
|
| 105 |
+
val CROP_MARGIN = mapOf("synthetic" to 1.2, "real" to 1.7)
|
| 106 |
+
|
| 107 |
+
/** The resize size of the author's first step: short side 1024, Python round() (half to even), at least 1. */
|
| 108 |
+
fun firstSize(width: Int, height: Int): Pair<Int, Int> {
|
| 109 |
+
val scale = IMAGE_SIZE.toDouble() / min(width, height)
|
| 110 |
+
// kotlin.math.round is Math.rint: ties to the even integer, as Python's round()
|
| 111 |
+
return max(1, round(width * scale).toInt()) to max(1, round(height * scale).toInt())
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
/** First step alone (the matting input of a picture without its own alpha): the resized picture. */
|
| 115 |
+
fun resized(image: Image8): Image8 {
|
| 116 |
+
val (w, h) = firstSize(image.width, image.height)
|
| 117 |
+
return Pil.resize(image, w, h)
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
/**
|
| 121 |
+
* scripts/idol_glue.py preprocess: resize the short side to 1024 (Lanczos), alpha from [alphaFn] unless the picture
|
| 122 |
+
* has its own, 3 x 3 minimum filter on the alpha, square crop of side margin x the box of the nonzero alpha,
|
| 123 |
+
* resize to 1024 (Lanczos), composite on black.
|
| 124 |
+
*
|
| 125 |
+
* The app adds guards the author's code does not have (it raises or crops nonsense there): a picture of one flat
|
| 126 |
+
* colour, an alpha with no pixel at 128 or above, and a crop square under one pixel are "no object".
|
| 127 |
+
*/
|
| 128 |
+
fun run(image: Image8, mode: String, alphaFn: (Image8) -> ByteArray): Preprocessed {
|
| 129 |
+
val margin = requireNotNull(CROP_MARGIN[mode]) { "mode $mode" }
|
| 130 |
+
var img = resized(image)
|
| 131 |
+
val w = img.width
|
| 132 |
+
val h = img.height
|
| 133 |
+
val ownAlpha = img.channels == 4 && minByte(img.channel(3)) < 255
|
| 134 |
+
if (!ownAlpha) {
|
| 135 |
+
val rgb = img.rgb()
|
| 136 |
+
if (isFlat(rgb)) throw NoObjectException("the picture is one flat colour")
|
| 137 |
+
img = rgb.withAlpha(alphaFn(rgb))
|
| 138 |
+
}
|
| 139 |
+
val alphaRaw = img.channel(3)
|
| 140 |
+
if (maxByte(alphaRaw) < 128) throw NoObjectException("no alpha value reaches 128")
|
| 141 |
+
val alpha = Pil.minFilter3(alphaRaw, w, h)
|
| 142 |
+
val px = img.data
|
| 143 |
+
for (i in 0 until w * h) px[i * 4 + 3] = alpha[i]
|
| 144 |
+
var left = Int.MAX_VALUE
|
| 145 |
+
var top = Int.MAX_VALUE
|
| 146 |
+
var right = -1
|
| 147 |
+
var bottom = -1
|
| 148 |
+
for (y in 0 until h) for (x in 0 until w) {
|
| 149 |
+
if (alpha[y * w + x].toInt() != 0) {
|
| 150 |
+
if (x < left) left = x
|
| 151 |
+
if (x > right) right = x
|
| 152 |
+
if (y < top) top = y
|
| 153 |
+
if (y > bottom) bottom = y
|
| 154 |
+
}
|
| 155 |
+
}
|
| 156 |
+
if (right < 0) throw NoObjectException("the alpha channel is empty")
|
| 157 |
+
val centerX = (left + right) / 2.0
|
| 158 |
+
val centerY = (top + bottom) / 2.0
|
| 159 |
+
val half = max(right - left, bottom - top) / 2.0 * margin
|
| 160 |
+
val box = intArrayOf(
|
| 161 |
+
(centerX - half).toInt(), (centerY - half).toInt(), (centerX + half).toInt(), (centerY + half).toInt(),
|
| 162 |
+
)
|
| 163 |
+
if (box[2] - box[0] < 1 || box[3] - box[1] < 1) throw NoObjectException("the object is under one pixel")
|
| 164 |
+
val cropped = Pil.crop(img, box[0], box[1], box[2], box[3])
|
| 165 |
+
val square = Pil.resize(cropped, IMAGE_SIZE, IMAGE_SIZE)
|
| 166 |
+
return Preprocessed(Pil.compositeOnBlack(square), alphaRaw, w, h, box, ownAlpha)
|
| 167 |
+
}
|
| 168 |
+
|
| 169 |
+
/** pixels_of: [1, 3, 1024, 1024] float32 in [0, 1]. */
|
| 170 |
+
fun pixels(composite: Image8): FloatArray {
|
| 171 |
+
val n = composite.width * composite.height
|
| 172 |
+
val out = FloatArray(3 * n)
|
| 173 |
+
val s = composite.data
|
| 174 |
+
for (i in 0 until n) for (c in 0 until 3) out[c * n + i] = (s[i * 3 + c].toInt() and 0xFF).toFloat() / 255.0f
|
| 175 |
+
return out
|
| 176 |
+
}
|
| 177 |
+
|
| 178 |
+
private fun minByte(a: ByteArray): Int {
|
| 179 |
+
var m = 255
|
| 180 |
+
for (b in a) {
|
| 181 |
+
val v = b.toInt() and 0xFF
|
| 182 |
+
if (v < m) m = v
|
| 183 |
+
}
|
| 184 |
+
return m
|
| 185 |
+
}
|
| 186 |
+
|
| 187 |
+
private fun maxByte(a: ByteArray): Int {
|
| 188 |
+
var m = 0
|
| 189 |
+
for (b in a) {
|
| 190 |
+
val v = b.toInt() and 0xFF
|
| 191 |
+
if (v > m) m = v
|
| 192 |
+
}
|
| 193 |
+
return m
|
| 194 |
+
}
|
| 195 |
+
|
| 196 |
+
private fun isFlat(rgb: Image8): Boolean {
|
| 197 |
+
val s = rgb.data
|
| 198 |
+
for (i in 3 until s.size) if (s[i] != s[i % 3]) return false
|
| 199 |
+
return true
|
| 200 |
+
}
|
| 201 |
+
}
|
android/app/src/main/java/com/idol/ResultViewer.kt
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import android.annotation.SuppressLint
|
| 4 |
+
import android.content.Context
|
| 5 |
+
import android.net.Uri
|
| 6 |
+
import android.view.ViewGroup
|
| 7 |
+
import android.webkit.WebResourceRequest
|
| 8 |
+
import android.webkit.WebResourceResponse
|
| 9 |
+
import android.webkit.WebView
|
| 10 |
+
import android.webkit.WebViewClient
|
| 11 |
+
import android.widget.FrameLayout
|
| 12 |
+
import androidx.webkit.WebViewAssetLoader
|
| 13 |
+
import java.io.File
|
| 14 |
+
import java.io.FileInputStream
|
| 15 |
+
|
| 16 |
+
/**
|
| 17 |
+
* The splat viewer (antimatter15/splat, vendored in assets/viewer/) in a WebView. The page and the .ply come from
|
| 18 |
+
* the app itself through WebViewAssetLoader (https://appassets.androidplatform.net/...): no network, no file:// access.
|
| 19 |
+
*/
|
| 20 |
+
object ResultViewer {
|
| 21 |
+
private const val HOST = "https://appassets.androidplatform.net"
|
| 22 |
+
|
| 23 |
+
@SuppressLint("SetJavaScriptEnabled")
|
| 24 |
+
fun show(context: Context, container: FrameLayout, resultsDir: File, ply: File): WebView {
|
| 25 |
+
val loader = WebViewAssetLoader.Builder()
|
| 26 |
+
.addPathHandler("/assets/", WebViewAssetLoader.AssetsPathHandler(context))
|
| 27 |
+
.addPathHandler("/results/", PlyHandler(resultsDir))
|
| 28 |
+
.build()
|
| 29 |
+
val web = WebView(context)
|
| 30 |
+
web.settings.javaScriptEnabled = true
|
| 31 |
+
web.settings.allowFileAccess = false
|
| 32 |
+
web.settings.allowContentAccess = false
|
| 33 |
+
web.webViewClient = object : WebViewClient() {
|
| 34 |
+
override fun shouldInterceptRequest(view: WebView, request: WebResourceRequest): WebResourceResponse? =
|
| 35 |
+
loader.shouldInterceptRequest(request.url)
|
| 36 |
+
|
| 37 |
+
// the viewer has no links; stay on the page
|
| 38 |
+
override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean = true
|
| 39 |
+
}
|
| 40 |
+
container.removeAllViews()
|
| 41 |
+
container.addView(web, ViewGroup.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
|
| 42 |
+
val plyUrl = "$HOST/results/" + Uri.encode(ply.name)
|
| 43 |
+
web.loadUrl("$HOST/assets/viewer/index.html?url=" + Uri.encode(plyUrl))
|
| 44 |
+
return web
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
fun destroy(container: FrameLayout, web: WebView?) {
|
| 48 |
+
if (web == null) return
|
| 49 |
+
container.removeView(web)
|
| 50 |
+
web.stopLoading()
|
| 51 |
+
web.destroy()
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
/** Serves files of [dir] with a Content-Length (the viewer sizes its buffer from it). */
|
| 55 |
+
private class PlyHandler(private val dir: File) : WebViewAssetLoader.PathHandler {
|
| 56 |
+
override fun handle(path: String): WebResourceResponse? {
|
| 57 |
+
val f = File(dir, path)
|
| 58 |
+
val inside = runCatching { f.canonicalPath.startsWith(dir.canonicalPath + File.separator) }.getOrDefault(false)
|
| 59 |
+
if (!inside || !f.isFile) {
|
| 60 |
+
return WebResourceResponse("text/plain", "utf-8", 404, "Not Found", emptyMap(), null)
|
| 61 |
+
}
|
| 62 |
+
val headers = mapOf("Content-Length" to f.length().toString(), "Cache-Control" to "no-store")
|
| 63 |
+
return WebResourceResponse("application/octet-stream", null, 200, "OK", headers, FileInputStream(f))
|
| 64 |
+
}
|
| 65 |
+
}
|
| 66 |
+
}
|
android/app/src/main/java/com/idol/TorchOps.kt
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.idol
|
| 2 |
+
|
| 3 |
+
import kotlin.math.floor
|
| 4 |
+
|
| 5 |
+
/**
|
| 6 |
+
* torch.nn.functional.interpolate(x, size, mode="bilinear", align_corners=...) on the CPU in float32, as ATen's
|
| 7 |
+
* generic N-d kernel computes it (UpSampleKernel.cpp: compute_indices_weights_linear + Interpolate, the path a
|
| 8 |
+
* 1- or 3-channel NCHW tensor takes): per output index the source index and weight from
|
| 9 |
+
* area_pixel_compute_scale / area_pixel_compute_source_index / guard_index_and_lambda in float32, then
|
| 10 |
+
* out = (row(y0) * wy0 + row(y1) * wy1) with row(y) = src(y, x0) * wx0 + src(y, x1) * wx1, each product and sum
|
| 11 |
+
* rounded to float32.
|
| 12 |
+
*/
|
| 13 |
+
object TorchOps {
|
| 14 |
+
private class Taps(val i0: IntArray, val i1: IntArray, val w0: FloatArray, val w1: FloatArray)
|
| 15 |
+
|
| 16 |
+
private fun taps(inSize: Int, outSize: Int, alignCorners: Boolean): Taps {
|
| 17 |
+
val i0 = IntArray(outSize)
|
| 18 |
+
val i1 = IntArray(outSize)
|
| 19 |
+
val w0 = FloatArray(outSize)
|
| 20 |
+
val w1 = FloatArray(outSize)
|
| 21 |
+
val scale: Float = if (alignCorners) {
|
| 22 |
+
if (outSize > 1) (inSize - 1).toFloat() / (outSize - 1).toFloat() else 0f
|
| 23 |
+
} else {
|
| 24 |
+
inSize.toFloat() / outSize.toFloat()
|
| 25 |
+
}
|
| 26 |
+
for (o in 0 until outSize) {
|
| 27 |
+
if (outSize == inSize) {
|
| 28 |
+
// scale factor 1: a copy
|
| 29 |
+
i0[o] = o
|
| 30 |
+
i1[o] = o
|
| 31 |
+
w0[o] = 1f
|
| 32 |
+
w1[o] = 0f
|
| 33 |
+
continue
|
| 34 |
+
}
|
| 35 |
+
val real: Float = if (alignCorners) {
|
| 36 |
+
scale * o.toFloat()
|
| 37 |
+
} else {
|
| 38 |
+
val src = scale * (o.toFloat() + 0.5f) - 0.5f
|
| 39 |
+
if (src < 0f) 0f else src
|
| 40 |
+
}
|
| 41 |
+
val idx = minOf(floor(real.toDouble()).toInt(), inSize - 1)
|
| 42 |
+
val lambda = minOf(maxOf(real - idx.toFloat(), 0f), 1f)
|
| 43 |
+
i0[o] = idx
|
| 44 |
+
i1[o] = idx + if (idx < inSize - 1) 1 else 0
|
| 45 |
+
w1[o] = lambda
|
| 46 |
+
w0[o] = 1f - lambda
|
| 47 |
+
}
|
| 48 |
+
return Taps(i0, i1, w0, w1)
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
/** [src] = channels x inH x inW (planar, float32) -> channels x outH x outW. */
|
| 52 |
+
fun bilinear(
|
| 53 |
+
src: FloatArray,
|
| 54 |
+
channels: Int,
|
| 55 |
+
inH: Int,
|
| 56 |
+
inW: Int,
|
| 57 |
+
outH: Int,
|
| 58 |
+
outW: Int,
|
| 59 |
+
alignCorners: Boolean,
|
| 60 |
+
): FloatArray {
|
| 61 |
+
require(src.size == channels * inH * inW)
|
| 62 |
+
val ty = taps(inH, outH, alignCorners)
|
| 63 |
+
val tx = taps(inW, outW, alignCorners)
|
| 64 |
+
val out = FloatArray(channels * outH * outW)
|
| 65 |
+
for (c in 0 until channels) {
|
| 66 |
+
val sBase = c * inH * inW
|
| 67 |
+
val oBase = c * outH * outW
|
| 68 |
+
for (y in 0 until outH) {
|
| 69 |
+
val r0 = sBase + ty.i0[y] * inW
|
| 70 |
+
val r1 = sBase + ty.i1[y] * inW
|
| 71 |
+
val wy0 = ty.w0[y]
|
| 72 |
+
val wy1 = ty.w1[y]
|
| 73 |
+
val o = oBase + y * outW
|
| 74 |
+
for (x in 0 until outW) {
|
| 75 |
+
val x0 = tx.i0[x]
|
| 76 |
+
val x1 = tx.i1[x]
|
| 77 |
+
val wx0 = tx.w0[x]
|
| 78 |
+
val wx1 = tx.w1[x]
|
| 79 |
+
var t0 = src[r0 + x0] * wx0
|
| 80 |
+
t0 += src[r0 + x1] * wx1
|
| 81 |
+
var t1 = src[r1 + x0] * wx0
|
| 82 |
+
t1 += src[r1 + x1] * wx1
|
| 83 |
+
var v = t0 * wy0
|
| 84 |
+
v += t1 * wy1
|
| 85 |
+
out[o + x] = v
|
| 86 |
+
}
|
| 87 |
+
}
|
| 88 |
+
}
|
| 89 |
+
return out
|
| 90 |
+
}
|
| 91 |
+
}
|
android/app/src/main/res/layout/activity_main.xml
ADDED
|
@@ -0,0 +1,216 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
|
| 3 |
+
android:id="@+id/root"
|
| 4 |
+
android:layout_width="match_parent"
|
| 5 |
+
android:layout_height="match_parent"
|
| 6 |
+
android:background="@android:color/black">
|
| 7 |
+
|
| 8 |
+
<!-- Home -->
|
| 9 |
+
<ScrollView
|
| 10 |
+
android:id="@+id/home"
|
| 11 |
+
android:layout_width="match_parent"
|
| 12 |
+
android:layout_height="match_parent"
|
| 13 |
+
android:fillViewport="true">
|
| 14 |
+
|
| 15 |
+
<LinearLayout
|
| 16 |
+
android:layout_width="match_parent"
|
| 17 |
+
android:layout_height="wrap_content"
|
| 18 |
+
android:orientation="vertical"
|
| 19 |
+
android:padding="24dp">
|
| 20 |
+
|
| 21 |
+
<TextView
|
| 22 |
+
android:layout_width="match_parent"
|
| 23 |
+
android:layout_height="wrap_content"
|
| 24 |
+
android:layout_marginTop="24dp"
|
| 25 |
+
android:text="@string/title"
|
| 26 |
+
android:textColor="@android:color/white"
|
| 27 |
+
android:textSize="26sp"
|
| 28 |
+
android:textStyle="bold" />
|
| 29 |
+
|
| 30 |
+
<TextView
|
| 31 |
+
android:layout_width="match_parent"
|
| 32 |
+
android:layout_height="wrap_content"
|
| 33 |
+
android:layout_marginTop="12dp"
|
| 34 |
+
android:text="@string/tagline"
|
| 35 |
+
android:textColor="#FFBDBDBD"
|
| 36 |
+
android:textSize="17sp" />
|
| 37 |
+
|
| 38 |
+
<RadioGroup
|
| 39 |
+
android:id="@+id/mode"
|
| 40 |
+
android:layout_width="match_parent"
|
| 41 |
+
android:layout_height="wrap_content"
|
| 42 |
+
android:layout_marginTop="32dp"
|
| 43 |
+
android:orientation="vertical">
|
| 44 |
+
|
| 45 |
+
<RadioButton
|
| 46 |
+
android:id="@+id/mode_photo"
|
| 47 |
+
android:layout_width="wrap_content"
|
| 48 |
+
android:layout_height="wrap_content"
|
| 49 |
+
android:checked="true"
|
| 50 |
+
android:text="@string/mode_photo"
|
| 51 |
+
android:textSize="16sp" />
|
| 52 |
+
|
| 53 |
+
<RadioButton
|
| 54 |
+
android:id="@+id/mode_render"
|
| 55 |
+
android:layout_width="wrap_content"
|
| 56 |
+
android:layout_height="wrap_content"
|
| 57 |
+
android:text="@string/mode_render"
|
| 58 |
+
android:textSize="16sp" />
|
| 59 |
+
</RadioGroup>
|
| 60 |
+
|
| 61 |
+
<Button
|
| 62 |
+
android:id="@+id/pick"
|
| 63 |
+
android:layout_width="match_parent"
|
| 64 |
+
android:layout_height="wrap_content"
|
| 65 |
+
android:layout_marginTop="24dp"
|
| 66 |
+
android:text="@string/pick" />
|
| 67 |
+
|
| 68 |
+
<Button
|
| 69 |
+
android:id="@+id/camera"
|
| 70 |
+
android:layout_width="match_parent"
|
| 71 |
+
android:layout_height="wrap_content"
|
| 72 |
+
android:layout_marginTop="8dp"
|
| 73 |
+
android:text="@string/camera" />
|
| 74 |
+
|
| 75 |
+
<TextView
|
| 76 |
+
android:id="@+id/home_message"
|
| 77 |
+
android:layout_width="match_parent"
|
| 78 |
+
android:layout_height="wrap_content"
|
| 79 |
+
android:layout_marginTop="20dp"
|
| 80 |
+
android:textColor="#FFFFB74D"
|
| 81 |
+
android:textSize="16sp"
|
| 82 |
+
android:visibility="gone" />
|
| 83 |
+
|
| 84 |
+
<View
|
| 85 |
+
android:layout_width="match_parent"
|
| 86 |
+
android:layout_height="0dp"
|
| 87 |
+
android:layout_weight="1" />
|
| 88 |
+
|
| 89 |
+
<CheckBox
|
| 90 |
+
android:id="@+id/dis"
|
| 91 |
+
android:layout_width="wrap_content"
|
| 92 |
+
android:layout_height="wrap_content"
|
| 93 |
+
android:layout_marginTop="24dp"
|
| 94 |
+
android:text="@string/dis"
|
| 95 |
+
android:textColor="#FF9E9E9E"
|
| 96 |
+
android:textSize="14sp" />
|
| 97 |
+
|
| 98 |
+
<TextView
|
| 99 |
+
android:id="@+id/home_status"
|
| 100 |
+
android:layout_width="match_parent"
|
| 101 |
+
android:layout_height="wrap_content"
|
| 102 |
+
android:layout_marginTop="8dp"
|
| 103 |
+
android:textColor="#FF9E9E9E"
|
| 104 |
+
android:textSize="13sp" />
|
| 105 |
+
</LinearLayout>
|
| 106 |
+
</ScrollView>
|
| 107 |
+
|
| 108 |
+
<!-- Run -->
|
| 109 |
+
<ScrollView
|
| 110 |
+
android:id="@+id/run"
|
| 111 |
+
android:layout_width="match_parent"
|
| 112 |
+
android:layout_height="match_parent"
|
| 113 |
+
android:visibility="gone">
|
| 114 |
+
|
| 115 |
+
<LinearLayout
|
| 116 |
+
android:layout_width="match_parent"
|
| 117 |
+
android:layout_height="wrap_content"
|
| 118 |
+
android:orientation="vertical"
|
| 119 |
+
android:padding="16dp">
|
| 120 |
+
|
| 121 |
+
<LinearLayout
|
| 122 |
+
android:layout_width="match_parent"
|
| 123 |
+
android:layout_height="wrap_content"
|
| 124 |
+
android:orientation="horizontal">
|
| 125 |
+
|
| 126 |
+
<ImageView
|
| 127 |
+
android:id="@+id/picture"
|
| 128 |
+
android:layout_width="0dp"
|
| 129 |
+
android:layout_height="180dp"
|
| 130 |
+
android:layout_marginEnd="8dp"
|
| 131 |
+
android:layout_weight="1"
|
| 132 |
+
android:contentDescription="@string/picture_description"
|
| 133 |
+
android:scaleType="fitCenter" />
|
| 134 |
+
|
| 135 |
+
<ImageView
|
| 136 |
+
android:id="@+id/cutout"
|
| 137 |
+
android:layout_width="0dp"
|
| 138 |
+
android:layout_height="180dp"
|
| 139 |
+
android:layout_weight="1"
|
| 140 |
+
android:contentDescription="@string/cutout_description"
|
| 141 |
+
android:scaleType="fitCenter"
|
| 142 |
+
android:visibility="invisible" />
|
| 143 |
+
</LinearLayout>
|
| 144 |
+
|
| 145 |
+
<LinearLayout
|
| 146 |
+
android:id="@+id/stages"
|
| 147 |
+
android:layout_width="match_parent"
|
| 148 |
+
android:layout_height="wrap_content"
|
| 149 |
+
android:layout_marginTop="20dp"
|
| 150 |
+
android:orientation="vertical" />
|
| 151 |
+
|
| 152 |
+
<TextView
|
| 153 |
+
android:id="@+id/total"
|
| 154 |
+
android:layout_width="match_parent"
|
| 155 |
+
android:layout_height="wrap_content"
|
| 156 |
+
android:layout_marginTop="16dp"
|
| 157 |
+
android:textColor="@android:color/white"
|
| 158 |
+
android:textSize="17sp"
|
| 159 |
+
android:textStyle="bold" />
|
| 160 |
+
|
| 161 |
+
<TextView
|
| 162 |
+
android:id="@+id/run_status"
|
| 163 |
+
android:layout_width="match_parent"
|
| 164 |
+
android:layout_height="wrap_content"
|
| 165 |
+
android:layout_marginTop="8dp"
|
| 166 |
+
android:textColor="#FF9E9E9E"
|
| 167 |
+
android:textSize="13sp" />
|
| 168 |
+
</LinearLayout>
|
| 169 |
+
</ScrollView>
|
| 170 |
+
|
| 171 |
+
<!-- Result -->
|
| 172 |
+
<FrameLayout
|
| 173 |
+
android:id="@+id/result"
|
| 174 |
+
android:layout_width="match_parent"
|
| 175 |
+
android:layout_height="match_parent"
|
| 176 |
+
android:visibility="gone">
|
| 177 |
+
|
| 178 |
+
<FrameLayout
|
| 179 |
+
android:id="@+id/viewer"
|
| 180 |
+
android:layout_width="match_parent"
|
| 181 |
+
android:layout_height="match_parent" />
|
| 182 |
+
|
| 183 |
+
<TextView
|
| 184 |
+
android:layout_width="wrap_content"
|
| 185 |
+
android:layout_height="wrap_content"
|
| 186 |
+
android:layout_gravity="top|center_horizontal"
|
| 187 |
+
android:layout_marginTop="12dp"
|
| 188 |
+
android:text="@string/viewer_hint"
|
| 189 |
+
android:textColor="#B3FFFFFF"
|
| 190 |
+
android:textSize="13sp" />
|
| 191 |
+
|
| 192 |
+
<LinearLayout
|
| 193 |
+
android:layout_width="match_parent"
|
| 194 |
+
android:layout_height="wrap_content"
|
| 195 |
+
android:layout_gravity="bottom"
|
| 196 |
+
android:gravity="center"
|
| 197 |
+
android:orientation="horizontal"
|
| 198 |
+
android:padding="12dp">
|
| 199 |
+
|
| 200 |
+
<Button
|
| 201 |
+
android:id="@+id/share"
|
| 202 |
+
android:layout_width="0dp"
|
| 203 |
+
android:layout_height="wrap_content"
|
| 204 |
+
android:layout_marginEnd="8dp"
|
| 205 |
+
android:layout_weight="1"
|
| 206 |
+
android:text="@string/share" />
|
| 207 |
+
|
| 208 |
+
<Button
|
| 209 |
+
android:id="@+id/new_run"
|
| 210 |
+
android:layout_width="0dp"
|
| 211 |
+
android:layout_height="wrap_content"
|
| 212 |
+
android:layout_weight="1"
|
| 213 |
+
android:text="@string/new_run" />
|
| 214 |
+
</LinearLayout>
|
| 215 |
+
</FrameLayout>
|
| 216 |
+
</FrameLayout>
|
android/app/src/main/res/values/strings.xml
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<resources>
|
| 3 |
+
<string name="app_name">Idol</string>
|
| 4 |
+
<string name="title">Idol — one photo to 3D</string>
|
| 5 |
+
<string name="tagline">One photo in, a 3D object out — including the sides the picture never showed.</string>
|
| 6 |
+
<string name="mode_photo">Photo</string>
|
| 7 |
+
<string name="mode_render">Render or illustration</string>
|
| 8 |
+
<string name="pick">Pick a photo</string>
|
| 9 |
+
<string name="camera">Take a photo</string>
|
| 10 |
+
<string name="dis">Background removal on the GPU (IS-Net DIS)</string>
|
| 11 |
+
<string name="picture_description">The picture</string>
|
| 12 |
+
<string name="cutout_description">The cut-out the 3D model is made from</string>
|
| 13 |
+
<string name="share">Share .ply</string>
|
| 14 |
+
<string name="new_run">New</string>
|
| 15 |
+
<string name="viewer_hint">Drag to turn, pinch to zoom</string>
|
| 16 |
+
<string name="no_object">No object found in this picture.</string>
|
| 17 |
+
<string name="camera_denied">Camera permission is off. You can still pick a photo.</string>
|
| 18 |
+
<string name="no_camera_app">No camera app is available. You can still pick a photo.</string>
|
| 19 |
+
<string name="no_share_app">No app can receive the .ply.</string>
|
| 20 |
+
<string name="stopping">Stopping after the current step…</string>
|
| 21 |
+
</resources>
|
android/app/src/main/res/values/themes.xml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<resources>
|
| 3 |
+
<style name="Theme.Idol" parent="android:Theme.Material.NoActionBar">
|
| 4 |
+
<item name="android:windowBackground">@android:color/black</item>
|
| 5 |
+
</style>
|
| 6 |
+
</resources>
|
android/app/src/main/res/xml/file_paths.xml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<paths>
|
| 3 |
+
<!-- the camera app writes the photo here -->
|
| 4 |
+
<cache-path name="camera" path="camera/" />
|
| 5 |
+
<!-- the .ply files "Share .ply" hands out -->
|
| 6 |
+
<files-path name="results" path="results/" />
|
| 7 |
+
</paths>
|
android/build.gradle.kts
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plugins {
|
| 2 |
+
alias(libs.plugins.android.application) apply false
|
| 3 |
+
alias(libs.plugins.kotlin.android) apply false
|
| 4 |
+
}
|
android/gradle.properties
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
android.useAndroidX=true
|
| 2 |
+
android.builder.sdkDownload=false
|
| 3 |
+
org.gradle.jvmargs=-Xmx3g -Dfile.encoding=UTF-8
|
| 4 |
+
org.gradle.daemon=false
|
| 5 |
+
org.gradle.workers.max=4
|
| 6 |
+
kotlin.compiler.execution.strategy=in-process
|
| 7 |
+
kotlin.code.style=official
|
android/gradle/libs.versions.toml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[versions]
|
| 2 |
+
agp = "8.9.1"
|
| 3 |
+
kotlin = "2.2.21"
|
| 4 |
+
litert = "2.3.0"
|
| 5 |
+
core = "1.15.0"
|
| 6 |
+
activity = "1.10.1"
|
| 7 |
+
webkit = "1.12.1"
|
| 8 |
+
|
| 9 |
+
[libraries]
|
| 10 |
+
litert = { module = "com.google.ai.edge.litert:litert", version.ref = "litert" }
|
| 11 |
+
litert-gpu = { module = "com.google.ai.edge.litert:litert-gpu", version.ref = "litert" }
|
| 12 |
+
androidx-core-ktx = { module = "androidx.core:core-ktx", version.ref = "core" }
|
| 13 |
+
androidx-activity-ktx = { module = "androidx.activity:activity-ktx", version.ref = "activity" }
|
| 14 |
+
androidx-webkit = { module = "androidx.webkit:webkit", version.ref = "webkit" }
|
| 15 |
+
|
| 16 |
+
[plugins]
|
| 17 |
+
android-application = { id = "com.android.application", version.ref = "agp" }
|
| 18 |
+
kotlin-android = { id = "org.jetbrains.kotlin.android", version.ref = "kotlin" }
|
android/gradle/wrapper/gradle-wrapper.jar
ADDED
|
Binary file (43.7 kB). View file
|
|
|
android/gradle/wrapper/gradle-wrapper.properties
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
distributionBase=GRADLE_USER_HOME
|
| 2 |
+
distributionPath=wrapper/dists
|
| 3 |
+
distributionUrl=https\://services.gradle.org/distributions/gradle-8.11.1-bin.zip
|
| 4 |
+
networkTimeout=10000
|
| 5 |
+
validateDistributionUrl=true
|
| 6 |
+
zipStoreBase=GRADLE_USER_HOME
|
| 7 |
+
zipStorePath=wrapper/dists
|
android/gradlew
ADDED
|
@@ -0,0 +1,251 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/sh
|
| 2 |
+
|
| 3 |
+
#
|
| 4 |
+
# Copyright © 2015-2021 the original authors.
|
| 5 |
+
#
|
| 6 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 7 |
+
# you may not use this file except in compliance with the License.
|
| 8 |
+
# You may obtain a copy of the License at
|
| 9 |
+
#
|
| 10 |
+
# https://www.apache.org/licenses/LICENSE-2.0
|
| 11 |
+
#
|
| 12 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 13 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 14 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 15 |
+
# See the License for the specific language governing permissions and
|
| 16 |
+
# limitations under the License.
|
| 17 |
+
#
|
| 18 |
+
# SPDX-License-Identifier: Apache-2.0
|
| 19 |
+
#
|
| 20 |
+
|
| 21 |
+
##############################################################################
|
| 22 |
+
#
|
| 23 |
+
# Gradle start up script for POSIX generated by Gradle.
|
| 24 |
+
#
|
| 25 |
+
# Important for running:
|
| 26 |
+
#
|
| 27 |
+
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
|
| 28 |
+
# noncompliant, but you have some other compliant shell such as ksh or
|
| 29 |
+
# bash, then to run this script, type that shell name before the whole
|
| 30 |
+
# command line, like:
|
| 31 |
+
#
|
| 32 |
+
# ksh Gradle
|
| 33 |
+
#
|
| 34 |
+
# Busybox and similar reduced shells will NOT work, because this script
|
| 35 |
+
# requires all of these POSIX shell features:
|
| 36 |
+
# * functions;
|
| 37 |
+
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
|
| 38 |
+
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
|
| 39 |
+
# * compound commands having a testable exit status, especially «case»;
|
| 40 |
+
# * various built-in commands including «command», «set», and «ulimit».
|
| 41 |
+
#
|
| 42 |
+
# Important for patching:
|
| 43 |
+
#
|
| 44 |
+
# (2) This script targets any POSIX shell, so it avoids extensions provided
|
| 45 |
+
# by Bash, Ksh, etc; in particular arrays are avoided.
|
| 46 |
+
#
|
| 47 |
+
# The "traditional" practice of packing multiple parameters into a
|
| 48 |
+
# space-separated string is a well documented source of bugs and security
|
| 49 |
+
# problems, so this is (mostly) avoided, by progressively accumulating
|
| 50 |
+
# options in "$@", and eventually passing that to Java.
|
| 51 |
+
#
|
| 52 |
+
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
|
| 53 |
+
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
|
| 54 |
+
# see the in-line comments for details.
|
| 55 |
+
#
|
| 56 |
+
# There are tweaks for specific operating systems such as AIX, CygWin,
|
| 57 |
+
# Darwin, MinGW, and NonStop.
|
| 58 |
+
#
|
| 59 |
+
# (3) This script is generated from the Groovy template
|
| 60 |
+
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
| 61 |
+
# within the Gradle project.
|
| 62 |
+
#
|
| 63 |
+
# You can find Gradle at https://github.com/gradle/gradle/.
|
| 64 |
+
#
|
| 65 |
+
##############################################################################
|
| 66 |
+
|
| 67 |
+
# Attempt to set APP_HOME
|
| 68 |
+
|
| 69 |
+
# Resolve links: $0 may be a link
|
| 70 |
+
app_path=$0
|
| 71 |
+
|
| 72 |
+
# Need this for daisy-chained symlinks.
|
| 73 |
+
while
|
| 74 |
+
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
|
| 75 |
+
[ -h "$app_path" ]
|
| 76 |
+
do
|
| 77 |
+
ls=$( ls -ld "$app_path" )
|
| 78 |
+
link=${ls#*' -> '}
|
| 79 |
+
case $link in #(
|
| 80 |
+
/*) app_path=$link ;; #(
|
| 81 |
+
*) app_path=$APP_HOME$link ;;
|
| 82 |
+
esac
|
| 83 |
+
done
|
| 84 |
+
|
| 85 |
+
# This is normally unused
|
| 86 |
+
# shellcheck disable=SC2034
|
| 87 |
+
APP_BASE_NAME=${0##*/}
|
| 88 |
+
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
|
| 89 |
+
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
|
| 90 |
+
|
| 91 |
+
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
| 92 |
+
MAX_FD=maximum
|
| 93 |
+
|
| 94 |
+
warn () {
|
| 95 |
+
echo "$*"
|
| 96 |
+
} >&2
|
| 97 |
+
|
| 98 |
+
die () {
|
| 99 |
+
echo
|
| 100 |
+
echo "$*"
|
| 101 |
+
echo
|
| 102 |
+
exit 1
|
| 103 |
+
} >&2
|
| 104 |
+
|
| 105 |
+
# OS specific support (must be 'true' or 'false').
|
| 106 |
+
cygwin=false
|
| 107 |
+
msys=false
|
| 108 |
+
darwin=false
|
| 109 |
+
nonstop=false
|
| 110 |
+
case "$( uname )" in #(
|
| 111 |
+
CYGWIN* ) cygwin=true ;; #(
|
| 112 |
+
Darwin* ) darwin=true ;; #(
|
| 113 |
+
MSYS* | MINGW* ) msys=true ;; #(
|
| 114 |
+
NONSTOP* ) nonstop=true ;;
|
| 115 |
+
esac
|
| 116 |
+
|
| 117 |
+
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
# Determine the Java command to use to start the JVM.
|
| 121 |
+
if [ -n "$JAVA_HOME" ] ; then
|
| 122 |
+
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
|
| 123 |
+
# IBM's JDK on AIX uses strange locations for the executables
|
| 124 |
+
JAVACMD=$JAVA_HOME/jre/sh/java
|
| 125 |
+
else
|
| 126 |
+
JAVACMD=$JAVA_HOME/bin/java
|
| 127 |
+
fi
|
| 128 |
+
if [ ! -x "$JAVACMD" ] ; then
|
| 129 |
+
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
|
| 130 |
+
|
| 131 |
+
Please set the JAVA_HOME variable in your environment to match the
|
| 132 |
+
location of your Java installation."
|
| 133 |
+
fi
|
| 134 |
+
else
|
| 135 |
+
JAVACMD=java
|
| 136 |
+
if ! command -v java >/dev/null 2>&1
|
| 137 |
+
then
|
| 138 |
+
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
|
| 139 |
+
|
| 140 |
+
Please set the JAVA_HOME variable in your environment to match the
|
| 141 |
+
location of your Java installation."
|
| 142 |
+
fi
|
| 143 |
+
fi
|
| 144 |
+
|
| 145 |
+
# Increase the maximum file descriptors if we can.
|
| 146 |
+
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
|
| 147 |
+
case $MAX_FD in #(
|
| 148 |
+
max*)
|
| 149 |
+
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
|
| 150 |
+
# shellcheck disable=SC2039,SC3045
|
| 151 |
+
MAX_FD=$( ulimit -H -n ) ||
|
| 152 |
+
warn "Could not query maximum file descriptor limit"
|
| 153 |
+
esac
|
| 154 |
+
case $MAX_FD in #(
|
| 155 |
+
'' | soft) :;; #(
|
| 156 |
+
*)
|
| 157 |
+
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
|
| 158 |
+
# shellcheck disable=SC2039,SC3045
|
| 159 |
+
ulimit -n "$MAX_FD" ||
|
| 160 |
+
warn "Could not set maximum file descriptor limit to $MAX_FD"
|
| 161 |
+
esac
|
| 162 |
+
fi
|
| 163 |
+
|
| 164 |
+
# Collect all arguments for the java command, stacking in reverse order:
|
| 165 |
+
# * args from the command line
|
| 166 |
+
# * the main class name
|
| 167 |
+
# * -classpath
|
| 168 |
+
# * -D...appname settings
|
| 169 |
+
# * --module-path (only if needed)
|
| 170 |
+
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
|
| 171 |
+
|
| 172 |
+
# For Cygwin or MSYS, switch paths to Windows format before running java
|
| 173 |
+
if "$cygwin" || "$msys" ; then
|
| 174 |
+
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
|
| 175 |
+
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
|
| 176 |
+
|
| 177 |
+
JAVACMD=$( cygpath --unix "$JAVACMD" )
|
| 178 |
+
|
| 179 |
+
# Now convert the arguments - kludge to limit ourselves to /bin/sh
|
| 180 |
+
for arg do
|
| 181 |
+
if
|
| 182 |
+
case $arg in #(
|
| 183 |
+
-*) false ;; # don't mess with options #(
|
| 184 |
+
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
|
| 185 |
+
[ -e "$t" ] ;; #(
|
| 186 |
+
*) false ;;
|
| 187 |
+
esac
|
| 188 |
+
then
|
| 189 |
+
arg=$( cygpath --path --ignore --mixed "$arg" )
|
| 190 |
+
fi
|
| 191 |
+
# Roll the args list around exactly as many times as the number of
|
| 192 |
+
# args, so each arg winds up back in the position where it started, but
|
| 193 |
+
# possibly modified.
|
| 194 |
+
#
|
| 195 |
+
# NB: a `for` loop captures its iteration list before it begins, so
|
| 196 |
+
# changing the positional parameters here affects neither the number of
|
| 197 |
+
# iterations, nor the values presented in `arg`.
|
| 198 |
+
shift # remove old arg
|
| 199 |
+
set -- "$@" "$arg" # push replacement arg
|
| 200 |
+
done
|
| 201 |
+
fi
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
| 205 |
+
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
| 206 |
+
|
| 207 |
+
# Collect all arguments for the java command:
|
| 208 |
+
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
|
| 209 |
+
# and any embedded shellness will be escaped.
|
| 210 |
+
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
|
| 211 |
+
# treated as '${Hostname}' itself on the command line.
|
| 212 |
+
|
| 213 |
+
set -- \
|
| 214 |
+
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
| 215 |
+
-classpath "$CLASSPATH" \
|
| 216 |
+
org.gradle.wrapper.GradleWrapperMain \
|
| 217 |
+
"$@"
|
| 218 |
+
|
| 219 |
+
# Stop when "xargs" is not available.
|
| 220 |
+
if ! command -v xargs >/dev/null 2>&1
|
| 221 |
+
then
|
| 222 |
+
die "xargs is not available"
|
| 223 |
+
fi
|
| 224 |
+
|
| 225 |
+
# Use "xargs" to parse quoted args.
|
| 226 |
+
#
|
| 227 |
+
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
|
| 228 |
+
#
|
| 229 |
+
# In Bash we could simply go:
|
| 230 |
+
#
|
| 231 |
+
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
|
| 232 |
+
# set -- "${ARGS[@]}" "$@"
|
| 233 |
+
#
|
| 234 |
+
# but POSIX shell has neither arrays nor command substitution, so instead we
|
| 235 |
+
# post-process each arg (as a line of input to sed) to backslash-escape any
|
| 236 |
+
# character that might be a shell metacharacter, then use eval to reverse
|
| 237 |
+
# that process (while maintaining the separation between arguments), and wrap
|
| 238 |
+
# the whole thing up as a single "set" statement.
|
| 239 |
+
#
|
| 240 |
+
# This will of course break if any of these variables contains a newline or
|
| 241 |
+
# an unmatched quote.
|
| 242 |
+
#
|
| 243 |
+
|
| 244 |
+
eval "set -- $(
|
| 245 |
+
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
|
| 246 |
+
xargs -n1 |
|
| 247 |
+
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
|
| 248 |
+
tr '\n' ' '
|
| 249 |
+
)" '"$@"'
|
| 250 |
+
|
| 251 |
+
exec "$JAVACMD" "$@"
|
android/gradlew.bat
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
@rem
|
| 2 |
+
@rem Copyright 2015 the original author or authors.
|
| 3 |
+
@rem
|
| 4 |
+
@rem Licensed under the Apache License, Version 2.0 (the "License");
|
| 5 |
+
@rem you may not use this file except in compliance with the License.
|
| 6 |
+
@rem You may obtain a copy of the License at
|
| 7 |
+
@rem
|
| 8 |
+
@rem https://www.apache.org/licenses/LICENSE-2.0
|
| 9 |
+
@rem
|
| 10 |
+
@rem Unless required by applicable law or agreed to in writing, software
|
| 11 |
+
@rem distributed under the License is distributed on an "AS IS" BASIS,
|
| 12 |
+
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 13 |
+
@rem See the License for the specific language governing permissions and
|
| 14 |
+
@rem limitations under the License.
|
| 15 |
+
@rem
|
| 16 |
+
@rem SPDX-License-Identifier: Apache-2.0
|
| 17 |
+
@rem
|
| 18 |
+
|
| 19 |
+
@if "%DEBUG%"=="" @echo off
|
| 20 |
+
@rem ##########################################################################
|
| 21 |
+
@rem
|
| 22 |
+
@rem Gradle startup script for Windows
|
| 23 |
+
@rem
|
| 24 |
+
@rem ##########################################################################
|
| 25 |
+
|
| 26 |
+
@rem Set local scope for the variables with windows NT shell
|
| 27 |
+
if "%OS%"=="Windows_NT" setlocal
|
| 28 |
+
|
| 29 |
+
set DIRNAME=%~dp0
|
| 30 |
+
if "%DIRNAME%"=="" set DIRNAME=.
|
| 31 |
+
@rem This is normally unused
|
| 32 |
+
set APP_BASE_NAME=%~n0
|
| 33 |
+
set APP_HOME=%DIRNAME%
|
| 34 |
+
|
| 35 |
+
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
|
| 36 |
+
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
|
| 37 |
+
|
| 38 |
+
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
| 39 |
+
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
|
| 40 |
+
|
| 41 |
+
@rem Find java.exe
|
| 42 |
+
if defined JAVA_HOME goto findJavaFromJavaHome
|
| 43 |
+
|
| 44 |
+
set JAVA_EXE=java.exe
|
| 45 |
+
%JAVA_EXE% -version >NUL 2>&1
|
| 46 |
+
if %ERRORLEVEL% equ 0 goto execute
|
| 47 |
+
|
| 48 |
+
echo. 1>&2
|
| 49 |
+
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
|
| 50 |
+
echo. 1>&2
|
| 51 |
+
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
|
| 52 |
+
echo location of your Java installation. 1>&2
|
| 53 |
+
|
| 54 |
+
goto fail
|
| 55 |
+
|
| 56 |
+
:findJavaFromJavaHome
|
| 57 |
+
set JAVA_HOME=%JAVA_HOME:"=%
|
| 58 |
+
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
|
| 59 |
+
|
| 60 |
+
if exist "%JAVA_EXE%" goto execute
|
| 61 |
+
|
| 62 |
+
echo. 1>&2
|
| 63 |
+
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
|
| 64 |
+
echo. 1>&2
|
| 65 |
+
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
|
| 66 |
+
echo location of your Java installation. 1>&2
|
| 67 |
+
|
| 68 |
+
goto fail
|
| 69 |
+
|
| 70 |
+
:execute
|
| 71 |
+
@rem Setup the command line
|
| 72 |
+
|
| 73 |
+
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
@rem Execute Gradle
|
| 77 |
+
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
|
| 78 |
+
|
| 79 |
+
:end
|
| 80 |
+
@rem End local scope for the variables with windows NT shell
|
| 81 |
+
if %ERRORLEVEL% equ 0 goto mainEnd
|
| 82 |
+
|
| 83 |
+
:fail
|
| 84 |
+
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
|
| 85 |
+
rem the _cmd.exe /c_ return code!
|
| 86 |
+
set EXIT_CODE=%ERRORLEVEL%
|
| 87 |
+
if %EXIT_CODE% equ 0 set EXIT_CODE=1
|
| 88 |
+
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
|
| 89 |
+
exit /b %EXIT_CODE%
|
| 90 |
+
|
| 91 |
+
:mainEnd
|
| 92 |
+
if "%OS%"=="Windows_NT" endlocal
|
| 93 |
+
|
| 94 |
+
:omega
|
android/licenses/Pillow-MIT-CMU.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
The Python Imaging Library (PIL) is
|
| 2 |
+
|
| 3 |
+
Copyright © 1997-2011 by Secret Labs AB
|
| 4 |
+
Copyright © 1995-2011 by Fredrik Lundh and contributors
|
| 5 |
+
|
| 6 |
+
Pillow is the friendly PIL fork. It is
|
| 7 |
+
|
| 8 |
+
Copyright © 2010 by Jeffrey 'Alex' Clark and contributors
|
| 9 |
+
|
| 10 |
+
Like PIL, Pillow is licensed under the open source MIT-CMU License:
|
| 11 |
+
|
| 12 |
+
By obtaining, using, and/or copying this software and/or its associated
|
| 13 |
+
documentation, you agree that you have read, understood, and will comply
|
| 14 |
+
with the following terms and conditions:
|
| 15 |
+
|
| 16 |
+
Permission to use, copy, modify and distribute this software and its
|
| 17 |
+
documentation for any purpose and without fee is hereby granted,
|
| 18 |
+
provided that the above copyright notice appears in all copies, and that
|
| 19 |
+
both that copyright notice and this permission notice appear in supporting
|
| 20 |
+
documentation, and that the name of Secret Labs AB or the author not be
|
| 21 |
+
used in advertising or publicity pertaining to distribution of the software
|
| 22 |
+
without specific, written prior permission.
|
| 23 |
+
|
| 24 |
+
SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
|
| 25 |
+
SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
|
| 26 |
+
IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR BE LIABLE FOR ANY SPECIAL,
|
| 27 |
+
INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
|
| 28 |
+
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
|
| 29 |
+
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
|
| 30 |
+
PERFORMANCE OF THIS SOFTWARE.
|
android/scripts/install_to_device.sh
ADDED
|
@@ -0,0 +1,165 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Copy Idol's graphs, side files and manifest into the app's private files/models/, one file at a time
|
| 3 |
+
# (adb push -> run-as cp -> rm), and with FIXTURES=1 also the test fixtures (device/fx/<tag>/ -> files/fx/<tag>/) and
|
| 4 |
+
# the autorun plans (plans/*.json -> files/plans/).
|
| 5 |
+
#
|
| 6 |
+
# The file list, sizes and SHA-256 come from manifest.json in IDOL_DIR. Sources:
|
| 7 |
+
# IDOL_DIR manifest.json, every graph except dino_* and dis, the side and JSON files
|
| 8 |
+
# DINO_DIR dino_0..3.tflite (and dino_rope_cos/sin.f32 when they are there, else IDOL_DIR's)
|
| 9 |
+
# DIS_DIR dis.tflite (optional: only the "Background removal on the GPU" setting uses it)
|
| 10 |
+
# Every source is checked by size before any device command; after each copy the SHA-256 is checked on the phone
|
| 11 |
+
# (the manifest's value for a graph, the local file's for the rest). A file whose copy on the phone already has the
|
| 12 |
+
# expected SHA-256 is not copied again. MODELS=0 skips the models (fixtures and plans only, with FIXTURES=1).
|
| 13 |
+
# Install the debug APK first: run-as needs a debuggable package. ANDROID_SERIAL selects the device. Needs python3
|
| 14 |
+
# (reads the manifest).
|
| 15 |
+
set -euo pipefail
|
| 16 |
+
|
| 17 |
+
MODULE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 18 |
+
|
| 19 |
+
# Defaults for local testing: this worktree's export (models/v2 holds every graph except dis.tflite).
|
| 20 |
+
# For the published release point them at the downloaded repos (see README), e.g.
|
| 21 |
+
# IDOL_DIR=~/Downloads/Idol DINO_DIR=~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT DIS_DIR=~/Downloads/DIS-ISNet-LiteRT
|
| 22 |
+
IDOL_DIR="${IDOL_DIR:-$MODULE_DIR/models/v2}"
|
| 23 |
+
DINO_DIR="${DINO_DIR:-$IDOL_DIR}"
|
| 24 |
+
DIS_DIR="${DIS_DIR:-$HOME/Downloads/DIS-ISNet-LiteRT}"
|
| 25 |
+
MODELS="${MODELS:-1}"
|
| 26 |
+
FIXTURES="${FIXTURES:-0}"
|
| 27 |
+
FX_TAGS="${FX_TAGS:-armchair axolotl succulent black}"
|
| 28 |
+
PACKAGE=com.idol
|
| 29 |
+
TEMP_DIR=/data/local/tmp/idol_install
|
| 30 |
+
|
| 31 |
+
if [[ $# -gt 0 ]]; then
|
| 32 |
+
printf 'Usage: [IDOL_DIR=...] [DINO_DIR=...] [DIS_DIR=...] [MODELS=0] [FIXTURES=1] %s\n' "$0" >&2
|
| 33 |
+
exit 2
|
| 34 |
+
fi
|
| 35 |
+
MANIFEST="$IDOL_DIR/manifest.json"
|
| 36 |
+
if [[ ! -f "$MANIFEST" ]]; then
|
| 37 |
+
printf 'Missing %s: set IDOL_DIR (and DINO_DIR, DIS_DIR) to the downloaded model folders (see README)\n' \
|
| 38 |
+
"$MANIFEST" >&2
|
| 39 |
+
exit 1
|
| 40 |
+
fi
|
| 41 |
+
|
| 42 |
+
sha_of() {
|
| 43 |
+
if command -v sha256sum > /dev/null; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
# "kind name file bytes sha" per line; bytes and sha are "-" when the manifest has none
|
| 47 |
+
ENTRIES="$(python3 -I - "$MANIFEST" <<'PY'
|
| 48 |
+
import json, sys
|
| 49 |
+
m = json.load(open(sys.argv[1]))
|
| 50 |
+
for name, g in m['graphs'].items():
|
| 51 |
+
print('graph', name, g['file'], g['bytes'], g.get('sha256') or '-')
|
| 52 |
+
for f, s in m.get('side_files', {}).items():
|
| 53 |
+
n = 4
|
| 54 |
+
for d in s['shape']:
|
| 55 |
+
n *= d
|
| 56 |
+
print('side', f, f, n, '-')
|
| 57 |
+
for f in m.get('json_files', []):
|
| 58 |
+
print('json', f, f, '-', '-')
|
| 59 |
+
PY
|
| 60 |
+
)"
|
| 61 |
+
|
| 62 |
+
# The source of each file.
|
| 63 |
+
source_of() {
|
| 64 |
+
local kind="$1" name="$2" file="$3"
|
| 65 |
+
case "$kind:$name" in
|
| 66 |
+
graph:dino_*) printf '%s/%s' "$DINO_DIR" "$file" ;;
|
| 67 |
+
graph:dis) printf '%s/%s' "$DIS_DIR" "$file" ;;
|
| 68 |
+
side:dino_rope_*)
|
| 69 |
+
if [[ -f "$DINO_DIR/$file" ]]; then printf '%s/%s' "$DINO_DIR" "$file"; else printf '%s/%s' "$IDOL_DIR" "$file"; fi
|
| 70 |
+
;;
|
| 71 |
+
*) printf '%s/%s' "$IDOL_DIR" "$file" ;;
|
| 72 |
+
esac
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
# Check every source before any device command to avoid a partial installation.
|
| 76 |
+
PLAN=()
|
| 77 |
+
if [[ "$MODELS" == 0 ]]; then ENTRIES=''; fi
|
| 78 |
+
while read -r kind name file bytes sha; do
|
| 79 |
+
[[ -z "$kind" ]] && continue
|
| 80 |
+
src="$(source_of "$kind" "$name" "$file")"
|
| 81 |
+
if [[ ! -f "$src" ]]; then
|
| 82 |
+
if [[ "$kind:$name" == graph:dis ]]; then
|
| 83 |
+
printf 'Skipping dis.tflite (not in %s): the GPU background removal setting will not work\n' "$DIS_DIR"
|
| 84 |
+
continue
|
| 85 |
+
fi
|
| 86 |
+
printf 'Missing source file: %s\n' "$src" >&2
|
| 87 |
+
exit 1
|
| 88 |
+
fi
|
| 89 |
+
if [[ "$bytes" != - ]]; then
|
| 90 |
+
actual="$(wc -c < "$src" | tr -d ' ')"
|
| 91 |
+
if [[ "$actual" != "$bytes" ]]; then
|
| 92 |
+
printf '%s is %s bytes, expected %s\n' "$src" "$actual" "$bytes" >&2
|
| 93 |
+
exit 1
|
| 94 |
+
fi
|
| 95 |
+
fi
|
| 96 |
+
# files without a manifest SHA-256 (side and JSON files) are checked against the local copy's
|
| 97 |
+
if [[ "$sha" == - ]]; then sha="$(sha_of "$src")"; fi
|
| 98 |
+
PLAN+=("$src|models/$file|$sha")
|
| 99 |
+
done <<< "$ENTRIES"
|
| 100 |
+
if [[ "$MODELS" != 0 ]]; then
|
| 101 |
+
PLAN+=("$MANIFEST|models/manifest.json|$(sha_of "$MANIFEST")")
|
| 102 |
+
fi
|
| 103 |
+
|
| 104 |
+
if [[ "$FIXTURES" == 1 ]]; then
|
| 105 |
+
for tag in $FX_TAGS; do
|
| 106 |
+
dir="$MODULE_DIR/device/fx/$tag"
|
| 107 |
+
if [[ ! -d "$dir" ]]; then
|
| 108 |
+
printf 'Missing fixture folder: %s\n' "$dir" >&2
|
| 109 |
+
exit 1
|
| 110 |
+
fi
|
| 111 |
+
for f in "$dir"/*; do
|
| 112 |
+
[[ -f "$f" ]] || continue
|
| 113 |
+
PLAN+=("$f|fx/$tag/$(basename "$f")|$(sha_of "$f")")
|
| 114 |
+
done
|
| 115 |
+
done
|
| 116 |
+
for f in "$MODULE_DIR"/plans/*.json; do
|
| 117 |
+
[[ -f "$f" ]] || continue
|
| 118 |
+
PLAN+=("$f|plans/$(basename "$f")|$(sha_of "$f")")
|
| 119 |
+
done
|
| 120 |
+
fi
|
| 121 |
+
|
| 122 |
+
pending=''
|
| 123 |
+
cleanup() {
|
| 124 |
+
if [[ -n "$pending" ]]; then
|
| 125 |
+
adb shell rm -f "$pending" < /dev/null
|
| 126 |
+
fi
|
| 127 |
+
}
|
| 128 |
+
trap cleanup EXIT
|
| 129 |
+
|
| 130 |
+
# Copies $1 (a local file) to files/$2 of the app and checks the SHA-256 $3 on the phone ("-": no check).
|
| 131 |
+
install_file() {
|
| 132 |
+
local src="$1" dst="$2" sha="$3"
|
| 133 |
+
pending="$TEMP_DIR/$$-$(basename "$dst")"
|
| 134 |
+
adb push "$src" "$pending" < /dev/null
|
| 135 |
+
adb shell run-as "$PACKAGE" mkdir -p "files/$(dirname "$dst")" < /dev/null
|
| 136 |
+
adb shell run-as "$PACKAGE" cp "$pending" "files/$dst" < /dev/null
|
| 137 |
+
adb shell rm "$pending" < /dev/null
|
| 138 |
+
pending=''
|
| 139 |
+
if [[ "$sha" != - ]]; then
|
| 140 |
+
local got
|
| 141 |
+
got="$(adb shell run-as "$PACKAGE" sha256sum "files/$dst" < /dev/null | cut -d' ' -f1 | tr -d '\r')"
|
| 142 |
+
if [[ "$got" != "$sha" ]]; then
|
| 143 |
+
printf 'files/%s on the device has SHA-256 %s, expected %s\n' "$dst" "$got" "$sha" >&2
|
| 144 |
+
exit 1
|
| 145 |
+
fi
|
| 146 |
+
fi
|
| 147 |
+
}
|
| 148 |
+
|
| 149 |
+
adb shell mkdir -p "$TEMP_DIR" < /dev/null
|
| 150 |
+
adb shell run-as "$PACKAGE" mkdir -p files/models < /dev/null
|
| 151 |
+
for entry in "${PLAN[@]}"; do
|
| 152 |
+
IFS='|' read -r src dst sha <<< "$entry"
|
| 153 |
+
if [[ "$sha" != - ]]; then
|
| 154 |
+
have="$(adb shell run-as "$PACKAGE" sha256sum "files/$dst" < /dev/null 2> /dev/null | cut -d' ' -f1 | tr -d '\r' || true)"
|
| 155 |
+
if [[ "$have" == "$sha" ]]; then
|
| 156 |
+
printf 'Up to date: %s\n' "$dst"
|
| 157 |
+
continue
|
| 158 |
+
fi
|
| 159 |
+
fi
|
| 160 |
+
printf 'Installing %s (%s bytes)\n' "$dst" "$(wc -c < "$src" | tr -d ' ')"
|
| 161 |
+
install_file "$src" "$dst" "$sha"
|
| 162 |
+
done
|
| 163 |
+
adb shell rmdir "$TEMP_DIR" < /dev/null 2> /dev/null || true
|
| 164 |
+
adb shell run-as "$PACKAGE" ls -la files/models < /dev/null
|
| 165 |
+
printf 'Files installed. Open Idol.\n'
|
android/settings.gradle.kts
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pluginManagement {
|
| 2 |
+
repositories {
|
| 3 |
+
google()
|
| 4 |
+
mavenCentral()
|
| 5 |
+
gradlePluginPortal()
|
| 6 |
+
}
|
| 7 |
+
}
|
| 8 |
+
|
| 9 |
+
dependencyResolutionManagement {
|
| 10 |
+
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
|
| 11 |
+
repositories {
|
| 12 |
+
google()
|
| 13 |
+
mavenCentral()
|
| 14 |
+
}
|
| 15 |
+
}
|
| 16 |
+
|
| 17 |
+
rootProject.name = "idol"
|
| 18 |
+
include(":app")
|
bir.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:42b98bf0e497dd596327c31883fec21fe7013a4c9f72d1f67ec8ff4d3f0dfd43
|
| 3 |
+
size 930410016
|
den_ctx.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:39278b27d00a3dcc70da5e2737f6055d5ba4e7e817423a4cf176aacbfb86230d
|
| 3 |
+
size 117711308
|
den_in.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1350c12da702a4a0abf53cdf528b62a8a75ea4d948c6a4764b264b927b419802
|
| 3 |
+
size 146782524
|
den_joint_00.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:37e269459ef1325bd4a84d1df1545afacf8e33d6d5c232c62f834f343a882f86
|
| 3 |
+
size 56134468
|
den_joint_01.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:103efe94ce85aeb29f084f85962d0e85d4a395c629101d3ea21d23e6cdc53e25
|
| 3 |
+
size 56134100
|
den_joint_02.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1ec688cf610156841a1523db9ed93ad268274f957132dcbc92539308143ce517
|
| 3 |
+
size 56134100
|
den_joint_03.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b5828ac85b0f42118e79f2e83cbe05021d2b68f9a3c167f0ac014a3026799cc2
|
| 3 |
+
size 56134100
|
den_joint_04.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ea5c57b89c8d8bbc5144d5d487e20fc49daf25308bccf853bb3c9d4581ae33fe
|
| 3 |
+
size 56134100
|
den_joint_05.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f339d8b79e5d4c2fc408c2d42f7ce959b81821cad37861b2a1294b8397e19997
|
| 3 |
+
size 56134100
|
den_joint_06.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:30d97e62ac4fc28882903961c88e65e6fe540d61e95504651b44d7cb8f341c4c
|
| 3 |
+
size 56134100
|
den_joint_07.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0c7b243e55c69b6cdd9c5e7bbef313f6cc6be0c0f7ad32e428187e22b2183a67
|
| 3 |
+
size 56134100
|
den_joint_08.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0eb17bea8ae04ea0d88d79b327515e1a106bb5339d72440a8ec63712865e3f27
|
| 3 |
+
size 56134100
|
den_joint_09.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:353ece07e6b4888d80ab08e2bdd6dc2dd456f213ca87694f6d77f987d5d4debc
|
| 3 |
+
size 56134100
|
den_joint_10.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0c7e15340ea4212369ae85f3f9db625f21ccd1747038e0725675c5a7ca32ac07
|
| 3 |
+
size 56134100
|