mlboydaisuke commited on
Commit
ed66dc1
·
verified ·
1 Parent(s): a8917cc

Idol on LiteRT: graphs, manifest, Python runner and Android sample

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +1 -0
  2. LICENSES.md +37 -0
  3. android/app/build.gradle.kts +67 -0
  4. android/app/src/main/AndroidManifest.xml +45 -0
  5. android/app/src/main/assets/viewer/LICENSE +21 -0
  6. android/app/src/main/assets/viewer/index.html +281 -0
  7. android/app/src/main/assets/viewer/main.js +1527 -0
  8. android/app/src/main/java/com/idol/AutoRun.kt +312 -0
  9. android/app/src/main/java/com/idol/CardSnippet.kt +75 -0
  10. android/app/src/main/java/com/idol/DeviceInfo.kt +102 -0
  11. android/app/src/main/java/com/idol/Fixture.kt +66 -0
  12. android/app/src/main/java/com/idol/Glue.kt +472 -0
  13. android/app/src/main/java/com/idol/Graphs.kt +339 -0
  14. android/app/src/main/java/com/idol/IdolEngine.kt +606 -0
  15. android/app/src/main/java/com/idol/MainActivity.kt +581 -0
  16. android/app/src/main/java/com/idol/Manifest.kt +103 -0
  17. android/app/src/main/java/com/idol/ModelFiles.kt +96 -0
  18. android/app/src/main/java/com/idol/Photo.kt +89 -0
  19. android/app/src/main/java/com/idol/Pil.kt +361 -0
  20. android/app/src/main/java/com/idol/Preprocess.kt +201 -0
  21. android/app/src/main/java/com/idol/ResultViewer.kt +66 -0
  22. android/app/src/main/java/com/idol/TorchOps.kt +91 -0
  23. android/app/src/main/res/layout/activity_main.xml +216 -0
  24. android/app/src/main/res/values/strings.xml +21 -0
  25. android/app/src/main/res/values/themes.xml +6 -0
  26. android/app/src/main/res/xml/file_paths.xml +7 -0
  27. android/build.gradle.kts +4 -0
  28. android/gradle.properties +7 -0
  29. android/gradle/libs.versions.toml +18 -0
  30. android/gradle/wrapper/gradle-wrapper.jar +0 -0
  31. android/gradle/wrapper/gradle-wrapper.properties +7 -0
  32. android/gradlew +251 -0
  33. android/gradlew.bat +94 -0
  34. android/licenses/Pillow-MIT-CMU.txt +30 -0
  35. android/scripts/install_to_device.sh +165 -0
  36. android/settings.gradle.kts +18 -0
  37. bir.tflite +3 -0
  38. den_ctx.tflite +3 -0
  39. den_in.tflite +3 -0
  40. den_joint_00.tflite +3 -0
  41. den_joint_01.tflite +3 -0
  42. den_joint_02.tflite +3 -0
  43. den_joint_03.tflite +3 -0
  44. den_joint_04.tflite +3 -0
  45. den_joint_05.tflite +3 -0
  46. den_joint_06.tflite +3 -0
  47. den_joint_07.tflite +3 -0
  48. den_joint_08.tflite +3 -0
  49. den_joint_09.tflite +3 -0
  50. den_joint_10.tflite +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ den_pos_emb.f32 filter=lfs diff=lfs merge=lfs -text
LICENSES.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Licences of the parts of the Idol pipeline (read 2026-11-09)
2
+
3
+ | Part | Weights used here | Licence | Source read |
4
+ |---|---|---|---|
5
+ | Denoiser, octree decoder, Gaussian decoder | speridlabs/idol @2731c229 (`denoiser`, `decoder`, `decoder_gs` .safetensors) | Apache-2.0 | HF card front matter line 2 `license: apache-2.0`; idol-sdk LICENSE @4702a7f0 (Apache License 2.0, lines 1-2) |
6
+ | Turbo LoRA (slow / fast experts) | speridlabs/idol-turbo @45cdd41f (`idol_turbo_lora.safetensors`) | Apache-2.0 | HF card front matter `license: apache-2.0`; card §License "Apache-2.0, inherited from speridlabs/idol" |
7
+ | Background removal (shipped: `bir.tflite`, LiteRT CPU) | speridlabs/idol @2731c229 `birefnet.safetensors` (BiRefNet, Swin-L) | MIT (BiRefNet); the idol repo that packages it is Apache-2.0 | github ZhengPeng7/BiRefNet licence = MIT (GitHub API); idol HF card front matter line 2 |
8
+ | Background removal, alternative in the app (off by default) | litert-community/DIS-ISNet-LiteRT @1b966dbe `dis.tflite` (IS-Net DIS general-use, xuebinqin/DIS) | Apache-2.0 | litert-community/DIS-ISNet-LiteRT card `license: apache-2.0` (Hub API) |
9
+ | FLUX.2 VAE (encoder side used) | black-forest-labs/FLUX.2-klein-4B @e7b7dc27 `vae/` (ungated) | Apache-2.0 | black-forest-labs/flux2 README @50fe5162, line 111: "The autoencoder is released under Apache 2.0"; klein-4B card `license: apache-2.0` |
10
+ | DINOv3 ViT-H+/16 | Comfy-Org/Idol @e907d8dc `clip_vision/dino_v3_vit_h_bf16.safetensors` (bf16 repack of facebook/dinov3-vith16plus-pretrain-lvd1689m, gated) | DINOv3 License (Meta, "Last Updated: August 19, 2025") | facebookresearch/dinov3 LICENSE.md @6876159a |
11
+ | Idol acknowledges TripoSplat | — (no TripoSplat weights used) | MIT | github VAST-AI-Research/TripoSplat licence = MIT (GitHub API) |
12
+
13
+ ## DINOv3 License — the clauses that decide redistribution (dinov3_LICENSE.md)
14
+
15
+ - Line 20: "By clicking "I Accept" below or by using or distributing any portion or element of the DINO Materials, you agree to be bound by this Agreement." → using the Comfy-Org repack binds the user of it to the Agreement as if accepted on the gated page.
16
+ - Line 24 (§1.a Grant of Rights): "non-exclusive, worldwide, non-transferable and royalty-free limited license … to use, reproduce, distribute, copy, create derivative works of, and make modifications to the DINO Materials." → a converted .tflite of the encoder is a derivative work and may be distributed.
17
+ - Line 28 (§1.b.i): "If you distribute or make the DINO Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement and you shall provide a copy of this Agreement with any such DINO Materials." → a hosted encoder file must carry the DINOv3 License text and be offered under it (not under Apache-2.0).
18
+ - Line 30 (§1.b.ii): acknowledgement in research publications.
19
+ - Line 34 (§1.b.iv): no use that involves or encourages reverse engineering / decompiling the DINO Materials.
20
+ - Line 36 (§1.b.v): trade-control limits (no ITAR / military / nuclear / espionage / weapons end uses; licensee not a target of trade controls).
21
+ - Line 52 (§5.a): derivative works made by the licensee are owned by the licensee (subject to Meta's ownership of the DINO Materials).
22
+ - No non-commercial clause anywhere in the text (66 lines read in full).
23
+
24
+ ## FLUX.2 VAE
25
+
26
+ - flux2 README line 111 (commit 50fe5162): the FLUX.2 autoencoder is released under Apache 2.0. The FLUX.2 [dev] Non-Commercial License of the FLUX.2-dev repo covers the [dev] transformer, not the autoencoder.
27
+ - The klein-4B `vae/` dir is ungated and tagged apache-2.0; the Idol pipeline loads the VAE in bf16 (`conditioning.py` `torch_dtype=torch.bfloat16`) and klein-4B's file is bf16.
28
+ - Verified 2026-11-09 14:04: klein-4B `vae/diffusion_pytorch_model.safetensors` (sha256 ca70d220…, bf16) = the FLUX.2-dev VAE cast to bf16. The FLUX.2-dev VAE was read from Comfy-Org/Idol `vae/flux2-vae.safetensors`, whose LFS sha256 868fe7b3… equals FLUX.2-dev's `ae.safetensors` (Hub API, FLUX.2-dev @26afe3a7): the same file. All 250 float tensors match bit for bit after the bf16 cast (84,046,372 values; 8 attention projections are [512, 512] in klein and [512, 512, 1, 1] in BFL, same values). The one other tensor differs: `bn.num_batches_tracked` (int64 training counter, 0 vs 400000), unused at inference.
29
+
30
+ ## Hosting
31
+
32
+ - litert-community/Idol (ungated, Apache-2.0, with the per-part provenance above): the denoiser, octree and Gaussian
33
+ decoder graphs, the VAE encoder graphs, BiRefNet (`bir.tflite`, MIT part named in the card), the side files, the
34
+ manifest, and the Python / Android sources.
35
+ - The DINOv3 encoder graphs and their RoPE tables: a separate litert-community repo, gated (automatic approval), carrying
36
+ the full DINOv3 License text and the provenance (base_model facebook/dinov3-vith16plus-pretrain-lvd1689m), the way
37
+ litert-community/Llama-3.2-1B carries the Llama licence.
android/app/build.gradle.kts ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plugins {
2
+ alias(libs.plugins.android.application)
3
+ alias(libs.plugins.kotlin.android)
4
+ }
5
+
6
+ android {
7
+ namespace = "com.idol"
8
+ compileSdk = 35
9
+ buildToolsVersion = "35.0.0"
10
+
11
+ defaultConfig {
12
+ applicationId = "com.idol"
13
+ minSdk = 26
14
+ targetSdk = 35
15
+ versionCode = 1
16
+ versionName = "0.1"
17
+ buildConfigField("String", "LITERT_VERSION", "\"${libs.versions.litert.get()}\"")
18
+ ndk {
19
+ abiFilters += "arm64-v8a"
20
+ }
21
+ }
22
+
23
+ buildTypes {
24
+ release {
25
+ isMinifyEnabled = false
26
+ // sample convenience: installable without a release keystore
27
+ signingConfig = signingConfigs.getByName("debug")
28
+ }
29
+ }
30
+
31
+ androidResources {
32
+ // graphs are never assets here (they are installed in files/models/), but keep any .tflite mmappable
33
+ noCompress += "tflite"
34
+ }
35
+
36
+ packaging {
37
+ // The GPU accelerator library (litert-gpu) is loaded from the extracted native library directory, as in the
38
+ // zoo's other GPU samples on the same phones.
39
+ jniLibs { useLegacyPackaging = true }
40
+ }
41
+
42
+ compileOptions {
43
+ sourceCompatibility = JavaVersion.VERSION_17
44
+ targetCompatibility = JavaVersion.VERSION_17
45
+ }
46
+ buildFeatures {
47
+ buildConfig = true
48
+ }
49
+ }
50
+
51
+ kotlin {
52
+ compilerOptions {
53
+ jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
54
+ // LiteRT 2.3.0's Kotlin API carries Kotlin 2.4 metadata; Google's own 2.3.0 samples on Kotlin 2.2.21 pass this
55
+ freeCompilerArgs.add("-Xskip-metadata-version-check")
56
+ }
57
+ }
58
+
59
+ dependencies {
60
+ // LiteRT CompiledModel (CPU / XNNPACK for BiRefNet) and its GPU accelerator (ML Drift, OpenCL) for the rest
61
+ implementation(libs.litert)
62
+ implementation(libs.litert.gpu)
63
+ implementation(libs.androidx.core.ktx)
64
+ implementation(libs.androidx.activity.ktx)
65
+ // WebViewAssetLoader: the splat viewer and the .ply served to the WebView from the app itself
66
+ implementation(libs.androidx.webkit)
67
+ }
android/app/src/main/AndroidManifest.xml ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <?xml version="1.0" encoding="utf-8"?>
2
+ <manifest xmlns:android="http://schemas.android.com/apk/res/android">
3
+
4
+ <!-- "Take a photo" asks for it first; without it the gallery still works. -->
5
+ <uses-permission android:name="android.permission.CAMERA" />
6
+
7
+ <uses-feature
8
+ android:name="android.hardware.camera"
9
+ android:required="false" />
10
+
11
+ <!-- The host side of a run (the 50 MB tensors between graphs, a photo's working copies) is on the Java heap. -->
12
+ <application
13
+ android:allowBackup="false"
14
+ android:label="@string/app_name"
15
+ android:largeHeap="true"
16
+ android:supportsRtl="true"
17
+ android:theme="@style/Theme.Idol">
18
+
19
+ <!-- The GPU accelerator opens the vendor OpenCL library. -->
20
+ <uses-native-library
21
+ android:name="libOpenCL.so"
22
+ android:required="false" />
23
+
24
+ <activity
25
+ android:name=".MainActivity"
26
+ android:configChanges="orientation|screenSize|screenLayout|keyboardHidden"
27
+ android:exported="true"
28
+ android:launchMode="singleTop">
29
+ <intent-filter>
30
+ <action android:name="android.intent.action.MAIN" />
31
+ <category android:name="android.intent.category.LAUNCHER" />
32
+ </intent-filter>
33
+ </activity>
34
+
35
+ <provider
36
+ android:name="androidx.core.content.FileProvider"
37
+ android:authorities="${applicationId}.files"
38
+ android:exported="false"
39
+ android:grantUriPermissions="true">
40
+ <meta-data
41
+ android:name="android.support.FILE_PROVIDER_PATHS"
42
+ android:resource="@xml/file_paths" />
43
+ </provider>
44
+ </application>
45
+ </manifest>
android/app/src/main/assets/viewer/LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2023 Kevin Kwok
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
android/app/src/main/assets/viewer/index.html ADDED
@@ -0,0 +1,281 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en" dir="ltr">
3
+ <head>
4
+ <title>WebGL Gaussian Splat Viewer</title>
5
+ <meta charset="utf-8" />
6
+ <meta
7
+ name="viewport"
8
+ content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=no"
9
+ />
10
+ <meta name="apple-mobile-web-app-capable" content="yes" />
11
+ <meta
12
+ name="apple-mobile-web-app-status-bar-style"
13
+ content="black-translucent"
14
+ />
15
+ <style>
16
+ body {
17
+ overflow: hidden;
18
+ margin: 0;
19
+ height: 100vh;
20
+ width: 100vw;
21
+ font-family: sans-serif;
22
+ background: black;
23
+ text-shadow: 0 0 3px black;
24
+ }
25
+ a, body {
26
+ color: white;
27
+ }
28
+ #info {
29
+ z-index: 100;
30
+ position: absolute;
31
+ top: 10px;
32
+ left: 15px;
33
+ }
34
+ h3 {
35
+ margin: 5px 0;
36
+ }
37
+ p {
38
+ margin: 5px 0;
39
+ font-size: small;
40
+ }
41
+
42
+ .cube-wrapper {
43
+ transform-style: preserve-3d;
44
+ }
45
+
46
+ .cube {
47
+ transform-style: preserve-3d;
48
+ transform: rotateX(45deg) rotateZ(45deg);
49
+ animation: rotation 2s infinite;
50
+ }
51
+
52
+ .cube-faces {
53
+ transform-style: preserve-3d;
54
+ height: 80px;
55
+ width: 80px;
56
+ position: relative;
57
+ transform-origin: 0 0;
58
+ transform: translateX(0) translateY(0) translateZ(-40px);
59
+ }
60
+
61
+ .cube-face {
62
+ position: absolute;
63
+ inset: 0;
64
+ background: #0017ff;
65
+ border: solid 1px #ffffff;
66
+ }
67
+ .cube-face.top {
68
+ transform: translateZ(80px);
69
+ }
70
+ .cube-face.front {
71
+ transform-origin: 0 50%;
72
+ transform: rotateY(-90deg);
73
+ }
74
+ .cube-face.back {
75
+ transform-origin: 0 50%;
76
+ transform: rotateY(-90deg) translateZ(-80px);
77
+ }
78
+ .cube-face.right {
79
+ transform-origin: 50% 0;
80
+ transform: rotateX(-90deg) translateY(-80px);
81
+ }
82
+ .cube-face.left {
83
+ transform-origin: 50% 0;
84
+ transform: rotateX(-90deg) translateY(-80px) translateZ(80px);
85
+ }
86
+
87
+ @keyframes rotation {
88
+ 0% {
89
+ transform: rotateX(45deg) rotateY(0) rotateZ(45deg);
90
+ animation-timing-function: cubic-bezier(
91
+ 0.17,
92
+ 0.84,
93
+ 0.44,
94
+ 1
95
+ );
96
+ }
97
+ 50% {
98
+ transform: rotateX(45deg) rotateY(0) rotateZ(225deg);
99
+ animation-timing-function: cubic-bezier(
100
+ 0.76,
101
+ 0.05,
102
+ 0.86,
103
+ 0.06
104
+ );
105
+ }
106
+ 100% {
107
+ transform: rotateX(45deg) rotateY(0) rotateZ(405deg);
108
+ animation-timing-function: cubic-bezier(
109
+ 0.17,
110
+ 0.84,
111
+ 0.44,
112
+ 1
113
+ );
114
+ }
115
+ }
116
+
117
+ .scene,
118
+ #message {
119
+ position: absolute;
120
+ display: flex;
121
+ top: 0;
122
+ right: 0;
123
+ left: 0;
124
+ bottom: 0;
125
+ z-index: 2;
126
+ height: 100%;
127
+ width: 100%;
128
+ align-items: center;
129
+ justify-content: center;
130
+ }
131
+ #message {
132
+ font-weight: bold;
133
+ font-size: large;
134
+ color: red;
135
+ pointer-events: none;
136
+ }
137
+
138
+ details {
139
+ font-size: small;
140
+
141
+ }
142
+
143
+ #progress {
144
+ position: absolute;
145
+ top: 0;
146
+ height: 5px;
147
+ background: blue;
148
+ z-index: 99;
149
+ transition: width 0.1s ease-in-out;
150
+ }
151
+
152
+ #quality {
153
+ position: absolute;
154
+ bottom: 10px;
155
+ z-index: 999;
156
+ right: 10px;
157
+ }
158
+
159
+ #caminfo {
160
+ position: absolute;
161
+ top: 10px;
162
+ z-index: 999;
163
+ right: 10px;
164
+ }
165
+ #canvas {
166
+ display: block;
167
+ position: absolute;
168
+ top: 0;
169
+ left: 0;
170
+ width: 100%;
171
+ height: 100%;
172
+ touch-action: none;
173
+ }
174
+
175
+ #instructions {
176
+ background: rgba(0,0,0,0.6);
177
+ white-space: pre-wrap;
178
+ padding: 10px;
179
+ border-radius: 10px;
180
+ font-size: x-small;
181
+ }
182
+ body.nohf .nohf {
183
+ display: none;
184
+ }
185
+ body.nohf #progress, body.nohf .cube-face {
186
+ background: #ff9d0d;
187
+ }
188
+ /* idol: in the app the keyboard and mouse help, its links and the fps counter are hidden */
189
+ #info, #quality, #caminfo {
190
+ display: none;
191
+ }
192
+ </style>
193
+ </head>
194
+ <body>
195
+ <script>
196
+ if(location.host.includes('hf.space')) document.body.classList.add('nohf');
197
+ </script>
198
+ <div id="info">
199
+ <h3 class="nohf">WebGL 3D Gaussian Splat Viewer</h3>
200
+ <p>
201
+ <small class="nohf">
202
+ By <a href="https://twitter.com/antimatter15">Kevin Kwok</a>.
203
+ Code on
204
+ <a href="https://github.com/antimatter15/splat">Github</a
205
+ >.
206
+ </small>
207
+ </p>
208
+
209
+ <details>
210
+ <summary>Use mouse or arrow keys to navigate.</summary>
211
+
212
+ <div id="instructions">movement (arrow keys)
213
+ - left/right arrow keys to strafe side to side
214
+ - up/down arrow keys to move forward/back
215
+ - space to jump
216
+
217
+ camera angle (wasd)
218
+ - a/d to turn camera left/right
219
+ - w/s to tilt camera up/down
220
+ - q/e to roll camera counterclockwise/clockwise
221
+ - i/k and j/l to orbit
222
+
223
+ trackpad
224
+ - scroll up/down/left/right to orbit
225
+ - pinch to move forward/back
226
+ - ctrl key + scroll to move forward/back
227
+ - shift + scroll to move up/down or strafe
228
+
229
+ mouse
230
+ - click and drag to orbit
231
+ - right click (or ctrl/cmd key) and drag up/down to move
232
+
233
+ touch (mobile)
234
+ - one finger to orbit
235
+ - two finger pinch to move forward/back
236
+ - two finger rotate to rotate camera clockwise/counterclockwise
237
+ - two finger pan to move side-to-side and up-down
238
+
239
+ gamepad
240
+ - if you have a game controller connected it should work
241
+
242
+ other
243
+ - press 0-9 to switch to one of the pre-loaded camera views
244
+ - press '-' or '+'key to cycle loaded cameras
245
+ - press p to resume default animation
246
+ - drag and drop .ply file to convert to .splat
247
+ - drag and drop cameras.json to load cameras
248
+ </div>
249
+
250
+ </details>
251
+
252
+ </div>
253
+
254
+ <div id="progress"></div>
255
+
256
+ <div id="message"></div>
257
+ <div class="scene" id="spinner">
258
+ <div class="cube-wrapper">
259
+ <div class="cube">
260
+ <div class="cube-faces">
261
+ <div class="cube-face bottom"></div>
262
+ <div class="cube-face top"></div>
263
+ <div class="cube-face left"></div>
264
+ <div class="cube-face right"></div>
265
+ <div class="cube-face back"></div>
266
+ <div class="cube-face front"></div>
267
+ </div>
268
+ </div>
269
+ </div>
270
+ </div>
271
+ <canvas id="canvas"></canvas>
272
+
273
+ <div id="quality">
274
+ <span id="fps"></span>
275
+ </div>
276
+ <div id="caminfo">
277
+ <span id="camid"></span>
278
+ </div>
279
+ <script src="main.js"></script>
280
+ </body>
281
+ </html>
android/app/src/main/assets/viewer/main.js ADDED
@@ -0,0 +1,1527 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ let cameras = [
2
+ {
3
+ id: 0,
4
+ img_name: "00001",
5
+ width: 1959,
6
+ height: 1090,
7
+ position: [
8
+ -3.0089893469241797, -0.11086489695181866, -3.7527640949141428,
9
+ ],
10
+ rotation: [
11
+ [0.876134201218856, 0.06925962026449776, 0.47706599800804744],
12
+ [-0.04747421839895102, 0.9972110940209488, -0.057586739349882114],
13
+ [-0.4797239414934443, 0.027805376500959853, 0.8769787916452908],
14
+ ],
15
+ fy: 1164.6601287484507,
16
+ fx: 1159.5880733038064,
17
+ },
18
+ {
19
+ id: 1,
20
+ img_name: "00009",
21
+ width: 1959,
22
+ height: 1090,
23
+ position: [
24
+ -2.5199776022057296, -0.09704735754873686, -3.6247725540304545,
25
+ ],
26
+ rotation: [
27
+ [0.9982731285632193, -0.011928707708098955, -0.05751927260507243],
28
+ [0.0065061360949636325, 0.9955928229282383, -0.09355533724430458],
29
+ [0.058381769258182864, 0.09301955098900708, 0.9939511719154457],
30
+ ],
31
+ fy: 1164.6601287484507,
32
+ fx: 1159.5880733038064,
33
+ },
34
+ {
35
+ id: 2,
36
+ img_name: "00017",
37
+ width: 1959,
38
+ height: 1090,
39
+ position: [
40
+ -0.7737533667465242, -0.3364271945329695, -2.9358969417573753,
41
+ ],
42
+ rotation: [
43
+ [0.9998813418672372, 0.013742375651625236, -0.0069605529394208224],
44
+ [-0.014268370388586709, 0.996512943252834, -0.08220929105659476],
45
+ [0.00580653013657589, 0.08229885200307129, 0.9965907801935302],
46
+ ],
47
+ fy: 1164.6601287484507,
48
+ fx: 1159.5880733038064,
49
+ },
50
+ {
51
+ id: 3,
52
+ img_name: "00025",
53
+ width: 1959,
54
+ height: 1090,
55
+ position: [
56
+ 1.2198221749590001, -0.2196687861401182, -2.3183162007028453,
57
+ ],
58
+ rotation: [
59
+ [0.9208648867765482, 0.0012010625395201253, 0.389880004297208],
60
+ [-0.06298204172269357, 0.987319521752825, 0.14571693239364383],
61
+ [-0.3847611242348369, -0.1587410451475895, 0.9092635249821667],
62
+ ],
63
+ fy: 1164.6601287484507,
64
+ fx: 1159.5880733038064,
65
+ },
66
+ {
67
+ id: 4,
68
+ img_name: "00033",
69
+ width: 1959,
70
+ height: 1090,
71
+ position: [
72
+ 1.742387858893817, -0.13848225198886954, -2.0566370113193146,
73
+ ],
74
+ rotation: [
75
+ [0.24669889292141334, -0.08370189346592856, -0.9654706879349405],
76
+ [0.11343747891376445, 0.9919082664242816, -0.05700815184573074],
77
+ [0.9624300466054861, -0.09545671285663988, 0.2541976029815521],
78
+ ],
79
+ fy: 1164.6601287484507,
80
+ fx: 1159.5880733038064,
81
+ },
82
+ {
83
+ id: 5,
84
+ img_name: "00041",
85
+ width: 1959,
86
+ height: 1090,
87
+ position: [
88
+ 3.6567309419223935, -0.16470990600750707, -1.3458085590422042,
89
+ ],
90
+ rotation: [
91
+ [0.2341293058324528, -0.02968330457755884, -0.9717522161434825],
92
+ [0.10270823606832301, 0.99469554638321, -0.005638106875665722],
93
+ [0.9667649592295676, -0.09848690996657204, 0.2359360976431732],
94
+ ],
95
+ fy: 1164.6601287484507,
96
+ fx: 1159.5880733038064,
97
+ },
98
+ {
99
+ id: 6,
100
+ img_name: "00049",
101
+ width: 1959,
102
+ height: 1090,
103
+ position: [
104
+ 3.9013554243203497, -0.2597500978038105, -0.8106154188297828,
105
+ ],
106
+ rotation: [
107
+ [0.6717235545638952, -0.015718162115524837, -0.7406351366386528],
108
+ [0.055627354673906296, 0.9980224478387622, 0.029270992841185218],
109
+ [0.7387104058127439, -0.060861588786650656, 0.6712695459756353],
110
+ ],
111
+ fy: 1164.6601287484507,
112
+ fx: 1159.5880733038064,
113
+ },
114
+ {
115
+ id: 7,
116
+ img_name: "00057",
117
+ width: 1959,
118
+ height: 1090,
119
+ position: [4.742994605467533, -0.05591660945412069, 0.9500365976084458],
120
+ rotation: [
121
+ [-0.17042655709210375, 0.01207080756938, -0.9852964448542146],
122
+ [0.1165090336695526, 0.9931575292530063, -0.00798543433078162],
123
+ [0.9784581921120181, -0.1161568667478904, -0.1706667764862097],
124
+ ],
125
+ fy: 1164.6601287484507,
126
+ fx: 1159.5880733038064,
127
+ },
128
+ {
129
+ id: 8,
130
+ img_name: "00065",
131
+ width: 1959,
132
+ height: 1090,
133
+ position: [4.34676307626522, 0.08168160516967145, 1.0876221470355405],
134
+ rotation: [
135
+ [-0.003575447631888379, -0.044792503246552894, -0.9989899137764799],
136
+ [0.10770152645126597, 0.9931680875192705, -0.04491693593046672],
137
+ [0.9941768441149182, -0.10775333677534978, 0.0012732004866391048],
138
+ ],
139
+ fy: 1164.6601287484507,
140
+ fx: 1159.5880733038064,
141
+ },
142
+ {
143
+ id: 9,
144
+ img_name: "00073",
145
+ width: 1959,
146
+ height: 1090,
147
+ position: [3.264984351114202, 0.078974937336732, 1.0117200284114904],
148
+ rotation: [
149
+ [-0.026919994628162257, -0.1565891128261527, -0.9872968974090509],
150
+ [0.08444552208239385, 0.983768234577625, -0.1583319754069128],
151
+ [0.9960643893290491, -0.0876350978794554, -0.013259786205163005],
152
+ ],
153
+ fy: 1164.6601287484507,
154
+ fx: 1159.5880733038064,
155
+ },
156
+ ];
157
+
158
+ let camera = cameras[0];
159
+
160
+ // idol: the camera of the contract renders (idol/scripts/render_views.py): it looks at the origin from distance 2.2,
161
+ // 20 degrees above the object, image up = -y (idol's PLY is y-down), field of view 40 degrees; the orbit turns it
162
+ // about the vertical axis. Azimuth 90 degrees faces the object's front (scripts/make_hero_app.py).
163
+ const IDOL_DISTANCE = 2.2;
164
+ const IDOL_ELEVATION = (20 * Math.PI) / 180;
165
+ const IDOL_FOV = (40 * Math.PI) / 180;
166
+ const IDOL_START_AZIMUTH = Math.PI / 2;
167
+ const IDOL_TURN_MS = 12000;
168
+
169
+ // idol: world -> camera (rows x, y, z = right, down, forward; t = -R c), column-major like getViewMatrix's result
170
+ function idolViewMatrix(azimuth) {
171
+ const ce = Math.cos(IDOL_ELEVATION);
172
+ const se = Math.sin(IDOL_ELEVATION);
173
+ const c = [
174
+ IDOL_DISTANCE * ce * Math.sin(azimuth),
175
+ -IDOL_DISTANCE * se,
176
+ IDOL_DISTANCE * ce * Math.cos(azimuth),
177
+ ];
178
+ const n = Math.hypot(c[0], c[1], c[2]);
179
+ const z = [-c[0] / n, -c[1] / n, -c[2] / n];
180
+ const up = [0, -1, 0];
181
+ let x = [
182
+ z[1] * up[2] - z[2] * up[1],
183
+ z[2] * up[0] - z[0] * up[2],
184
+ z[0] * up[1] - z[1] * up[0],
185
+ ];
186
+ const xn = Math.hypot(x[0], x[1], x[2]);
187
+ x = [x[0] / xn, x[1] / xn, x[2] / xn];
188
+ const y = [
189
+ z[1] * x[2] - z[2] * x[1],
190
+ z[2] * x[0] - z[0] * x[2],
191
+ z[0] * x[1] - z[1] * x[0],
192
+ ];
193
+ const t = [
194
+ -(x[0] * c[0] + x[1] * c[1] + x[2] * c[2]),
195
+ -(y[0] * c[0] + y[1] * c[1] + y[2] * c[2]),
196
+ -(z[0] * c[0] + z[1] * c[1] + z[2] * c[2]),
197
+ ];
198
+ return [x[0], y[0], z[0], 0, x[1], y[1], z[1], 0, x[2], y[2], z[2], 0, t[0], t[1], t[2], 1];
199
+ }
200
+
201
+ function getProjectionMatrix(fx, fy, width, height) {
202
+ const znear = 0.2;
203
+ const zfar = 200;
204
+ return [
205
+ [(2 * fx) / width, 0, 0, 0],
206
+ [0, -(2 * fy) / height, 0, 0],
207
+ [0, 0, zfar / (zfar - znear), 1],
208
+ [0, 0, -(zfar * znear) / (zfar - znear), 0],
209
+ ].flat();
210
+ }
211
+
212
+ function getViewMatrix(camera) {
213
+ const R = camera.rotation.flat();
214
+ const t = camera.position;
215
+ const camToWorld = [
216
+ [R[0], R[1], R[2], 0],
217
+ [R[3], R[4], R[5], 0],
218
+ [R[6], R[7], R[8], 0],
219
+ [
220
+ -t[0] * R[0] - t[1] * R[3] - t[2] * R[6],
221
+ -t[0] * R[1] - t[1] * R[4] - t[2] * R[7],
222
+ -t[0] * R[2] - t[1] * R[5] - t[2] * R[8],
223
+ 1,
224
+ ],
225
+ ].flat();
226
+ return camToWorld;
227
+ }
228
+ // function translate4(a, x, y, z) {
229
+ // return [
230
+ // ...a.slice(0, 12),
231
+ // a[0] * x + a[4] * y + a[8] * z + a[12],
232
+ // a[1] * x + a[5] * y + a[9] * z + a[13],
233
+ // a[2] * x + a[6] * y + a[10] * z + a[14],
234
+ // a[3] * x + a[7] * y + a[11] * z + a[15],
235
+ // ];
236
+ // }
237
+
238
+ function multiply4(a, b) {
239
+ return [
240
+ b[0] * a[0] + b[1] * a[4] + b[2] * a[8] + b[3] * a[12],
241
+ b[0] * a[1] + b[1] * a[5] + b[2] * a[9] + b[3] * a[13],
242
+ b[0] * a[2] + b[1] * a[6] + b[2] * a[10] + b[3] * a[14],
243
+ b[0] * a[3] + b[1] * a[7] + b[2] * a[11] + b[3] * a[15],
244
+ b[4] * a[0] + b[5] * a[4] + b[6] * a[8] + b[7] * a[12],
245
+ b[4] * a[1] + b[5] * a[5] + b[6] * a[9] + b[7] * a[13],
246
+ b[4] * a[2] + b[5] * a[6] + b[6] * a[10] + b[7] * a[14],
247
+ b[4] * a[3] + b[5] * a[7] + b[6] * a[11] + b[7] * a[15],
248
+ b[8] * a[0] + b[9] * a[4] + b[10] * a[8] + b[11] * a[12],
249
+ b[8] * a[1] + b[9] * a[5] + b[10] * a[9] + b[11] * a[13],
250
+ b[8] * a[2] + b[9] * a[6] + b[10] * a[10] + b[11] * a[14],
251
+ b[8] * a[3] + b[9] * a[7] + b[10] * a[11] + b[11] * a[15],
252
+ b[12] * a[0] + b[13] * a[4] + b[14] * a[8] + b[15] * a[12],
253
+ b[12] * a[1] + b[13] * a[5] + b[14] * a[9] + b[15] * a[13],
254
+ b[12] * a[2] + b[13] * a[6] + b[14] * a[10] + b[15] * a[14],
255
+ b[12] * a[3] + b[13] * a[7] + b[14] * a[11] + b[15] * a[15],
256
+ ];
257
+ }
258
+
259
+ function invert4(a) {
260
+ let b00 = a[0] * a[5] - a[1] * a[4];
261
+ let b01 = a[0] * a[6] - a[2] * a[4];
262
+ let b02 = a[0] * a[7] - a[3] * a[4];
263
+ let b03 = a[1] * a[6] - a[2] * a[5];
264
+ let b04 = a[1] * a[7] - a[3] * a[5];
265
+ let b05 = a[2] * a[7] - a[3] * a[6];
266
+ let b06 = a[8] * a[13] - a[9] * a[12];
267
+ let b07 = a[8] * a[14] - a[10] * a[12];
268
+ let b08 = a[8] * a[15] - a[11] * a[12];
269
+ let b09 = a[9] * a[14] - a[10] * a[13];
270
+ let b10 = a[9] * a[15] - a[11] * a[13];
271
+ let b11 = a[10] * a[15] - a[11] * a[14];
272
+ let det =
273
+ b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;
274
+ if (!det) return null;
275
+ return [
276
+ (a[5] * b11 - a[6] * b10 + a[7] * b09) / det,
277
+ (a[2] * b10 - a[1] * b11 - a[3] * b09) / det,
278
+ (a[13] * b05 - a[14] * b04 + a[15] * b03) / det,
279
+ (a[10] * b04 - a[9] * b05 - a[11] * b03) / det,
280
+ (a[6] * b08 - a[4] * b11 - a[7] * b07) / det,
281
+ (a[0] * b11 - a[2] * b08 + a[3] * b07) / det,
282
+ (a[14] * b02 - a[12] * b05 - a[15] * b01) / det,
283
+ (a[8] * b05 - a[10] * b02 + a[11] * b01) / det,
284
+ (a[4] * b10 - a[5] * b08 + a[7] * b06) / det,
285
+ (a[1] * b08 - a[0] * b10 - a[3] * b06) / det,
286
+ (a[12] * b04 - a[13] * b02 + a[15] * b00) / det,
287
+ (a[9] * b02 - a[8] * b04 - a[11] * b00) / det,
288
+ (a[5] * b07 - a[4] * b09 - a[6] * b06) / det,
289
+ (a[0] * b09 - a[1] * b07 + a[2] * b06) / det,
290
+ (a[13] * b01 - a[12] * b03 - a[14] * b00) / det,
291
+ (a[8] * b03 - a[9] * b01 + a[10] * b00) / det,
292
+ ];
293
+ }
294
+
295
+ function rotate4(a, rad, x, y, z) {
296
+ let len = Math.hypot(x, y, z);
297
+ x /= len;
298
+ y /= len;
299
+ z /= len;
300
+ let s = Math.sin(rad);
301
+ let c = Math.cos(rad);
302
+ let t = 1 - c;
303
+ let b00 = x * x * t + c;
304
+ let b01 = y * x * t + z * s;
305
+ let b02 = z * x * t - y * s;
306
+ let b10 = x * y * t - z * s;
307
+ let b11 = y * y * t + c;
308
+ let b12 = z * y * t + x * s;
309
+ let b20 = x * z * t + y * s;
310
+ let b21 = y * z * t - x * s;
311
+ let b22 = z * z * t + c;
312
+ return [
313
+ a[0] * b00 + a[4] * b01 + a[8] * b02,
314
+ a[1] * b00 + a[5] * b01 + a[9] * b02,
315
+ a[2] * b00 + a[6] * b01 + a[10] * b02,
316
+ a[3] * b00 + a[7] * b01 + a[11] * b02,
317
+ a[0] * b10 + a[4] * b11 + a[8] * b12,
318
+ a[1] * b10 + a[5] * b11 + a[9] * b12,
319
+ a[2] * b10 + a[6] * b11 + a[10] * b12,
320
+ a[3] * b10 + a[7] * b11 + a[11] * b12,
321
+ a[0] * b20 + a[4] * b21 + a[8] * b22,
322
+ a[1] * b20 + a[5] * b21 + a[9] * b22,
323
+ a[2] * b20 + a[6] * b21 + a[10] * b22,
324
+ a[3] * b20 + a[7] * b21 + a[11] * b22,
325
+ ...a.slice(12, 16),
326
+ ];
327
+ }
328
+
329
+ function translate4(a, x, y, z) {
330
+ return [
331
+ ...a.slice(0, 12),
332
+ a[0] * x + a[4] * y + a[8] * z + a[12],
333
+ a[1] * x + a[5] * y + a[9] * z + a[13],
334
+ a[2] * x + a[6] * y + a[10] * z + a[14],
335
+ a[3] * x + a[7] * y + a[11] * z + a[15],
336
+ ];
337
+ }
338
+
339
+ function createWorker(self) {
340
+ let buffer;
341
+ let vertexCount = 0;
342
+ let viewProj;
343
+ // 6*4 + 4 + 4 = 8*4
344
+ // XYZ - Position (Float32)
345
+ // XYZ - Scale (Float32)
346
+ // RGBA - colors (uint8)
347
+ // IJKL - quaternion/rot (uint8)
348
+ const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
349
+ let lastProj = [];
350
+ let depthIndex = new Uint32Array();
351
+ let lastVertexCount = 0;
352
+
353
+ var _floatView = new Float32Array(1);
354
+ var _int32View = new Int32Array(_floatView.buffer);
355
+
356
+ function floatToHalf(float) {
357
+ _floatView[0] = float;
358
+ var f = _int32View[0];
359
+
360
+ var sign = (f >> 31) & 0x0001;
361
+ var exp = (f >> 23) & 0x00ff;
362
+ var frac = f & 0x007fffff;
363
+
364
+ var newExp;
365
+ if (exp == 0) {
366
+ newExp = 0;
367
+ } else if (exp < 113) {
368
+ newExp = 0;
369
+ frac |= 0x00800000;
370
+ frac = frac >> (113 - exp);
371
+ if (frac & 0x01000000) {
372
+ newExp = 1;
373
+ frac = 0;
374
+ }
375
+ } else if (exp < 142) {
376
+ newExp = exp - 112;
377
+ } else {
378
+ newExp = 31;
379
+ frac = 0;
380
+ }
381
+
382
+ return (sign << 15) | (newExp << 10) | (frac >> 13);
383
+ }
384
+
385
+ function packHalf2x16(x, y) {
386
+ return (floatToHalf(x) | (floatToHalf(y) << 16)) >>> 0;
387
+ }
388
+
389
+ function generateTexture() {
390
+ if (!buffer) return;
391
+ const f_buffer = new Float32Array(buffer);
392
+ const u_buffer = new Uint8Array(buffer);
393
+
394
+ var texwidth = 1024 * 2; // Set to your desired width
395
+ var texheight = Math.ceil((2 * vertexCount) / texwidth); // Set to your desired height
396
+ var texdata = new Uint32Array(texwidth * texheight * 4); // 4 components per pixel (RGBA)
397
+ var texdata_c = new Uint8Array(texdata.buffer);
398
+ var texdata_f = new Float32Array(texdata.buffer);
399
+
400
+ // Here we convert from a .splat file buffer into a texture
401
+ // With a little bit more foresight perhaps this texture file
402
+ // should have been the native format as it'd be very easy to
403
+ // load it into webgl.
404
+ for (let i = 0; i < vertexCount; i++) {
405
+ // x, y, z
406
+ texdata_f[8 * i + 0] = f_buffer[8 * i + 0];
407
+ texdata_f[8 * i + 1] = f_buffer[8 * i + 1];
408
+ texdata_f[8 * i + 2] = f_buffer[8 * i + 2];
409
+
410
+ // r, g, b, a
411
+ texdata_c[4 * (8 * i + 7) + 0] = u_buffer[32 * i + 24 + 0];
412
+ texdata_c[4 * (8 * i + 7) + 1] = u_buffer[32 * i + 24 + 1];
413
+ texdata_c[4 * (8 * i + 7) + 2] = u_buffer[32 * i + 24 + 2];
414
+ texdata_c[4 * (8 * i + 7) + 3] = u_buffer[32 * i + 24 + 3];
415
+
416
+ // quaternions
417
+ let scale = [
418
+ f_buffer[8 * i + 3 + 0],
419
+ f_buffer[8 * i + 3 + 1],
420
+ f_buffer[8 * i + 3 + 2],
421
+ ];
422
+ let rot = [
423
+ (u_buffer[32 * i + 28 + 0] - 128) / 128,
424
+ (u_buffer[32 * i + 28 + 1] - 128) / 128,
425
+ (u_buffer[32 * i + 28 + 2] - 128) / 128,
426
+ (u_buffer[32 * i + 28 + 3] - 128) / 128,
427
+ ];
428
+
429
+ // Compute the matrix product of S and R (M = S * R)
430
+ const M = [
431
+ 1.0 - 2.0 * (rot[2] * rot[2] + rot[3] * rot[3]),
432
+ 2.0 * (rot[1] * rot[2] + rot[0] * rot[3]),
433
+ 2.0 * (rot[1] * rot[3] - rot[0] * rot[2]),
434
+
435
+ 2.0 * (rot[1] * rot[2] - rot[0] * rot[3]),
436
+ 1.0 - 2.0 * (rot[1] * rot[1] + rot[3] * rot[3]),
437
+ 2.0 * (rot[2] * rot[3] + rot[0] * rot[1]),
438
+
439
+ 2.0 * (rot[1] * rot[3] + rot[0] * rot[2]),
440
+ 2.0 * (rot[2] * rot[3] - rot[0] * rot[1]),
441
+ 1.0 - 2.0 * (rot[1] * rot[1] + rot[2] * rot[2]),
442
+ ].map((k, i) => k * scale[Math.floor(i / 3)]);
443
+
444
+ const sigma = [
445
+ M[0] * M[0] + M[3] * M[3] + M[6] * M[6],
446
+ M[0] * M[1] + M[3] * M[4] + M[6] * M[7],
447
+ M[0] * M[2] + M[3] * M[5] + M[6] * M[8],
448
+ M[1] * M[1] + M[4] * M[4] + M[7] * M[7],
449
+ M[1] * M[2] + M[4] * M[5] + M[7] * M[8],
450
+ M[2] * M[2] + M[5] * M[5] + M[8] * M[8],
451
+ ];
452
+
453
+ texdata[8 * i + 4] = packHalf2x16(4 * sigma[0], 4 * sigma[1]);
454
+ texdata[8 * i + 5] = packHalf2x16(4 * sigma[2], 4 * sigma[3]);
455
+ texdata[8 * i + 6] = packHalf2x16(4 * sigma[4], 4 * sigma[5]);
456
+ }
457
+
458
+ self.postMessage({ texdata, texwidth, texheight }, [texdata.buffer]);
459
+ }
460
+
461
+ function runSort(viewProj) {
462
+ if (!buffer) return;
463
+ const f_buffer = new Float32Array(buffer);
464
+ if (lastVertexCount == vertexCount) {
465
+ let dot =
466
+ lastProj[2] * viewProj[2] +
467
+ lastProj[6] * viewProj[6] +
468
+ lastProj[10] * viewProj[10];
469
+ if (Math.abs(dot - 1) < 0.01) {
470
+ return;
471
+ }
472
+ } else {
473
+ generateTexture();
474
+ lastVertexCount = vertexCount;
475
+ }
476
+
477
+ console.time("sort");
478
+ let maxDepth = -Infinity;
479
+ let minDepth = Infinity;
480
+ let sizeList = new Int32Array(vertexCount);
481
+ for (let i = 0; i < vertexCount; i++) {
482
+ let depth =
483
+ ((viewProj[2] * f_buffer[8 * i + 0] +
484
+ viewProj[6] * f_buffer[8 * i + 1] +
485
+ viewProj[10] * f_buffer[8 * i + 2]) *
486
+ 4096) |
487
+ 0;
488
+ sizeList[i] = depth;
489
+ if (depth > maxDepth) maxDepth = depth;
490
+ if (depth < minDepth) minDepth = depth;
491
+ }
492
+
493
+ // This is a 16 bit single-pass counting sort
494
+ let depthInv = (256 * 256 - 1) / (maxDepth - minDepth);
495
+ let counts0 = new Uint32Array(256 * 256);
496
+ for (let i = 0; i < vertexCount; i++) {
497
+ sizeList[i] = ((sizeList[i] - minDepth) * depthInv) | 0;
498
+ counts0[sizeList[i]]++;
499
+ }
500
+ let starts0 = new Uint32Array(256 * 256);
501
+ for (let i = 1; i < 256 * 256; i++)
502
+ starts0[i] = starts0[i - 1] + counts0[i - 1];
503
+ depthIndex = new Uint32Array(vertexCount);
504
+ for (let i = 0; i < vertexCount; i++)
505
+ depthIndex[starts0[sizeList[i]]++] = i;
506
+
507
+ console.timeEnd("sort");
508
+
509
+ lastProj = viewProj;
510
+ self.postMessage({ depthIndex, viewProj, vertexCount }, [
511
+ depthIndex.buffer,
512
+ ]);
513
+ }
514
+
515
+ function processPlyBuffer(inputBuffer) {
516
+ const ubuf = new Uint8Array(inputBuffer);
517
+ // 10KB ought to be enough for a header...
518
+ const header = new TextDecoder().decode(ubuf.slice(0, 1024 * 10));
519
+ const header_end = "end_header\n";
520
+ const header_end_index = header.indexOf(header_end);
521
+ if (header_end_index < 0)
522
+ throw new Error("Unable to read .ply file header");
523
+ const vertexCount = parseInt(/element vertex (\d+)\n/.exec(header)[1]);
524
+ console.log("Vertex Count", vertexCount);
525
+ let row_offset = 0,
526
+ offsets = {},
527
+ types = {};
528
+ const TYPE_MAP = {
529
+ double: "getFloat64",
530
+ int: "getInt32",
531
+ uint: "getUint32",
532
+ float: "getFloat32",
533
+ short: "getInt16",
534
+ ushort: "getUint16",
535
+ uchar: "getUint8",
536
+ };
537
+ for (let prop of header
538
+ .slice(0, header_end_index)
539
+ .split("\n")
540
+ .filter((k) => k.startsWith("property "))) {
541
+ const [p, type, name] = prop.split(" ");
542
+ const arrayType = TYPE_MAP[type] || "getInt8";
543
+ types[name] = arrayType;
544
+ offsets[name] = row_offset;
545
+ row_offset += parseInt(arrayType.replace(/[^\d]/g, "")) / 8;
546
+ }
547
+ console.log("Bytes per row", row_offset, types, offsets);
548
+
549
+ let dataView = new DataView(
550
+ inputBuffer,
551
+ header_end_index + header_end.length,
552
+ );
553
+ let row = 0;
554
+ const attrs = new Proxy(
555
+ {},
556
+ {
557
+ get(target, prop) {
558
+ if (!types[prop]) throw new Error(prop + " not found");
559
+ return dataView[types[prop]](
560
+ row * row_offset + offsets[prop],
561
+ true,
562
+ );
563
+ },
564
+ },
565
+ );
566
+
567
+ console.time("calculate importance");
568
+ let sizeList = new Float32Array(vertexCount);
569
+ let sizeIndex = new Uint32Array(vertexCount);
570
+ for (row = 0; row < vertexCount; row++) {
571
+ sizeIndex[row] = row;
572
+ if (!types["scale_0"]) continue;
573
+ const size =
574
+ Math.exp(attrs.scale_0) *
575
+ Math.exp(attrs.scale_1) *
576
+ Math.exp(attrs.scale_2);
577
+ const opacity = 1 / (1 + Math.exp(-attrs.opacity));
578
+ sizeList[row] = size * opacity;
579
+ }
580
+ console.timeEnd("calculate importance");
581
+
582
+ console.time("sort");
583
+ sizeIndex.sort((b, a) => sizeList[a] - sizeList[b]);
584
+ console.timeEnd("sort");
585
+
586
+ // 6*4 + 4 + 4 = 8*4
587
+ // XYZ - Position (Float32)
588
+ // XYZ - Scale (Float32)
589
+ // RGBA - colors (uint8)
590
+ // IJKL - quaternion/rot (uint8)
591
+ const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
592
+ const buffer = new ArrayBuffer(rowLength * vertexCount);
593
+
594
+ console.time("build buffer");
595
+ for (let j = 0; j < vertexCount; j++) {
596
+ row = sizeIndex[j];
597
+
598
+ const position = new Float32Array(buffer, j * rowLength, 3);
599
+ const scales = new Float32Array(buffer, j * rowLength + 4 * 3, 3);
600
+ const rgba = new Uint8ClampedArray(
601
+ buffer,
602
+ j * rowLength + 4 * 3 + 4 * 3,
603
+ 4,
604
+ );
605
+ const rot = new Uint8ClampedArray(
606
+ buffer,
607
+ j * rowLength + 4 * 3 + 4 * 3 + 4,
608
+ 4,
609
+ );
610
+
611
+ if (types["scale_0"]) {
612
+ const qlen = Math.sqrt(
613
+ attrs.rot_0 ** 2 +
614
+ attrs.rot_1 ** 2 +
615
+ attrs.rot_2 ** 2 +
616
+ attrs.rot_3 ** 2,
617
+ );
618
+
619
+ rot[0] = (attrs.rot_0 / qlen) * 128 + 128;
620
+ rot[1] = (attrs.rot_1 / qlen) * 128 + 128;
621
+ rot[2] = (attrs.rot_2 / qlen) * 128 + 128;
622
+ rot[3] = (attrs.rot_3 / qlen) * 128 + 128;
623
+
624
+ scales[0] = Math.exp(attrs.scale_0);
625
+ scales[1] = Math.exp(attrs.scale_1);
626
+ scales[2] = Math.exp(attrs.scale_2);
627
+ } else {
628
+ scales[0] = 0.01;
629
+ scales[1] = 0.01;
630
+ scales[2] = 0.01;
631
+
632
+ rot[0] = 255;
633
+ rot[1] = 0;
634
+ rot[2] = 0;
635
+ rot[3] = 0;
636
+ }
637
+
638
+ position[0] = attrs.x;
639
+ position[1] = attrs.y;
640
+ position[2] = attrs.z;
641
+
642
+ if (types["f_dc_0"]) {
643
+ const SH_C0 = 0.28209479177387814;
644
+ rgba[0] = (0.5 + SH_C0 * attrs.f_dc_0) * 255;
645
+ rgba[1] = (0.5 + SH_C0 * attrs.f_dc_1) * 255;
646
+ rgba[2] = (0.5 + SH_C0 * attrs.f_dc_2) * 255;
647
+ } else {
648
+ rgba[0] = attrs.red;
649
+ rgba[1] = attrs.green;
650
+ rgba[2] = attrs.blue;
651
+ }
652
+ if (types["opacity"]) {
653
+ rgba[3] = (1 / (1 + Math.exp(-attrs.opacity))) * 255;
654
+ } else {
655
+ rgba[3] = 255;
656
+ }
657
+ }
658
+ console.timeEnd("build buffer");
659
+ return buffer;
660
+ }
661
+
662
+ const throttledSort = () => {
663
+ if (!sortRunning) {
664
+ sortRunning = true;
665
+ let lastView = viewProj;
666
+ runSort(lastView);
667
+ setTimeout(() => {
668
+ sortRunning = false;
669
+ if (lastView !== viewProj) {
670
+ throttledSort();
671
+ }
672
+ }, 0);
673
+ }
674
+ };
675
+
676
+ let sortRunning;
677
+ self.onmessage = (e) => {
678
+ if (e.data.ply) {
679
+ vertexCount = 0;
680
+ runSort(viewProj);
681
+ buffer = processPlyBuffer(e.data.ply);
682
+ vertexCount = Math.floor(buffer.byteLength / rowLength);
683
+ postMessage({ buffer: buffer, save: !!e.data.save });
684
+ } else if (e.data.buffer) {
685
+ buffer = e.data.buffer;
686
+ vertexCount = e.data.vertexCount;
687
+ } else if (e.data.vertexCount) {
688
+ vertexCount = e.data.vertexCount;
689
+ } else if (e.data.view) {
690
+ viewProj = e.data.view;
691
+ throttledSort();
692
+ }
693
+ };
694
+ }
695
+
696
+ const vertexShaderSource = `
697
+ #version 300 es
698
+ precision highp float;
699
+ precision highp int;
700
+
701
+ uniform highp usampler2D u_texture;
702
+ uniform mat4 projection, view;
703
+ uniform vec2 focal;
704
+ uniform vec2 viewport;
705
+
706
+ in vec2 position;
707
+ in int index;
708
+
709
+ out vec4 vColor;
710
+ out vec2 vPosition;
711
+
712
+ void main () {
713
+ uvec4 cen = texelFetch(u_texture, ivec2((uint(index) & 0x3ffu) << 1, uint(index) >> 10), 0);
714
+ vec4 cam = view * vec4(uintBitsToFloat(cen.xyz), 1);
715
+ vec4 pos2d = projection * cam;
716
+
717
+ float clip = 1.2 * pos2d.w;
718
+ if (pos2d.z < -clip || pos2d.x < -clip || pos2d.x > clip || pos2d.y < -clip || pos2d.y > clip) {
719
+ gl_Position = vec4(0.0, 0.0, 2.0, 1.0);
720
+ return;
721
+ }
722
+
723
+ uvec4 cov = texelFetch(u_texture, ivec2(((uint(index) & 0x3ffu) << 1) | 1u, uint(index) >> 10), 0);
724
+ vec2 u1 = unpackHalf2x16(cov.x), u2 = unpackHalf2x16(cov.y), u3 = unpackHalf2x16(cov.z);
725
+ mat3 Vrk = mat3(u1.x, u1.y, u2.x, u1.y, u2.y, u3.x, u2.x, u3.x, u3.y);
726
+
727
+ mat3 J = mat3(
728
+ focal.x / cam.z, 0., -(focal.x * cam.x) / (cam.z * cam.z),
729
+ 0., -focal.y / cam.z, (focal.y * cam.y) / (cam.z * cam.z),
730
+ 0., 0., 0.
731
+ );
732
+
733
+ mat3 T = transpose(mat3(view)) * J;
734
+ mat3 cov2d = transpose(T) * Vrk * T;
735
+
736
+ float mid = (cov2d[0][0] + cov2d[1][1]) / 2.0;
737
+ float radius = length(vec2((cov2d[0][0] - cov2d[1][1]) / 2.0, cov2d[0][1]));
738
+ float lambda1 = mid + radius, lambda2 = mid - radius;
739
+
740
+ if(lambda2 < 0.0) return;
741
+ vec2 diagonalVector = normalize(vec2(cov2d[0][1], lambda1 - cov2d[0][0]));
742
+ vec2 majorAxis = min(sqrt(2.0 * lambda1), 1024.0) * diagonalVector;
743
+ vec2 minorAxis = min(sqrt(2.0 * lambda2), 1024.0) * vec2(diagonalVector.y, -diagonalVector.x);
744
+
745
+ vColor = clamp(pos2d.z/pos2d.w+1.0, 0.0, 1.0) * vec4((cov.w) & 0xffu, (cov.w >> 8) & 0xffu, (cov.w >> 16) & 0xffu, (cov.w >> 24) & 0xffu) / 255.0;
746
+ vPosition = position;
747
+
748
+ vec2 vCenter = vec2(pos2d) / pos2d.w;
749
+ gl_Position = vec4(
750
+ vCenter
751
+ + position.x * majorAxis / viewport
752
+ + position.y * minorAxis / viewport, 0.0, 1.0);
753
+
754
+ }
755
+ `.trim();
756
+
757
+ const fragmentShaderSource = `
758
+ #version 300 es
759
+ precision highp float;
760
+
761
+ in vec4 vColor;
762
+ in vec2 vPosition;
763
+
764
+ out vec4 fragColor;
765
+
766
+ void main () {
767
+ float A = -dot(vPosition, vPosition);
768
+ if (A < -4.0) discard;
769
+ float B = exp(A) * vColor.a;
770
+ fragColor = vec4(B * vColor.rgb, B);
771
+ }
772
+
773
+ `.trim();
774
+
775
+ // idol: start at the contract camera facing the object's front (was a fixed matrix for the demo scenes)
776
+ let defaultViewMatrix = idolViewMatrix(IDOL_START_AZIMUTH);
777
+ let viewMatrix = defaultViewMatrix;
778
+ async function main() {
779
+ let carousel = true;
780
+ const params = new URLSearchParams(location.search);
781
+ try {
782
+ viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
783
+ carousel = false;
784
+ } catch (err) {}
785
+ const url = new URL(
786
+ // "nike.splat",
787
+ // location.href,
788
+ params.get("url") || "train.splat",
789
+ "https://huggingface.co/cakewalk/splat-data/resolve/main/",
790
+ );
791
+ const req = await fetch(url, {
792
+ mode: "cors", // no-cors, *cors, same-origin
793
+ credentials: "omit", // include, *same-origin, omit
794
+ });
795
+ console.log(req);
796
+ if (req.status != 200)
797
+ throw new Error(req.status + " Unable to load " + req.url);
798
+
799
+ const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
800
+ const reader = req.body.getReader();
801
+ let splatData = new Uint8Array(req.headers.get("content-length"));
802
+
803
+ const downsample =
804
+ splatData.length / rowLength > 500000 ? 1 : 1 / devicePixelRatio;
805
+ console.log(splatData.length / rowLength, downsample);
806
+
807
+ const worker = new Worker(
808
+ URL.createObjectURL(
809
+ new Blob(["(", createWorker.toString(), ")(self)"], {
810
+ type: "application/javascript",
811
+ }),
812
+ ),
813
+ );
814
+
815
+ const canvas = document.getElementById("canvas");
816
+ const fps = document.getElementById("fps");
817
+ const camid = document.getElementById("camid");
818
+
819
+ let projectionMatrix;
820
+
821
+ const gl = canvas.getContext("webgl2", {
822
+ antialias: false,
823
+ });
824
+
825
+ const vertexShader = gl.createShader(gl.VERTEX_SHADER);
826
+ gl.shaderSource(vertexShader, vertexShaderSource);
827
+ gl.compileShader(vertexShader);
828
+ if (!gl.getShaderParameter(vertexShader, gl.COMPILE_STATUS))
829
+ console.error(gl.getShaderInfoLog(vertexShader));
830
+
831
+ const fragmentShader = gl.createShader(gl.FRAGMENT_SHADER);
832
+ gl.shaderSource(fragmentShader, fragmentShaderSource);
833
+ gl.compileShader(fragmentShader);
834
+ if (!gl.getShaderParameter(fragmentShader, gl.COMPILE_STATUS))
835
+ console.error(gl.getShaderInfoLog(fragmentShader));
836
+
837
+ const program = gl.createProgram();
838
+ gl.attachShader(program, vertexShader);
839
+ gl.attachShader(program, fragmentShader);
840
+ gl.linkProgram(program);
841
+ gl.useProgram(program);
842
+
843
+ if (!gl.getProgramParameter(program, gl.LINK_STATUS))
844
+ console.error(gl.getProgramInfoLog(program));
845
+
846
+ gl.disable(gl.DEPTH_TEST); // Disable depth testing
847
+
848
+ // Enable blending
849
+ gl.enable(gl.BLEND);
850
+ gl.blendFuncSeparate(
851
+ gl.ONE_MINUS_DST_ALPHA,
852
+ gl.ONE,
853
+ gl.ONE_MINUS_DST_ALPHA,
854
+ gl.ONE,
855
+ );
856
+ gl.blendEquationSeparate(gl.FUNC_ADD, gl.FUNC_ADD);
857
+
858
+ const u_projection = gl.getUniformLocation(program, "projection");
859
+ const u_viewport = gl.getUniformLocation(program, "viewport");
860
+ const u_focal = gl.getUniformLocation(program, "focal");
861
+ const u_view = gl.getUniformLocation(program, "view");
862
+
863
+ // positions
864
+ const triangleVertices = new Float32Array([-2, -2, 2, -2, 2, 2, -2, 2]);
865
+ const vertexBuffer = gl.createBuffer();
866
+ gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
867
+ gl.bufferData(gl.ARRAY_BUFFER, triangleVertices, gl.STATIC_DRAW);
868
+ const a_position = gl.getAttribLocation(program, "position");
869
+ gl.enableVertexAttribArray(a_position);
870
+ gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
871
+ gl.vertexAttribPointer(a_position, 2, gl.FLOAT, false, 0, 0);
872
+
873
+ var texture = gl.createTexture();
874
+ gl.bindTexture(gl.TEXTURE_2D, texture);
875
+
876
+ var u_textureLocation = gl.getUniformLocation(program, "u_texture");
877
+ gl.uniform1i(u_textureLocation, 0);
878
+
879
+ const indexBuffer = gl.createBuffer();
880
+ const a_index = gl.getAttribLocation(program, "index");
881
+ gl.enableVertexAttribArray(a_index);
882
+ gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
883
+ gl.vertexAttribIPointer(a_index, 1, gl.INT, false, 0, 0);
884
+ gl.vertexAttribDivisor(a_index, 1);
885
+
886
+ const resize = () => {
887
+ // idol: focal length for a 40 degree field of view across the shorter side of the screen (the square
888
+ // contract render's framing, whole object in view in portrait and landscape)
889
+ camera.fx = camera.fy =
890
+ Math.min(innerWidth, innerHeight) / 2 / Math.tan(IDOL_FOV / 2);
891
+ gl.uniform2fv(u_focal, new Float32Array([camera.fx, camera.fy]));
892
+
893
+ projectionMatrix = getProjectionMatrix(
894
+ camera.fx,
895
+ camera.fy,
896
+ innerWidth,
897
+ innerHeight,
898
+ );
899
+
900
+ gl.uniform2fv(u_viewport, new Float32Array([innerWidth, innerHeight]));
901
+
902
+ gl.canvas.width = Math.round(innerWidth / downsample);
903
+ gl.canvas.height = Math.round(innerHeight / downsample);
904
+ gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
905
+
906
+ gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
907
+ };
908
+
909
+ window.addEventListener("resize", resize);
910
+ resize();
911
+
912
+ worker.onmessage = (e) => {
913
+ if (e.data.buffer) {
914
+ splatData = new Uint8Array(e.data.buffer);
915
+ if (e.data.save) {
916
+ const blob = new Blob([splatData.buffer], {
917
+ type: "application/octet-stream",
918
+ });
919
+ const link = document.createElement("a");
920
+ link.download = "model.splat";
921
+ link.href = URL.createObjectURL(blob);
922
+ document.body.appendChild(link);
923
+ link.click();
924
+ }
925
+ } else if (e.data.texdata) {
926
+ const { texdata, texwidth, texheight } = e.data;
927
+ // console.log(texdata)
928
+ gl.bindTexture(gl.TEXTURE_2D, texture);
929
+ gl.texParameteri(
930
+ gl.TEXTURE_2D,
931
+ gl.TEXTURE_WRAP_S,
932
+ gl.CLAMP_TO_EDGE,
933
+ );
934
+ gl.texParameteri(
935
+ gl.TEXTURE_2D,
936
+ gl.TEXTURE_WRAP_T,
937
+ gl.CLAMP_TO_EDGE,
938
+ );
939
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
940
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
941
+
942
+ gl.texImage2D(
943
+ gl.TEXTURE_2D,
944
+ 0,
945
+ gl.RGBA32UI,
946
+ texwidth,
947
+ texheight,
948
+ 0,
949
+ gl.RGBA_INTEGER,
950
+ gl.UNSIGNED_INT,
951
+ texdata,
952
+ );
953
+ gl.activeTexture(gl.TEXTURE0);
954
+ gl.bindTexture(gl.TEXTURE_2D, texture);
955
+ } else if (e.data.depthIndex) {
956
+ const { depthIndex, viewProj } = e.data;
957
+ gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
958
+ gl.bufferData(gl.ARRAY_BUFFER, depthIndex, gl.DYNAMIC_DRAW);
959
+ vertexCount = e.data.vertexCount;
960
+ }
961
+ };
962
+
963
+ let activeKeys = [];
964
+ let currentCameraIndex = 0;
965
+
966
+ window.addEventListener("keydown", (e) => {
967
+ // if (document.activeElement != document.body) return;
968
+ carousel = false;
969
+ if (!activeKeys.includes(e.code)) activeKeys.push(e.code);
970
+ if (/\d/.test(e.key)) {
971
+ currentCameraIndex = parseInt(e.key);
972
+ camera = cameras[currentCameraIndex];
973
+ viewMatrix = getViewMatrix(camera);
974
+ }
975
+ if (["-", "_"].includes(e.key)) {
976
+ currentCameraIndex =
977
+ (currentCameraIndex + cameras.length - 1) % cameras.length;
978
+ viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
979
+ }
980
+ if (["+", "="].includes(e.key)) {
981
+ currentCameraIndex = (currentCameraIndex + 1) % cameras.length;
982
+ viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
983
+ }
984
+ camid.innerText = "cam " + currentCameraIndex;
985
+ if (e.code == "KeyV") {
986
+ location.hash =
987
+ "#" +
988
+ JSON.stringify(
989
+ viewMatrix.map((k) => Math.round(k * 100) / 100),
990
+ );
991
+ camid.innerText = "";
992
+ } else if (e.code === "KeyP") {
993
+ carousel = true;
994
+ camid.innerText = "";
995
+ }
996
+ });
997
+ window.addEventListener("keyup", (e) => {
998
+ activeKeys = activeKeys.filter((k) => k !== e.code);
999
+ });
1000
+ window.addEventListener("blur", () => {
1001
+ activeKeys = [];
1002
+ });
1003
+
1004
+ window.addEventListener(
1005
+ "wheel",
1006
+ (e) => {
1007
+ carousel = false;
1008
+ e.preventDefault();
1009
+ const lineHeight = 10;
1010
+ const scale =
1011
+ e.deltaMode == 1
1012
+ ? lineHeight
1013
+ : e.deltaMode == 2
1014
+ ? innerHeight
1015
+ : 1;
1016
+ let inv = invert4(viewMatrix);
1017
+ if (e.shiftKey) {
1018
+ inv = translate4(
1019
+ inv,
1020
+ (e.deltaX * scale) / innerWidth,
1021
+ (e.deltaY * scale) / innerHeight,
1022
+ 0,
1023
+ );
1024
+ } else if (e.ctrlKey || e.metaKey) {
1025
+ // inv = rotate4(inv, (e.deltaX * scale) / innerWidth, 0, 0, 1);
1026
+ // inv = translate4(inv, 0, (e.deltaY * scale) / innerHeight, 0);
1027
+ // let preY = inv[13];
1028
+ inv = translate4(
1029
+ inv,
1030
+ 0,
1031
+ 0,
1032
+ (-10 * (e.deltaY * scale)) / innerHeight,
1033
+ );
1034
+ // inv[13] = preY;
1035
+ } else {
1036
+ let d = 4;
1037
+ inv = translate4(inv, 0, 0, d);
1038
+ inv = rotate4(inv, -(e.deltaX * scale) / innerWidth, 0, 1, 0);
1039
+ inv = rotate4(inv, (e.deltaY * scale) / innerHeight, 1, 0, 0);
1040
+ inv = translate4(inv, 0, 0, -d);
1041
+ }
1042
+
1043
+ viewMatrix = invert4(inv);
1044
+ },
1045
+ { passive: false },
1046
+ );
1047
+
1048
+ let startX, startY, down;
1049
+ canvas.addEventListener("mousedown", (e) => {
1050
+ carousel = false;
1051
+ e.preventDefault();
1052
+ startX = e.clientX;
1053
+ startY = e.clientY;
1054
+ down = e.ctrlKey || e.metaKey ? 2 : 1;
1055
+ });
1056
+ canvas.addEventListener("contextmenu", (e) => {
1057
+ carousel = false;
1058
+ e.preventDefault();
1059
+ startX = e.clientX;
1060
+ startY = e.clientY;
1061
+ down = 2;
1062
+ });
1063
+
1064
+ canvas.addEventListener("mousemove", (e) => {
1065
+ e.preventDefault();
1066
+ if (down == 1) {
1067
+ let inv = invert4(viewMatrix);
1068
+ let dx = (5 * (e.clientX - startX)) / innerWidth;
1069
+ let dy = (5 * (e.clientY - startY)) / innerHeight;
1070
+ let d = 4;
1071
+
1072
+ inv = translate4(inv, 0, 0, d);
1073
+ inv = rotate4(inv, dx, 0, 1, 0);
1074
+ inv = rotate4(inv, -dy, 1, 0, 0);
1075
+ inv = translate4(inv, 0, 0, -d);
1076
+ // let postAngle = Math.atan2(inv[0], inv[10])
1077
+ // inv = rotate4(inv, postAngle - preAngle, 0, 0, 1)
1078
+ // console.log(postAngle)
1079
+ viewMatrix = invert4(inv);
1080
+
1081
+ startX = e.clientX;
1082
+ startY = e.clientY;
1083
+ } else if (down == 2) {
1084
+ let inv = invert4(viewMatrix);
1085
+ // inv = rotateY(inv, );
1086
+ // let preY = inv[13];
1087
+ inv = translate4(
1088
+ inv,
1089
+ (-10 * (e.clientX - startX)) / innerWidth,
1090
+ 0,
1091
+ (10 * (e.clientY - startY)) / innerHeight,
1092
+ );
1093
+ // inv[13] = preY;
1094
+ viewMatrix = invert4(inv);
1095
+
1096
+ startX = e.clientX;
1097
+ startY = e.clientY;
1098
+ }
1099
+ });
1100
+ canvas.addEventListener("mouseup", (e) => {
1101
+ e.preventDefault();
1102
+ down = false;
1103
+ startX = 0;
1104
+ startY = 0;
1105
+ });
1106
+
1107
+ let altX = 0,
1108
+ altY = 0;
1109
+ canvas.addEventListener(
1110
+ "touchstart",
1111
+ (e) => {
1112
+ e.preventDefault();
1113
+ if (e.touches.length === 1) {
1114
+ carousel = false;
1115
+ startX = e.touches[0].clientX;
1116
+ startY = e.touches[0].clientY;
1117
+ down = 1;
1118
+ } else if (e.touches.length === 2) {
1119
+ // console.log('beep')
1120
+ carousel = false;
1121
+ startX = e.touches[0].clientX;
1122
+ altX = e.touches[1].clientX;
1123
+ startY = e.touches[0].clientY;
1124
+ altY = e.touches[1].clientY;
1125
+ down = 1;
1126
+ }
1127
+ },
1128
+ { passive: false },
1129
+ );
1130
+ canvas.addEventListener(
1131
+ "touchmove",
1132
+ (e) => {
1133
+ e.preventDefault();
1134
+ if (e.touches.length === 1 && down) {
1135
+ let inv = invert4(viewMatrix);
1136
+ let dx = (4 * (e.touches[0].clientX - startX)) / innerWidth;
1137
+ let dy = (4 * (e.touches[0].clientY - startY)) / innerHeight;
1138
+
1139
+ // idol: orbit about the object at the origin (was a point 4 units ahead)
1140
+ let d = Math.hypot(inv[12], inv[13], inv[14]);
1141
+ inv = translate4(inv, 0, 0, d);
1142
+ // inv = translate4(inv, -x, -y, -z);
1143
+ // inv = translate4(inv, x, y, z);
1144
+ inv = rotate4(inv, dx, 0, 1, 0);
1145
+ inv = rotate4(inv, -dy, 1, 0, 0);
1146
+ inv = translate4(inv, 0, 0, -d);
1147
+
1148
+ viewMatrix = invert4(inv);
1149
+
1150
+ startX = e.touches[0].clientX;
1151
+ startY = e.touches[0].clientY;
1152
+ } else if (e.touches.length === 2) {
1153
+ // alert('beep')
1154
+ const dtheta =
1155
+ Math.atan2(startY - altY, startX - altX) -
1156
+ Math.atan2(
1157
+ e.touches[0].clientY - e.touches[1].clientY,
1158
+ e.touches[0].clientX - e.touches[1].clientX,
1159
+ );
1160
+ const dscale =
1161
+ Math.hypot(startX - altX, startY - altY) /
1162
+ Math.hypot(
1163
+ e.touches[0].clientX - e.touches[1].clientX,
1164
+ e.touches[0].clientY - e.touches[1].clientY,
1165
+ );
1166
+ const dx =
1167
+ (e.touches[0].clientX +
1168
+ e.touches[1].clientX -
1169
+ (startX + altX)) /
1170
+ 2;
1171
+ const dy =
1172
+ (e.touches[0].clientY +
1173
+ e.touches[1].clientY -
1174
+ (startY + altY)) /
1175
+ 2;
1176
+ let inv = invert4(viewMatrix);
1177
+ // inv = translate4(inv, 0, 0, d);
1178
+ inv = rotate4(inv, dtheta, 0, 0, 1);
1179
+
1180
+ inv = translate4(inv, -dx / innerWidth, -dy / innerHeight, 0);
1181
+
1182
+ // let preY = inv[13];
1183
+ inv = translate4(inv, 0, 0, 3 * (1 - dscale));
1184
+ // inv[13] = preY;
1185
+
1186
+ viewMatrix = invert4(inv);
1187
+
1188
+ startX = e.touches[0].clientX;
1189
+ altX = e.touches[1].clientX;
1190
+ startY = e.touches[0].clientY;
1191
+ altY = e.touches[1].clientY;
1192
+ }
1193
+ },
1194
+ { passive: false },
1195
+ );
1196
+ canvas.addEventListener(
1197
+ "touchend",
1198
+ (e) => {
1199
+ e.preventDefault();
1200
+ down = false;
1201
+ startX = 0;
1202
+ startY = 0;
1203
+ },
1204
+ { passive: false },
1205
+ );
1206
+
1207
+ let jumpDelta = 0;
1208
+ let vertexCount = 0;
1209
+
1210
+ let lastFrame = 0;
1211
+ let avgFps = 0;
1212
+ let start = 0;
1213
+
1214
+ window.addEventListener("gamepadconnected", (e) => {
1215
+ const gp = navigator.getGamepads()[e.gamepad.index];
1216
+ console.log(
1217
+ `Gamepad connected at index ${gp.index}: ${gp.id}. It has ${gp.buttons.length} buttons and ${gp.axes.length} axes.`,
1218
+ );
1219
+ });
1220
+ window.addEventListener("gamepaddisconnected", (e) => {
1221
+ console.log("Gamepad disconnected");
1222
+ });
1223
+
1224
+ let leftGamepadTrigger, rightGamepadTrigger;
1225
+
1226
+ const frame = (now) => {
1227
+ let inv = invert4(viewMatrix);
1228
+ let shiftKey =
1229
+ activeKeys.includes("Shift") ||
1230
+ activeKeys.includes("ShiftLeft") ||
1231
+ activeKeys.includes("ShiftRight");
1232
+
1233
+ if (activeKeys.includes("ArrowUp")) {
1234
+ if (shiftKey) {
1235
+ inv = translate4(inv, 0, -0.03, 0);
1236
+ } else {
1237
+ inv = translate4(inv, 0, 0, 0.1);
1238
+ }
1239
+ }
1240
+ if (activeKeys.includes("ArrowDown")) {
1241
+ if (shiftKey) {
1242
+ inv = translate4(inv, 0, 0.03, 0);
1243
+ } else {
1244
+ inv = translate4(inv, 0, 0, -0.1);
1245
+ }
1246
+ }
1247
+ if (activeKeys.includes("ArrowLeft"))
1248
+ inv = translate4(inv, -0.03, 0, 0);
1249
+ //
1250
+ if (activeKeys.includes("ArrowRight"))
1251
+ inv = translate4(inv, 0.03, 0, 0);
1252
+ // inv = rotate4(inv, 0.01, 0, 1, 0);
1253
+ if (activeKeys.includes("KeyA")) inv = rotate4(inv, -0.01, 0, 1, 0);
1254
+ if (activeKeys.includes("KeyD")) inv = rotate4(inv, 0.01, 0, 1, 0);
1255
+ if (activeKeys.includes("KeyQ")) inv = rotate4(inv, 0.01, 0, 0, 1);
1256
+ if (activeKeys.includes("KeyE")) inv = rotate4(inv, -0.01, 0, 0, 1);
1257
+ if (activeKeys.includes("KeyW")) inv = rotate4(inv, 0.005, 1, 0, 0);
1258
+ if (activeKeys.includes("KeyS")) inv = rotate4(inv, -0.005, 1, 0, 0);
1259
+
1260
+ const gamepads = navigator.getGamepads ? navigator.getGamepads() : [];
1261
+ let isJumping = activeKeys.includes("Space");
1262
+ for (let gamepad of gamepads) {
1263
+ if (!gamepad) continue;
1264
+
1265
+ const axisThreshold = 0.1; // Threshold to detect when the axis is intentionally moved
1266
+ const moveSpeed = 0.06;
1267
+ const rotateSpeed = 0.02;
1268
+
1269
+ // Assuming the left stick controls translation (axes 0 and 1)
1270
+ if (Math.abs(gamepad.axes[0]) > axisThreshold) {
1271
+ inv = translate4(inv, moveSpeed * gamepad.axes[0], 0, 0);
1272
+ carousel = false;
1273
+ }
1274
+ if (Math.abs(gamepad.axes[1]) > axisThreshold) {
1275
+ inv = translate4(inv, 0, 0, -moveSpeed * gamepad.axes[1]);
1276
+ carousel = false;
1277
+ }
1278
+ if (gamepad.buttons[12].pressed || gamepad.buttons[13].pressed) {
1279
+ inv = translate4(
1280
+ inv,
1281
+ 0,
1282
+ -moveSpeed *
1283
+ (gamepad.buttons[12].pressed -
1284
+ gamepad.buttons[13].pressed),
1285
+ 0,
1286
+ );
1287
+ carousel = false;
1288
+ }
1289
+
1290
+ if (gamepad.buttons[14].pressed || gamepad.buttons[15].pressed) {
1291
+ inv = translate4(
1292
+ inv,
1293
+ -moveSpeed *
1294
+ (gamepad.buttons[14].pressed -
1295
+ gamepad.buttons[15].pressed),
1296
+ 0,
1297
+ 0,
1298
+ );
1299
+ carousel = false;
1300
+ }
1301
+
1302
+ // Assuming the right stick controls rotation (axes 2 and 3)
1303
+ if (Math.abs(gamepad.axes[2]) > axisThreshold) {
1304
+ inv = rotate4(inv, rotateSpeed * gamepad.axes[2], 0, 1, 0);
1305
+ carousel = false;
1306
+ }
1307
+ if (Math.abs(gamepad.axes[3]) > axisThreshold) {
1308
+ inv = rotate4(inv, -rotateSpeed * gamepad.axes[3], 1, 0, 0);
1309
+ carousel = false;
1310
+ }
1311
+
1312
+ let tiltAxis = gamepad.buttons[6].value - gamepad.buttons[7].value;
1313
+ if (Math.abs(tiltAxis) > axisThreshold) {
1314
+ inv = rotate4(inv, rotateSpeed * tiltAxis, 0, 0, 1);
1315
+ carousel = false;
1316
+ }
1317
+ if (gamepad.buttons[4].pressed && !leftGamepadTrigger) {
1318
+ camera =
1319
+ cameras[(cameras.indexOf(camera) + 1) % cameras.length];
1320
+ inv = invert4(getViewMatrix(camera));
1321
+ carousel = false;
1322
+ }
1323
+ if (gamepad.buttons[5].pressed && !rightGamepadTrigger) {
1324
+ camera =
1325
+ cameras[
1326
+ (cameras.indexOf(camera) + cameras.length - 1) %
1327
+ cameras.length
1328
+ ];
1329
+ inv = invert4(getViewMatrix(camera));
1330
+ carousel = false;
1331
+ }
1332
+ leftGamepadTrigger = gamepad.buttons[4].pressed;
1333
+ rightGamepadTrigger = gamepad.buttons[5].pressed;
1334
+ if (gamepad.buttons[0].pressed) {
1335
+ isJumping = true;
1336
+ carousel = false;
1337
+ }
1338
+ if (gamepad.buttons[3].pressed) {
1339
+ carousel = true;
1340
+ }
1341
+ }
1342
+
1343
+ if (
1344
+ ["KeyJ", "KeyK", "KeyL", "KeyI"].some((k) => activeKeys.includes(k))
1345
+ ) {
1346
+ let d = 4;
1347
+ inv = translate4(inv, 0, 0, d);
1348
+ inv = rotate4(
1349
+ inv,
1350
+ activeKeys.includes("KeyJ")
1351
+ ? -0.05
1352
+ : activeKeys.includes("KeyL")
1353
+ ? 0.05
1354
+ : 0,
1355
+ 0,
1356
+ 1,
1357
+ 0,
1358
+ );
1359
+ inv = rotate4(
1360
+ inv,
1361
+ activeKeys.includes("KeyI")
1362
+ ? 0.05
1363
+ : activeKeys.includes("KeyK")
1364
+ ? -0.05
1365
+ : 0,
1366
+ 1,
1367
+ 0,
1368
+ 0,
1369
+ );
1370
+ inv = translate4(inv, 0, 0, -d);
1371
+ }
1372
+
1373
+ viewMatrix = invert4(inv);
1374
+
1375
+ if (carousel) {
1376
+ // idol: a full turn about the vertical axis, about 12 s per turn, until the first touch (was a swing
1377
+ // left and right for the demo scenes)
1378
+ const azimuth =
1379
+ IDOL_START_AZIMUTH +
1380
+ (2 * Math.PI * Math.max(0, Date.now() - start)) / IDOL_TURN_MS;
1381
+ viewMatrix = idolViewMatrix(azimuth);
1382
+ }
1383
+
1384
+ if (isJumping) {
1385
+ jumpDelta = Math.min(1, jumpDelta + 0.05);
1386
+ } else {
1387
+ jumpDelta = Math.max(0, jumpDelta - 0.05);
1388
+ }
1389
+
1390
+ let inv2 = invert4(viewMatrix);
1391
+ inv2 = translate4(inv2, 0, -jumpDelta, 0);
1392
+ inv2 = rotate4(inv2, -0.1 * jumpDelta, 1, 0, 0);
1393
+ let actualViewMatrix = invert4(inv2);
1394
+
1395
+ const viewProj = multiply4(projectionMatrix, actualViewMatrix);
1396
+ worker.postMessage({ view: viewProj });
1397
+
1398
+ const currentFps = 1000 / (now - lastFrame) || 0;
1399
+ avgFps = avgFps * 0.9 + currentFps * 0.1;
1400
+
1401
+ if (vertexCount > 0) {
1402
+ document.getElementById("spinner").style.display = "none";
1403
+ gl.uniformMatrix4fv(u_view, false, actualViewMatrix);
1404
+ gl.clear(gl.COLOR_BUFFER_BIT);
1405
+ gl.drawArraysInstanced(gl.TRIANGLE_FAN, 0, 4, vertexCount);
1406
+ } else {
1407
+ gl.clear(gl.COLOR_BUFFER_BIT);
1408
+ document.getElementById("spinner").style.display = "";
1409
+ start = Date.now() + 2000;
1410
+ }
1411
+ const progress = (100 * vertexCount) / (splatData.length / rowLength);
1412
+ if (progress < 100) {
1413
+ document.getElementById("progress").style.width = progress + "%";
1414
+ } else {
1415
+ document.getElementById("progress").style.display = "none";
1416
+ }
1417
+ fps.innerText = Math.round(avgFps) + " fps";
1418
+ if (isNaN(currentCameraIndex)) {
1419
+ camid.innerText = "";
1420
+ }
1421
+ lastFrame = now;
1422
+ requestAnimationFrame(frame);
1423
+ };
1424
+
1425
+ frame();
1426
+
1427
+ const isPly = (splatData) =>
1428
+ splatData[0] == 112 &&
1429
+ splatData[1] == 108 &&
1430
+ splatData[2] == 121 &&
1431
+ splatData[3] == 10;
1432
+
1433
+ const selectFile = (file) => {
1434
+ const fr = new FileReader();
1435
+ if (/\.json$/i.test(file.name)) {
1436
+ fr.onload = () => {
1437
+ cameras = JSON.parse(fr.result);
1438
+ viewMatrix = getViewMatrix(cameras[0]);
1439
+ projectionMatrix = getProjectionMatrix(
1440
+ camera.fx / downsample,
1441
+ camera.fy / downsample,
1442
+ canvas.width,
1443
+ canvas.height,
1444
+ );
1445
+ gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
1446
+
1447
+ console.log("Loaded Cameras");
1448
+ };
1449
+ fr.readAsText(file);
1450
+ } else {
1451
+ stopLoading = true;
1452
+ fr.onload = () => {
1453
+ splatData = new Uint8Array(fr.result);
1454
+ console.log("Loaded", Math.floor(splatData.length / rowLength));
1455
+
1456
+ if (isPly(splatData)) {
1457
+ // ply file magic header means it should be handled differently
1458
+ worker.postMessage({ ply: splatData.buffer, save: true });
1459
+ } else {
1460
+ worker.postMessage({
1461
+ buffer: splatData.buffer,
1462
+ vertexCount: Math.floor(splatData.length / rowLength),
1463
+ });
1464
+ }
1465
+ };
1466
+ fr.readAsArrayBuffer(file);
1467
+ }
1468
+ };
1469
+
1470
+ window.addEventListener("hashchange", (e) => {
1471
+ try {
1472
+ viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
1473
+ carousel = false;
1474
+ } catch (err) {}
1475
+ });
1476
+
1477
+ const preventDefault = (e) => {
1478
+ e.preventDefault();
1479
+ e.stopPropagation();
1480
+ };
1481
+ document.addEventListener("dragenter", preventDefault);
1482
+ document.addEventListener("dragover", preventDefault);
1483
+ document.addEventListener("dragleave", preventDefault);
1484
+ document.addEventListener("drop", (e) => {
1485
+ e.preventDefault();
1486
+ e.stopPropagation();
1487
+ selectFile(e.dataTransfer.files[0]);
1488
+ });
1489
+
1490
+ let bytesRead = 0;
1491
+ let lastVertexCount = -1;
1492
+ let stopLoading = false;
1493
+
1494
+ while (true) {
1495
+ const { done, value } = await reader.read();
1496
+ if (done || stopLoading) break;
1497
+
1498
+ splatData.set(value, bytesRead);
1499
+ bytesRead += value.length;
1500
+
1501
+ if (vertexCount > lastVertexCount) {
1502
+ if (!isPly(splatData)) {
1503
+ worker.postMessage({
1504
+ buffer: splatData.buffer,
1505
+ vertexCount: Math.floor(bytesRead / rowLength),
1506
+ });
1507
+ }
1508
+ lastVertexCount = vertexCount;
1509
+ }
1510
+ }
1511
+ if (!stopLoading) {
1512
+ if (isPly(splatData)) {
1513
+ // ply file magic header means it should be handled differently
1514
+ worker.postMessage({ ply: splatData.buffer, save: false });
1515
+ } else {
1516
+ worker.postMessage({
1517
+ buffer: splatData.buffer,
1518
+ vertexCount: Math.floor(bytesRead / rowLength),
1519
+ });
1520
+ }
1521
+ }
1522
+ }
1523
+
1524
+ main().catch((err) => {
1525
+ document.getElementById("spinner").style.display = "none";
1526
+ document.getElementById("message").innerText = err.toString();
1527
+ });
android/app/src/main/java/com/idol/AutoRun.kt ADDED
@@ -0,0 +1,312 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.os.Build
4
+ import android.os.Process
5
+ import android.os.SystemClock
6
+ import android.util.Log
7
+ import com.google.ai.edge.litert.CompiledModel
8
+ import org.json.JSONArray
9
+ import org.json.JSONObject
10
+ import java.io.File
11
+
12
+ /**
13
+ * Debug builds only: runs a plan of rows through the app and writes what each produced, for the device lines of
14
+ * CONTRACT.md ((a) parity, (b) the .ply, (c) speed, (d) failure paths, (e) memory).
15
+ *
16
+ * adb shell am start -n com.idol/.MainActivity --es autorun plans/<plan>.json --es out runs/<name>
17
+ *
18
+ * Paths are under the app's files dir. Plan: {"preload": true, "reserve_mb": 2500, "mem_check": true,
19
+ * "handoff": "host" | "direct", "matting": "bir" | "dis", "cpu_threads": 6, "gpu_priority": "default", "rows": [...]}
20
+ * (every key optional; a row may override "preload"). Rows:
21
+ * {"id", "kind": "e2e", "fixture": "fx/<tag>", "runs": 4, "preload": true}: the app's whole pipeline from the
22
+ * fixture's raw image and mode with its noise arrays; <id>.<run>.ply + <id>.<run>.json per run (run 0 is the
23
+ * warm-up), <id>.alpha.u8 once (raw uint8 matting alpha), <id>.json for the row.
24
+ * {"id", "kind": "parity", "fixture": "fx/<tag>"}: each part on the fixture's reference input, <id>.<name>.f32.
25
+ * {"id", "kind": "image", "image": "<path>", "mode": "real"}: a normal run of a picture (failure paths);
26
+ * <id>.json with the outcome and the message on the screen.
27
+ * {"id", "kind": "snippet", "fixture": "fx/<tag>"}: the model card's Kotlin block (CardSnippet.kt) on the
28
+ * fixture's picture; its line goes to <id>.json, DONE.json and logcat.
29
+ * Output: <out>/header.json (device, options, the load and compile seconds of each preload), the row files,
30
+ * <out>/DONE.json; logcat tag IdolAutoRun: START / ROW / DONE lines. A file <out>/STOP ends the plan after the
31
+ * current run.
32
+ */
33
+ class AutoRun(private val activity: MainActivity, private val planFile: File, private val outDir: File) {
34
+ private val header = JSONObject()
35
+ private val loads = JSONArray()
36
+
37
+ fun start() {
38
+ Thread({
39
+ try {
40
+ runAll()
41
+ } catch (e: Throwable) {
42
+ Log.e(TAG, "IDOL_AUTORUN FAILED", e)
43
+ runCatching {
44
+ outDir.mkdirs()
45
+ File(outDir, "DONE.json").writeText(
46
+ JSONObject().put("status", "FAILED").put("error", e.toString()).toString(1),
47
+ )
48
+ }
49
+ Log.i(TAG, "IDOL_AUTORUN DONE status=FAILED pid=${Process.myPid()}")
50
+ } finally {
51
+ activity.autorunFinished()
52
+ }
53
+ }, "IdolAutoRun").start()
54
+ }
55
+
56
+ private fun stopped() = File(outDir, "STOP").exists()
57
+
58
+ private fun writeHeader() = File(outDir, "header.json").writeText(header.toString(1))
59
+
60
+ private fun runAll() {
61
+ val plan = JSONObject(planFile.readText())
62
+ outDir.mkdirs()
63
+ File(outDir, "DONE.json").delete()
64
+ header.put("plan", planFile.name).put("plan_json", plan)
65
+ .put("device", DeviceInfo.displayName()).put("model", Build.MODEL).put("android_sdk", Build.VERSION.SDK_INT)
66
+ .put("fingerprint", Build.FINGERPRINT).put("litert", BuildConfig.LITERT_VERSION)
67
+ .put("app_version", BuildConfig.VERSION_NAME).put("pid", Process.myPid())
68
+ .put("mem_total_kb", DeviceInfo.memTotalKb()).put("mem_available_kb", DeviceInfo.memAvailableKb())
69
+ .put("thermal", DeviceInfo.thermalStatus(activity)).put("cpuset", DeviceInfo.cpuset())
70
+ .put("start_ms", System.currentTimeMillis()).put("load", loads)
71
+ writeHeader()
72
+ Log.i(TAG, "IDOL_AUTORUN START plan=${planFile.name} out=${outDir.name} pid=${Process.myPid()}")
73
+ val rows = plan.getJSONArray("rows")
74
+ val summary = JSONArray()
75
+ val done = JSONObject().put("status", "DONE")
76
+ for (r in 0 until rows.length()) {
77
+ if (stopped()) {
78
+ done.put("stopped", true)
79
+ break
80
+ }
81
+ val row = rows.getJSONObject(r)
82
+ val id = row.getString("id")
83
+ val kind = row.optString("kind", "e2e")
84
+ Log.i(TAG, "IDOL_AUTORUN ROW_START id=$id kind=$kind")
85
+ val rec = try {
86
+ when (kind) {
87
+ "e2e" -> e2e(plan, row)
88
+ "parity" -> parity(plan, row)
89
+ "image" -> image(plan, row)
90
+ "snippet" -> snippet(row)
91
+ else -> JSONObject().put("id", id).put("kind", kind).put("status", "unknown_kind")
92
+ }
93
+ } catch (e: Throwable) {
94
+ Log.e(TAG, "row $id failed", e)
95
+ JSONObject().put("id", id).put("kind", kind).put("status", "failed").put("error", e.toString())
96
+ }
97
+ summary.put(rec)
98
+ if (kind == "snippet") done.put("snippet", rec.optString("line"))
99
+ Log.i(TAG, "IDOL_AUTORUN ROW id=$id kind=$kind status=${rec.optString("status")} ${rec.optString("summary")}")
100
+ }
101
+ done.put("rows", summary).put("pid", Process.myPid()).put("end_ms", System.currentTimeMillis())
102
+ .put("self_status_kb", JSONObject(DeviceInfo.selfStatusKb() as Map<*, *>))
103
+ File(outDir, "DONE.json").writeText(done.toString(1))
104
+ Log.i(TAG, "IDOL_AUTORUN DONE rows=${summary.length()} pid=${Process.myPid()}")
105
+ }
106
+
107
+ /** The engine options of [row] in [plan] (the row's keys first). */
108
+ private fun options(plan: JSONObject, row: JSONObject): EngineOptions {
109
+ fun has(k: String) = row.has(k) || plan.has(k)
110
+ fun src(k: String) = if (row.has(k)) row else plan
111
+ val defaults = EngineOptions()
112
+ val manifestMatting = ModelFiles.manifest(ModelFiles.dir(activity.filesDir))?.defaultMatting ?: "bir"
113
+ val priority = when (if (has("gpu_priority")) src("gpu_priority").getString("gpu_priority") else "default") {
114
+ "low" -> CompiledModel.GpuOptions.Priority.LOW
115
+ "normal" -> CompiledModel.GpuOptions.Priority.NORMAL
116
+ "high" -> CompiledModel.GpuOptions.Priority.HIGH
117
+ else -> null
118
+ }
119
+ return EngineOptions(
120
+ preload = if (has("preload")) src("preload").getBoolean("preload") else defaults.preload,
121
+ reserveMb = if (has("reserve_mb")) src("reserve_mb").getLong("reserve_mb") else defaults.reserveMb,
122
+ memoryCheck = if (has("mem_check")) src("mem_check").getBoolean("mem_check") else defaults.memoryCheck,
123
+ directHandoff = has("handoff") && src("handoff").getString("handoff") == "direct",
124
+ matting = if (has("matting")) src("matting").getString("matting") else manifestMatting,
125
+ cpuThreads = if (has("cpu_threads")) src("cpu_threads").getInt("cpu_threads") else defaults.cpuThreads,
126
+ gpuPriority = priority,
127
+ disFloor = if (has("dis_floor")) src("dis_floor").getInt("dis_floor") else defaults.disFloor,
128
+ )
129
+ }
130
+
131
+ private fun json(o: EngineOptions) = JSONObject().put("preload", o.preload).put("reserve_mb", o.reserveMb)
132
+ .put("mem_check", o.memoryCheck).put("handoff", if (o.directHandoff) "direct" else "host")
133
+ .put("matting", o.matting).put("cpu_threads", o.cpuThreads)
134
+ .put("gpu_priority", o.gpuPriority?.name ?: "default").put("dis_floor", o.disFloor)
135
+
136
+ private fun outcomeFields(rec: JSONObject, outcome: Outcome) {
137
+ when (outcome) {
138
+ is Outcome.Done -> rec.put("status", "done")
139
+ is Outcome.NoObject -> rec.put("status", "no_object").put("reason", outcome.reason)
140
+ is Outcome.Missing -> rec.put("status", "missing_models").put("message", ModelFiles.message(outcome.problems))
141
+ is Outcome.Failed -> rec.put("status", outcome.kind).put("message", outcome.message)
142
+ }
143
+ }
144
+
145
+ private fun e2e(plan: JSONObject, row: JSONObject): JSONObject {
146
+ val id = row.getString("id")
147
+ val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
148
+ val runs = row.optInt("runs", 4)
149
+ val opts = options(plan, row)
150
+ val rowRec = JSONObject().put("id", id).put("kind", "e2e").put("fixture", row.getString("fixture"))
151
+ .put("mode", fx.mode).put("runs", runs).put("options", json(opts))
152
+ val problems = activity.ensureEngine(opts)
153
+ if (problems.isNotEmpty()) {
154
+ return rowRec.put("status", "missing_models").put("message", ModelFiles.message(problems))
155
+ }
156
+ if (opts.preload) {
157
+ val load = activity.onModel { requireNotNull(Idol.engine).preload(activity.progress) }
158
+ loads.put(JSONObject().put("row", id).put("record", load))
159
+ writeHeader()
160
+ rowRec.put("load_total_s", load.optDouble("total_s")).put("load_loaded", load.optInt("loaded"))
161
+ .put("load_of", load.optInt("of"))
162
+ }
163
+ val photo = Photo.fromFile(fx.image)
164
+ val noise = fx.noise()
165
+ val statuses = JSONArray()
166
+ val measured = ArrayList<Double>()
167
+ var alphaWritten = false
168
+ for (k in 0 until runs) {
169
+ if (stopped()) {
170
+ rowRec.put("stopped_after", k)
171
+ break
172
+ }
173
+ val ply = File(outDir, "$id.$k.ply")
174
+ val rec = JSONObject().put("id", id).put("run", k).put("warmup", k == 0)
175
+ .put("fixture", row.getString("fixture")).put("mode", fx.mode).put("noise", noise.source)
176
+ .put("options", json(opts))
177
+ .put("thermal_start", DeviceInfo.thermalStatus(activity))
178
+ .put("mem_available_kb_start", DeviceInfo.memAvailableKb())
179
+ .put("vmhwm_reset", DeviceInfo.resetPeakRss())
180
+ .put("cpuset", DeviceInfo.cpuset())
181
+ val sampler = GpuPageSampler().also { it.start() }
182
+ val t0 = SystemClock.elapsedRealtimeNanos()
183
+ val outcome = activity.runAndWait(photo, fx.mode, noise, ply, showResult = false)
184
+ val wall = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
185
+ val gpuMax = sampler.stop()
186
+ val st = DeviceInfo.selfStatusKb()
187
+ rec.put("wall_s", wall).put("thermal_end", DeviceInfo.thermalStatus(activity))
188
+ .put("vmhwm_kb", st["VmHWM"] ?: -1L).put("vmrss_kb", st["VmRSS"] ?: -1L)
189
+ .put("vmswap_kb", st["VmSwap"] ?: -1L).put("gpu_page_alloc_max_bytes", gpuMax)
190
+ .put("mem_available_kb_end", DeviceInfo.memAvailableKb())
191
+ outcomeFields(rec, outcome)
192
+ if (outcome is Outcome.Done) {
193
+ val r = outcome.result
194
+ val stages = JSONObject()
195
+ for ((key, v) in r.times) if (key != "generate" && key != "all") stages.put(key, v)
196
+ rec.put("stages", stages).put("generate", r.times["generate"]).put("total", r.times["all"])
197
+ .put("ply", ply.name).put("ply_bytes", ply.length()).put("gaussians", r.gaussians)
198
+ .put("box", JSONArray(r.pre.box.toList()))
199
+ .put("alpha_size", JSONArray(listOf(r.pre.alphaWidth, r.pre.alphaHeight)))
200
+ .put("own_alpha", r.pre.ownAlpha).put("handoff", r.handoff)
201
+ .put("compiles_in_run", JSONObject().put("count", r.compiles.size)
202
+ .put("seconds", r.compiles.sumOf { it.seconds })
203
+ .put("graphs", JSONArray(r.compiles.map { it.json() })))
204
+ .put("memory_events", JSONArray(r.events))
205
+ if (!alphaWritten) {
206
+ File(outDir, "$id.alpha.u8").writeBytes(r.pre.alphaRaw)
207
+ rowRec.put("alpha", JSONObject().put("file", "$id.alpha.u8").put("width", r.pre.alphaWidth)
208
+ .put("height", r.pre.alphaHeight).put("box", JSONArray(r.pre.box.toList())))
209
+ alphaWritten = true
210
+ }
211
+ if (k > 0) r.times["generate"]?.let { measured.add(it) }
212
+ }
213
+ File(outDir, "$id.$k.json").writeText(rec.toString(1))
214
+ statuses.put(rec.optString("status"))
215
+ Log.i(TAG, "IDOL_AUTORUN RUN id=$id run=$k warmup=${k == 0} status=${rec.optString("status")} " +
216
+ "generate_s=${rec.optDouble("generate", -1.0)} vmhwm_kb=${rec.optLong("vmhwm_kb")}")
217
+ }
218
+ val allDone = (0 until statuses.length()).all { statuses.getString(it) == "done" }
219
+ rowRec.put("statuses", statuses).put("status", if (allDone && statuses.length() > 0) "done" else "incomplete")
220
+ if (measured.isNotEmpty()) {
221
+ val sorted = measured.sorted()
222
+ val median = if (sorted.size % 2 == 1) sorted[sorted.size / 2]
223
+ else (sorted[sorted.size / 2 - 1] + sorted[sorted.size / 2]) / 2
224
+ rowRec.put("generate_measured_s", JSONArray(measured)).put("generate_median_s", median)
225
+ .put("summary", "generate median %.2f s over %d measured runs".format(median, measured.size))
226
+ }
227
+ File(outDir, "$id.json").writeText(rowRec.toString(1))
228
+ return rowRec
229
+ }
230
+
231
+ private fun parity(plan: JSONObject, row: JSONObject): JSONObject {
232
+ val id = row.getString("id")
233
+ val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
234
+ val opts = options(plan, row).copy(preload = false)
235
+ val problems = activity.ensureEngine(opts)
236
+ if (problems.isNotEmpty()) {
237
+ return JSONObject().put("id", id).put("kind", "parity").put("status", "missing_models")
238
+ .put("message", ModelFiles.message(problems))
239
+ }
240
+ activity.showRunScreen("Part agreement: ${fx.dir.name}")
241
+ val t0 = SystemClock.elapsedRealtimeNanos()
242
+ val rec = activity.onModel { requireNotNull(Idol.engine).parity(fx, outDir, id, activity.progress) }
243
+ rec.put("kind", "parity").put("status", "done").put("options", json(opts))
244
+ .put("wall_s", (SystemClock.elapsedRealtimeNanos() - t0) / 1e9)
245
+ .put("summary", "${rec.getJSONObject("arrays").length()} arrays")
246
+ File(outDir, "$id.json").writeText(rec.toString(1))
247
+ return rec
248
+ }
249
+
250
+ private fun image(plan: JSONObject, row: JSONObject): JSONObject {
251
+ val id = row.getString("id")
252
+ val path = row.getString("image")
253
+ val mode = row.optString("mode", "real")
254
+ val rec = JSONObject().put("id", id).put("kind", "image").put("image", path).put("mode", mode)
255
+ activity.ensureEngine(options(plan, row))
256
+ val photo = try {
257
+ Photo.fromFile(File(activity.filesDir, path))
258
+ } catch (e: Throwable) {
259
+ return rec.put("status", "decode_failed").put("error", e.toString())
260
+ }
261
+ val outcome = activity.runAndWait(
262
+ photo, mode, Noise.random(MainActivity.DEFAULT_SEED), File(outDir, "$id.ply"),
263
+ showResult = row.optBoolean("show_result", false),
264
+ )
265
+ outcomeFields(rec, outcome)
266
+ if (outcome is Outcome.Done) rec.put("generate", outcome.result.times["generate"])
267
+ // the outcome is on the screen once the UI thread has run the posts before this one
268
+ rec.put("message_on_screen", activity.homeMessageText()).put("pid", Process.myPid())
269
+ rec.put("summary", rec.optString("message_on_screen").lineSequence().firstOrNull() ?: "")
270
+ File(outDir, "$id.json").writeText(rec.toString(1))
271
+ return rec
272
+ }
273
+
274
+ private fun snippet(row: JSONObject): JSONObject {
275
+ val id = row.getString("id")
276
+ val fx = Fixture(File(activity.filesDir, row.getString("fixture")))
277
+ val photo = Photo.fromFile(fx.image)
278
+ val rgb = Preprocess.resized(photo.image).rgb()
279
+ val image = Matting.birefnetInput(rgb)
280
+ val dir = ModelFiles.dir(activity.filesDir)
281
+ val t0 = SystemClock.elapsedRealtimeNanos()
282
+ val line = activity.onModel {
283
+ // the card's block builds its own environment and graph; free the engine's graphs first
284
+ Idol.engine?.pool?.releaseAll()
285
+ val bir = cardSnippet(dir, image)
286
+ if (!fx.has("pixels") || !fx.has("feature1")) return@onModel bir
287
+ // the DINOv3 card's block on the fixture's preprocessed picture, against the reference's feature1
288
+ val got = dinoCardSnippet(dir, fx.floats("pixels"))
289
+ val want = fx.floats("feature1")
290
+ var dot = 0.0
291
+ var ng = 0.0
292
+ var nw = 0.0
293
+ for (i in want.indices) {
294
+ dot += got[i].toDouble() * want[i]
295
+ ng += got[i].toDouble() * got[i]
296
+ nw += want[i].toDouble() * want[i]
297
+ }
298
+ "$bir | dino card snippet: ${got.size} values, cos vs the reference feature1 %.6f"
299
+ .format(dot / Math.sqrt(ng * nw))
300
+ }
301
+ Log.i(TAG, "IDOL_AUTORUN SNIPPET $line")
302
+ val rec = JSONObject().put("id", id).put("kind", "snippet").put("status", "done").put("line", line)
303
+ .put("picture", JSONArray(listOf(rgb.width, rgb.height)))
304
+ .put("seconds", (SystemClock.elapsedRealtimeNanos() - t0) / 1e9).put("summary", line)
305
+ File(outDir, "$id.json").writeText(rec.toString(1))
306
+ return rec
307
+ }
308
+
309
+ companion object {
310
+ const val TAG = "IdolAutoRun"
311
+ }
312
+ }
android/app/src/main/java/com/idol/CardSnippet.kt ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import com.google.ai.edge.litert.Accelerator
4
+ import com.google.ai.edge.litert.CompiledModel
5
+ import com.google.ai.edge.litert.Environment
6
+ import java.io.File
7
+
8
+ /**
9
+ * The model card's Kotlin block, run on the phone as written (autorun plan plans/snippet.json). [dir] is the app's
10
+ * model folder; [image] is the card's `image`: Matting.birefnetInput(the picture resized to a short side of 1024),
11
+ * [1, 3, 1024, 1024]. Returns one line about the alpha it produced.
12
+ */
13
+ fun cardSnippet(dir: File, image: FloatArray): String {
14
+ // ---- the card's block, verbatim
15
+ val env = Environment.create()
16
+ val cpu = CompiledModel.Options(Accelerator.CPU) // bir.tflite: BiRefNet, the one CPU graph
17
+ val bir = CompiledModel.create(File(dir, "bir.tflite").absolutePath, cpu, env)
18
+ val inputs = bir.createInputBuffers()
19
+ val outputs = bir.createOutputBuffers()
20
+ inputs[0].writeFloat(image) // [1, 3, 1024, 1024]: RGB / 255, ImageNet mean / std
21
+ bir.run(inputs, outputs)
22
+ val alpha = outputs[0].readFloat() // [1024 * 1024] sigmoid map, 1 = the object
23
+ bir.close()
24
+ val gpu = CompiledModel.Options(Accelerator.GPU).apply { // every other graph
25
+ gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM)
26
+ }
27
+ // ---- end of the card's block
28
+ for (b in inputs + outputs) runCatching { b.close() }
29
+ runCatching { env.close() }
30
+ var mn = Float.POSITIVE_INFINITY
31
+ var mx = Float.NEGATIVE_INFINITY
32
+ var sum = 0.0
33
+ var above = 0
34
+ for (v in alpha) {
35
+ if (v < mn) mn = v
36
+ if (v > mx) mx = v
37
+ sum += v
38
+ if (v > 0.5f) above++
39
+ }
40
+ return "card snippet: alpha ${alpha.size} values, min %.4f max %.4f mean %.4f, fraction > 0.5 %.4f; gpu options built (%s)"
41
+ .format(mn, mx, sum / alpha.size, above.toDouble() / alpha.size, gpu.gpuOptions?.precision)
42
+ }
43
+
44
+ /**
45
+ * The DINOv3 repo card's Kotlin block, run on the phone as written (autorun plan plans/snippet.json, after
46
+ * [cardSnippet]). [dir] holds dino_0..3.tflite and the RoPE tables; [pixels] is the card's `pixels`: the picture Idol
47
+ * feeds, [1, 3, 1024, 1024] RGB in [0, 1]. Returns the feature (for the check against the fixture's feature1).
48
+ */
49
+ fun dinoCardSnippet(dir: File, pixels: FloatArray): FloatArray {
50
+ // ---- the DINOv3 card's block, verbatim
51
+ fun floats(f: File): FloatArray {
52
+ val b = java.nio.ByteBuffer.wrap(f.readBytes()).order(java.nio.ByteOrder.LITTLE_ENDIAN).asFloatBuffer()
53
+ return FloatArray(b.remaining()).also { b.get(it) }
54
+ }
55
+ val env = Environment.create()
56
+ val gpu = CompiledModel.Options(Accelerator.GPU).apply {
57
+ gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM)
58
+ }
59
+ val rope = mapOf("args_1" to floats(File(dir, "dino_rope_cos.f32")), "args_2" to floats(File(dir, "dino_rope_sin.f32")))
60
+ var x = pixels // [1, 3, 1024, 1024] RGB in [0, 1]
61
+ for (i in 0..3) { // 8 layers per graph
62
+ val model = CompiledModel.create(File(dir, "dino_$i.tflite").absolutePath, gpu, env)
63
+ val inputs = listOf("args_0", "args_1", "args_2").associateWith { model.createInputBuffer(it, "serving_default") }
64
+ val output = model.createOutputBuffer("output_0", "serving_default")
65
+ for ((name, buffer) in inputs) buffer.writeFloat(if (name == "args_0") x else rope.getValue(name))
66
+ model.run(inputs, mapOf("output_0" to output), "serving_default")
67
+ x = output.readFloat() // [1, 4101, 1280]
68
+ for (b in inputs.values + output) b.close()
69
+ model.close()
70
+ }
71
+ env.close()
72
+ // x: the image feature Idol's denoiser reads (1 class, 4 register and 64 x 64 patch tokens)
73
+ // ---- end of the DINOv3 card's block
74
+ return x
75
+ }
android/app/src/main/java/com/idol/DeviceInfo.kt ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.content.Context
4
+ import android.os.Build
5
+ import android.os.PowerManager
6
+ import java.io.File
7
+
8
+ /** Device facts for the status line and the run JSON. */
9
+ object DeviceInfo {
10
+ /** `MemAvailable` of `/proc/meminfo` in kB (what the memory guard compares with), or -1. */
11
+ fun memAvailableKb(): Long = meminfo("MemAvailable:")
12
+
13
+ fun memTotalKb(): Long = meminfo("MemTotal:")
14
+
15
+ private fun meminfo(key: String): Long = runCatching {
16
+ File("/proc/meminfo").useLines { lines ->
17
+ lines.first { it.startsWith(key) }.trim().split(Regex("\\s+"))[1].toLong()
18
+ }
19
+ }.getOrDefault(-1L)
20
+
21
+ /** VmHWM / VmRSS / VmSwap of this process in kB (`/proc/self/status`). */
22
+ fun selfStatusKb(): Map<String, Long> = runCatching {
23
+ File("/proc/self/status").readLines().mapNotNull { line ->
24
+ val key = line.substringBefore(':')
25
+ if (key in setOf("VmHWM", "VmRSS", "VmSwap")) key to line.substringAfter(':').trim().split(' ')[0].toLong()
26
+ else null
27
+ }.toMap()
28
+ }.getOrDefault(emptyMap())
29
+
30
+ /**
31
+ * Resets this process's VmHWM to its current RSS (`echo 5 > /proc/self/clear_refs`), so that VmHWM read after a run
32
+ * is that run's peak. Returns false when the system does not allow it (the VmHWM is then the process's peak).
33
+ */
34
+ fun resetPeakRss(): Boolean = runCatching {
35
+ File("/proc/self/clear_refs").writeText("5")
36
+ true
37
+ }.getOrDefault(false)
38
+
39
+ /** The system's GPU pages (`/sys/class/kgsl/kgsl/page_alloc`, bytes), or -1 when the file cannot be read. */
40
+ fun gpuPageAllocBytes(): Long = runCatching {
41
+ File("/sys/class/kgsl/kgsl/page_alloc").readText().trim().split(Regex("\\s+"))[0].toLong()
42
+ }.getOrDefault(-1L)
43
+
44
+ /** "Galaxy S26 (SM-S942Q)" for the phones this sample was measured on, else `Build.MODEL`. */
45
+ fun displayName(): String {
46
+ val model = Build.MODEL
47
+ val market = MARKET_NAMES.entries.firstOrNull { model.startsWith(it.key) }?.value
48
+ return if (market == null || market == model) model else "$market ($model)"
49
+ }
50
+
51
+ fun thermalStatus(context: Context): Int =
52
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
53
+ context.getSystemService(PowerManager::class.java).currentThermalStatus
54
+ } else {
55
+ -1
56
+ }
57
+
58
+ /** The cpuset of this process (`/top-app` while the app is in front). */
59
+ fun cpuset(): String = runCatching {
60
+ File("/proc/self/cgroup").readLines().firstOrNull { ":cpuset:" in it } ?: "none"
61
+ }.getOrDefault("unreadable")
62
+
63
+ private val MARKET_NAMES = mapOf("SM-S942" to "Galaxy S26")
64
+ }
65
+
66
+ /**
67
+ * Samples the system's GPU pages every [periodMs] while a run is in progress and keeps the maximum (CONTRACT.md
68
+ * line (e)). Best effort: when the app cannot read the file, [maxBytes] stays -1.
69
+ */
70
+ class GpuPageSampler(private val periodMs: Long = 100) {
71
+ @Volatile var maxBytes = -1L
72
+ private set
73
+ @Volatile private var running = false
74
+ private var thread: Thread? = null
75
+
76
+ fun start() {
77
+ maxBytes = DeviceInfo.gpuPageAllocBytes()
78
+ if (maxBytes < 0) return
79
+ running = true
80
+ thread = Thread({
81
+ while (running) {
82
+ val v = DeviceInfo.gpuPageAllocBytes()
83
+ if (v > maxBytes) maxBytes = v
84
+ try {
85
+ Thread.sleep(periodMs)
86
+ } catch (_: InterruptedException) {
87
+ break
88
+ }
89
+ }
90
+ }, "IdolGpuPages").apply { isDaemon = true; start() }
91
+ }
92
+
93
+ fun stop(): Long {
94
+ running = false
95
+ thread?.interrupt()
96
+ thread?.join(500)
97
+ thread = null
98
+ val v = DeviceInfo.gpuPageAllocBytes()
99
+ if (v > maxBytes) maxBytes = v
100
+ return maxBytes
101
+ }
102
+ }
android/app/src/main/java/com/idol/Fixture.kt ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import org.json.JSONObject
4
+ import java.io.File
5
+ import java.io.FileOutputStream
6
+ import java.io.RandomAccessFile
7
+ import java.nio.ByteBuffer
8
+ import java.nio.ByteOrder
9
+ import java.nio.channels.FileChannel
10
+
11
+ /**
12
+ * A fixture directory `files/fx/<tag>/` as scripts/make_fixtures.py writes it: fixture.json
13
+ * {"tag", "mode", "seed", "image", "arrays": {name: {"file", "shape"}}}, the raw input image and little-endian
14
+ * float32 arrays (the reference run's shared randomness and each part's own reference input).
15
+ */
16
+ class Fixture(val dir: File) {
17
+ private val json = JSONObject(File(dir, "fixture.json").readText())
18
+ val tag: String = json.optString("tag", dir.name)
19
+ val mode: String = json.getString("mode")
20
+ val seed: Long = json.optLong("seed", 0L)
21
+ val image: File = File(dir, json.getString("image"))
22
+ private val arrays = json.getJSONObject("arrays")
23
+
24
+ fun has(name: String) = arrays.has(name)
25
+
26
+ fun shape(name: String): IntArray {
27
+ val a = arrays.getJSONObject(name).getJSONArray("shape")
28
+ return IntArray(a.length()) { a.getInt(it) }
29
+ }
30
+
31
+ fun floats(name: String): FloatArray {
32
+ val e = arrays.getJSONObject(name)
33
+ val shape = shape(name)
34
+ val n = shape.fold(1) { a, b -> a * b }
35
+ return readF32(File(dir, e.getString("file")), n)
36
+ }
37
+
38
+ /** The five noise arrays, in place of the app's random draws. */
39
+ fun noise(): Noise = Noise(
40
+ vae = floats("vae_noise"),
41
+ latent = floats("latent_noise"),
42
+ camera = floats("camera_noise"),
43
+ pool = floats("octree_pool"),
44
+ leaf = floats("leaf_pool"),
45
+ source = "fixture ${dir.name}",
46
+ )
47
+
48
+ companion object {
49
+ /** [n] little-endian float32 values from [file] (its size must be n x 4 bytes). */
50
+ fun readF32(file: File, n: Int): FloatArray {
51
+ require(file.length() == n * 4L) { "${file.name} is ${file.length()} bytes, expected ${n * 4L}" }
52
+ val out = FloatArray(n)
53
+ RandomAccessFile(file, "r").use { raf ->
54
+ val map = raf.channel.map(FileChannel.MapMode.READ_ONLY, 0, n * 4L)
55
+ map.order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().get(out)
56
+ }
57
+ return out
58
+ }
59
+
60
+ fun writeF32(file: File, data: FloatArray) {
61
+ val buf = ByteBuffer.allocate(data.size * 4).order(ByteOrder.LITTLE_ENDIAN)
62
+ buf.asFloatBuffer().put(data)
63
+ FileOutputStream(file).use { it.write(buf.array()) }
64
+ }
65
+ }
66
+ }
android/app/src/main/java/com/idol/Glue.kt ADDED
@@ -0,0 +1,472 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import org.json.JSONObject
4
+ import java.io.BufferedOutputStream
5
+ import java.io.File
6
+ import java.io.FileOutputStream
7
+ import java.nio.ByteBuffer
8
+ import java.nio.ByteOrder
9
+ import kotlin.math.PI
10
+ import kotlin.math.cos
11
+ import kotlin.math.exp
12
+ import kotlin.math.expm1
13
+ import kotlin.math.ln
14
+ import kotlin.math.ln1p
15
+ import kotlin.math.pow
16
+ import kotlin.math.sin
17
+ import kotlin.math.sqrt
18
+ import kotlin.math.tanh
19
+
20
+ /** The five arrays of shared randomness (CONTRACT.md "Shared randomness"), flat in C order. */
21
+ class Noise(
22
+ /** [1, 32, 128, 128] normal: the VAE posterior noise. */
23
+ val vae: FloatArray,
24
+ /** [1, 8192, 16] normal: x at t = 1. */
25
+ val latent: FloatArray,
26
+ /** [1, 1, 5] normal: the camera at t = 1. */
27
+ val camera: FloatArray,
28
+ /** [8, 8192] uniform: the octree sampler's per-row offsets. */
29
+ val pool: FloatArray,
30
+ /** [1, 8192, 3] uniform: the leaf positions. */
31
+ val leaf: FloatArray,
32
+ /** "seed <n>" or the fixture it came from. */
33
+ val source: String,
34
+ ) {
35
+ init {
36
+ require(vae.size == 32 * 128 * 128 && latent.size == Glue.NUM_POINTS * Glue.IN_CH && camera.size == Glue.CAM_CH)
37
+ require(pool.size == Glue.MAX_LEVEL * Glue.NUM_POINTS && leaf.size == Glue.NUM_POINTS * 3)
38
+ }
39
+
40
+ companion object {
41
+ /**
42
+ * The app's own draws for a run from a picture: same distributions as the reference (normal, uniform [0, 1)),
43
+ * from java.util.Random, not torch's generator; the fixtures carry the reference's own numbers.
44
+ */
45
+ fun random(seed: Long): Noise {
46
+ val r = java.util.Random(seed)
47
+ fun normal(n: Int) = FloatArray(n) { r.nextGaussian().toFloat() }
48
+ fun uniform(n: Int) = FloatArray(n) { r.nextFloat() }
49
+ val vae = normal(32 * 128 * 128)
50
+ val latent = normal(Glue.NUM_POINTS * Glue.IN_CH)
51
+ val camera = normal(Glue.CAM_CH)
52
+ val pool = uniform(Glue.MAX_LEVEL * Glue.NUM_POINTS)
53
+ val leaf = uniform(Glue.NUM_POINTS * 3)
54
+ return Noise(vae, latent, camera, pool, leaf, "seed $seed")
55
+ }
56
+ }
57
+ }
58
+
59
+ /** Activated Gaussians (z-up, as GaussianFixedlenDecoder.to_representation returns them), N = 8192 x 32. */
60
+ class Gaussians(
61
+ val count: Int,
62
+ val xyz: FloatArray,
63
+ val featuresDc: FloatArray,
64
+ val scaling: FloatArray,
65
+ /** (w, x, y, z) */
66
+ val rotation: FloatArray,
67
+ val opacity: FloatArray,
68
+ )
69
+
70
+ /** gs_rep.json: the Gaussian decoder's representation constants. */
71
+ class GaussianRep(
72
+ val baseOffsetScale: Float,
73
+ val lrXyz: Float,
74
+ val lrFeaturesDc: Float,
75
+ val lrOpacity: Float,
76
+ val lrScaling: Float,
77
+ val lrRotation: Float,
78
+ val perturbeSize: Float,
79
+ val filterKernelSize: Double,
80
+ val scalingBias: Float,
81
+ val opacityBias: Float,
82
+ ) {
83
+ companion object {
84
+ fun parse(o: JSONObject): GaussianRep {
85
+ val rep = o.getJSONObject("rep")
86
+ val lr = rep.getJSONObject("lr")
87
+ return GaussianRep(
88
+ baseOffsetScale = o.getDouble("base_offset_scale").toFloat(),
89
+ lrXyz = lr.getDouble("_xyz").toFloat(),
90
+ lrFeaturesDc = lr.getDouble("_features_dc").toFloat(),
91
+ lrOpacity = lr.getDouble("_opacity").toFloat(),
92
+ lrScaling = lr.getDouble("_scaling").toFloat(),
93
+ lrRotation = lr.getDouble("_rotation").toFloat(),
94
+ perturbeSize = rep.getDouble("perturbe_size").toFloat(),
95
+ filterKernelSize = rep.getDouble("3d_filter_kernel_size"),
96
+ scalingBias = rep.getDouble("scaling_bias").toFloat(),
97
+ opacityBias = rep.getDouble("opacity_bias").toFloat(),
98
+ )
99
+ }
100
+ }
101
+ }
102
+
103
+ /**
104
+ * The host glue between the graphs, ported line by line from scripts/idol_glue.py, scripts/idol_graphs.py
105
+ * (sinusoid, point_pe, lora_gate) and scripts/idol_runner.py (vae_tokens): the same order of float32 operations as
106
+ * the torch / numpy code (each transcendental is computed in double and rounded to float32).
107
+ */
108
+ object Glue {
109
+ const val MAX_LEVEL = 8
110
+ const val PER_ANCHOR = 32
111
+ const val PHASE_THRESHOLD = 0.5
112
+ const val NUM_POINTS = 8192
113
+ const val IN_CH = 16
114
+ const val CAM_CH = 5
115
+ const val COND_TOKENS = 4101
116
+ const val LORA_SCALE = 128.0f / 32.0f
117
+ val OCT_BUCKETS = intArrayOf(64, 256, 1024, 4096, 8192)
118
+ const val PLY_FLOATS = 17
119
+ val PLY_NAMES = listOf(
120
+ "x", "y", "z", "nx", "ny", "nz", "f_dc_0", "f_dc_1", "f_dc_2", "opacity",
121
+ "scale_0", "scale_1", "scale_2", "rot_0", "rot_1", "rot_2", "rot_3",
122
+ )
123
+
124
+ private val PI_F = PI.toFloat()
125
+
126
+ private fun expf(x: Float) = exp(x.toDouble()).toFloat()
127
+ private fun logf(x: Float) = ln(x.toDouble()).toFloat()
128
+
129
+ /** time_grid: t = r s / (1 + (r - 1) s), s = numpy.linspace(1, 0, steps + 1), float64. */
130
+ fun timeGrid(steps: Int = 5, rescale: Double = 3.0): DoubleArray {
131
+ val num = steps + 1
132
+ val step = (0.0 - 1.0) / (num - 1)
133
+ val s = DoubleArray(num) { i -> i * step + 1.0 }
134
+ s[num - 1] = 0.0
135
+ return DoubleArray(num) { rescale * s[it] / (1 + (rescale - 1) * s[it]) }
136
+ }
137
+
138
+ /** lora_gate: [1, 64], the slow expert's half (t >= 0.5) or the fast one's at LORA_SCALE. */
139
+ fun loraGate(slow: Boolean): FloatArray = FloatArray(64) { i ->
140
+ if ((i < 32) == slow) LORA_SCALE else 0f
141
+ }
142
+
143
+ /** sinusoid: SinusoidalEmbedder's frequency features of one scalar -> [1, dim]. */
144
+ fun sinusoid(value: Double, dim: Int = 256, maxPeriod: Double = 10000.0, twoPi: Boolean = false): FloatArray {
145
+ val half = dim / 2
146
+ val c = (-ln(maxPeriod)).toFloat()
147
+ val v = value.toFloat()
148
+ val out = FloatArray(dim)
149
+ for (k in 0 until half) {
150
+ val f = expf((c * k.toFloat()) / half.toFloat())
151
+ var a = v * f
152
+ if (twoPi) {
153
+ a *= 2f
154
+ a *= PI_F
155
+ }
156
+ out[k] = cos(a.toDouble()).toFloat()
157
+ out[half + k] = sin(a.toDouble()).toFloat()
158
+ }
159
+ return out
160
+ }
161
+
162
+ /**
163
+ * torch.linspace(start, end, steps) in float32 as the reference runs' torch computes it: start + step * i below
164
+ * halfway, end - step * (steps - 1 - i) from there, each one fused multiply-add (one rounding). Only called with
165
+ * (0, 10, 170): there the product and the sum are exact in double, so the double expression rounds once too.
166
+ */
167
+ private fun linspaceF(start: Float, end: Float, steps: Int): FloatArray {
168
+ val step = (end - start) / (steps - 1).toFloat()
169
+ val halfway = steps / 2
170
+ return FloatArray(steps) { i ->
171
+ if (i < halfway) (step.toDouble() * i + start).toFloat() else (end - step.toDouble() * (steps - i - 1)).toFloat()
172
+ }
173
+ }
174
+
175
+ /**
176
+ * The frequencies where torch's float32 pow(2, x) is one ulp off the correctly rounded value, read from the Mac
177
+ * reference run's point embedding (device/fx/g1_gs_0): with them, all 170 frequency columns of that embedding agree
178
+ * to one ulp of sin / cos (without them six columns are off by up to 1.1e-3).
179
+ */
180
+ private val TORCH_POW_ULP = mapOf(43 to -1, 72 to -1, 95 to -1, 113 to 1, 138 to -1, 159 to 1)
181
+
182
+ /** torch.pow(2.0, torch.linspace(0.0, 10.0, 170)), float32. */
183
+ private val PE_FREQS: FloatArray by lazy {
184
+ val lin = linspaceF(0f, 10f, 1024 / 3 / 2)
185
+ FloatArray(lin.size) {
186
+ val f = 2.0.pow(lin[it].toDouble()).toFloat()
187
+ Float.fromBits(f.toRawBits() + (TORCH_POW_ULP[it] ?: 0))
188
+ }
189
+ }
190
+
191
+ /** point_pe: PcdAbsolutePositionEmbedderv2 of [n] points (x, y, z) -> [n, 1024]. */
192
+ fun pointPe(points: FloatArray, n: Int, channels: Int = 1024): FloatArray {
193
+ val freqs = PE_FREQS
194
+ val fd = freqs.size
195
+ val out = FloatArray(n * channels)
196
+ for (p in 0 until n) {
197
+ for (c in 0 until 3) {
198
+ val v = points[p * 3 + c]
199
+ val base = p * channels + c * 2 * fd
200
+ for (j in 0 until fd) {
201
+ val ang = (v * freqs[j]) * PI_F
202
+ out[base + j] = sin(ang.toDouble()).toFloat()
203
+ out[base + fd + j] = cos(ang.toDouble()).toFloat()
204
+ }
205
+ }
206
+ }
207
+ return out
208
+ }
209
+
210
+ /**
211
+ * vae_tokens: posterior sample with the shared noise (logvar clipped to [-30, 20]), 2 x 2 patchify, bn ->
212
+ * [1, 4096, 128]. moments [1, 64, 128, 128] (mean, logvar), noise [1, 32, 128, 128].
213
+ */
214
+ fun vaeTokens(moments: FloatArray, noise: FloatArray, bnMean: FloatArray, bnStd: FloatArray): FloatArray {
215
+ val hw = 128
216
+ val plane = hw * hw
217
+ val out = FloatArray(4096 * 128)
218
+ for (c in 0 until 32) for (y in 0 until hw) for (x in 0 until hw) {
219
+ val i = c * plane + y * hw + x
220
+ var lv = moments[32 * plane + i]
221
+ lv = if (lv < -30f) -30f else if (lv > 20f) 20f else lv
222
+ val lat = moments[i] + expf(0.5f * lv) * noise[i]
223
+ val k = c * 4 + (y % 2) * 2 + (x % 2)
224
+ val t = (y / 2) * 64 + (x / 2)
225
+ out[t * 128 + k] = (lat - bnMean[k]) / bnStd[k]
226
+ }
227
+ return out
228
+ }
229
+
230
+ /** torch.softmax over the last dimension (8) of [rows, 8] logits: exp(x - max) times 1 / sum. */
231
+ fun softmax8(logits: FloatArray, rows: Int): FloatArray {
232
+ val out = FloatArray(rows * 8)
233
+ for (r in 0 until rows) {
234
+ val b = r * 8
235
+ var mx = logits[b]
236
+ for (k in 1 until 8) if (logits[b + k] > mx) mx = logits[b + k]
237
+ var sum = 0f
238
+ for (k in 0 until 8) {
239
+ val e = expf(logits[b + k] - mx)
240
+ out[b + k] = e
241
+ sum += e
242
+ }
243
+ val inv = 1f / sum
244
+ for (k in 0 until 8) out[b + k] = out[b + k] * inv
245
+ }
246
+ return out
247
+ }
248
+
249
+ /**
250
+ * systematic_rows: idol's systematic_sample for one batch, row i drawing counts[i] samples from its 8 child
251
+ * probabilities with the offset u[i] / counts[i] -> [rows, 8] sample counts. The cumulative sum accumulates in
252
+ * double as torch's CPU cumsum does for float32.
253
+ */
254
+ fun systematicRows(probs: FloatArray, counts: IntArray, u: FloatArray, rows: Int): IntArray {
255
+ val P = 8
256
+ val out = IntArray(rows * P)
257
+ val p = FloatArray(P)
258
+ val cdf = FloatArray(P)
259
+ for (r in 0 until rows) {
260
+ val n = counts[r]
261
+ if (n == 0) continue
262
+ var s = 0f
263
+ for (k in 0 until P) {
264
+ val v = probs[r * P + k]
265
+ p[k] = if (v < 0f) 0f else v
266
+ s += p[k]
267
+ }
268
+ if (s == 0f) {
269
+ for (k in 0 until P) p[k] = 1.0f / P
270
+ } else {
271
+ val d = if (s < 1f) 1f else s
272
+ for (k in 0 until P) p[k] = p[k] / d
273
+ }
274
+ var acc = 0.0
275
+ for (k in 0 until P) {
276
+ acc += p[k]
277
+ val c = acc.toFloat()
278
+ cdf[k] = if (c > 1f) 1f else c
279
+ }
280
+ val nf = n.toFloat()
281
+ val u0 = u[r] / nf
282
+ for (j in 0 until n) {
283
+ var us = u0 + j.toFloat() / nf
284
+ if (us > 1f) us = 1f
285
+ var idx = 0
286
+ while (idx < P && cdf[idx] < us) idx++
287
+ if (idx > P - 1) idx = P - 1
288
+ out[r * P + idx]++
289
+ }
290
+ }
291
+ return out
292
+ }
293
+
294
+ /**
295
+ * octree_descend: idol OctreeProbabilityFixedlenDecoder.sample for one batch. [probsFn] (cells [rows, 3], rows,
296
+ * level) -> child probabilities [rows, 8]. pool [8, 8192], leaf [8192, 3] -> anchors [8192, 3] in [0, 1]^3.
297
+ * [cellsOut], when given, receives each level's cells (the parity record).
298
+ */
299
+ fun octreeDescend(
300
+ probsFn: (FloatArray, Int, Int) -> FloatArray,
301
+ pool: FloatArray,
302
+ leaf: FloatArray,
303
+ numPoints: Int = NUM_POINTS,
304
+ levels: Int = MAX_LEVEL,
305
+ ): FloatArray {
306
+ var prev = IntArray(3)
307
+ var rows = 1
308
+ var counts = intArrayOf(numPoints)
309
+ for (l in 1..levels) {
310
+ val res = (1 shl (l - 1)).toFloat()
311
+ val cells = FloatArray(rows * 3) { (prev[it].toFloat() + 0.5f) / res }
312
+ val probs = probsFn(cells, rows, l)
313
+ val u = FloatArray(rows) { pool[(l - 1) * numPoints + it] }
314
+ val sampled = systematicRows(probs, counts, u, rows)
315
+ var m = 0
316
+ for (v in sampled) if (v > 0) m++
317
+ val next = IntArray(m * 3)
318
+ val nextCounts = IntArray(m)
319
+ var o = 0
320
+ for (r in 0 until rows) for (c in 0 until 8) {
321
+ val s = sampled[r * 8 + c]
322
+ if (s <= 0) continue
323
+ // CHILD_OFFSET[c] = (i, j, k) with c = k * 4 + j * 2 + i
324
+ next[o * 3] = prev[r * 3] * 2 + (c and 1)
325
+ next[o * 3 + 1] = prev[r * 3 + 1] * 2 + ((c shr 1) and 1)
326
+ next[o * 3 + 2] = prev[r * 3 + 2] * 2 + ((c shr 2) and 1)
327
+ nextCounts[o] = s
328
+ o++
329
+ }
330
+ prev = next
331
+ counts = nextCounts
332
+ rows = m
333
+ }
334
+ val out = FloatArray(numPoints * 3)
335
+ val scale = (1 shl levels).toFloat()
336
+ var p = 0
337
+ for (r in 0 until rows) for (k in 0 until counts[r]) {
338
+ check(p < numPoints) { "octree sampling produced more than $numPoints points" }
339
+ for (d in 0 until 3) out[p * 3 + d] = (prev[r * 3 + d].toFloat() + leaf[p * 3 + d]) / scale
340
+ p++
341
+ }
342
+ check(p == numPoints) { "octree sampling produced $p points, not $numPoints" }
343
+ return out
344
+ }
345
+
346
+ private fun softplus(x: Float): Float = if (x > 20f) x else ln1p(expf(x).toDouble()).toFloat()
347
+
348
+ private fun sigmoid(x: Float): Float = 1f / (1f + expf(-x))
349
+
350
+ /**
351
+ * to_gaussians: GaussianFixedlenDecoder.to_representation + Gaussian.__init__ for one batch. anchors [L, 3],
352
+ * features [L, 480] (per anchor: xyz 32x3, f_dc 32x3, scaling 32x3, rotation 32x4, opacity 32, offset scale 32),
353
+ * perturbation [32, 3].
354
+ */
355
+ fun toGaussians(anchors: FloatArray, features: FloatArray, rep: GaussianRep, perturbation: FloatArray): Gaussians {
356
+ val L = anchors.size / 3
357
+ val n = L * PER_ANCHOR
358
+ val width = PER_ANCHOR * (3 + 3 + 3 + 4 + 1 + 1)
359
+ require(features.size == L * width) { "features hold ${features.size} values, not $L x $width" }
360
+ val xyz = FloatArray(n * 3)
361
+ val fdc = FloatArray(n * 3)
362
+ val scaling = FloatArray(n * 3)
363
+ val rotation = FloatArray(n * 4)
364
+ val opacity = FloatArray(n)
365
+ // inverse_softplus(scaling_bias), inverse_sigmoid(opacity_bias): float32 tensors in the reference
366
+ val sb = rep.scalingBias
367
+ val scalingOffset = sb + logf(-expm1(-sb.toDouble()).toFloat())
368
+ val ob = rep.opacityBias
369
+ val opacityOffset = logf(ob / (1f - ob))
370
+ val kernel2 = (rep.filterKernelSize * rep.filterKernelSize).toFloat()
371
+ val xyzAt = 0
372
+ val fdcAt = PER_ANCHOR * 3
373
+ val scaleAt = PER_ANCHOR * 6
374
+ val rotAt = PER_ANCHOR * 9
375
+ val opaAt = PER_ANCHOR * 13
376
+ val oscAt = PER_ANCHOR * 14
377
+ for (a in 0 until L) {
378
+ val f = a * width
379
+ for (j in 0 until PER_ANCHOR) {
380
+ val g = a * PER_ANCHOR + j
381
+ val offsetScale = softplus(features[f + oscAt + j] + rep.baseOffsetScale)
382
+ for (c in 0 until 3) {
383
+ var off = features[f + xyzAt + j * 3 + c] * rep.lrXyz + perturbation[j * 3 + c]
384
+ off = tanh(off.toDouble()).toFloat() * 0.5f * rep.perturbeSize * offsetScale
385
+ val raw = off + anchors[a * 3 + c]
386
+ xyz[g * 3 + c] = raw * 1.0f - 0.5f
387
+ fdc[g * 3 + c] = features[f + fdcAt + j * 3 + c] * rep.lrFeaturesDc
388
+ val s = softplus(features[f + scaleAt + j * 3 + c] * rep.lrScaling + scalingOffset)
389
+ scaling[g * 3 + c] = sqrt((s * s + kernel2).toDouble()).toFloat()
390
+ }
391
+ val r0 = features[f + rotAt + j * 4] * rep.lrRotation + 1.0f
392
+ val r1 = features[f + rotAt + j * 4 + 1] * rep.lrRotation + 0.0f
393
+ val r2 = features[f + rotAt + j * 4 + 2] * rep.lrRotation + 0.0f
394
+ val r3 = features[f + rotAt + j * 4 + 3] * rep.lrRotation + 0.0f
395
+ var ss = r0 * r0
396
+ ss += r1 * r1
397
+ ss += r2 * r2
398
+ ss += r3 * r3
399
+ val norm = maxOf(sqrt(ss.toDouble()).toFloat(), 1e-12f)
400
+ rotation[g * 4] = r0 / norm
401
+ rotation[g * 4 + 1] = r1 / norm
402
+ rotation[g * 4 + 2] = r2 / norm
403
+ rotation[g * 4 + 3] = r3 / norm
404
+ opacity[g] = sigmoid(features[f + opaAt + j] * rep.lrOpacity + opacityOffset)
405
+ }
406
+ }
407
+ return Gaussians(n, xyz, fdc, scaling, rotation, opacity)
408
+ }
409
+
410
+ /** ply_rows: idol Gaussian.save_ply's 17 attributes per Gaussian (z-up turned to y-up), float32. */
411
+ fun plyRows(g: Gaussians): FloatArray {
412
+ val n = g.count
413
+ val rows = FloatArray(n * PLY_FLOATS)
414
+ val eps = 1e-6f
415
+ val hi = (1.0 - 1e-6).toFloat()
416
+ // quaternion_multiply([sqrt(.5), sqrt(.5), 0, 0], rotation) in float64 (numpy promotes), rounded at the end
417
+ val aw = sqrt(0.5)
418
+ val ax = sqrt(0.5)
419
+ val ay = 0.0
420
+ val az = 0.0
421
+ for (i in 0 until n) {
422
+ val o = i * PLY_FLOATS
423
+ rows[o] = g.xyz[i * 3]
424
+ rows[o + 1] = -g.xyz[i * 3 + 2]
425
+ rows[o + 2] = g.xyz[i * 3 + 1]
426
+ rows[o + 6] = g.featuresDc[i * 3]
427
+ rows[o + 7] = g.featuresDc[i * 3 + 1]
428
+ rows[o + 8] = g.featuresDc[i * 3 + 2]
429
+ var op = g.opacity[i]
430
+ op = if (op < eps) eps else if (op > hi) hi else op
431
+ rows[o + 9] = logf(op / (1f - op))
432
+ for (c in 0 until 3) rows[o + 10 + c] = logf(maxOf(g.scaling[i * 3 + c], eps))
433
+ val bw = g.rotation[i * 4].toDouble()
434
+ val bx = g.rotation[i * 4 + 1].toDouble()
435
+ val by = g.rotation[i * 4 + 2].toDouble()
436
+ val bz = g.rotation[i * 4 + 3].toDouble()
437
+ rows[o + 13] = (aw * bw - ax * bx - ay * by - az * bz).toFloat()
438
+ rows[o + 14] = (aw * bx + ax * bw + ay * bz - az * by).toFloat()
439
+ rows[o + 15] = (aw * by - ax * bz + ay * bw + az * bx).toFloat()
440
+ rows[o + 16] = (aw * bz + ax * by - ay * bx + az * bw).toFloat()
441
+ }
442
+ return rows
443
+ }
444
+
445
+ /** write_ply: binary little-endian PLY with the 17 float properties (the bytes plyfile writes). */
446
+ fun writePly(rows: FloatArray, file: File) {
447
+ val n = rows.size / PLY_FLOATS
448
+ val header = buildString {
449
+ append("ply\nformat binary_little_endian 1.0\nelement vertex ").append(n).append('\n')
450
+ for (name in PLY_NAMES) append("property float ").append(name).append('\n')
451
+ append("end_header\n")
452
+ }
453
+ val tmp = File(file.parentFile, file.name + ".tmp")
454
+ BufferedOutputStream(FileOutputStream(tmp), 1 shl 20).use { out ->
455
+ out.write(header.toByteArray(Charsets.US_ASCII))
456
+ val chunk = 4096 * PLY_FLOATS
457
+ val buf = ByteBuffer.allocate(chunk * 4).order(ByteOrder.LITTLE_ENDIAN)
458
+ var i = 0
459
+ while (i < rows.size) {
460
+ val m = minOf(chunk, rows.size - i)
461
+ buf.clear()
462
+ buf.asFloatBuffer().put(rows, i, m)
463
+ out.write(buf.array(), 0, m * 4)
464
+ i += m
465
+ }
466
+ }
467
+ if (!tmp.renameTo(file)) {
468
+ tmp.copyTo(file, overwrite = true)
469
+ tmp.delete()
470
+ }
471
+ }
472
+ }
android/app/src/main/java/com/idol/Graphs.kt ADDED
@@ -0,0 +1,339 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.os.SystemClock
4
+ import android.util.Log
5
+ import com.google.ai.edge.litert.Accelerator
6
+ import com.google.ai.edge.litert.CompiledModel
7
+ import com.google.ai.edge.litert.Environment
8
+ import com.google.ai.edge.litert.TensorBuffer
9
+ import org.json.JSONObject
10
+ import java.io.File
11
+
12
+ /**
13
+ * One compiled graph with its I/O buffers, made once per signature and reused. Inputs written with a tag are not
14
+ * written again while the buffer still holds that tag (the RoPE tables, the position embedding, the LoRA gate, the
15
+ * conditioning of the current phase...).
16
+ */
17
+ class Graph(val spec: GraphSpec, private val model: CompiledModel) : AutoCloseable {
18
+ val name: String get() = spec.name
19
+ private val inputs = HashMap<String, List<TensorBuffer>>()
20
+ private val outputs = HashMap<String, List<TensorBuffer>>()
21
+ private val tags = HashMap<String, Array<Any?>>()
22
+ var runs = 0
23
+ private set
24
+
25
+ init {
26
+ try {
27
+ for (sig in spec.signatures.values) {
28
+ inputs[sig.key] = sig.inputs.map { model.createInputBuffer(it.name, sig.key) }
29
+ outputs[sig.key] = sig.outputs.map { model.createOutputBuffer(it.name, sig.key) }
30
+ tags[sig.key] = arrayOfNulls(sig.inputs.size)
31
+ }
32
+ } catch (e: Throwable) {
33
+ close()
34
+ throw e
35
+ }
36
+ }
37
+
38
+ private fun sig(key: String?): SignatureSpec = spec.signature(key)
39
+
40
+ /** Writes [data] to input [index] of signature [key], unless the buffer already holds [tag]. */
41
+ fun write(index: Int, data: FloatArray, tag: Any? = null, key: String? = null) {
42
+ val s = sig(key)
43
+ val t = tags.getValue(s.key)
44
+ if (tag != null && t[index] == tag) return
45
+ val want = s.inputs[index].size
46
+ require(data.size == want) { "$name ${s.key} input $index: ${data.size} values, expected $want" }
47
+ inputs.getValue(s.key)[index].writeFloat(data)
48
+ t[index] = tag
49
+ }
50
+
51
+ /**
52
+ * Runs signature [key]. [overrides] replace input buffers by index (another graph's output buffer: the direct
53
+ * handoff); the graph's own buffer at that index keeps its content and loses its tag.
54
+ */
55
+ fun run(key: String? = null, overrides: Map<Int, TensorBuffer> = emptyMap()) {
56
+ val s = sig(key)
57
+ val ins = inputs.getValue(s.key)
58
+ val outs = outputs.getValue(s.key)
59
+ val inMap = HashMap<String, TensorBuffer>()
60
+ for ((i, t) in s.inputs.withIndex()) inMap[t.name] = overrides[i] ?: ins[i]
61
+ val outMap = HashMap<String, TensorBuffer>()
62
+ for ((i, t) in s.outputs.withIndex()) outMap[t.name] = outs[i]
63
+ model.run(inMap, outMap, s.key)
64
+ runs++
65
+ }
66
+
67
+ /** Output [index] of signature [key] as float32 (waits for the GPU). */
68
+ fun read(index: Int, key: String? = null): FloatArray {
69
+ val s = sig(key)
70
+ val want = s.outputs[index].size
71
+ val v = outputs.getValue(s.key)[index].readFloat()
72
+ check(v.size >= want) { "$name ${s.key} output $index: ${v.size} values, expected $want" }
73
+ return if (v.size == want) v else v.copyOf(want)
74
+ }
75
+
76
+ fun outputBuffer(index: Int, key: String? = null): TensorBuffer = outputs.getValue(sig(key).key)[index]
77
+
78
+ /** Forget the tag of input [index] (its buffer was not written by us). */
79
+ fun untag(index: Int, key: String? = null) {
80
+ tags.getValue(sig(key).key)[index] = null
81
+ }
82
+
83
+ override fun close() {
84
+ for (l in inputs.values) for (b in l) runCatching { b.close() }
85
+ for (l in outputs.values) for (b in l) runCatching { b.close() }
86
+ inputs.clear()
87
+ outputs.clear()
88
+ runCatching { model.close() }
89
+ }
90
+ }
91
+
92
+ /** The memory guard found no graph left to close and the next one does not fit above the reserve. */
93
+ class NotEnoughMemoryException(message: String) : Exception(message)
94
+
95
+ /**
96
+ * The compiled graphs the app keeps, all on the model thread. All 46 graphs do not fit in a phone's memory at once
97
+ * (one denoiser block alone holds about 0.7 GB of GPU memory), so the pool keeps as many as fit above a reserve of
98
+ * `MemAvailable` and, when the next graph does not fit, closes the resident graph whose next use in the run's
99
+ * schedule is farthest away (Belady's rule over the fixed graph order of a run, read cyclically for the next run).
100
+ * Every compile, close and memory shortfall is recorded.
101
+ */
102
+ class GraphPool(
103
+ private val env: Environment,
104
+ private val dir: File,
105
+ private val manifest: Manifest,
106
+ var reserveKb: Long,
107
+ var memoryCheck: Boolean,
108
+ var cpuThreads: Int,
109
+ var gpuPriority: CompiledModel.GpuOptions.Priority?,
110
+ ) : AutoCloseable {
111
+ class CompileRecord(
112
+ val graph: String,
113
+ val seconds: Double,
114
+ val memAvailableBeforeKb: Long,
115
+ val memAvailableAfterKb: Long,
116
+ val phase: String,
117
+ ) {
118
+ fun json(): JSONObject = JSONObject().put("graph", graph).put("seconds", seconds)
119
+ .put("mem_available_before_kb", memAvailableBeforeKb).put("mem_available_after_kb", memAvailableAfterKb)
120
+ .put("phase", phase)
121
+ }
122
+
123
+ private val resident = LinkedHashMap<String, Graph>()
124
+ private val lastUse = HashMap<String, Long>()
125
+ private val pinned = HashSet<String>()
126
+ private val footprintKb = HashMap<String, Long>()
127
+ private var schedule: List<String> = emptyList()
128
+ private var cursor = -1
129
+ private var clock = 0L
130
+
131
+ /** What the compile records are filed under ("preload", "run", "parity", ...). */
132
+ var phase = "run"
133
+ val compiles = ArrayList<CompileRecord>()
134
+ val events = ArrayList<JSONObject>()
135
+
136
+ val residentNames: List<String> get() = resident.keys.toList()
137
+
138
+ fun isResident(name: String) = name in resident
139
+
140
+ /** The order the next run will ask for graphs in (resets the position in it). */
141
+ fun setSchedule(order: List<String>) {
142
+ schedule = order
143
+ cursor = -1
144
+ }
145
+
146
+ fun pin(name: String) {
147
+ pinned.add(name)
148
+ }
149
+
150
+ fun unpin(name: String) {
151
+ pinned.remove(name)
152
+ }
153
+
154
+ /** The graph [name], compiled now if it is not resident (closing others first when memory is short). */
155
+ fun get(name: String): Graph {
156
+ advance(name)
157
+ lastUse[name] = clock++
158
+ resident[name]?.let { return it }
159
+ makeRoom(name, evict = true)
160
+ return compile(name)
161
+ }
162
+
163
+ /** Compiles [name] if it fits without closing another graph; false when it does not fit. */
164
+ fun preload(name: String): Boolean {
165
+ if (name in resident) return true
166
+ val need = estimateKb(name) + reserveKb
167
+ val avail = DeviceInfo.memAvailableKb()
168
+ if (avail in 0 until need) {
169
+ event("preload_stop", name, avail, need)
170
+ return false
171
+ }
172
+ lastUse[name] = clock++
173
+ compile(name)
174
+ return true
175
+ }
176
+
177
+ fun release(name: String) {
178
+ resident.remove(name)?.let {
179
+ it.close()
180
+ event("close", name, DeviceInfo.memAvailableKb(), null)
181
+ }
182
+ }
183
+
184
+ fun releaseIf(predicate: (String) -> Boolean) {
185
+ for (n in resident.keys.filter(predicate)) release(n)
186
+ }
187
+
188
+ fun releaseAll() = releaseIf { true }
189
+
190
+ override fun close() = releaseAll()
191
+
192
+ private fun advance(name: String) {
193
+ if (schedule.isEmpty()) return
194
+ for (i in cursor + 1 until schedule.size) if (schedule[i] == name) {
195
+ cursor = i
196
+ return
197
+ }
198
+ for (i in 0..minOf(cursor, schedule.size - 1)) if (schedule[i] == name) {
199
+ cursor = i
200
+ return
201
+ }
202
+ }
203
+
204
+ /** Distance in the schedule to the next use of [name] after the current position (cyclic); MAX when never. */
205
+ private fun nextUse(name: String): Int {
206
+ if (schedule.isEmpty()) return Int.MAX_VALUE
207
+ for (i in cursor + 1 until schedule.size) if (schedule[i] == name) return i - cursor
208
+ for (i in 0..minOf(cursor, schedule.size - 1)) if (schedule[i] == name) return schedule.size - cursor + i
209
+ return Int.MAX_VALUE
210
+ }
211
+
212
+ private fun victim(exclude: String): String? {
213
+ val candidates = resident.keys.filter { it != exclude && it !in pinned }
214
+ if (candidates.isEmpty()) return null
215
+ if (schedule.isEmpty()) return candidates.minByOrNull { lastUse[it] ?: 0L }
216
+ return candidates.maxWithOrNull(compareBy<String>({ nextUse(it) }, { -(lastUse[it] ?: 0L) }))
217
+ }
218
+
219
+ /**
220
+ * Closes graphs until `MemAvailable` covers the estimate for [name] plus the reserve. With [memoryCheck], fails
221
+ * when nothing is left to close and even the reserve is not there (or the measured size of [name] is not).
222
+ */
223
+ private fun makeRoom(name: String, evict: Boolean) {
224
+ val est = estimateKb(name)
225
+ while (true) {
226
+ val avail = DeviceInfo.memAvailableKb()
227
+ if (avail < 0 || avail >= est + reserveKb) return
228
+ val v = if (evict) victim(name) else null
229
+ if (v == null) {
230
+ event("short", name, avail, est + reserveKb)
231
+ val measured = footprintKb[name]
232
+ val floor = reserveKb + (measured ?: 0L)
233
+ if (memoryCheck && avail < floor) {
234
+ val now = awaitAvailable(floor, MAX_WAIT_SHORT_MS)
235
+ event("waited", name, now, floor)
236
+ if (now in 0 until floor) {
237
+ throw NotEnoughMemoryException(
238
+ "Not enough memory for $name: ${now / 1024} MB available, ${floor / 1024} MB needed " +
239
+ "(reserve ${reserveKb / 1024} MB). Close other apps and try again.",
240
+ )
241
+ }
242
+ }
243
+ return
244
+ }
245
+ event("evict", v, avail, est + reserveKb, "for" to name)
246
+ val freed = footprintKb[v]
247
+ release(v)
248
+ // A closed graph can return its GPU memory a few seconds late (Pixel 11 Pro, PowerVR: MemAvailable rose
249
+ // from 1.6 to 8.9 GB within 8 s of the closes); without this wait the loop closed every resident graph.
250
+ if (freed != null && freed > 0) awaitAvailable(avail + freed / 2, MAX_WAIT_EVICT_MS)
251
+ }
252
+ }
253
+
254
+ /** Polls `MemAvailable` until it reaches [targetKb] or [maxMs] pass; returns the last value read. */
255
+ private fun awaitAvailable(targetKb: Long, maxMs: Long): Long {
256
+ val t0 = SystemClock.elapsedRealtime()
257
+ var now = DeviceInfo.memAvailableKb()
258
+ while (now in 0 until targetKb && SystemClock.elapsedRealtime() - t0 < maxMs) {
259
+ Thread.sleep(POLL_MS)
260
+ now = DeviceInfo.memAvailableKb()
261
+ }
262
+ return now
263
+ }
264
+
265
+ private fun compile(name: String): Graph {
266
+ val spec = manifest.graph(name)
267
+ val file = File(dir, spec.file)
268
+ val options = if (spec.onCpu) {
269
+ CompiledModel.Options(Accelerator.CPU).apply {
270
+ cpuOptions = CompiledModel.CpuOptions(numThreads = cpuThreads)
271
+ }
272
+ } else {
273
+ CompiledModel.Options(Accelerator.GPU).apply {
274
+ gpuOptions = CompiledModel.GpuOptions(
275
+ // one copy of the weights for a graph with several signatures (the octree decoder's buckets)
276
+ constantTensorSharing = if (spec.signatures.size > 1) true else null,
277
+ precision = precisionOf(spec.precision),
278
+ priority = gpuPriority,
279
+ )
280
+ }
281
+ }
282
+ val before = DeviceInfo.memAvailableKb()
283
+ val t0 = SystemClock.elapsedRealtimeNanos()
284
+ val model = CompiledModel.create(file.absolutePath, options, env)
285
+ val graph = try {
286
+ Graph(spec, model)
287
+ } catch (e: Throwable) {
288
+ runCatching { model.close() }
289
+ throw e
290
+ }
291
+ val seconds = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
292
+ val after = DeviceInfo.memAvailableKb()
293
+ if (before >= 0 && after >= 0) footprintKb[name] = maxOf(footprintKb[name] ?: 0L, before - after)
294
+ resident[name] = graph
295
+ compiles.add(CompileRecord(name, seconds, before, after, phase))
296
+ event("compile", name, after, null, "seconds" to seconds)
297
+ Log.i(TAG, "compiled $name in ${"%.2f".format(seconds)} s (${spec.accelerator}, ${spec.precision}), " +
298
+ "MemAvailable ${before / 1024} -> ${after / 1024} MB, resident ${resident.size}")
299
+ return graph
300
+ }
301
+
302
+ /**
303
+ * The memory [name] is expected to take: what it took when it was last compiled, else an estimate from the
304
+ * file (weights at the run precision) and its largest tensor (a transformer block keeps a few dozen of those).
305
+ */
306
+ fun estimateKb(name: String): Long {
307
+ footprintKb[name]?.let { if (it > 0) return it }
308
+ val spec = manifest.graph(name)
309
+ val elementBytes = if (spec.precision == "fp32" || spec.onCpu) 4L else 2L
310
+ val weights = if (elementBytes == 4L) spec.bytes else spec.bytes / 2
311
+ val transformer = name.startsWith("dino_") || name.startsWith("den_") || name == "oct" ||
312
+ name.startsWith("gs_") || name == "bir"
313
+ val activations = spec.maxIoElements * elementBytes * (if (transformer) 26 else 8)
314
+ return (weights + activations) / 1024
315
+ }
316
+
317
+ private fun event(kind: String, graph: String, availKb: Long, needKb: Long?, vararg extra: Pair<String, Any>) {
318
+ val e = JSONObject().put("t_ms", SystemClock.elapsedRealtime()).put("event", kind).put("graph", graph)
319
+ .put("mem_available_kb", availKb).put("resident", resident.size)
320
+ if (needKb != null) e.put("need_kb", needKb)
321
+ for ((k, v) in extra) e.put(k, v)
322
+ events.add(e)
323
+ if (kind != "compile") Log.i(TAG, "memory: $e")
324
+ }
325
+
326
+ companion object {
327
+ const val TAG = "IdolGraphs"
328
+ private const val POLL_MS = 100L
329
+ private const val MAX_WAIT_EVICT_MS = 1_500L
330
+ private const val MAX_WAIT_SHORT_MS = 8_000L
331
+
332
+ fun precisionOf(p: String): CompiledModel.GpuOptions.Precision = when (p) {
333
+ "fp32" -> CompiledModel.GpuOptions.Precision.FP32
334
+ "fp16" -> CompiledModel.GpuOptions.Precision.FP16
335
+ "fp16_with_fp32_accum" -> CompiledModel.GpuOptions.Precision.FP16_WITH_FP32_ACCUM
336
+ else -> CompiledModel.GpuOptions.Precision.DEFAULT
337
+ }
338
+ }
339
+ }
android/app/src/main/java/com/idol/IdolEngine.kt ADDED
@@ -0,0 +1,606 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.os.SystemClock
4
+ import android.util.Log
5
+ import com.google.ai.edge.litert.CompiledModel
6
+ import com.google.ai.edge.litert.Environment
7
+ import com.google.ai.edge.litert.LiteRtException
8
+ import org.json.JSONArray
9
+ import org.json.JSONObject
10
+ import java.io.File
11
+ import java.util.concurrent.atomic.AtomicLong
12
+ import kotlin.math.sqrt
13
+
14
+ /** The stages the Run screen lists, in order. */
15
+ enum class Stage(val label: String) {
16
+ MATTING("Removing background"),
17
+ DINO("Image features (DINOv3)"),
18
+ VAE("Image latent (VAE)"),
19
+ DENOISER("Generating the 3D shape"),
20
+ OCTREE("Placing points"),
21
+ GAUSSIANS("Decoding Gaussians"),
22
+ PLY("Saving the .ply"),
23
+ }
24
+
25
+ /** What the engine tells the screen while it works (called on the model thread). */
26
+ interface Progress {
27
+ fun stage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {}
28
+ fun status(text: String) {}
29
+ fun cutout(composite: Image8) {}
30
+ }
31
+
32
+ class CancelledException : Exception("cancelled")
33
+
34
+ data class EngineOptions(
35
+ /** Compile the graphs before the first run and keep them (true), or per stage, closed after it (false). */
36
+ val preload: Boolean = true,
37
+ /**
38
+ * MemAvailable the memory guard keeps free. MemAvailable counts the page cache of the mapped graph files, and with
39
+ * 1 GB kept a compile burst outran its reclaim: lmkd killed the app on the Galaxy S26 at about 1.9 GB available.
40
+ */
41
+ val reserveMb: Long = 2500,
42
+ /** Fail a run with a message when a graph does not fit even after closing all others (instead of trying). */
43
+ val memoryCheck: Boolean = true,
44
+ /** Pass a graph's output buffer straight to the next graph instead of copying it through the host. */
45
+ val directHandoff: Boolean = false,
46
+ /** "bir" (BiRefNet on the CPU, the author's matting) or "dis" (IS-Net DIS on the GPU). */
47
+ val matting: String = "bir",
48
+ val cpuThreads: Int = 6,
49
+ val gpuPriority: CompiledModel.GpuOptions.Priority? = null,
50
+ val disFloor: Int = Matting.DIS_ALPHA_FLOOR,
51
+ )
52
+
53
+ class RunResult(
54
+ val ply: File,
55
+ val gaussians: Int,
56
+ /** Seconds: matting, preprocess (matting included, as idol_runner times it), dinov3, vae, denoiser, octree,
57
+ * gaussian_decoder, ply, generate (preprocessed pixels ready -> .ply written), all. */
58
+ val times: LinkedHashMap<String, Double>,
59
+ val pre: Preprocessed,
60
+ val compiles: List<GraphPool.CompileRecord>,
61
+ val events: List<JSONObject>,
62
+ val noiseSource: String,
63
+ val handoff: String,
64
+ )
65
+
66
+ /**
67
+ * The phone pipeline, step for step scripts/idol_runner.py: matting -> preprocessing -> DINOv3 (4 graphs) + VAE
68
+ * (6 graphs) -> 5 Euler steps (den_ctx once per LoRA phase, den_in, 24 joint graphs) -> octree (8 levels, row
69
+ * buckets) -> Gaussian decoder (8 graphs) -> .ply. Every graph runs on its own GPU call (the 2 s rule of F0.md
70
+ * finding 2), every call on the one model thread. Create, use and close it on that thread.
71
+ */
72
+ class IdolEngine(val modelDir: File, val manifest: Manifest, val options: EngineOptions) : AutoCloseable {
73
+ private val env: Environment = Environment.create()
74
+ val pool = GraphPool(
75
+ env, modelDir, manifest, options.reserveMb * 1024, options.memoryCheck, options.cpuThreads, options.gpuPriority,
76
+ )
77
+
78
+ private val posEmb = side("den_pos_emb.f32")
79
+ private val ropeCos = side("dino_rope_cos.f32")
80
+ private val ropeSin = side("dino_rope_sin.f32")
81
+ private val perturbation = side("gs_perturbation.f32")
82
+ private val bnMean: FloatArray
83
+ private val bnStd: FloatArray
84
+ private val rep: GaussianRep
85
+ private val gateSlow = Glue.loraGate(true)
86
+ private val gateFast = Glue.loraGate(false)
87
+ private val timeGrid = Glue.timeGrid()
88
+
89
+ /** Set from the screen thread; the model thread stops at the next graph. */
90
+ @Volatile var cancel = false
91
+
92
+ /** Direct handoff, until the first failure (then host copies, and the reason is kept). */
93
+ var direct = options.directHandoff
94
+ private set
95
+ var directFailure: String? = null
96
+ private set
97
+
98
+ private var progress: Progress = object : Progress {}
99
+
100
+ init {
101
+ val bn = JSONObject(File(modelDir, "vae_bn.json").readText())
102
+ val mean = bn.getJSONArray("bn_mean")
103
+ val variance = bn.getJSONArray("bn_var")
104
+ val eps = bn.getDouble("bn_eps").toFloat()
105
+ bnMean = FloatArray(mean.length()) { mean.getDouble(it).toFloat() }
106
+ bnStd = FloatArray(variance.length()) { sqrt((variance.getDouble(it).toFloat() + eps).toDouble()).toFloat() }
107
+ rep = GaussianRep.parse(JSONObject(File(modelDir, "gs_rep.json").readText()))
108
+ }
109
+
110
+ private fun side(name: String): FloatArray {
111
+ val s = requireNotNull(manifest.sideFiles[name]) { "manifest.json has no side file $name" }
112
+ return Fixture.readF32(File(modelDir, name), s.elements.toInt())
113
+ }
114
+
115
+ override fun close() {
116
+ pool.close()
117
+ runCatching { env.close() }
118
+ }
119
+
120
+ // ------------------------------------------------------------------------------------------------ schedule
121
+ // Each chain's graphs in manifest order (manifest.json lists them in pipeline order: dino_0..3, vae_0..3, vae_attn,
122
+ // vae_4, den_joint_00..23, gs_0..7), so a stage split into more or fewer graphs needs a new manifest, not new code.
123
+ private fun chain(prefix: String): List<String> = manifest.graphs.keys.filter { it.startsWith(prefix) }.also {
124
+ require(it.isNotEmpty()) { "manifest.json has no $prefix graphs" }
125
+ }
126
+ private val dinoNames = chain("dino_")
127
+ private val vaeNames = chain("vae_")
128
+ private val jointNames = chain("den_joint_")
129
+ private val gsNames = chain("gs_")
130
+
131
+ /** The order a run asks for graphs in. */
132
+ fun runSchedule(): List<String> {
133
+ val s = ArrayList<String>()
134
+ s.add(options.matting)
135
+ s.addAll(dinoNames)
136
+ s.addAll(vaeNames)
137
+ var phase: Boolean? = null
138
+ for (i in 0 until timeGrid.size - 1) {
139
+ val slow = timeGrid[i] >= Glue.PHASE_THRESHOLD
140
+ if (slow != phase) s.add("den_ctx")
141
+ phase = slow
142
+ s.add("den_in")
143
+ s.addAll(jointNames)
144
+ }
145
+ repeat(Glue.MAX_LEVEL) { s.add("oct") }
146
+ s.addAll(gsNames)
147
+ return s
148
+ }
149
+
150
+ /** The GPU graphs of a run in first-use order (the CPU matting graph is built for each matting and closed). */
151
+ fun preloadOrder(): List<String> = runSchedule().distinct().filter { !manifest.graph(it).onCpu }
152
+
153
+ /**
154
+ * preload = true: compiles the graphs before the first run, in first-use order, as many as fit above the reserve.
155
+ * Returns the load record (seconds per graph and the total).
156
+ */
157
+ fun preload(progress: Progress): JSONObject {
158
+ cancel = false
159
+ val names = preloadOrder()
160
+ pool.phase = "preload"
161
+ pool.setSchedule(runSchedule())
162
+ val first = pool.compiles.size
163
+ val t0 = SystemClock.elapsedRealtimeNanos()
164
+ var loaded = 0
165
+ var stoppedAt: String? = null
166
+ try {
167
+ for (n in names) {
168
+ if (cancel) throw CancelledException()
169
+ progress.status("Loading models ${loaded + 1}/${names.size}")
170
+ if (!pool.preload(n)) {
171
+ stoppedAt = n
172
+ break
173
+ }
174
+ loaded++
175
+ }
176
+ } finally {
177
+ pool.phase = "run"
178
+ }
179
+ val total = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
180
+ progress.status(
181
+ if (stoppedAt == null) "Models loaded ($loaded/${names.size})"
182
+ else "Loaded $loaded/${names.size} models; the rest load during the run (memory)",
183
+ )
184
+ val graphs = JSONArray()
185
+ for (r in pool.compiles.subList(first, pool.compiles.size)) graphs.put(r.json())
186
+ return JSONObject().put("graphs", graphs).put("loaded", loaded).put("of", names.size)
187
+ .put("stopped_at", stoppedAt ?: JSONObject.NULL).put("total_s", total)
188
+ .put("compile_s", pool.compiles.subList(first, pool.compiles.size).sumOf { it.seconds })
189
+ .put("resident", JSONArray(pool.residentNames))
190
+ }
191
+
192
+ // ------------------------------------------------------------------------------------------------ the run
193
+ /** A picture (or a fixture's raw image) -> .ply at [outPly]. */
194
+ fun generate(image: Image8, mode: String, noise: Noise, outPly: File, progress: Progress): RunResult {
195
+ cancel = false
196
+ this.progress = progress
197
+ val firstCompile = pool.compiles.size
198
+ val firstEvent = pool.events.size
199
+ pool.phase = "run"
200
+ pool.setSchedule(runSchedule())
201
+ val times = LinkedHashMap<String, Double>()
202
+ val tAll = now()
203
+
204
+ progress.stage(Stage.MATTING, null, null, false)
205
+ var mattingS = 0.0
206
+ val tPre = now()
207
+ val pre = Preprocess.run(image, mode) { rgb ->
208
+ val t = now()
209
+ val a = alpha(rgb)
210
+ mattingS = since(t)
211
+ a
212
+ }
213
+ times["matting"] = mattingS
214
+ times["preprocess"] = since(tPre)
215
+ pool.release(options.matting)
216
+ progress.cutout(pre.composite)
217
+ progress.stage(Stage.MATTING, null, times["preprocess"], true)
218
+ val pixels = Preprocess.pixels(pre.composite)
219
+
220
+ val tGen = now()
221
+ progress.stage(Stage.DINO, null, null, false)
222
+ var t = now()
223
+ val f1 = dinov3(pixels)
224
+ times["dinov3"] = since(t)
225
+ progress.stage(Stage.DINO, null, times["dinov3"], true)
226
+ if (!options.preload) pool.releaseIf { it.startsWith("dino_") }
227
+
228
+ progress.stage(Stage.VAE, null, null, false)
229
+ t = now()
230
+ val moments = vaeMoments(pixels)
231
+ times["vae"] = since(t)
232
+ val f2 = cond2(Glue.vaeTokens(moments, noise.vae, bnMean, bnStd))
233
+ progress.stage(Stage.VAE, null, times["vae"], true)
234
+ if (!options.preload) pool.releaseIf { it.startsWith("vae_") }
235
+
236
+ val tDen = now()
237
+ val latent = sample(noise, f1, f2, times)
238
+ progress.stage(Stage.DENOISER, null, since(tDen), true)
239
+ if (!options.preload) pool.releaseIf { it.startsWith("den_") }
240
+
241
+ progress.stage(Stage.OCTREE, null, null, false)
242
+ t = now()
243
+ val latentTag = nextTag()
244
+ val anchors = Glue.octreeDescend({ cells, rows, level ->
245
+ progress.stage(Stage.OCTREE, "level $level of ${Glue.MAX_LEVEL}", null, false)
246
+ Glue.softmax8(octreeLogits(cells, rows, level, latent, latentTag), rows)
247
+ }, noise.pool, noise.leaf)
248
+ times["octree"] = since(t)
249
+ progress.stage(Stage.OCTREE, null, times["octree"], true)
250
+ if (!options.preload) pool.release("oct")
251
+
252
+ progress.stage(Stage.GAUSSIANS, null, null, false)
253
+ t = now()
254
+ val features = gaussianFeatures(anchors, latent, latentTag)
255
+ times["gaussian_decoder"] = since(t)
256
+ if (!options.preload) pool.releaseIf { it.startsWith("gs_") }
257
+ val g = Glue.toGaussians(anchors, features, rep, perturbation)
258
+ progress.stage(Stage.GAUSSIANS, null, since(t), true)
259
+
260
+ progress.stage(Stage.PLY, null, null, false)
261
+ t = now()
262
+ Glue.writePly(Glue.plyRows(g), outPly)
263
+ times["ply"] = since(t)
264
+ progress.stage(Stage.PLY, null, times["ply"], true)
265
+ times["generate"] = since(tGen)
266
+ times["all"] = since(tAll)
267
+ Log.i(TAG, "run done: " + times.entries.joinToString { "${it.key} ${"%.2f".format(it.value)}" })
268
+ return RunResult(
269
+ outPly, g.count, times, pre,
270
+ pool.compiles.subList(firstCompile, pool.compiles.size).toList(),
271
+ pool.events.subList(firstEvent, pool.events.size).toList(),
272
+ noise.source, handoffName(),
273
+ )
274
+ }
275
+
276
+ fun handoffName(): String = when {
277
+ direct -> "direct"
278
+ options.directHandoff -> "host (direct failed: $directFailure)"
279
+ else -> "host"
280
+ }
281
+
282
+ /** feature2 = 5 zero tokens + the 4,096 VAE tokens: [1, 4101, 128]. */
283
+ private fun cond2(tokens: FloatArray): FloatArray {
284
+ val f2 = FloatArray(Glue.COND_TOKENS * 128)
285
+ System.arraycopy(tokens, 0, f2, 5 * 128, tokens.size)
286
+ return f2
287
+ }
288
+
289
+ // ------------------------------------------------------------------------------------------------ stages
290
+ private fun graph(name: String): Graph {
291
+ if (cancel) throw CancelledException()
292
+ if (!pool.isResident(name)) progress.status("Loading $name")
293
+ val g = pool.get(name)
294
+ progress.status("")
295
+ return g
296
+ }
297
+
298
+ /** The matting graph's uint8 alpha for the resized RGB picture; the graph is closed right after. */
299
+ private fun alpha(rgb: Image8): ByteArray {
300
+ val m = options.matting
301
+ val input = if (m == "bir") Matting.birefnetInput(rgb) else Matting.disInput(rgb)
302
+ try {
303
+ val g = graph(m)
304
+ g.write(0, input)
305
+ g.run()
306
+ val map = g.read(0)
307
+ pool.release(m)
308
+ return if (m == "bir") Matting.birefnetAlpha(map, rgb.width, rgb.height)
309
+ else Matting.disAlpha(map, rgb.width, rgb.height, options.disFloor)
310
+ } finally {
311
+ pool.release(m)
312
+ }
313
+ }
314
+
315
+ private class Link(val from: Graph, val fromOut: Int, val to: Graph, val toIn: Int, val direct: Boolean)
316
+
317
+ /**
318
+ * The next graph [toName] of a chain, fed [from]'s output [fromOut] at input [toIn]: copied through the host now
319
+ * (the read waits for [from]'s GPU work), or handed over as the buffer itself at [run] with direct handoff.
320
+ */
321
+ private fun link(from: Graph, fromOut: Int, toName: String, toIn: Int = 0): Link {
322
+ if (direct) {
323
+ pool.pin(from.name)
324
+ try {
325
+ return Link(from, fromOut, graph(toName), toIn, true)
326
+ } catch (e: Throwable) {
327
+ pool.unpin(from.name)
328
+ throw e
329
+ }
330
+ }
331
+ val data = from.read(fromOut)
332
+ val to = graph(toName)
333
+ to.write(toIn, data)
334
+ return Link(from, fromOut, to, toIn, false)
335
+ }
336
+
337
+ private fun run(l: Link) {
338
+ if (!l.direct) {
339
+ l.to.run()
340
+ return
341
+ }
342
+ try {
343
+ l.to.run(overrides = mapOf(l.toIn to l.from.outputBuffer(l.fromOut)))
344
+ l.to.untag(l.toIn)
345
+ } catch (e: LiteRtException) {
346
+ Log.w(TAG, "direct handoff ${l.from.name} -> ${l.to.name} failed, host copies from now on", e)
347
+ direct = false
348
+ directFailure = "${l.from.name} -> ${l.to.name}: ${e.message}"
349
+ l.to.write(l.toIn, l.from.read(l.fromOut))
350
+ l.to.run()
351
+ } finally {
352
+ pool.unpin(l.from.name)
353
+ }
354
+ }
355
+
356
+ private fun dinov3(pixels: FloatArray): FloatArray {
357
+ var g = graph(dinoNames[0])
358
+ g.write(0, pixels)
359
+ g.write(1, ropeCos, "rope_cos")
360
+ g.write(2, ropeSin, "rope_sin")
361
+ g.run()
362
+ for (name in dinoNames.drop(1)) {
363
+ val l = link(g, 0, name)
364
+ l.to.write(1, ropeCos, "rope_cos")
365
+ l.to.write(2, ropeSin, "rope_sin")
366
+ run(l)
367
+ g = l.to
368
+ }
369
+ return g.read(0)
370
+ }
371
+
372
+ private fun vaeMoments(pixels: FloatArray): FloatArray {
373
+ var g = graph(vaeNames[0])
374
+ g.write(0, pixels)
375
+ g.run()
376
+ for (name in vaeNames.drop(1)) {
377
+ val l = link(g, 0, name)
378
+ run(l)
379
+ g = l.to
380
+ }
381
+ return g.read(0)
382
+ }
383
+
384
+ /** The conditioning of one LoRA phase (den_ctx), kept for the steps of that phase. */
385
+ private class Ctx(val h: FloatArray, val tag: Long)
386
+
387
+ /** sample: 5 Euler steps from the shared noise -> the final latent [1, 8192, 16]. */
388
+ private fun sample(noise: Noise, f1: FloatArray, f2: FloatArray, times: MutableMap<String, Double>): FloatArray {
389
+ val xLat = noise.latent.copyOf()
390
+ val xCam = noise.camera.copyOf()
391
+ val ctx = HashMap<Boolean, Ctx>()
392
+ val condTag = nextTag()
393
+ var den = 0.0
394
+ val steps = timeGrid.size - 1
395
+ for (i in 0 until steps) {
396
+ progress.stage(Stage.DENOISER, "step ${i + 1} of $steps", null, false)
397
+ val t0 = now()
398
+ val (v, vc) = velocity(xLat, xCam, timeGrid[i], f1, f2, condTag, ctx)
399
+ den += since(t0)
400
+ val dt = (timeGrid[i] - timeGrid[i + 1]).toFloat()
401
+ for (k in xLat.indices) xLat[k] = xLat[k] - v[k] * dt
402
+ for (k in xCam.indices) xCam[k] = xCam[k] - vc[k] * dt
403
+ }
404
+ times["denoiser"] = den
405
+ return xLat
406
+ }
407
+
408
+ /** velocity: one denoiser call at time t -> (latent velocity [1, 8192, 16], camera velocity [1, 1, 5]). */
409
+ private fun velocity(
410
+ xLat: FloatArray,
411
+ xCam: FloatArray,
412
+ t: Double,
413
+ f1: FloatArray,
414
+ f2: FloatArray,
415
+ condTag: Long,
416
+ ctx: HashMap<Boolean, Ctx>,
417
+ ): Pair<FloatArray, FloatArray> {
418
+ val slow = t >= Glue.PHASE_THRESHOLD
419
+ val gate = if (slow) gateSlow else gateFast
420
+ val gateTag = if (slow) "gate_slow" else "gate_fast"
421
+ val c = ctx.getOrPut(slow) {
422
+ val g = graph("den_ctx")
423
+ g.write(0, f1, "f1_$condTag")
424
+ g.write(1, f2, "f2_$condTag")
425
+ g.write(2, gate, gateTag)
426
+ g.run()
427
+ Ctx(g.read(0), nextTag())
428
+ }
429
+ val din = graph("den_in")
430
+ din.write(0, xLat)
431
+ din.write(1, xCam)
432
+ din.write(2, Glue.sinusoid(1000 * t))
433
+ din.write(3, posEmb, "pos_emb")
434
+ din.write(4, gate, gateTag)
435
+ din.run()
436
+ val hCam = din.read(1)
437
+ val tMod = din.read(2)
438
+ val tEmb = din.read(3)
439
+ val tModTag = nextTag()
440
+ var prev = din
441
+ for ((k, name) in jointNames.withIndex()) {
442
+ val l = link(prev, 0, name)
443
+ val j = l.to
444
+ when (k) {
445
+ 0 -> {
446
+ j.write(1, c.h, c.tag)
447
+ j.write(2, hCam)
448
+ j.write(3, tMod, tModTag)
449
+ j.write(4, gate, gateTag)
450
+ }
451
+ jointNames.size - 1 -> {
452
+ j.write(1, tMod, tModTag)
453
+ j.write(2, tEmb)
454
+ j.write(3, gate, gateTag)
455
+ }
456
+ else -> {
457
+ j.write(1, tMod, tModTag)
458
+ j.write(2, gate, gateTag)
459
+ }
460
+ }
461
+ run(l)
462
+ prev = j
463
+ }
464
+ return prev.read(0) to prev.read(1)
465
+ }
466
+
467
+ /** octree_logits: the decoder's child logits for [n] cells at [level], rows padded to the bucket -> [n, 8]. */
468
+ private fun octreeLogits(cells: FloatArray, n: Int, level: Int, latent: FloatArray, latentTag: Any?): FloatArray {
469
+ val b = Glue.OCT_BUCKETS.first { n <= it }
470
+ val key = "L$b"
471
+ val c = FloatArray(b * 3)
472
+ System.arraycopy(cells, 0, c, 0, n * 3)
473
+ val pe = Glue.pointPe(c, b)
474
+ val lf = Glue.sinusoid((1 shl level).toDouble(), maxPeriod = 1024.0, twoPi = true)
475
+ val g = graph("oct")
476
+ g.write(0, c, key = key)
477
+ g.write(1, pe, key = key)
478
+ g.write(2, lf, key = key)
479
+ g.write(3, latent, latentTag, key)
480
+ g.run(key)
481
+ return g.read(0, key).copyOf(n * 8)
482
+ }
483
+
484
+ /** The Gaussian decoder's 8 graphs: anchors [8192, 3] + latent -> features [8192, 480]. */
485
+ private fun gaussianFeatures(anchors: FloatArray, latent: FloatArray, latentTag: Any?): FloatArray {
486
+ val pe = Glue.pointPe(anchors, Glue.NUM_POINTS)
487
+ var g = graph(gsNames[0])
488
+ g.write(0, anchors)
489
+ g.write(1, pe)
490
+ g.write(2, latent, latentTag)
491
+ g.run()
492
+ for (name in gsNames.drop(1)) {
493
+ val l = link(g, 0, name)
494
+ l.to.write(1, latent, latentTag)
495
+ run(l)
496
+ g = l.to
497
+ }
498
+ return g.read(0)
499
+ }
500
+
501
+ // ------------------------------------------------------------------------------------------------ parity
502
+ /**
503
+ * CONTRACT.md line (a) on the phone: each part gets the fixture's reference input for it (no chaining) and its
504
+ * output is written as `<id>.<name>.f32` (little-endian float32) in [out]; the matting alpha as `<id>.alpha.u8`.
505
+ */
506
+ fun parity(fx: Fixture, out: File, id: String, progress: Progress): JSONObject {
507
+ cancel = false
508
+ this.progress = progress
509
+ pool.phase = "parity"
510
+ pool.setSchedule(emptyList())
511
+ val arrays = JSONObject()
512
+ val seconds = JSONObject()
513
+ fun dump(name: String, data: FloatArray, shape: IntArray) {
514
+ Fixture.writeF32(File(out, "$id.$name.f32"), data)
515
+ arrays.put(name, JSONObject().put("file", "$id.$name.f32").put("shape", JSONArray(shape.toList())))
516
+ }
517
+ val result = JSONObject().put("id", id).put("fixture", fx.dir.name).put("matting", options.matting)
518
+
519
+ progress.stage(Stage.MATTING, null, null, false)
520
+ var t = now()
521
+ val photo = Photo.fromFile(fx.image)
522
+ val pre = Preprocess.run(photo.image, fx.mode) { alpha(it) }
523
+ File(out, "$id.alpha.u8").writeBytes(pre.alphaRaw)
524
+ result.put("alpha", JSONObject().put("file", "$id.alpha.u8").put("width", pre.alphaWidth)
525
+ .put("height", pre.alphaHeight).put("box", JSONArray(pre.box.toList())))
526
+ seconds.put("matting", since(t))
527
+ progress.stage(Stage.MATTING, null, since(t), true)
528
+
529
+ val pixels = fx.floats("pixels")
530
+ progress.stage(Stage.DINO, null, null, false)
531
+ t = now()
532
+ dump("feature1", dinov3(pixels), intArrayOf(1, Glue.COND_TOKENS, 1280))
533
+ seconds.put("dinov3", since(t))
534
+ pool.releaseIf { it.startsWith("dino_") }
535
+ progress.stage(Stage.DINO, null, since(t), true)
536
+
537
+ progress.stage(Stage.VAE, null, null, false)
538
+ t = now()
539
+ val moments = vaeMoments(pixels)
540
+ dump("vae_moments", moments, intArrayOf(1, 64, 128, 128))
541
+ dump("vae_tokens", Glue.vaeTokens(moments, fx.floats("vae_noise"), bnMean, bnStd), intArrayOf(1, 4096, 128))
542
+ seconds.put("vae", since(t))
543
+ pool.releaseIf { it.startsWith("vae_") }
544
+ progress.stage(Stage.VAE, null, since(t), true)
545
+
546
+ t = now()
547
+ val f1 = fx.floats("feature1")
548
+ val f2 = fx.floats("feature2")
549
+ // t comes from the engine's float64 grid, as the reference passes float(ts[i]) (1000 t rounds differently from
550
+ // the fixture's float32 copy); the fixture's grid only confirms it is the same schedule
551
+ val tg = fx.floats("t_grid")
552
+ result.put("t_grid_matches_fixture", tg.size == timeGrid.size && tg.indices.all { tg[it] == timeGrid[it].toFloat() })
553
+ val ctx = HashMap<Boolean, Ctx>()
554
+ val condTag = nextTag()
555
+ for (i in 0 until timeGrid.size - 1) {
556
+ progress.stage(Stage.DENOISER, "step ${i + 1} of ${timeGrid.size - 1}", null, false)
557
+ val (v, vc) = velocity(fx.floats("x_latent_$i"), fx.floats("x_camera_$i"), timeGrid[i], f1, f2, condTag, ctx)
558
+ dump("v_latent_$i", v, intArrayOf(1, Glue.NUM_POINTS, Glue.IN_CH))
559
+ dump("v_camera_$i", vc, intArrayOf(1, 1, Glue.CAM_CH))
560
+ }
561
+ seconds.put("denoiser", since(t))
562
+ pool.releaseIf { it.startsWith("den_") }
563
+ progress.stage(Stage.DENOISER, null, since(t), true)
564
+
565
+ t = now()
566
+ val latent = fx.floats("latent")
567
+ val latentTag = nextTag()
568
+ for (level in 1..Glue.MAX_LEVEL) {
569
+ progress.stage(Stage.OCTREE, "level $level of ${Glue.MAX_LEVEL}", null, false)
570
+ val cells = fx.floats("oct_cells_$level")
571
+ val n = cells.size / 3
572
+ dump("oct_logits_$level", octreeLogits(cells, n, level, latent, latentTag), intArrayOf(1, n, 8))
573
+ }
574
+ seconds.put("octree", since(t))
575
+ pool.release("oct")
576
+ progress.stage(Stage.OCTREE, null, since(t), true)
577
+
578
+ progress.stage(Stage.GAUSSIANS, null, null, false)
579
+ t = now()
580
+ val anchors = fx.floats("anchors")
581
+ val feats = gaussianFeatures(anchors, latent, latentTag)
582
+ seconds.put("gaussian_decoder", since(t))
583
+ pool.releaseIf { it.startsWith("gs_") }
584
+ dump("gs_features", feats, intArrayOf(1, Glue.NUM_POINTS, 480))
585
+ val g = Glue.toGaussians(anchors, feats, rep, perturbation)
586
+ dump("g_xyz", g.xyz, intArrayOf(g.count, 3))
587
+ dump("g_features_dc", g.featuresDc, intArrayOf(g.count, 1, 3))
588
+ dump("g_scaling", g.scaling, intArrayOf(g.count, 3))
589
+ dump("g_rotation", g.rotation, intArrayOf(g.count, 4))
590
+ dump("g_opacity", g.opacity, intArrayOf(g.count, 1))
591
+ progress.stage(Stage.GAUSSIANS, null, since(t), true)
592
+ pool.phase = "run"
593
+ return result.put("arrays", arrays).put("seconds", seconds).put("handoff", handoffName())
594
+ }
595
+
596
+ companion object {
597
+ const val TAG = "IdolEngine"
598
+ private val tags = AtomicLong(1)
599
+
600
+ fun nextTag(): Long = tags.getAndIncrement()
601
+
602
+ private fun now() = SystemClock.elapsedRealtimeNanos()
603
+
604
+ private fun since(t0: Long) = (SystemClock.elapsedRealtimeNanos() - t0) / 1e9
605
+ }
606
+ }
android/app/src/main/java/com/idol/MainActivity.kt ADDED
@@ -0,0 +1,581 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.Manifest
4
+ import android.content.ActivityNotFoundException
5
+ import android.content.ComponentCallbacks2
6
+ import android.content.Intent
7
+ import android.content.pm.ApplicationInfo
8
+ import android.content.pm.PackageManager
9
+ import android.net.Uri
10
+ import android.os.Build
11
+ import android.os.Bundle
12
+ import android.os.Handler
13
+ import android.os.Looper
14
+ import android.os.SystemClock
15
+ import android.provider.OpenableColumns
16
+ import android.util.Log
17
+ import android.util.TypedValue
18
+ import android.view.View
19
+ import android.view.WindowManager
20
+ import android.webkit.WebView
21
+ import android.widget.Button
22
+ import android.widget.CheckBox
23
+ import android.widget.FrameLayout
24
+ import android.widget.ImageView
25
+ import android.widget.LinearLayout
26
+ import android.widget.RadioGroup
27
+ import android.widget.TextView
28
+ import androidx.activity.ComponentActivity
29
+ import androidx.activity.OnBackPressedCallback
30
+ import androidx.activity.result.PickVisualMediaRequest
31
+ import androidx.activity.result.contract.ActivityResultContracts
32
+ import androidx.core.content.FileProvider
33
+ import androidx.core.view.ViewCompat
34
+ import androidx.core.view.WindowInsetsCompat
35
+ import org.json.JSONObject
36
+ import java.io.File
37
+ import java.text.SimpleDateFormat
38
+ import java.util.Date
39
+ import java.util.Locale
40
+ import java.util.concurrent.Callable
41
+ import java.util.concurrent.CountDownLatch
42
+ import java.util.concurrent.ExecutorService
43
+ import java.util.concurrent.Executors
44
+
45
+ /** How a run ended, for the screen and the autorun. */
46
+ sealed class Outcome {
47
+ class Done(val result: RunResult) : Outcome()
48
+ class NoObject(val reason: String) : Outcome()
49
+ class Missing(val problems: List<ModelFiles.Problem>) : Outcome()
50
+ class Failed(val kind: String, val message: String) : Outcome()
51
+ }
52
+
53
+ /** The model state of the process: one model thread and one engine, kept across activity instances. */
54
+ object Idol {
55
+ /** Every LiteRT call (compile, run, close) happens on this one thread. */
56
+ val model: ExecutorService = Executors.newSingleThreadExecutor { r -> Thread(r, "IdolModel") }
57
+ @Volatile var engine: IdolEngine? = null
58
+ @Volatile var problems: List<ModelFiles.Problem>? = null
59
+ @Volatile var started = false
60
+ }
61
+
62
+ /**
63
+ * Home (pick or take a photo) -> Run (the stages with their seconds) -> Result (the splat viewer). Every run, from a
64
+ * button or from the debug autorun ([AutoRun]), goes through [startRun]; the work is on the model thread, never on
65
+ * the UI thread.
66
+ */
67
+ class MainActivity : ComponentActivity() {
68
+ private lateinit var home: View
69
+ private lateinit var runScreen: View
70
+ private lateinit var resultScreen: View
71
+ private lateinit var modeGroup: RadioGroup
72
+ private lateinit var disCheck: CheckBox
73
+ private lateinit var homeMessage: TextView
74
+ private lateinit var homeStatus: TextView
75
+ private lateinit var picture: ImageView
76
+ private lateinit var cutout: ImageView
77
+ private lateinit var stageList: LinearLayout
78
+ private lateinit var totalText: TextView
79
+ private lateinit var runStatus: TextView
80
+ private lateinit var viewerBox: FrameLayout
81
+ private var web: WebView? = null
82
+
83
+ private val ui = Handler(Looper.getMainLooper())
84
+ private val decoder = Executors.newSingleThreadExecutor { r -> Thread(r, "IdolDecode") }
85
+ private val stageRows = ArrayList<TextView>()
86
+
87
+ private class StageState {
88
+ var detail: String? = null
89
+ var seconds: Double? = null
90
+ var startedAt = 0L
91
+ var running = false
92
+ var done = false
93
+ }
94
+
95
+ private val stages = Array(Stage.entries.size) { StageState() }
96
+ @Volatile private var running = false
97
+ @Volatile var autorunActive = false
98
+ private set
99
+ private var lastPly: File? = null
100
+ private var deviceLine = ""
101
+
102
+ /** Engine progress, posted to the screen. */
103
+ val progress = object : Progress {
104
+ override fun stage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {
105
+ ui.post { onStage(stage, detail, seconds, done) }
106
+ }
107
+
108
+ override fun status(text: String) {
109
+ ui.post {
110
+ runStatus.text = text
111
+ homeStatus.text = if (text.isEmpty()) deviceLine else "$text\n$deviceLine"
112
+ }
113
+ }
114
+
115
+ override fun cutout(composite: Image8) {
116
+ val bmp = Photo.displayBitmap(composite, 512)
117
+ ui.post {
118
+ cutout.setImageBitmap(bmp)
119
+ cutout.visibility = View.VISIBLE
120
+ }
121
+ }
122
+ }
123
+
124
+ private val pickPhoto = registerForActivityResult(ActivityResultContracts.PickVisualMedia()) { uri ->
125
+ if (uri != null) startFromPicture { Photo.fromUri(contentResolver, uri, displayName(uri)) }
126
+ }
127
+
128
+ private val takePhoto = registerForActivityResult(ActivityResultContracts.TakePicture()) { ok ->
129
+ // the camera app can take enough memory for Android to stop this process while it is open; the activity is
130
+ // then created again for the result, so the photo's path is a fixed one, not a field
131
+ val file = cameraFile()
132
+ if (ok && file.isFile) startFromPicture { Photo.fromFile(file) }
133
+ }
134
+
135
+ private val cameraPermission = registerForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
136
+ if (granted) launchCamera() else showHome(getString(R.string.camera_denied))
137
+ }
138
+
139
+ override fun onCreate(savedInstanceState: Bundle?) {
140
+ super.onCreate(savedInstanceState)
141
+ if (isDebuggable()) {
142
+ // the autorun drives a phone nobody unlocked: stay in front (cpuset top-app) with the screen on
143
+ window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
144
+ if (Build.VERSION.SDK_INT >= 27) {
145
+ setShowWhenLocked(true)
146
+ setTurnScreenOn(true)
147
+ }
148
+ }
149
+ setContentView(R.layout.activity_main)
150
+ // targetSdk 35 draws edge to edge: keep the screens clear of the status and navigation bars
151
+ val root = findViewById<View>(R.id.root)
152
+ ViewCompat.setOnApplyWindowInsetsListener(root) { v, insets ->
153
+ val bars = insets.getInsets(WindowInsetsCompat.Type.systemBars())
154
+ v.setPadding(bars.left, bars.top, bars.right, bars.bottom)
155
+ insets
156
+ }
157
+ home = findViewById(R.id.home)
158
+ runScreen = findViewById(R.id.run)
159
+ resultScreen = findViewById(R.id.result)
160
+ modeGroup = findViewById(R.id.mode)
161
+ disCheck = findViewById(R.id.dis)
162
+ homeMessage = findViewById(R.id.home_message)
163
+ homeStatus = findViewById(R.id.home_status)
164
+ picture = findViewById(R.id.picture)
165
+ cutout = findViewById(R.id.cutout)
166
+ stageList = findViewById(R.id.stages)
167
+ totalText = findViewById(R.id.total)
168
+ runStatus = findViewById(R.id.run_status)
169
+ viewerBox = findViewById(R.id.viewer)
170
+
171
+ for (s in Stage.entries) {
172
+ val tv = TextView(this)
173
+ tv.setTextSize(TypedValue.COMPLEX_UNIT_SP, 16f)
174
+ tv.setPadding(0, dp(6), 0, dp(6))
175
+ stageList.addView(tv)
176
+ stageRows.add(tv)
177
+ }
178
+
179
+ findViewById<Button>(R.id.pick).setOnClickListener {
180
+ pickPhoto.launch(PickVisualMediaRequest(ActivityResultContracts.PickVisualMedia.ImageOnly))
181
+ }
182
+ findViewById<Button>(R.id.camera).setOnClickListener {
183
+ if (checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
184
+ launchCamera()
185
+ } else {
186
+ cameraPermission.launch(Manifest.permission.CAMERA)
187
+ }
188
+ }
189
+ findViewById<Button>(R.id.share).setOnClickListener { share() }
190
+ findViewById<Button>(R.id.new_run).setOnClickListener { showHome(null) }
191
+
192
+ val prefs = getSharedPreferences("settings", MODE_PRIVATE)
193
+ disCheck.isChecked = prefs.getBoolean("dis", false)
194
+ disCheck.setOnCheckedChangeListener { _, checked ->
195
+ prefs.edit().putBoolean("dis", checked).apply()
196
+ if (!running && !autorunActive) Idol.model.execute { setup(uiOptions(), preload = true) }
197
+ }
198
+
199
+ onBackPressedDispatcher.addCallback(this, object : OnBackPressedCallback(true) {
200
+ override fun handleOnBackPressed() = onBack()
201
+ })
202
+
203
+ deviceLine = "${DeviceInfo.displayName()} · LiteRT ${BuildConfig.LITERT_VERSION}"
204
+ homeStatus.text = deviceLine
205
+ showHome(null)
206
+ val autorun = isDebuggable() && intent?.getStringExtra("autorun") != null
207
+ if (!Idol.started) {
208
+ Idol.started = true
209
+ // an autorun builds its own engine with its plan's options: no preload here then
210
+ val opts = uiOptions()
211
+ Idol.model.execute { setup(opts, preload = !autorun) }
212
+ } else {
213
+ Idol.problems?.takeIf { it.isNotEmpty() }?.let { showMissing(it) }
214
+ }
215
+ handleIntent(intent)
216
+ }
217
+
218
+ override fun onNewIntent(intent: Intent) {
219
+ super.onNewIntent(intent)
220
+ handleIntent(intent)
221
+ }
222
+
223
+ private fun handleIntent(intent: Intent?) {
224
+ val plan = intent?.getStringExtra("autorun") ?: return
225
+ if (!isDebuggable() || autorunActive) return
226
+ autorunActive = true
227
+ AutoRun(this, File(filesDir, plan), File(filesDir, intent.getStringExtra("out") ?: "runs/autorun")).start()
228
+ }
229
+
230
+ fun autorunFinished() {
231
+ ui.post { autorunActive = false }
232
+ }
233
+
234
+ override fun onTrimMemory(level: Int) {
235
+ super.onTrimMemory(level)
236
+ // In the background (the camera app in front, for one) the resident graphs make this process the first one
237
+ // Android stops: give their memory back while nothing runs; the next run compiles them again.
238
+ if (level >= ComponentCallbacks2.TRIM_MEMORY_BACKGROUND && !running && !autorunActive) {
239
+ Idol.model.execute { if (!running) Idol.engine?.pool?.releaseAll() }
240
+ }
241
+ }
242
+
243
+ override fun onDestroy() {
244
+ ResultViewer.destroy(viewerBox, web)
245
+ web = null
246
+ if (isFinishing && !running && !autorunActive) {
247
+ Idol.model.execute {
248
+ Idol.engine?.close()
249
+ Idol.engine = null
250
+ Idol.started = false
251
+ }
252
+ }
253
+ super.onDestroy()
254
+ }
255
+
256
+ private fun isDebuggable() = applicationInfo.flags and ApplicationInfo.FLAG_DEBUGGABLE != 0
257
+
258
+ private fun dp(v: Int) = (v * resources.displayMetrics.density + 0.5f).toInt()
259
+
260
+ // ------------------------------------------------------------------------------------------------ model thread
261
+ /** The options of a run from the buttons: preload on, the matting the setting picks (else the manifest's). */
262
+ private fun uiOptions(): EngineOptions {
263
+ val matting = if (getSharedPreferences("settings", MODE_PRIVATE).getBoolean("dis", false)) "dis" else ""
264
+ return EngineOptions(preload = true, matting = matting)
265
+ }
266
+
267
+ /**
268
+ * On the model thread: checks the model files (sizes; SHA-256 once per install), then builds the engine with
269
+ * [options] (an empty matting = the manifest's default) and, with [preload], compiles the graphs.
270
+ */
271
+ private fun setup(options: EngineOptions, preload: Boolean) {
272
+ Idol.engine?.close()
273
+ Idol.engine = null
274
+ val dir = ModelFiles.dir(filesDir)
275
+ val manifest = ModelFiles.manifest(dir)
276
+ val matting = options.matting.ifEmpty { manifest?.defaultMatting ?: "bir" }
277
+ val problems = ModelFiles.check(this, dir, matting, hash = true) { k, n ->
278
+ progress.status("Checking model files ${k + 1}/$n")
279
+ }
280
+ Idol.problems = problems
281
+ if (problems.isNotEmpty() || manifest == null) {
282
+ progress.status("")
283
+ ui.post { if (!running) showMissing(problems) }
284
+ return
285
+ }
286
+ ui.post { if (homeMessage.text.startsWith("Model files")) homeMessage.text = "" }
287
+ val engine = try {
288
+ IdolEngine(dir, manifest, options.copy(matting = matting))
289
+ } catch (e: Throwable) {
290
+ Log.e(TAG, "engine", e)
291
+ progress.status("")
292
+ ui.post { showHome("The model files could not be read: ${e.message}") }
293
+ return
294
+ }
295
+ Idol.engine = engine
296
+ if (preload && options.preload) {
297
+ try {
298
+ engine.preload(progress)
299
+ } catch (e: Throwable) {
300
+ Log.e(TAG, "preload", e)
301
+ progress.status("Loading stopped: ${e.message}")
302
+ }
303
+ } else {
304
+ progress.status("")
305
+ }
306
+ }
307
+
308
+ /** Runs [block] on the model thread and waits for it (the autorun's engine calls). */
309
+ fun <T> onModel(block: () -> T): T = Idol.model.submit(Callable { block() }).get()
310
+
311
+ /** The autorun's engine: rebuilt (without preloading) unless the current one has [options]. */
312
+ fun ensureEngine(options: EngineOptions): List<ModelFiles.Problem> = onModel {
313
+ val e = Idol.engine
314
+ if (e != null && e.options == options) {
315
+ emptyList()
316
+ } else {
317
+ setup(options, preload = false)
318
+ Idol.problems ?: emptyList()
319
+ }
320
+ }
321
+
322
+ private fun generate(photo: Photo, mode: String, noise: Noise, ply: File): Outcome {
323
+ val e = Idol.engine ?: return Idol.problems?.takeIf { it.isNotEmpty() }?.let { Outcome.Missing(it) }
324
+ ?: Outcome.Failed("not_ready", "The models are not loaded.")
325
+ return try {
326
+ ply.parentFile?.mkdirs()
327
+ Outcome.Done(e.generate(photo.image, mode, noise, ply, progress))
328
+ } catch (x: NoObjectException) {
329
+ Outcome.NoObject(x.reason)
330
+ } catch (x: NotEnoughMemoryException) {
331
+ Outcome.Failed("memory", x.message ?: "Not enough memory.")
332
+ } catch (x: CancelledException) {
333
+ Outcome.Failed("cancelled", "Stopped.")
334
+ } catch (x: OutOfMemoryError) {
335
+ Log.e(TAG, "run", x)
336
+ Outcome.Failed("memory", "Not enough memory for this picture.")
337
+ } catch (x: Throwable) {
338
+ Log.e(TAG, "run", x)
339
+ Outcome.Failed("error", "Something went wrong: ${x.message ?: x.javaClass.simpleName}")
340
+ } finally {
341
+ progress.status("")
342
+ }
343
+ }
344
+
345
+ // ------------------------------------------------------------------------------------------------ runs
346
+ private fun startFromPicture(decode: () -> Photo) {
347
+ if (running) return
348
+ val mode = currentMode()
349
+ showRun(null)
350
+ decoder.execute {
351
+ val photo = try {
352
+ decode()
353
+ } catch (e: Throwable) {
354
+ Log.e(TAG, "decode", e)
355
+ ui.post { showHome("This picture could not be opened: ${e.message ?: e.javaClass.simpleName}") }
356
+ return@execute
357
+ }
358
+ ui.post { startRun(photo, mode, Noise.random(DEFAULT_SEED), newResultFile(), showResult = true) {} }
359
+ }
360
+ }
361
+
362
+ /**
363
+ * Starts a run of [photo] on the model thread; [done] gets the outcome on the UI thread. With [showResult] a
364
+ * finished run opens the viewer; without (the autorun) the Run screen stays with its times.
365
+ */
366
+ fun startRun(photo: Photo, mode: String, noise: Noise, ply: File, showResult: Boolean, done: (Outcome) -> Unit) {
367
+ running = true
368
+ showRun(photo)
369
+ Idol.model.execute {
370
+ val outcome = generate(photo, mode, noise, ply)
371
+ ui.post {
372
+ running = false
373
+ when (outcome) {
374
+ is Outcome.Done -> {
375
+ val t = outcome.result.times
376
+ totalText.text = "Total %.1f s (3D generation %.1f s)".format(t["all"] ?: 0.0, t["generate"] ?: 0.0)
377
+ if (showResult) showResult(outcome.result.ply)
378
+ }
379
+ is Outcome.NoObject -> showHome(getString(R.string.no_object))
380
+ is Outcome.Missing -> showMissing(outcome.problems)
381
+ is Outcome.Failed -> showHome(if (outcome.kind == "cancelled") null else outcome.message)
382
+ }
383
+ done(outcome)
384
+ }
385
+ }
386
+ }
387
+
388
+ /** The autorun's run: [startRun] from another thread, waiting for the outcome. */
389
+ fun runAndWait(photo: Photo, mode: String, noise: Noise, ply: File, showResult: Boolean): Outcome {
390
+ val latch = CountDownLatch(1)
391
+ var result: Outcome? = null
392
+ ui.post {
393
+ if (running) {
394
+ result = Outcome.Failed("busy", "A run is already in progress.")
395
+ latch.countDown()
396
+ } else {
397
+ startRun(photo, mode, noise, ply, showResult) {
398
+ result = it
399
+ latch.countDown()
400
+ }
401
+ }
402
+ }
403
+ latch.await()
404
+ return result!!
405
+ }
406
+
407
+ /** The Run screen for a run the autorun drives without a picture (the parity row). */
408
+ fun showRunScreen(title: String) {
409
+ ui.post {
410
+ showRun(null)
411
+ runStatus.text = title
412
+ }
413
+ }
414
+
415
+ /** The message the Home screen shows now (the autorun records it). */
416
+ fun homeMessageText(): String {
417
+ val latch = CountDownLatch(1)
418
+ var text = ""
419
+ ui.post {
420
+ text = if (home.visibility == View.VISIBLE) homeMessage.text.toString() else ""
421
+ latch.countDown()
422
+ }
423
+ latch.await()
424
+ return text
425
+ }
426
+
427
+ private fun currentMode() = if (modeGroup.checkedRadioButtonId == R.id.mode_render) "synthetic" else "real"
428
+
429
+ private fun cameraFile() = File(File(cacheDir, "camera"), "photo.jpg")
430
+
431
+ private fun launchCamera() {
432
+ val file = cameraFile().apply { parentFile?.mkdirs() }
433
+ try {
434
+ takePhoto.launch(FileProvider.getUriForFile(this, "$packageName.files", file))
435
+ } catch (e: ActivityNotFoundException) {
436
+ showHome(getString(R.string.no_camera_app))
437
+ }
438
+ }
439
+
440
+ private fun resultsDir() = File(filesDir, "results").apply { mkdirs() }
441
+
442
+ /** A new file in files/results/ (the last few results are kept). */
443
+ private fun newResultFile(): File {
444
+ val dir = resultsDir()
445
+ dir.listFiles { f -> f.name.endsWith(".ply") }?.sortedByDescending { it.lastModified() }?.drop(KEEP_RESULTS - 1)
446
+ ?.forEach { it.delete() }
447
+ val stamp = SimpleDateFormat("yyyyMMdd_HHmmss", Locale.US).format(Date())
448
+ return File(dir, "idol_$stamp.ply")
449
+ }
450
+
451
+ private fun displayName(uri: Uri): String =
452
+ runCatching {
453
+ contentResolver.query(uri, arrayOf(OpenableColumns.DISPLAY_NAME), null, null, null)?.use { c ->
454
+ if (c.moveToFirst()) c.getString(0) else null
455
+ }
456
+ }.getOrNull() ?: uri.lastPathSegment ?: "photo"
457
+
458
+ private fun share() {
459
+ val ply = lastPly ?: return
460
+ val uri = FileProvider.getUriForFile(this, "$packageName.files", ply)
461
+ val send = Intent(Intent.ACTION_SEND).apply {
462
+ type = "application/octet-stream"
463
+ putExtra(Intent.EXTRA_STREAM, uri)
464
+ addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION)
465
+ }
466
+ try {
467
+ startActivity(Intent.createChooser(send, getString(R.string.share)))
468
+ } catch (e: ActivityNotFoundException) {
469
+ runStatus.text = getString(R.string.no_share_app)
470
+ }
471
+ }
472
+
473
+ // ------------------------------------------------------------------------------------------------ screens
474
+ private fun onBack() {
475
+ when {
476
+ resultScreen.visibility == View.VISIBLE -> showHome(null)
477
+ runScreen.visibility == View.VISIBLE && running -> {
478
+ Idol.engine?.cancel = true
479
+ runStatus.text = getString(R.string.stopping)
480
+ }
481
+ runScreen.visibility == View.VISIBLE -> showHome(null)
482
+ else -> finish()
483
+ }
484
+ }
485
+
486
+ private fun showHome(message: String?) {
487
+ ResultViewer.destroy(viewerBox, web)
488
+ web = null
489
+ stopTicker()
490
+ home.visibility = View.VISIBLE
491
+ runScreen.visibility = View.GONE
492
+ resultScreen.visibility = View.GONE
493
+ homeMessage.text = message ?: ""
494
+ homeMessage.visibility = if (message.isNullOrEmpty()) View.GONE else View.VISIBLE
495
+ }
496
+
497
+ private fun showMissing(problems: List<ModelFiles.Problem>) {
498
+ showHome(ModelFiles.message(problems))
499
+ }
500
+
501
+ private fun showRun(photo: Photo?) {
502
+ ResultViewer.destroy(viewerBox, web)
503
+ web = null
504
+ home.visibility = View.GONE
505
+ resultScreen.visibility = View.GONE
506
+ runScreen.visibility = View.VISIBLE
507
+ if (photo != null) picture.setImageBitmap(photo.displayBitmap()) else picture.setImageDrawable(null)
508
+ cutout.setImageDrawable(null)
509
+ cutout.visibility = View.INVISIBLE
510
+ for (s in stages) {
511
+ s.detail = null
512
+ s.seconds = null
513
+ s.running = false
514
+ s.done = false
515
+ }
516
+ totalText.text = ""
517
+ runStatus.text = ""
518
+ renderStages()
519
+ startTicker()
520
+ }
521
+
522
+ private fun showResult(ply: File) {
523
+ stopTicker()
524
+ lastPly = ply
525
+ home.visibility = View.GONE
526
+ runScreen.visibility = View.GONE
527
+ resultScreen.visibility = View.VISIBLE
528
+ web = ResultViewer.show(this, viewerBox, resultsDir(), ply)
529
+ }
530
+
531
+ private fun onStage(stage: Stage, detail: String?, seconds: Double?, done: Boolean) {
532
+ val s = stages[stage.ordinal]
533
+ s.detail = detail
534
+ if (done) {
535
+ s.running = false
536
+ s.done = true
537
+ s.seconds = seconds
538
+ } else if (!s.running) {
539
+ s.running = true
540
+ s.startedAt = SystemClock.elapsedRealtime()
541
+ }
542
+ renderStages()
543
+ }
544
+
545
+ private fun renderStages() {
546
+ val now = SystemClock.elapsedRealtime()
547
+ for ((i, st) in Stage.entries.withIndex()) {
548
+ val s = stages[i]
549
+ val label = if (s.detail != null) "${st.label}, ${s.detail}" else st.label
550
+ stageRows[i].text = when {
551
+ s.done -> "✓ $label — %.1f s".format(s.seconds ?: 0.0)
552
+ s.running -> "● $label — %.1f s".format((now - s.startedAt) / 1000.0)
553
+ else -> "○ $label"
554
+ }
555
+ stageRows[i].alpha = if (s.done || s.running) 1f else 0.5f
556
+ }
557
+ }
558
+
559
+ private val ticker = object : Runnable {
560
+ override fun run() {
561
+ renderStages()
562
+ ui.postDelayed(this, 250)
563
+ }
564
+ }
565
+
566
+ private fun startTicker() {
567
+ ui.removeCallbacks(ticker)
568
+ ui.postDelayed(ticker, 250)
569
+ }
570
+
571
+ private fun stopTicker() {
572
+ ui.removeCallbacks(ticker)
573
+ }
574
+
575
+ companion object {
576
+ const val TAG = "Idol"
577
+ /** The seed of a run from a picture (scripts/run_idol.py's default; the app draws with java.util.Random). */
578
+ const val DEFAULT_SEED = 42L
579
+ const val KEEP_RESULTS = 5
580
+ }
581
+ }
android/app/src/main/java/com/idol/Manifest.kt ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import org.json.JSONArray
4
+ import org.json.JSONObject
5
+ import java.io.File
6
+
7
+ /** One input or output tensor of a graph signature: its name in the flatbuffer and its shape. */
8
+ class TensorSpec(val name: String, val shape: IntArray) {
9
+ val size: Int = shape.fold(1) { a, b -> a * b }
10
+ }
11
+
12
+ /** A signature of a graph: inputs in signature order, outputs sorted by their trailing index (output_0, output_1, ...). */
13
+ class SignatureSpec(val key: String, val inputs: List<TensorSpec>, val outputs: List<TensorSpec>)
14
+
15
+ class GraphSpec(
16
+ val name: String,
17
+ val file: String,
18
+ val bytes: Long,
19
+ val sha256: String?,
20
+ /** "gpu" (ML Drift) or "cpu" (XNNPACK: BiRefNet, whose deformable convolutions have no GPU path). */
21
+ val accelerator: String,
22
+ /** "fp16_with_fp32_accum", "fp16" or "fp32". */
23
+ val precision: String,
24
+ val signatures: Map<String, SignatureSpec>,
25
+ ) {
26
+ val onCpu: Boolean get() = accelerator == "cpu"
27
+
28
+ /** The largest input or output tensor, in elements. */
29
+ val maxIoElements: Long =
30
+ signatures.values.flatMap { it.inputs + it.outputs }.maxOfOrNull { it.size.toLong() } ?: 0L
31
+
32
+ fun signature(key: String? = null): SignatureSpec =
33
+ if (key == null) signatures.values.first() else requireNotNull(signatures[key]) { "$name has no signature $key" }
34
+ }
35
+
36
+ class SideFile(val name: String, val shape: IntArray) {
37
+ val elements: Long = shape.fold(1L) { a, b -> a * b }
38
+ val bytes: Long = elements * 4
39
+ }
40
+
41
+ /**
42
+ * `models/manifest.json` as `scripts/make_manifest.py` writes it: every graph's file, size, SHA-256, accelerator,
43
+ * precision and signatures, the raw side files (little-endian float32, with their shapes) and the JSON files. The
44
+ * app builds its file checks, its buffers and its run calls from it.
45
+ */
46
+ class Manifest(
47
+ val graphs: Map<String, GraphSpec>,
48
+ val sideFiles: Map<String, SideFile>,
49
+ val jsonFiles: List<String>,
50
+ /** The matting graph the app uses unless the setting picks the alternative ("bir" or "dis"). */
51
+ val defaultMatting: String,
52
+ ) {
53
+ fun graph(name: String): GraphSpec = requireNotNull(graphs[name]) { "manifest.json has no graph $name" }
54
+
55
+ companion object {
56
+ const val FILE = "manifest.json"
57
+
58
+ fun read(file: File): Manifest = parse(JSONObject(file.readText()))
59
+
60
+ fun parse(o: JSONObject): Manifest {
61
+ val graphs = LinkedHashMap<String, GraphSpec>()
62
+ val g = o.getJSONObject("graphs")
63
+ for (name in g.keys()) {
64
+ val e = g.getJSONObject(name)
65
+ val sigs = LinkedHashMap<String, SignatureSpec>()
66
+ val s = e.getJSONObject("signatures")
67
+ for (key in s.keys()) {
68
+ val sig = s.getJSONObject(key)
69
+ val inputs = tensors(sig.getJSONArray("inputs"))
70
+ val outputs = tensors(sig.getJSONArray("outputs")).sortedBy { trailingIndex(it.name) }
71
+ sigs[key] = SignatureSpec(key, inputs, outputs)
72
+ }
73
+ graphs[name] = GraphSpec(
74
+ name = name,
75
+ file = e.getString("file"),
76
+ bytes = e.getLong("bytes"),
77
+ sha256 = e.optString("sha256", "").ifEmpty { null },
78
+ accelerator = e.optString("accelerator", "gpu"),
79
+ precision = e.optString("precision", "fp16_with_fp32_accum"),
80
+ signatures = sigs,
81
+ )
82
+ }
83
+ val side = LinkedHashMap<String, SideFile>()
84
+ o.optJSONObject("side_files")?.let { sf ->
85
+ for (name in sf.keys()) side[name] = SideFile(name, ints(sf.getJSONObject(name).getJSONArray("shape")))
86
+ }
87
+ val json = ArrayList<String>()
88
+ o.optJSONArray("json_files")?.let { a -> for (i in 0 until a.length()) json.add(a.getString(i)) }
89
+ val matting = o.optJSONObject("matting")?.optString("default", "")?.ifEmpty { null }
90
+ ?: if ("bir" in graphs) "bir" else "dis"
91
+ return Manifest(graphs, side, json, matting)
92
+ }
93
+
94
+ private fun tensors(a: JSONArray): List<TensorSpec> = (0 until a.length()).map { i ->
95
+ val t = a.getJSONArray(i)
96
+ TensorSpec(t.getString(0), ints(t.getJSONArray(1)))
97
+ }
98
+
99
+ private fun ints(a: JSONArray) = IntArray(a.length()) { a.getInt(it) }
100
+
101
+ private fun trailingIndex(name: String): Int = name.substringAfterLast('_').toIntOrNull() ?: 0
102
+ }
103
+ }
android/app/src/main/java/com/idol/ModelFiles.kt ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.content.Context
4
+ import java.io.File
5
+ import java.io.FileInputStream
6
+ import java.security.MessageDigest
7
+
8
+ /**
9
+ * The files `scripts/install_to_device.sh` puts in the app's `files/models/`: `manifest.json`, every graph it lists
10
+ * (sizes and SHA-256 from the manifest), the side files (float32, size from their shapes) and the JSON files. The
11
+ * app checks them before it builds a graph.
12
+ */
13
+ object ModelFiles {
14
+ /** The two matting graphs; a run needs only the one the setting picks. */
15
+ val MATTING_GRAPHS = setOf("bir", "dis")
16
+
17
+ fun dir(filesDir: File): File = File(filesDir, "models")
18
+
19
+ /** A file problem the app reports instead of building a graph. */
20
+ class Problem(val file: String, val message: String)
21
+
22
+ /** The graphs a run with [matting] needs, in the manifest's order. */
23
+ fun neededGraphs(manifest: Manifest, matting: String): List<String> =
24
+ manifest.graphs.keys.filter { it == matting || it !in MATTING_GRAPHS }
25
+
26
+ /** Reads the manifest, or returns null when it is missing or unreadable (reported by [check]). */
27
+ fun manifest(dir: File): Manifest? = runCatching { Manifest.read(File(dir, Manifest.FILE)) }.getOrNull()
28
+
29
+ /**
30
+ * Missing files and wrong sizes (cheap); with [hash] also the SHA-256 of every graph the manifest gives one for,
31
+ * once per install: a match is remembered by size and modification time in the app's preferences.
32
+ * [progress] gets (files checked, files to check) while hashing.
33
+ */
34
+ fun check(
35
+ context: Context,
36
+ dir: File,
37
+ matting: String,
38
+ hash: Boolean,
39
+ progress: (Int, Int) -> Unit = { _, _ -> },
40
+ ): List<Problem> {
41
+ val manifestFile = File(dir, Manifest.FILE)
42
+ if (!manifestFile.isFile) return listOf(Problem(Manifest.FILE, "${Manifest.FILE} (missing)"))
43
+ val manifest = manifest(dir) ?: return listOf(Problem(Manifest.FILE, "${Manifest.FILE} (unreadable)"))
44
+ val out = ArrayList<Problem>()
45
+ if (matting !in manifest.graphs) out.add(Problem(matting, "the manifest has no matting graph \"$matting\""))
46
+ val graphs = neededGraphs(manifest, matting).map { manifest.graph(it) }
47
+ for (g in graphs) sizeProblem(File(dir, g.file), g.bytes)?.let { out.add(it) }
48
+ for (s in manifest.sideFiles.values) sizeProblem(File(dir, s.name), s.bytes)?.let { out.add(it) }
49
+ for (j in manifest.jsonFiles) {
50
+ val f = File(dir, j)
51
+ if (!f.isFile || f.length() == 0L) out.add(Problem(j, "$j (missing)"))
52
+ }
53
+ if (!hash || out.isNotEmpty()) return out
54
+ val prefs = context.getSharedPreferences("verified", Context.MODE_PRIVATE)
55
+ val toHash = graphs.filter { it.sha256 != null }
56
+ for ((k, g) in toHash.withIndex()) {
57
+ progress(k, toHash.size)
58
+ val f = File(dir, g.file)
59
+ val key = "${f.length()}:${f.lastModified()}:${g.sha256}"
60
+ if (prefs.getString(g.file, null) == key) continue
61
+ val digest = sha256(f)
62
+ if (digest != g.sha256) {
63
+ out.add(Problem(g.file, "${g.file} (SHA-256 ${digest.take(12)}..., expected ${g.sha256!!.take(12)}...)"))
64
+ } else {
65
+ prefs.edit().putString(g.file, key).apply()
66
+ }
67
+ }
68
+ progress(toHash.size, toHash.size)
69
+ return out
70
+ }
71
+
72
+ private fun sizeProblem(f: File, bytes: Long): Problem? = when {
73
+ !f.isFile -> Problem(f.name, "${f.name} (missing)")
74
+ f.length() != bytes -> Problem(f.name, "${f.name} (${f.length()} bytes, expected $bytes)")
75
+ else -> null
76
+ }
77
+
78
+ fun sha256(file: File): String {
79
+ val md = MessageDigest.getInstance("SHA-256")
80
+ FileInputStream(file).use { input ->
81
+ val buf = ByteArray(1 shl 20)
82
+ while (true) {
83
+ val n = input.read(buf)
84
+ if (n < 0) break
85
+ md.update(buf, 0, n)
86
+ }
87
+ }
88
+ return md.digest().joinToString("") { "%02x".format(it) }
89
+ }
90
+
91
+ /** The message the Home screen shows for [problems]. */
92
+ fun message(problems: List<Problem>): String =
93
+ "Model files are missing. Run scripts/install_to_device.sh (see README).\n" +
94
+ problems.take(12).joinToString("\n") { "• ${it.message}" } +
95
+ if (problems.size > 12) "\n• ... and ${problems.size - 12} more" else ""
96
+ }
android/app/src/main/java/com/idol/Photo.kt ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.content.ContentResolver
4
+ import android.graphics.Bitmap
5
+ import android.graphics.BitmapFactory
6
+ import android.net.Uri
7
+ import java.io.File
8
+ import java.nio.ByteBuffer
9
+
10
+ /**
11
+ * A picture as the pipeline reads it: the decoded bytes at full size, unpremultiplied, without colour conversion and
12
+ * without the EXIF rotation (what PIL's Image.open gives the author's pipeline: idol/pipeline.py opens the file and
13
+ * does not transpose it). RGBA when the file has an alpha channel, else RGB.
14
+ */
15
+ class Photo(val name: String, val image: Image8) {
16
+ val width: Int get() = image.width
17
+ val height: Int get() = image.height
18
+
19
+ /** A screen copy of at most [side] pixels a side (nearest sampling), opaque. */
20
+ fun displayBitmap(side: Int = 1024): Bitmap = Photo.displayBitmap(image, side)
21
+
22
+ companion object {
23
+ /** Pictures above this many pixels are decoded at half size per side (and again) to bound the memory. */
24
+ const val MAX_PIXELS = 40_000_000L
25
+
26
+ fun fromUri(resolver: ContentResolver, uri: Uri, name: String): Photo {
27
+ val bytes = requireNotNull(resolver.openInputStream(uri)) { "cannot open $uri" }.use { it.readBytes() }
28
+ return Photo(name, decode(bytes))
29
+ }
30
+
31
+ fun fromFile(file: File): Photo = Photo(file.name, decode(file.readBytes()))
32
+
33
+ private fun decode(bytes: ByteArray): Image8 {
34
+ val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
35
+ BitmapFactory.decodeByteArray(bytes, 0, bytes.size, bounds)
36
+ require(bounds.outWidth > 0 && bounds.outHeight > 0) { "not an image the phone can decode" }
37
+ var sample = 1
38
+ while (bounds.outWidth.toLong() * bounds.outHeight / (sample.toLong() * sample) > MAX_PIXELS) sample *= 2
39
+ val options = BitmapFactory.Options().apply {
40
+ inPremultiplied = false
41
+ inPreferredConfig = Bitmap.Config.ARGB_8888
42
+ inSampleSize = sample
43
+ }
44
+ val decoded = requireNotNull(BitmapFactory.decodeByteArray(bytes, 0, bytes.size, options)) {
45
+ "not an image the phone can decode"
46
+ }
47
+ val bitmap = if (decoded.config == Bitmap.Config.ARGB_8888) decoded else decoded.copy(Bitmap.Config.ARGB_8888, false)
48
+ val w = bitmap.width
49
+ val h = bitmap.height
50
+ // copyPixelsToBuffer keeps the stored values (getPixels would convert colour spaces); ARGB_8888 is R, G, B, A
51
+ val rgba = ByteBuffer.allocateDirect(w * h * 4)
52
+ bitmap.copyPixelsToBuffer(rgba)
53
+ rgba.rewind()
54
+ val hasAlpha = bitmap.hasAlpha()
55
+ val ch = if (hasAlpha) 4 else 3
56
+ val out = ByteArray(w * h * ch)
57
+ val row = ByteArray(w * 4)
58
+ for (y in 0 until h) {
59
+ rgba.get(row)
60
+ var o = y * w * ch
61
+ for (x in 0 until w) {
62
+ out[o] = row[x * 4]
63
+ out[o + 1] = row[x * 4 + 1]
64
+ out[o + 2] = row[x * 4 + 2]
65
+ if (hasAlpha) out[o + 3] = row[x * 4 + 3]
66
+ o += ch
67
+ }
68
+ }
69
+ if (bitmap !== decoded) bitmap.recycle()
70
+ decoded.recycle()
71
+ return Image8(w, h, ch, out)
72
+ }
73
+
74
+ fun displayBitmap(image: Image8, side: Int): Bitmap {
75
+ val step = maxOf(1, (maxOf(image.width, image.height) + side - 1) / side)
76
+ val dw = maxOf(1, image.width / step)
77
+ val dh = maxOf(1, image.height / step)
78
+ val ch = image.channels
79
+ val s = image.data
80
+ val px = IntArray(dw * dh)
81
+ for (y in 0 until dh) for (x in 0 until dw) {
82
+ val i = ((y * step) * image.width + x * step) * ch
83
+ px[y * dw + x] = (0xFF shl 24) or ((s[i].toInt() and 0xFF) shl 16) or
84
+ ((s[i + 1].toInt() and 0xFF) shl 8) or (s[i + 2].toInt() and 0xFF)
85
+ }
86
+ return Bitmap.createBitmap(px, dw, dh, Bitmap.Config.ARGB_8888)
87
+ }
88
+ }
89
+ }
android/app/src/main/java/com/idol/Pil.kt ADDED
@@ -0,0 +1,361 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import kotlin.math.PI
4
+ import kotlin.math.ceil
5
+ import kotlin.math.sin
6
+
7
+ /** An 8-bit image, channels interleaved, rows top to bottom: what a PIL image of mode RGB (3) or RGBA (4) holds. */
8
+ class Image8(val width: Int, val height: Int, val channels: Int, val data: ByteArray) {
9
+ init {
10
+ require(data.size == width * height * channels) { "${data.size} bytes for ${width}x${height}x$channels" }
11
+ }
12
+
13
+ fun copy() = Image8(width, height, channels, data.copyOf())
14
+
15
+ /** One channel as a width x height plane. */
16
+ fun channel(c: Int): ByteArray {
17
+ val out = ByteArray(width * height)
18
+ var p = c
19
+ for (i in out.indices) {
20
+ out[i] = data[p]
21
+ p += channels
22
+ }
23
+ return out
24
+ }
25
+
26
+ /** PIL's convert("RGB"): the first three channels. */
27
+ fun rgb(): Image8 {
28
+ if (channels == 3) return this
29
+ val out = ByteArray(width * height * 3)
30
+ var p = 0
31
+ var o = 0
32
+ for (i in 0 until width * height) {
33
+ out[o] = data[p]
34
+ out[o + 1] = data[p + 1]
35
+ out[o + 2] = data[p + 2]
36
+ p += channels
37
+ o += 3
38
+ }
39
+ return Image8(width, height, 3, out)
40
+ }
41
+
42
+ /** PIL's putalpha(L): an RGBA image with this image's colours and [alpha] as its alpha channel. */
43
+ fun withAlpha(alpha: ByteArray): Image8 {
44
+ require(alpha.size == width * height)
45
+ val out = ByteArray(width * height * 4)
46
+ var p = 0
47
+ var o = 0
48
+ for (i in 0 until width * height) {
49
+ out[o] = data[p]
50
+ out[o + 1] = data[p + 1]
51
+ out[o + 2] = data[p + 2]
52
+ out[o + 3] = alpha[i]
53
+ p += channels
54
+ o += 4
55
+ }
56
+ return Image8(width, height, 4, out)
57
+ }
58
+ }
59
+
60
+ /**
61
+ * The parts of Pillow 13.0.0 the author's preprocessing uses (idol/preprocess.py, scripts/idol_glue.py preprocess),
62
+ * pixel for pixel, from Pillow's src/libImaging: Resample.c (Lanczos, 22-bit fixed-point coefficients, the pass order
63
+ * of ImagingResampleInner), Convert.c (RGBA <-> RGBa around a resize), RankFilter.c + Filter.c (MinFilter(3) with edge
64
+ * replication), Crop.c (zero fill outside the image) and Paste.c (paste with an L mask).
65
+ * Pillow: Copyright (c) 1997-2011 by Secret Labs AB, Copyright (c) 1995-2011 by Fredrik Lundh and contributors,
66
+ * Copyright (c) 2010 by Jeffrey 'Alex' Clark and contributors, under the MIT-CMU License (licenses/Pillow-MIT-CMU.txt).
67
+ */
68
+ object Pil {
69
+ private const val PRECISION_BITS = 32 - 8 - 2
70
+
71
+ private class Coeffs(val ksize: Int, val bounds: IntArray, val kk: IntArray)
72
+
73
+ private fun sinc(x0: Double): Double {
74
+ if (x0 == 0.0) return 1.0
75
+ val x = x0 * PI
76
+ return sin(x) / x
77
+ }
78
+
79
+ /** lanczos_filter: truncated sinc, support 3. */
80
+ private fun lanczos(x: Double): Double = if (-3.0 <= x && x < 3.0) sinc(x) * sinc(x / 3) else 0.0
81
+
82
+ /** precompute_coeffs for the box [0, inSize) -> outSize, then normalize_coeffs_8bpc. */
83
+ private fun coeffs(inSize: Int, outSize: Int): Coeffs {
84
+ val in0 = 0.0f
85
+ val in1 = inSize.toFloat()
86
+ val scale = (in1 - in0).toDouble() / outSize
87
+ val filterscale = if (scale < 1.0) 1.0 else scale
88
+ val support = 3.0 * filterscale
89
+ val ksize = ceil(support).toInt() * 2 + 1
90
+ val pre = DoubleArray(outSize * ksize)
91
+ val bounds = IntArray(outSize * 2)
92
+ val invFilterscale = 1.0 / filterscale
93
+ for (xx in 0 until outSize) {
94
+ val center = in0 + (xx + 0.5) * scale
95
+ var ww = 0.0
96
+ var xmin = (center - support + 0.5).toInt()
97
+ if (xmin < 0) xmin = 0
98
+ var xmax = (center + support + 0.5).toInt()
99
+ if (xmax > inSize) xmax = inSize
100
+ xmax -= xmin
101
+ val base = xx * ksize
102
+ for (x in 0 until xmax) {
103
+ val w = lanczos((x + xmin - center + 0.5) * invFilterscale)
104
+ pre[base + x] = w
105
+ ww += w
106
+ }
107
+ if (ww != 0.0) for (x in 0 until xmax) pre[base + x] /= ww
108
+ bounds[xx * 2] = xmin
109
+ bounds[xx * 2 + 1] = xmax
110
+ }
111
+ val one = (1 shl PRECISION_BITS).toDouble()
112
+ val kk = IntArray(pre.size) { i ->
113
+ val v = pre[i]
114
+ if (v < 0) (-0.5 + v * one).toInt() else (0.5 + v * one).toInt()
115
+ }
116
+ return Coeffs(ksize, bounds, kk)
117
+ }
118
+
119
+ private fun clip8(ss: Int): Byte {
120
+ val v = ss shr PRECISION_BITS
121
+ return (if (v < 0) 0 else if (v > 255) 255 else v).toByte()
122
+ }
123
+
124
+ /** _ImagingResampleHorizontal_8bpc: [rows] output rows from input rows offset.. */
125
+ private fun horizontal(src: Image8, outWidth: Int, rows: Int, offset: Int, c: Coeffs): Image8 {
126
+ val ch = src.channels
127
+ val inW = src.width
128
+ val s = src.data
129
+ val out = ByteArray(outWidth * rows * ch)
130
+ val half = 1 shl (PRECISION_BITS - 1)
131
+ val acc = IntArray(ch)
132
+ for (yy in 0 until rows) {
133
+ val inRow = (yy + offset) * inW * ch
134
+ val outRow = yy * outWidth * ch
135
+ for (xx in 0 until outWidth) {
136
+ val xmin = c.bounds[xx * 2]
137
+ val xmax = c.bounds[xx * 2 + 1]
138
+ val k = xx * c.ksize
139
+ val o = outRow + xx * ch
140
+ when (ch) {
141
+ 3 -> {
142
+ var s0 = half
143
+ var s1 = half
144
+ var s2 = half
145
+ var p = inRow + xmin * 3
146
+ for (x in 0 until xmax) {
147
+ val w = c.kk[k + x]
148
+ s0 += (s[p].toInt() and 0xFF) * w
149
+ s1 += (s[p + 1].toInt() and 0xFF) * w
150
+ s2 += (s[p + 2].toInt() and 0xFF) * w
151
+ p += 3
152
+ }
153
+ out[o] = clip8(s0)
154
+ out[o + 1] = clip8(s1)
155
+ out[o + 2] = clip8(s2)
156
+ }
157
+ 4 -> {
158
+ var s0 = half
159
+ var s1 = half
160
+ var s2 = half
161
+ var s3 = half
162
+ var p = inRow + xmin * 4
163
+ for (x in 0 until xmax) {
164
+ val w = c.kk[k + x]
165
+ s0 += (s[p].toInt() and 0xFF) * w
166
+ s1 += (s[p + 1].toInt() and 0xFF) * w
167
+ s2 += (s[p + 2].toInt() and 0xFF) * w
168
+ s3 += (s[p + 3].toInt() and 0xFF) * w
169
+ p += 4
170
+ }
171
+ out[o] = clip8(s0)
172
+ out[o + 1] = clip8(s1)
173
+ out[o + 2] = clip8(s2)
174
+ out[o + 3] = clip8(s3)
175
+ }
176
+ else -> {
177
+ acc.fill(half)
178
+ var p = inRow + xmin * ch
179
+ for (x in 0 until xmax) {
180
+ val w = c.kk[k + x]
181
+ for (b in 0 until ch) acc[b] += (s[p + b].toInt() and 0xFF) * w
182
+ p += ch
183
+ }
184
+ for (b in 0 until ch) out[o + b] = clip8(acc[b])
185
+ }
186
+ }
187
+ }
188
+ }
189
+ return Image8(outWidth, rows, ch, out)
190
+ }
191
+
192
+ /** _ImagingResampleVertical_8bpc over every column of [src]. */
193
+ private fun vertical(src: Image8, outHeight: Int, c: Coeffs): Image8 {
194
+ val ch = src.channels
195
+ val w = src.width
196
+ val stride = w * ch
197
+ val s = src.data
198
+ val out = ByteArray(w * outHeight * ch)
199
+ val half = 1 shl (PRECISION_BITS - 1)
200
+ val acc = IntArray(stride)
201
+ for (yy in 0 until outHeight) {
202
+ val ymin = c.bounds[yy * 2]
203
+ val ymax = c.bounds[yy * 2 + 1]
204
+ val k = yy * c.ksize
205
+ // the same sums as Pillow's per-pixel loop (integer adds, so the order does not change them), row by row
206
+ acc.fill(half)
207
+ for (y in 0 until ymax) {
208
+ val wgt = c.kk[k + y]
209
+ val row = (y + ymin) * stride
210
+ for (i in 0 until stride) acc[i] += (s[row + i].toInt() and 0xFF) * wgt
211
+ }
212
+ val o = yy * stride
213
+ for (i in 0 until stride) out[o + i] = clip8(acc[i])
214
+ }
215
+ return Image8(w, outHeight, ch, out)
216
+ }
217
+
218
+ /** ImagingResample with LANCZOS on an 8-bit image (any channel count): no premultiplication here. */
219
+ fun resizeLanczos(img: Image8, outWidth: Int, outHeight: Int): Image8 {
220
+ val inW = img.width
221
+ val inH = img.height
222
+ val needHorizontal = outWidth != inW
223
+ val needVertical = outHeight != inH
224
+ if (!needHorizontal && !needVertical) return img.copy()
225
+ // If height is being scaled down more than twice the amount the width is, run the vertical pass first.
226
+ val horizontalFirst = !((inH - outHeight) > 0 && (inH - outHeight) > (inW - outWidth) * 2)
227
+ var vert: Coeffs? = null
228
+ if ((needHorizontal && horizontalFirst) || needVertical) vert = coeffs(inH, outHeight)
229
+ var yFirst = 0
230
+ var yLast = 0
231
+ var horiz: Coeffs? = null
232
+ if (needHorizontal) {
233
+ if (horizontalFirst) {
234
+ val v = vert!!
235
+ yFirst = v.bounds[0]
236
+ yLast = v.bounds[outHeight * 2 - 2] + v.bounds[outHeight * 2 - 1]
237
+ if (yFirst != 0) for (i in 0 until outHeight) v.bounds[i * 2] -= yFirst
238
+ }
239
+ horiz = coeffs(inW, outWidth)
240
+ }
241
+ var cur = img
242
+ if (horizontalFirst) {
243
+ if (needHorizontal) cur = horizontal(cur, outWidth, yLast - yFirst, yFirst, horiz!!)
244
+ if (needVertical) cur = vertical(cur, outHeight, vert!!)
245
+ } else {
246
+ if (needVertical) cur = vertical(cur, outHeight, vert!!)
247
+ if (needHorizontal) cur = horizontal(cur, outWidth, outHeight, 0, horiz!!)
248
+ }
249
+ return cur
250
+ }
251
+
252
+ /**
253
+ * Image.resize(size, LANCZOS): an RGBA image goes through RGBa (premultiplied) and back, as Pillow does; RGB and
254
+ * single-channel images are resized directly; the same size returns a copy.
255
+ */
256
+ fun resize(img: Image8, outWidth: Int, outHeight: Int): Image8 {
257
+ if (outWidth == img.width && outHeight == img.height) return img.copy()
258
+ if (img.channels != 4) return resizeLanczos(img, outWidth, outHeight)
259
+ return unpremultiply(resizeLanczos(premultiply(img), outWidth, outHeight))
260
+ }
261
+
262
+ /** Convert.c rgbA2rgba (RGBA -> RGBa): MULDIV255(c, a). */
263
+ fun premultiply(img: Image8): Image8 {
264
+ require(img.channels == 4)
265
+ val s = img.data
266
+ val out = ByteArray(s.size)
267
+ var p = 0
268
+ while (p < s.size) {
269
+ val a = s[p + 3].toInt() and 0xFF
270
+ for (b in 0 until 3) {
271
+ val t = (s[p + b].toInt() and 0xFF) * a + 128
272
+ out[p + b] = (((t shr 8) + t) shr 8).toByte()
273
+ }
274
+ out[p + 3] = s[p + 3]
275
+ p += 4
276
+ }
277
+ return Image8(img.width, img.height, 4, out)
278
+ }
279
+
280
+ /** Convert.c rgba2rgbA (RGBa -> RGBA): CLIP8(255 * c / a), alpha 0 and 255 unchanged. */
281
+ fun unpremultiply(img: Image8): Image8 {
282
+ require(img.channels == 4)
283
+ val s = img.data
284
+ val out = ByteArray(s.size)
285
+ var p = 0
286
+ while (p < s.size) {
287
+ val a = s[p + 3].toInt() and 0xFF
288
+ if (a == 255 || a == 0) {
289
+ out[p] = s[p]
290
+ out[p + 1] = s[p + 1]
291
+ out[p + 2] = s[p + 2]
292
+ } else {
293
+ for (b in 0 until 3) {
294
+ val v = (255 * (s[p + b].toInt() and 0xFF)) / a
295
+ out[p + b] = (if (v <= 0) 0 else if (v < 256) v else 255).toByte()
296
+ }
297
+ }
298
+ out[p + 3] = s[p + 3]
299
+ p += 4
300
+ }
301
+ return Image8(img.width, img.height, 4, out)
302
+ }
303
+
304
+ /** ImageFilter.MinFilter(3) on an L plane: the 3 x 3 minimum, edges replicated (ImagingExpand + RankFilter). */
305
+ fun minFilter3(src: ByteArray, w: Int, h: Int): ByteArray {
306
+ val out = ByteArray(w * h)
307
+ for (y in 0 until h) {
308
+ val y0 = if (y > 0) y - 1 else 0
309
+ val y1 = if (y < h - 1) y + 1 else h - 1
310
+ for (x in 0 until w) {
311
+ val x0 = if (x > 0) x - 1 else 0
312
+ val x1 = if (x < w - 1) x + 1 else w - 1
313
+ var m = 255
314
+ for (yy in y0..y1) {
315
+ val row = yy * w
316
+ for (xx in x0..x1) {
317
+ val v = src[row + xx].toInt() and 0xFF
318
+ if (v < m) m = v
319
+ }
320
+ }
321
+ out[y * w + x] = m.toByte()
322
+ }
323
+ }
324
+ return out
325
+ }
326
+
327
+ /** Image.crop((x0, y0, x1, y1)): the region, zeros where it leaves the image. */
328
+ fun crop(img: Image8, x0: Int, y0: Int, x1: Int, y1: Int): Image8 {
329
+ val w = maxOf(0, x1 - x0)
330
+ val h = maxOf(0, y1 - y0)
331
+ val ch = img.channels
332
+ val out = ByteArray(w * h * ch)
333
+ val xs = maxOf(0, x0)
334
+ val xe = minOf(img.width, x1)
335
+ if (xe > xs) {
336
+ for (y in maxOf(0, y0) until minOf(img.height, y1)) {
337
+ System.arraycopy(img.data, (y * img.width + xs) * ch, out, ((y - y0) * w + (xs - x0)) * ch, (xe - xs) * ch)
338
+ }
339
+ }
340
+ return Image8(w, h, ch, out)
341
+ }
342
+
343
+ /**
344
+ * Image.new("RGB", size, (0, 0, 0)).paste(rgba, mask=rgba.split()[3]): Paste.c paste_mask_L's BLEND on black,
345
+ * DIV255(c * a).
346
+ */
347
+ fun compositeOnBlack(rgba: Image8): Image8 {
348
+ require(rgba.channels == 4)
349
+ val n = rgba.width * rgba.height
350
+ val s = rgba.data
351
+ val out = ByteArray(n * 3)
352
+ for (i in 0 until n) {
353
+ val a = s[i * 4 + 3].toInt() and 0xFF
354
+ for (b in 0 until 3) {
355
+ val t = (s[i * 4 + b].toInt() and 0xFF) * a + 128
356
+ out[i * 3 + b] = (((t shr 8) + t) shr 8).toByte()
357
+ }
358
+ }
359
+ return Image8(rgba.width, rgba.height, 3, out)
360
+ }
361
+ }
android/app/src/main/java/com/idol/Preprocess.kt ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import kotlin.math.max
4
+ import kotlin.math.min
5
+ import kotlin.math.round
6
+
7
+ /** The picture has nothing to cut out (the app's "No object found in this picture."). */
8
+ class NoObjectException(val reason: String) : Exception("no object found: $reason")
9
+
10
+ /** What the encoders see, and how it was cut out (scripts/idol_glue.py preprocess). */
11
+ class Preprocessed(
12
+ /** 1024 x 1024 RGB: the object on black. */
13
+ val composite: Image8,
14
+ /** The uint8 alpha at the resized size, before the minimum filter (the matting graph's, or the picture's own). */
15
+ val alphaRaw: ByteArray,
16
+ val alphaWidth: Int,
17
+ val alphaHeight: Int,
18
+ /** The crop square (left, top, right, bottom) in the resized picture. */
19
+ val box: IntArray,
20
+ /** True when the picture brought its own alpha (RGBA with a pixel below 255) and no matting ran. */
21
+ val ownAlpha: Boolean,
22
+ )
23
+
24
+ /**
25
+ * The host steps around the two matting graphs, ported line by line from scripts/idol_glue.py: birefnet_alpha (the
26
+ * author's BiRefNet.remove_background) and dis_alpha (IS-Net DIS as its authors run it, plus the alpha floor).
27
+ */
28
+ object Matting {
29
+ const val SIZE = 1024
30
+ private val BIREFNET_MEAN = floatArrayOf(0.485f, 0.456f, 0.406f)
31
+ private val BIREFNET_STD = floatArrayOf(0.229f, 0.224f, 0.225f)
32
+
33
+ /** uint8 DIS alpha values below this are set to 0 (idol_glue.DIS_ALPHA_FLOOR, DIS only). */
34
+ const val DIS_ALPHA_FLOOR = 8
35
+
36
+ /** The RGB picture as float32 planes [3, h, w], each value c (or c / 255 with [divide]). */
37
+ private fun planes(rgb: Image8, divide: Boolean): FloatArray {
38
+ val n = rgb.width * rgb.height
39
+ val out = FloatArray(3 * n)
40
+ val s = rgb.data
41
+ for (i in 0 until n) {
42
+ for (c in 0 until 3) {
43
+ val v = (s[i * 3 + c].toInt() and 0xFF).toFloat()
44
+ out[c * n + i] = if (divide) v / 255.0f else v
45
+ }
46
+ }
47
+ return out
48
+ }
49
+
50
+ /** BiRefNet's input [1, 3, 1024, 1024]: RGB / 255, bilinear (align_corners) to 1024 x 1024, ImageNet mean / std. */
51
+ fun birefnetInput(rgb: Image8): FloatArray {
52
+ val t = TorchOps.bilinear(planes(rgb, true), 3, rgb.height, rgb.width, SIZE, SIZE, alignCorners = true)
53
+ val n = SIZE * SIZE
54
+ for (c in 0 until 3) {
55
+ val m = BIREFNET_MEAN[c]
56
+ val s = BIREFNET_STD[c]
57
+ for (i in c * n until (c + 1) * n) t[i] = (t[i] - m) / s
58
+ }
59
+ return t
60
+ }
61
+
62
+ /** BiRefNet's sigmoid map [1, 1, 1024, 1024] -> uint8 alpha at w x h: bilinear (align_corners), x 255, truncated. */
63
+ fun birefnetAlpha(map: FloatArray, w: Int, h: Int): ByteArray {
64
+ val a = TorchOps.bilinear(map, 1, SIZE, SIZE, h, w, alignCorners = true)
65
+ return ByteArray(a.size) { i -> toUint8(a[i]) }
66
+ }
67
+
68
+ /** DIS's input [1, 3, 1024, 1024]: bilinear to 1024 x 1024 on 0..255, then x / 255 - 0.5. */
69
+ fun disInput(rgb: Image8): FloatArray {
70
+ val t = TorchOps.bilinear(planes(rgb, false), 3, rgb.height, rgb.width, SIZE, SIZE, alignCorners = false)
71
+ for (i in t.indices) t[i] = t[i] / 255.0f - 0.5f
72
+ return t
73
+ }
74
+
75
+ /**
76
+ * DIS's sigmoid map -> uint8 alpha at w x h: bilinear back, min-max normalised (the division guarded by 1e-12 as in
77
+ * the authors' code), x 255 truncated, values below [floor] set to 0.
78
+ */
79
+ fun disAlpha(map: FloatArray, w: Int, h: Int, floor: Int = DIS_ALPHA_FLOOR): ByteArray {
80
+ val a = TorchOps.bilinear(map, 1, SIZE, SIZE, h, w, alignCorners = false)
81
+ var mi = Float.POSITIVE_INFINITY
82
+ var ma = Float.NEGATIVE_INFINITY
83
+ for (v in a) {
84
+ if (v < mi) mi = v
85
+ if (v > ma) ma = v
86
+ }
87
+ val d = max(ma - mi, 1e-12f)
88
+ val out = ByteArray(a.size)
89
+ for (i in a.indices) {
90
+ val u = toUint8((a[i] - mi) / d).toInt() and 0xFF
91
+ out[i] = (if (u >= floor) u else 0).toByte()
92
+ }
93
+ return out
94
+ }
95
+
96
+ /** torch: (a.clamp(0, 1) * 255).to(torch.uint8), the cast truncating. */
97
+ private fun toUint8(v: Float): Byte {
98
+ val c = if (v < 0f) 0f else if (v > 1f) 1f else v
99
+ return (c * 255f).toInt().toByte()
100
+ }
101
+ }
102
+
103
+ object Preprocess {
104
+ const val IMAGE_SIZE = 1024
105
+ val CROP_MARGIN = mapOf("synthetic" to 1.2, "real" to 1.7)
106
+
107
+ /** The resize size of the author's first step: short side 1024, Python round() (half to even), at least 1. */
108
+ fun firstSize(width: Int, height: Int): Pair<Int, Int> {
109
+ val scale = IMAGE_SIZE.toDouble() / min(width, height)
110
+ // kotlin.math.round is Math.rint: ties to the even integer, as Python's round()
111
+ return max(1, round(width * scale).toInt()) to max(1, round(height * scale).toInt())
112
+ }
113
+
114
+ /** First step alone (the matting input of a picture without its own alpha): the resized picture. */
115
+ fun resized(image: Image8): Image8 {
116
+ val (w, h) = firstSize(image.width, image.height)
117
+ return Pil.resize(image, w, h)
118
+ }
119
+
120
+ /**
121
+ * scripts/idol_glue.py preprocess: resize the short side to 1024 (Lanczos), alpha from [alphaFn] unless the picture
122
+ * has its own, 3 x 3 minimum filter on the alpha, square crop of side margin x the box of the nonzero alpha,
123
+ * resize to 1024 (Lanczos), composite on black.
124
+ *
125
+ * The app adds guards the author's code does not have (it raises or crops nonsense there): a picture of one flat
126
+ * colour, an alpha with no pixel at 128 or above, and a crop square under one pixel are "no object".
127
+ */
128
+ fun run(image: Image8, mode: String, alphaFn: (Image8) -> ByteArray): Preprocessed {
129
+ val margin = requireNotNull(CROP_MARGIN[mode]) { "mode $mode" }
130
+ var img = resized(image)
131
+ val w = img.width
132
+ val h = img.height
133
+ val ownAlpha = img.channels == 4 && minByte(img.channel(3)) < 255
134
+ if (!ownAlpha) {
135
+ val rgb = img.rgb()
136
+ if (isFlat(rgb)) throw NoObjectException("the picture is one flat colour")
137
+ img = rgb.withAlpha(alphaFn(rgb))
138
+ }
139
+ val alphaRaw = img.channel(3)
140
+ if (maxByte(alphaRaw) < 128) throw NoObjectException("no alpha value reaches 128")
141
+ val alpha = Pil.minFilter3(alphaRaw, w, h)
142
+ val px = img.data
143
+ for (i in 0 until w * h) px[i * 4 + 3] = alpha[i]
144
+ var left = Int.MAX_VALUE
145
+ var top = Int.MAX_VALUE
146
+ var right = -1
147
+ var bottom = -1
148
+ for (y in 0 until h) for (x in 0 until w) {
149
+ if (alpha[y * w + x].toInt() != 0) {
150
+ if (x < left) left = x
151
+ if (x > right) right = x
152
+ if (y < top) top = y
153
+ if (y > bottom) bottom = y
154
+ }
155
+ }
156
+ if (right < 0) throw NoObjectException("the alpha channel is empty")
157
+ val centerX = (left + right) / 2.0
158
+ val centerY = (top + bottom) / 2.0
159
+ val half = max(right - left, bottom - top) / 2.0 * margin
160
+ val box = intArrayOf(
161
+ (centerX - half).toInt(), (centerY - half).toInt(), (centerX + half).toInt(), (centerY + half).toInt(),
162
+ )
163
+ if (box[2] - box[0] < 1 || box[3] - box[1] < 1) throw NoObjectException("the object is under one pixel")
164
+ val cropped = Pil.crop(img, box[0], box[1], box[2], box[3])
165
+ val square = Pil.resize(cropped, IMAGE_SIZE, IMAGE_SIZE)
166
+ return Preprocessed(Pil.compositeOnBlack(square), alphaRaw, w, h, box, ownAlpha)
167
+ }
168
+
169
+ /** pixels_of: [1, 3, 1024, 1024] float32 in [0, 1]. */
170
+ fun pixels(composite: Image8): FloatArray {
171
+ val n = composite.width * composite.height
172
+ val out = FloatArray(3 * n)
173
+ val s = composite.data
174
+ for (i in 0 until n) for (c in 0 until 3) out[c * n + i] = (s[i * 3 + c].toInt() and 0xFF).toFloat() / 255.0f
175
+ return out
176
+ }
177
+
178
+ private fun minByte(a: ByteArray): Int {
179
+ var m = 255
180
+ for (b in a) {
181
+ val v = b.toInt() and 0xFF
182
+ if (v < m) m = v
183
+ }
184
+ return m
185
+ }
186
+
187
+ private fun maxByte(a: ByteArray): Int {
188
+ var m = 0
189
+ for (b in a) {
190
+ val v = b.toInt() and 0xFF
191
+ if (v > m) m = v
192
+ }
193
+ return m
194
+ }
195
+
196
+ private fun isFlat(rgb: Image8): Boolean {
197
+ val s = rgb.data
198
+ for (i in 3 until s.size) if (s[i] != s[i % 3]) return false
199
+ return true
200
+ }
201
+ }
android/app/src/main/java/com/idol/ResultViewer.kt ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import android.annotation.SuppressLint
4
+ import android.content.Context
5
+ import android.net.Uri
6
+ import android.view.ViewGroup
7
+ import android.webkit.WebResourceRequest
8
+ import android.webkit.WebResourceResponse
9
+ import android.webkit.WebView
10
+ import android.webkit.WebViewClient
11
+ import android.widget.FrameLayout
12
+ import androidx.webkit.WebViewAssetLoader
13
+ import java.io.File
14
+ import java.io.FileInputStream
15
+
16
+ /**
17
+ * The splat viewer (antimatter15/splat, vendored in assets/viewer/) in a WebView. The page and the .ply come from
18
+ * the app itself through WebViewAssetLoader (https://appassets.androidplatform.net/...): no network, no file:// access.
19
+ */
20
+ object ResultViewer {
21
+ private const val HOST = "https://appassets.androidplatform.net"
22
+
23
+ @SuppressLint("SetJavaScriptEnabled")
24
+ fun show(context: Context, container: FrameLayout, resultsDir: File, ply: File): WebView {
25
+ val loader = WebViewAssetLoader.Builder()
26
+ .addPathHandler("/assets/", WebViewAssetLoader.AssetsPathHandler(context))
27
+ .addPathHandler("/results/", PlyHandler(resultsDir))
28
+ .build()
29
+ val web = WebView(context)
30
+ web.settings.javaScriptEnabled = true
31
+ web.settings.allowFileAccess = false
32
+ web.settings.allowContentAccess = false
33
+ web.webViewClient = object : WebViewClient() {
34
+ override fun shouldInterceptRequest(view: WebView, request: WebResourceRequest): WebResourceResponse? =
35
+ loader.shouldInterceptRequest(request.url)
36
+
37
+ // the viewer has no links; stay on the page
38
+ override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean = true
39
+ }
40
+ container.removeAllViews()
41
+ container.addView(web, ViewGroup.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
42
+ val plyUrl = "$HOST/results/" + Uri.encode(ply.name)
43
+ web.loadUrl("$HOST/assets/viewer/index.html?url=" + Uri.encode(plyUrl))
44
+ return web
45
+ }
46
+
47
+ fun destroy(container: FrameLayout, web: WebView?) {
48
+ if (web == null) return
49
+ container.removeView(web)
50
+ web.stopLoading()
51
+ web.destroy()
52
+ }
53
+
54
+ /** Serves files of [dir] with a Content-Length (the viewer sizes its buffer from it). */
55
+ private class PlyHandler(private val dir: File) : WebViewAssetLoader.PathHandler {
56
+ override fun handle(path: String): WebResourceResponse? {
57
+ val f = File(dir, path)
58
+ val inside = runCatching { f.canonicalPath.startsWith(dir.canonicalPath + File.separator) }.getOrDefault(false)
59
+ if (!inside || !f.isFile) {
60
+ return WebResourceResponse("text/plain", "utf-8", 404, "Not Found", emptyMap(), null)
61
+ }
62
+ val headers = mapOf("Content-Length" to f.length().toString(), "Cache-Control" to "no-store")
63
+ return WebResourceResponse("application/octet-stream", null, 200, "OK", headers, FileInputStream(f))
64
+ }
65
+ }
66
+ }
android/app/src/main/java/com/idol/TorchOps.kt ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ package com.idol
2
+
3
+ import kotlin.math.floor
4
+
5
+ /**
6
+ * torch.nn.functional.interpolate(x, size, mode="bilinear", align_corners=...) on the CPU in float32, as ATen's
7
+ * generic N-d kernel computes it (UpSampleKernel.cpp: compute_indices_weights_linear + Interpolate, the path a
8
+ * 1- or 3-channel NCHW tensor takes): per output index the source index and weight from
9
+ * area_pixel_compute_scale / area_pixel_compute_source_index / guard_index_and_lambda in float32, then
10
+ * out = (row(y0) * wy0 + row(y1) * wy1) with row(y) = src(y, x0) * wx0 + src(y, x1) * wx1, each product and sum
11
+ * rounded to float32.
12
+ */
13
+ object TorchOps {
14
+ private class Taps(val i0: IntArray, val i1: IntArray, val w0: FloatArray, val w1: FloatArray)
15
+
16
+ private fun taps(inSize: Int, outSize: Int, alignCorners: Boolean): Taps {
17
+ val i0 = IntArray(outSize)
18
+ val i1 = IntArray(outSize)
19
+ val w0 = FloatArray(outSize)
20
+ val w1 = FloatArray(outSize)
21
+ val scale: Float = if (alignCorners) {
22
+ if (outSize > 1) (inSize - 1).toFloat() / (outSize - 1).toFloat() else 0f
23
+ } else {
24
+ inSize.toFloat() / outSize.toFloat()
25
+ }
26
+ for (o in 0 until outSize) {
27
+ if (outSize == inSize) {
28
+ // scale factor 1: a copy
29
+ i0[o] = o
30
+ i1[o] = o
31
+ w0[o] = 1f
32
+ w1[o] = 0f
33
+ continue
34
+ }
35
+ val real: Float = if (alignCorners) {
36
+ scale * o.toFloat()
37
+ } else {
38
+ val src = scale * (o.toFloat() + 0.5f) - 0.5f
39
+ if (src < 0f) 0f else src
40
+ }
41
+ val idx = minOf(floor(real.toDouble()).toInt(), inSize - 1)
42
+ val lambda = minOf(maxOf(real - idx.toFloat(), 0f), 1f)
43
+ i0[o] = idx
44
+ i1[o] = idx + if (idx < inSize - 1) 1 else 0
45
+ w1[o] = lambda
46
+ w0[o] = 1f - lambda
47
+ }
48
+ return Taps(i0, i1, w0, w1)
49
+ }
50
+
51
+ /** [src] = channels x inH x inW (planar, float32) -> channels x outH x outW. */
52
+ fun bilinear(
53
+ src: FloatArray,
54
+ channels: Int,
55
+ inH: Int,
56
+ inW: Int,
57
+ outH: Int,
58
+ outW: Int,
59
+ alignCorners: Boolean,
60
+ ): FloatArray {
61
+ require(src.size == channels * inH * inW)
62
+ val ty = taps(inH, outH, alignCorners)
63
+ val tx = taps(inW, outW, alignCorners)
64
+ val out = FloatArray(channels * outH * outW)
65
+ for (c in 0 until channels) {
66
+ val sBase = c * inH * inW
67
+ val oBase = c * outH * outW
68
+ for (y in 0 until outH) {
69
+ val r0 = sBase + ty.i0[y] * inW
70
+ val r1 = sBase + ty.i1[y] * inW
71
+ val wy0 = ty.w0[y]
72
+ val wy1 = ty.w1[y]
73
+ val o = oBase + y * outW
74
+ for (x in 0 until outW) {
75
+ val x0 = tx.i0[x]
76
+ val x1 = tx.i1[x]
77
+ val wx0 = tx.w0[x]
78
+ val wx1 = tx.w1[x]
79
+ var t0 = src[r0 + x0] * wx0
80
+ t0 += src[r0 + x1] * wx1
81
+ var t1 = src[r1 + x0] * wx0
82
+ t1 += src[r1 + x1] * wx1
83
+ var v = t0 * wy0
84
+ v += t1 * wy1
85
+ out[o + x] = v
86
+ }
87
+ }
88
+ }
89
+ return out
90
+ }
91
+ }
android/app/src/main/res/layout/activity_main.xml ADDED
@@ -0,0 +1,216 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <?xml version="1.0" encoding="utf-8"?>
2
+ <FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
3
+ android:id="@+id/root"
4
+ android:layout_width="match_parent"
5
+ android:layout_height="match_parent"
6
+ android:background="@android:color/black">
7
+
8
+ <!-- Home -->
9
+ <ScrollView
10
+ android:id="@+id/home"
11
+ android:layout_width="match_parent"
12
+ android:layout_height="match_parent"
13
+ android:fillViewport="true">
14
+
15
+ <LinearLayout
16
+ android:layout_width="match_parent"
17
+ android:layout_height="wrap_content"
18
+ android:orientation="vertical"
19
+ android:padding="24dp">
20
+
21
+ <TextView
22
+ android:layout_width="match_parent"
23
+ android:layout_height="wrap_content"
24
+ android:layout_marginTop="24dp"
25
+ android:text="@string/title"
26
+ android:textColor="@android:color/white"
27
+ android:textSize="26sp"
28
+ android:textStyle="bold" />
29
+
30
+ <TextView
31
+ android:layout_width="match_parent"
32
+ android:layout_height="wrap_content"
33
+ android:layout_marginTop="12dp"
34
+ android:text="@string/tagline"
35
+ android:textColor="#FFBDBDBD"
36
+ android:textSize="17sp" />
37
+
38
+ <RadioGroup
39
+ android:id="@+id/mode"
40
+ android:layout_width="match_parent"
41
+ android:layout_height="wrap_content"
42
+ android:layout_marginTop="32dp"
43
+ android:orientation="vertical">
44
+
45
+ <RadioButton
46
+ android:id="@+id/mode_photo"
47
+ android:layout_width="wrap_content"
48
+ android:layout_height="wrap_content"
49
+ android:checked="true"
50
+ android:text="@string/mode_photo"
51
+ android:textSize="16sp" />
52
+
53
+ <RadioButton
54
+ android:id="@+id/mode_render"
55
+ android:layout_width="wrap_content"
56
+ android:layout_height="wrap_content"
57
+ android:text="@string/mode_render"
58
+ android:textSize="16sp" />
59
+ </RadioGroup>
60
+
61
+ <Button
62
+ android:id="@+id/pick"
63
+ android:layout_width="match_parent"
64
+ android:layout_height="wrap_content"
65
+ android:layout_marginTop="24dp"
66
+ android:text="@string/pick" />
67
+
68
+ <Button
69
+ android:id="@+id/camera"
70
+ android:layout_width="match_parent"
71
+ android:layout_height="wrap_content"
72
+ android:layout_marginTop="8dp"
73
+ android:text="@string/camera" />
74
+
75
+ <TextView
76
+ android:id="@+id/home_message"
77
+ android:layout_width="match_parent"
78
+ android:layout_height="wrap_content"
79
+ android:layout_marginTop="20dp"
80
+ android:textColor="#FFFFB74D"
81
+ android:textSize="16sp"
82
+ android:visibility="gone" />
83
+
84
+ <View
85
+ android:layout_width="match_parent"
86
+ android:layout_height="0dp"
87
+ android:layout_weight="1" />
88
+
89
+ <CheckBox
90
+ android:id="@+id/dis"
91
+ android:layout_width="wrap_content"
92
+ android:layout_height="wrap_content"
93
+ android:layout_marginTop="24dp"
94
+ android:text="@string/dis"
95
+ android:textColor="#FF9E9E9E"
96
+ android:textSize="14sp" />
97
+
98
+ <TextView
99
+ android:id="@+id/home_status"
100
+ android:layout_width="match_parent"
101
+ android:layout_height="wrap_content"
102
+ android:layout_marginTop="8dp"
103
+ android:textColor="#FF9E9E9E"
104
+ android:textSize="13sp" />
105
+ </LinearLayout>
106
+ </ScrollView>
107
+
108
+ <!-- Run -->
109
+ <ScrollView
110
+ android:id="@+id/run"
111
+ android:layout_width="match_parent"
112
+ android:layout_height="match_parent"
113
+ android:visibility="gone">
114
+
115
+ <LinearLayout
116
+ android:layout_width="match_parent"
117
+ android:layout_height="wrap_content"
118
+ android:orientation="vertical"
119
+ android:padding="16dp">
120
+
121
+ <LinearLayout
122
+ android:layout_width="match_parent"
123
+ android:layout_height="wrap_content"
124
+ android:orientation="horizontal">
125
+
126
+ <ImageView
127
+ android:id="@+id/picture"
128
+ android:layout_width="0dp"
129
+ android:layout_height="180dp"
130
+ android:layout_marginEnd="8dp"
131
+ android:layout_weight="1"
132
+ android:contentDescription="@string/picture_description"
133
+ android:scaleType="fitCenter" />
134
+
135
+ <ImageView
136
+ android:id="@+id/cutout"
137
+ android:layout_width="0dp"
138
+ android:layout_height="180dp"
139
+ android:layout_weight="1"
140
+ android:contentDescription="@string/cutout_description"
141
+ android:scaleType="fitCenter"
142
+ android:visibility="invisible" />
143
+ </LinearLayout>
144
+
145
+ <LinearLayout
146
+ android:id="@+id/stages"
147
+ android:layout_width="match_parent"
148
+ android:layout_height="wrap_content"
149
+ android:layout_marginTop="20dp"
150
+ android:orientation="vertical" />
151
+
152
+ <TextView
153
+ android:id="@+id/total"
154
+ android:layout_width="match_parent"
155
+ android:layout_height="wrap_content"
156
+ android:layout_marginTop="16dp"
157
+ android:textColor="@android:color/white"
158
+ android:textSize="17sp"
159
+ android:textStyle="bold" />
160
+
161
+ <TextView
162
+ android:id="@+id/run_status"
163
+ android:layout_width="match_parent"
164
+ android:layout_height="wrap_content"
165
+ android:layout_marginTop="8dp"
166
+ android:textColor="#FF9E9E9E"
167
+ android:textSize="13sp" />
168
+ </LinearLayout>
169
+ </ScrollView>
170
+
171
+ <!-- Result -->
172
+ <FrameLayout
173
+ android:id="@+id/result"
174
+ android:layout_width="match_parent"
175
+ android:layout_height="match_parent"
176
+ android:visibility="gone">
177
+
178
+ <FrameLayout
179
+ android:id="@+id/viewer"
180
+ android:layout_width="match_parent"
181
+ android:layout_height="match_parent" />
182
+
183
+ <TextView
184
+ android:layout_width="wrap_content"
185
+ android:layout_height="wrap_content"
186
+ android:layout_gravity="top|center_horizontal"
187
+ android:layout_marginTop="12dp"
188
+ android:text="@string/viewer_hint"
189
+ android:textColor="#B3FFFFFF"
190
+ android:textSize="13sp" />
191
+
192
+ <LinearLayout
193
+ android:layout_width="match_parent"
194
+ android:layout_height="wrap_content"
195
+ android:layout_gravity="bottom"
196
+ android:gravity="center"
197
+ android:orientation="horizontal"
198
+ android:padding="12dp">
199
+
200
+ <Button
201
+ android:id="@+id/share"
202
+ android:layout_width="0dp"
203
+ android:layout_height="wrap_content"
204
+ android:layout_marginEnd="8dp"
205
+ android:layout_weight="1"
206
+ android:text="@string/share" />
207
+
208
+ <Button
209
+ android:id="@+id/new_run"
210
+ android:layout_width="0dp"
211
+ android:layout_height="wrap_content"
212
+ android:layout_weight="1"
213
+ android:text="@string/new_run" />
214
+ </LinearLayout>
215
+ </FrameLayout>
216
+ </FrameLayout>
android/app/src/main/res/values/strings.xml ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <?xml version="1.0" encoding="utf-8"?>
2
+ <resources>
3
+ <string name="app_name">Idol</string>
4
+ <string name="title">Idol — one photo to 3D</string>
5
+ <string name="tagline">One photo in, a 3D object out — including the sides the picture never showed.</string>
6
+ <string name="mode_photo">Photo</string>
7
+ <string name="mode_render">Render or illustration</string>
8
+ <string name="pick">Pick a photo</string>
9
+ <string name="camera">Take a photo</string>
10
+ <string name="dis">Background removal on the GPU (IS-Net DIS)</string>
11
+ <string name="picture_description">The picture</string>
12
+ <string name="cutout_description">The cut-out the 3D model is made from</string>
13
+ <string name="share">Share .ply</string>
14
+ <string name="new_run">New</string>
15
+ <string name="viewer_hint">Drag to turn, pinch to zoom</string>
16
+ <string name="no_object">No object found in this picture.</string>
17
+ <string name="camera_denied">Camera permission is off. You can still pick a photo.</string>
18
+ <string name="no_camera_app">No camera app is available. You can still pick a photo.</string>
19
+ <string name="no_share_app">No app can receive the .ply.</string>
20
+ <string name="stopping">Stopping after the current step…</string>
21
+ </resources>
android/app/src/main/res/values/themes.xml ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ <?xml version="1.0" encoding="utf-8"?>
2
+ <resources>
3
+ <style name="Theme.Idol" parent="android:Theme.Material.NoActionBar">
4
+ <item name="android:windowBackground">@android:color/black</item>
5
+ </style>
6
+ </resources>
android/app/src/main/res/xml/file_paths.xml ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ <?xml version="1.0" encoding="utf-8"?>
2
+ <paths>
3
+ <!-- the camera app writes the photo here -->
4
+ <cache-path name="camera" path="camera/" />
5
+ <!-- the .ply files "Share .ply" hands out -->
6
+ <files-path name="results" path="results/" />
7
+ </paths>
android/build.gradle.kts ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ plugins {
2
+ alias(libs.plugins.android.application) apply false
3
+ alias(libs.plugins.kotlin.android) apply false
4
+ }
android/gradle.properties ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ android.useAndroidX=true
2
+ android.builder.sdkDownload=false
3
+ org.gradle.jvmargs=-Xmx3g -Dfile.encoding=UTF-8
4
+ org.gradle.daemon=false
5
+ org.gradle.workers.max=4
6
+ kotlin.compiler.execution.strategy=in-process
7
+ kotlin.code.style=official
android/gradle/libs.versions.toml ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [versions]
2
+ agp = "8.9.1"
3
+ kotlin = "2.2.21"
4
+ litert = "2.3.0"
5
+ core = "1.15.0"
6
+ activity = "1.10.1"
7
+ webkit = "1.12.1"
8
+
9
+ [libraries]
10
+ litert = { module = "com.google.ai.edge.litert:litert", version.ref = "litert" }
11
+ litert-gpu = { module = "com.google.ai.edge.litert:litert-gpu", version.ref = "litert" }
12
+ androidx-core-ktx = { module = "androidx.core:core-ktx", version.ref = "core" }
13
+ androidx-activity-ktx = { module = "androidx.activity:activity-ktx", version.ref = "activity" }
14
+ androidx-webkit = { module = "androidx.webkit:webkit", version.ref = "webkit" }
15
+
16
+ [plugins]
17
+ android-application = { id = "com.android.application", version.ref = "agp" }
18
+ kotlin-android = { id = "org.jetbrains.kotlin.android", version.ref = "kotlin" }
android/gradle/wrapper/gradle-wrapper.jar ADDED
Binary file (43.7 kB). View file
 
android/gradle/wrapper/gradle-wrapper.properties ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ distributionBase=GRADLE_USER_HOME
2
+ distributionPath=wrapper/dists
3
+ distributionUrl=https\://services.gradle.org/distributions/gradle-8.11.1-bin.zip
4
+ networkTimeout=10000
5
+ validateDistributionUrl=true
6
+ zipStoreBase=GRADLE_USER_HOME
7
+ zipStorePath=wrapper/dists
android/gradlew ADDED
@@ -0,0 +1,251 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/sh
2
+
3
+ #
4
+ # Copyright © 2015-2021 the original authors.
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # https://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+ #
18
+ # SPDX-License-Identifier: Apache-2.0
19
+ #
20
+
21
+ ##############################################################################
22
+ #
23
+ # Gradle start up script for POSIX generated by Gradle.
24
+ #
25
+ # Important for running:
26
+ #
27
+ # (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
28
+ # noncompliant, but you have some other compliant shell such as ksh or
29
+ # bash, then to run this script, type that shell name before the whole
30
+ # command line, like:
31
+ #
32
+ # ksh Gradle
33
+ #
34
+ # Busybox and similar reduced shells will NOT work, because this script
35
+ # requires all of these POSIX shell features:
36
+ # * functions;
37
+ # * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
38
+ # «${var#prefix}», «${var%suffix}», and «$( cmd )»;
39
+ # * compound commands having a testable exit status, especially «case»;
40
+ # * various built-in commands including «command», «set», and «ulimit».
41
+ #
42
+ # Important for patching:
43
+ #
44
+ # (2) This script targets any POSIX shell, so it avoids extensions provided
45
+ # by Bash, Ksh, etc; in particular arrays are avoided.
46
+ #
47
+ # The "traditional" practice of packing multiple parameters into a
48
+ # space-separated string is a well documented source of bugs and security
49
+ # problems, so this is (mostly) avoided, by progressively accumulating
50
+ # options in "$@", and eventually passing that to Java.
51
+ #
52
+ # Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
53
+ # and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
54
+ # see the in-line comments for details.
55
+ #
56
+ # There are tweaks for specific operating systems such as AIX, CygWin,
57
+ # Darwin, MinGW, and NonStop.
58
+ #
59
+ # (3) This script is generated from the Groovy template
60
+ # https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
61
+ # within the Gradle project.
62
+ #
63
+ # You can find Gradle at https://github.com/gradle/gradle/.
64
+ #
65
+ ##############################################################################
66
+
67
+ # Attempt to set APP_HOME
68
+
69
+ # Resolve links: $0 may be a link
70
+ app_path=$0
71
+
72
+ # Need this for daisy-chained symlinks.
73
+ while
74
+ APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
75
+ [ -h "$app_path" ]
76
+ do
77
+ ls=$( ls -ld "$app_path" )
78
+ link=${ls#*' -> '}
79
+ case $link in #(
80
+ /*) app_path=$link ;; #(
81
+ *) app_path=$APP_HOME$link ;;
82
+ esac
83
+ done
84
+
85
+ # This is normally unused
86
+ # shellcheck disable=SC2034
87
+ APP_BASE_NAME=${0##*/}
88
+ # Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
89
+ APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
90
+
91
+ # Use the maximum available, or set MAX_FD != -1 to use that value.
92
+ MAX_FD=maximum
93
+
94
+ warn () {
95
+ echo "$*"
96
+ } >&2
97
+
98
+ die () {
99
+ echo
100
+ echo "$*"
101
+ echo
102
+ exit 1
103
+ } >&2
104
+
105
+ # OS specific support (must be 'true' or 'false').
106
+ cygwin=false
107
+ msys=false
108
+ darwin=false
109
+ nonstop=false
110
+ case "$( uname )" in #(
111
+ CYGWIN* ) cygwin=true ;; #(
112
+ Darwin* ) darwin=true ;; #(
113
+ MSYS* | MINGW* ) msys=true ;; #(
114
+ NONSTOP* ) nonstop=true ;;
115
+ esac
116
+
117
+ CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
118
+
119
+
120
+ # Determine the Java command to use to start the JVM.
121
+ if [ -n "$JAVA_HOME" ] ; then
122
+ if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
123
+ # IBM's JDK on AIX uses strange locations for the executables
124
+ JAVACMD=$JAVA_HOME/jre/sh/java
125
+ else
126
+ JAVACMD=$JAVA_HOME/bin/java
127
+ fi
128
+ if [ ! -x "$JAVACMD" ] ; then
129
+ die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
130
+
131
+ Please set the JAVA_HOME variable in your environment to match the
132
+ location of your Java installation."
133
+ fi
134
+ else
135
+ JAVACMD=java
136
+ if ! command -v java >/dev/null 2>&1
137
+ then
138
+ die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
139
+
140
+ Please set the JAVA_HOME variable in your environment to match the
141
+ location of your Java installation."
142
+ fi
143
+ fi
144
+
145
+ # Increase the maximum file descriptors if we can.
146
+ if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
147
+ case $MAX_FD in #(
148
+ max*)
149
+ # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
150
+ # shellcheck disable=SC2039,SC3045
151
+ MAX_FD=$( ulimit -H -n ) ||
152
+ warn "Could not query maximum file descriptor limit"
153
+ esac
154
+ case $MAX_FD in #(
155
+ '' | soft) :;; #(
156
+ *)
157
+ # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
158
+ # shellcheck disable=SC2039,SC3045
159
+ ulimit -n "$MAX_FD" ||
160
+ warn "Could not set maximum file descriptor limit to $MAX_FD"
161
+ esac
162
+ fi
163
+
164
+ # Collect all arguments for the java command, stacking in reverse order:
165
+ # * args from the command line
166
+ # * the main class name
167
+ # * -classpath
168
+ # * -D...appname settings
169
+ # * --module-path (only if needed)
170
+ # * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
171
+
172
+ # For Cygwin or MSYS, switch paths to Windows format before running java
173
+ if "$cygwin" || "$msys" ; then
174
+ APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
175
+ CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
176
+
177
+ JAVACMD=$( cygpath --unix "$JAVACMD" )
178
+
179
+ # Now convert the arguments - kludge to limit ourselves to /bin/sh
180
+ for arg do
181
+ if
182
+ case $arg in #(
183
+ -*) false ;; # don't mess with options #(
184
+ /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
185
+ [ -e "$t" ] ;; #(
186
+ *) false ;;
187
+ esac
188
+ then
189
+ arg=$( cygpath --path --ignore --mixed "$arg" )
190
+ fi
191
+ # Roll the args list around exactly as many times as the number of
192
+ # args, so each arg winds up back in the position where it started, but
193
+ # possibly modified.
194
+ #
195
+ # NB: a `for` loop captures its iteration list before it begins, so
196
+ # changing the positional parameters here affects neither the number of
197
+ # iterations, nor the values presented in `arg`.
198
+ shift # remove old arg
199
+ set -- "$@" "$arg" # push replacement arg
200
+ done
201
+ fi
202
+
203
+
204
+ # Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
205
+ DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
206
+
207
+ # Collect all arguments for the java command:
208
+ # * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
209
+ # and any embedded shellness will be escaped.
210
+ # * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
211
+ # treated as '${Hostname}' itself on the command line.
212
+
213
+ set -- \
214
+ "-Dorg.gradle.appname=$APP_BASE_NAME" \
215
+ -classpath "$CLASSPATH" \
216
+ org.gradle.wrapper.GradleWrapperMain \
217
+ "$@"
218
+
219
+ # Stop when "xargs" is not available.
220
+ if ! command -v xargs >/dev/null 2>&1
221
+ then
222
+ die "xargs is not available"
223
+ fi
224
+
225
+ # Use "xargs" to parse quoted args.
226
+ #
227
+ # With -n1 it outputs one arg per line, with the quotes and backslashes removed.
228
+ #
229
+ # In Bash we could simply go:
230
+ #
231
+ # readarray ARGS < <( xargs -n1 <<<"$var" ) &&
232
+ # set -- "${ARGS[@]}" "$@"
233
+ #
234
+ # but POSIX shell has neither arrays nor command substitution, so instead we
235
+ # post-process each arg (as a line of input to sed) to backslash-escape any
236
+ # character that might be a shell metacharacter, then use eval to reverse
237
+ # that process (while maintaining the separation between arguments), and wrap
238
+ # the whole thing up as a single "set" statement.
239
+ #
240
+ # This will of course break if any of these variables contains a newline or
241
+ # an unmatched quote.
242
+ #
243
+
244
+ eval "set -- $(
245
+ printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
246
+ xargs -n1 |
247
+ sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
248
+ tr '\n' ' '
249
+ )" '"$@"'
250
+
251
+ exec "$JAVACMD" "$@"
android/gradlew.bat ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ @rem
2
+ @rem Copyright 2015 the original author or authors.
3
+ @rem
4
+ @rem Licensed under the Apache License, Version 2.0 (the "License");
5
+ @rem you may not use this file except in compliance with the License.
6
+ @rem You may obtain a copy of the License at
7
+ @rem
8
+ @rem https://www.apache.org/licenses/LICENSE-2.0
9
+ @rem
10
+ @rem Unless required by applicable law or agreed to in writing, software
11
+ @rem distributed under the License is distributed on an "AS IS" BASIS,
12
+ @rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ @rem See the License for the specific language governing permissions and
14
+ @rem limitations under the License.
15
+ @rem
16
+ @rem SPDX-License-Identifier: Apache-2.0
17
+ @rem
18
+
19
+ @if "%DEBUG%"=="" @echo off
20
+ @rem ##########################################################################
21
+ @rem
22
+ @rem Gradle startup script for Windows
23
+ @rem
24
+ @rem ##########################################################################
25
+
26
+ @rem Set local scope for the variables with windows NT shell
27
+ if "%OS%"=="Windows_NT" setlocal
28
+
29
+ set DIRNAME=%~dp0
30
+ if "%DIRNAME%"=="" set DIRNAME=.
31
+ @rem This is normally unused
32
+ set APP_BASE_NAME=%~n0
33
+ set APP_HOME=%DIRNAME%
34
+
35
+ @rem Resolve any "." and ".." in APP_HOME to make it shorter.
36
+ for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
37
+
38
+ @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
39
+ set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
40
+
41
+ @rem Find java.exe
42
+ if defined JAVA_HOME goto findJavaFromJavaHome
43
+
44
+ set JAVA_EXE=java.exe
45
+ %JAVA_EXE% -version >NUL 2>&1
46
+ if %ERRORLEVEL% equ 0 goto execute
47
+
48
+ echo. 1>&2
49
+ echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
50
+ echo. 1>&2
51
+ echo Please set the JAVA_HOME variable in your environment to match the 1>&2
52
+ echo location of your Java installation. 1>&2
53
+
54
+ goto fail
55
+
56
+ :findJavaFromJavaHome
57
+ set JAVA_HOME=%JAVA_HOME:"=%
58
+ set JAVA_EXE=%JAVA_HOME%/bin/java.exe
59
+
60
+ if exist "%JAVA_EXE%" goto execute
61
+
62
+ echo. 1>&2
63
+ echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
64
+ echo. 1>&2
65
+ echo Please set the JAVA_HOME variable in your environment to match the 1>&2
66
+ echo location of your Java installation. 1>&2
67
+
68
+ goto fail
69
+
70
+ :execute
71
+ @rem Setup the command line
72
+
73
+ set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
74
+
75
+
76
+ @rem Execute Gradle
77
+ "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
78
+
79
+ :end
80
+ @rem End local scope for the variables with windows NT shell
81
+ if %ERRORLEVEL% equ 0 goto mainEnd
82
+
83
+ :fail
84
+ rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
85
+ rem the _cmd.exe /c_ return code!
86
+ set EXIT_CODE=%ERRORLEVEL%
87
+ if %EXIT_CODE% equ 0 set EXIT_CODE=1
88
+ if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
89
+ exit /b %EXIT_CODE%
90
+
91
+ :mainEnd
92
+ if "%OS%"=="Windows_NT" endlocal
93
+
94
+ :omega
android/licenses/Pillow-MIT-CMU.txt ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The Python Imaging Library (PIL) is
2
+
3
+ Copyright © 1997-2011 by Secret Labs AB
4
+ Copyright © 1995-2011 by Fredrik Lundh and contributors
5
+
6
+ Pillow is the friendly PIL fork. It is
7
+
8
+ Copyright © 2010 by Jeffrey 'Alex' Clark and contributors
9
+
10
+ Like PIL, Pillow is licensed under the open source MIT-CMU License:
11
+
12
+ By obtaining, using, and/or copying this software and/or its associated
13
+ documentation, you agree that you have read, understood, and will comply
14
+ with the following terms and conditions:
15
+
16
+ Permission to use, copy, modify and distribute this software and its
17
+ documentation for any purpose and without fee is hereby granted,
18
+ provided that the above copyright notice appears in all copies, and that
19
+ both that copyright notice and this permission notice appear in supporting
20
+ documentation, and that the name of Secret Labs AB or the author not be
21
+ used in advertising or publicity pertaining to distribution of the software
22
+ without specific, written prior permission.
23
+
24
+ SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
25
+ SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
26
+ IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR BE LIABLE FOR ANY SPECIAL,
27
+ INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
28
+ LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
29
+ OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
30
+ PERFORMANCE OF THIS SOFTWARE.
android/scripts/install_to_device.sh ADDED
@@ -0,0 +1,165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Copy Idol's graphs, side files and manifest into the app's private files/models/, one file at a time
3
+ # (adb push -> run-as cp -> rm), and with FIXTURES=1 also the test fixtures (device/fx/<tag>/ -> files/fx/<tag>/) and
4
+ # the autorun plans (plans/*.json -> files/plans/).
5
+ #
6
+ # The file list, sizes and SHA-256 come from manifest.json in IDOL_DIR. Sources:
7
+ # IDOL_DIR manifest.json, every graph except dino_* and dis, the side and JSON files
8
+ # DINO_DIR dino_0..3.tflite (and dino_rope_cos/sin.f32 when they are there, else IDOL_DIR's)
9
+ # DIS_DIR dis.tflite (optional: only the "Background removal on the GPU" setting uses it)
10
+ # Every source is checked by size before any device command; after each copy the SHA-256 is checked on the phone
11
+ # (the manifest's value for a graph, the local file's for the rest). A file whose copy on the phone already has the
12
+ # expected SHA-256 is not copied again. MODELS=0 skips the models (fixtures and plans only, with FIXTURES=1).
13
+ # Install the debug APK first: run-as needs a debuggable package. ANDROID_SERIAL selects the device. Needs python3
14
+ # (reads the manifest).
15
+ set -euo pipefail
16
+
17
+ MODULE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
18
+
19
+ # Defaults for local testing: this worktree's export (models/v2 holds every graph except dis.tflite).
20
+ # For the published release point them at the downloaded repos (see README), e.g.
21
+ # IDOL_DIR=~/Downloads/Idol DINO_DIR=~/Downloads/DINOv3-ViT-H16plus-Idol-LiteRT DIS_DIR=~/Downloads/DIS-ISNet-LiteRT
22
+ IDOL_DIR="${IDOL_DIR:-$MODULE_DIR/models/v2}"
23
+ DINO_DIR="${DINO_DIR:-$IDOL_DIR}"
24
+ DIS_DIR="${DIS_DIR:-$HOME/Downloads/DIS-ISNet-LiteRT}"
25
+ MODELS="${MODELS:-1}"
26
+ FIXTURES="${FIXTURES:-0}"
27
+ FX_TAGS="${FX_TAGS:-armchair axolotl succulent black}"
28
+ PACKAGE=com.idol
29
+ TEMP_DIR=/data/local/tmp/idol_install
30
+
31
+ if [[ $# -gt 0 ]]; then
32
+ printf 'Usage: [IDOL_DIR=...] [DINO_DIR=...] [DIS_DIR=...] [MODELS=0] [FIXTURES=1] %s\n' "$0" >&2
33
+ exit 2
34
+ fi
35
+ MANIFEST="$IDOL_DIR/manifest.json"
36
+ if [[ ! -f "$MANIFEST" ]]; then
37
+ printf 'Missing %s: set IDOL_DIR (and DINO_DIR, DIS_DIR) to the downloaded model folders (see README)\n' \
38
+ "$MANIFEST" >&2
39
+ exit 1
40
+ fi
41
+
42
+ sha_of() {
43
+ if command -v sha256sum > /dev/null; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi
44
+ }
45
+
46
+ # "kind name file bytes sha" per line; bytes and sha are "-" when the manifest has none
47
+ ENTRIES="$(python3 -I - "$MANIFEST" <<'PY'
48
+ import json, sys
49
+ m = json.load(open(sys.argv[1]))
50
+ for name, g in m['graphs'].items():
51
+ print('graph', name, g['file'], g['bytes'], g.get('sha256') or '-')
52
+ for f, s in m.get('side_files', {}).items():
53
+ n = 4
54
+ for d in s['shape']:
55
+ n *= d
56
+ print('side', f, f, n, '-')
57
+ for f in m.get('json_files', []):
58
+ print('json', f, f, '-', '-')
59
+ PY
60
+ )"
61
+
62
+ # The source of each file.
63
+ source_of() {
64
+ local kind="$1" name="$2" file="$3"
65
+ case "$kind:$name" in
66
+ graph:dino_*) printf '%s/%s' "$DINO_DIR" "$file" ;;
67
+ graph:dis) printf '%s/%s' "$DIS_DIR" "$file" ;;
68
+ side:dino_rope_*)
69
+ if [[ -f "$DINO_DIR/$file" ]]; then printf '%s/%s' "$DINO_DIR" "$file"; else printf '%s/%s' "$IDOL_DIR" "$file"; fi
70
+ ;;
71
+ *) printf '%s/%s' "$IDOL_DIR" "$file" ;;
72
+ esac
73
+ }
74
+
75
+ # Check every source before any device command to avoid a partial installation.
76
+ PLAN=()
77
+ if [[ "$MODELS" == 0 ]]; then ENTRIES=''; fi
78
+ while read -r kind name file bytes sha; do
79
+ [[ -z "$kind" ]] && continue
80
+ src="$(source_of "$kind" "$name" "$file")"
81
+ if [[ ! -f "$src" ]]; then
82
+ if [[ "$kind:$name" == graph:dis ]]; then
83
+ printf 'Skipping dis.tflite (not in %s): the GPU background removal setting will not work\n' "$DIS_DIR"
84
+ continue
85
+ fi
86
+ printf 'Missing source file: %s\n' "$src" >&2
87
+ exit 1
88
+ fi
89
+ if [[ "$bytes" != - ]]; then
90
+ actual="$(wc -c < "$src" | tr -d ' ')"
91
+ if [[ "$actual" != "$bytes" ]]; then
92
+ printf '%s is %s bytes, expected %s\n' "$src" "$actual" "$bytes" >&2
93
+ exit 1
94
+ fi
95
+ fi
96
+ # files without a manifest SHA-256 (side and JSON files) are checked against the local copy's
97
+ if [[ "$sha" == - ]]; then sha="$(sha_of "$src")"; fi
98
+ PLAN+=("$src|models/$file|$sha")
99
+ done <<< "$ENTRIES"
100
+ if [[ "$MODELS" != 0 ]]; then
101
+ PLAN+=("$MANIFEST|models/manifest.json|$(sha_of "$MANIFEST")")
102
+ fi
103
+
104
+ if [[ "$FIXTURES" == 1 ]]; then
105
+ for tag in $FX_TAGS; do
106
+ dir="$MODULE_DIR/device/fx/$tag"
107
+ if [[ ! -d "$dir" ]]; then
108
+ printf 'Missing fixture folder: %s\n' "$dir" >&2
109
+ exit 1
110
+ fi
111
+ for f in "$dir"/*; do
112
+ [[ -f "$f" ]] || continue
113
+ PLAN+=("$f|fx/$tag/$(basename "$f")|$(sha_of "$f")")
114
+ done
115
+ done
116
+ for f in "$MODULE_DIR"/plans/*.json; do
117
+ [[ -f "$f" ]] || continue
118
+ PLAN+=("$f|plans/$(basename "$f")|$(sha_of "$f")")
119
+ done
120
+ fi
121
+
122
+ pending=''
123
+ cleanup() {
124
+ if [[ -n "$pending" ]]; then
125
+ adb shell rm -f "$pending" < /dev/null
126
+ fi
127
+ }
128
+ trap cleanup EXIT
129
+
130
+ # Copies $1 (a local file) to files/$2 of the app and checks the SHA-256 $3 on the phone ("-": no check).
131
+ install_file() {
132
+ local src="$1" dst="$2" sha="$3"
133
+ pending="$TEMP_DIR/$$-$(basename "$dst")"
134
+ adb push "$src" "$pending" < /dev/null
135
+ adb shell run-as "$PACKAGE" mkdir -p "files/$(dirname "$dst")" < /dev/null
136
+ adb shell run-as "$PACKAGE" cp "$pending" "files/$dst" < /dev/null
137
+ adb shell rm "$pending" < /dev/null
138
+ pending=''
139
+ if [[ "$sha" != - ]]; then
140
+ local got
141
+ got="$(adb shell run-as "$PACKAGE" sha256sum "files/$dst" < /dev/null | cut -d' ' -f1 | tr -d '\r')"
142
+ if [[ "$got" != "$sha" ]]; then
143
+ printf 'files/%s on the device has SHA-256 %s, expected %s\n' "$dst" "$got" "$sha" >&2
144
+ exit 1
145
+ fi
146
+ fi
147
+ }
148
+
149
+ adb shell mkdir -p "$TEMP_DIR" < /dev/null
150
+ adb shell run-as "$PACKAGE" mkdir -p files/models < /dev/null
151
+ for entry in "${PLAN[@]}"; do
152
+ IFS='|' read -r src dst sha <<< "$entry"
153
+ if [[ "$sha" != - ]]; then
154
+ have="$(adb shell run-as "$PACKAGE" sha256sum "files/$dst" < /dev/null 2> /dev/null | cut -d' ' -f1 | tr -d '\r' || true)"
155
+ if [[ "$have" == "$sha" ]]; then
156
+ printf 'Up to date: %s\n' "$dst"
157
+ continue
158
+ fi
159
+ fi
160
+ printf 'Installing %s (%s bytes)\n' "$dst" "$(wc -c < "$src" | tr -d ' ')"
161
+ install_file "$src" "$dst" "$sha"
162
+ done
163
+ adb shell rmdir "$TEMP_DIR" < /dev/null 2> /dev/null || true
164
+ adb shell run-as "$PACKAGE" ls -la files/models < /dev/null
165
+ printf 'Files installed. Open Idol.\n'
android/settings.gradle.kts ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pluginManagement {
2
+ repositories {
3
+ google()
4
+ mavenCentral()
5
+ gradlePluginPortal()
6
+ }
7
+ }
8
+
9
+ dependencyResolutionManagement {
10
+ repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
11
+ repositories {
12
+ google()
13
+ mavenCentral()
14
+ }
15
+ }
16
+
17
+ rootProject.name = "idol"
18
+ include(":app")
bir.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42b98bf0e497dd596327c31883fec21fe7013a4c9f72d1f67ec8ff4d3f0dfd43
3
+ size 930410016
den_ctx.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39278b27d00a3dcc70da5e2737f6055d5ba4e7e817423a4cf176aacbfb86230d
3
+ size 117711308
den_in.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1350c12da702a4a0abf53cdf528b62a8a75ea4d948c6a4764b264b927b419802
3
+ size 146782524
den_joint_00.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:37e269459ef1325bd4a84d1df1545afacf8e33d6d5c232c62f834f343a882f86
3
+ size 56134468
den_joint_01.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:103efe94ce85aeb29f084f85962d0e85d4a395c629101d3ea21d23e6cdc53e25
3
+ size 56134100
den_joint_02.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ec688cf610156841a1523db9ed93ad268274f957132dcbc92539308143ce517
3
+ size 56134100
den_joint_03.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b5828ac85b0f42118e79f2e83cbe05021d2b68f9a3c167f0ac014a3026799cc2
3
+ size 56134100
den_joint_04.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ea5c57b89c8d8bbc5144d5d487e20fc49daf25308bccf853bb3c9d4581ae33fe
3
+ size 56134100
den_joint_05.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f339d8b79e5d4c2fc408c2d42f7ce959b81821cad37861b2a1294b8397e19997
3
+ size 56134100
den_joint_06.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30d97e62ac4fc28882903961c88e65e6fe540d61e95504651b44d7cb8f341c4c
3
+ size 56134100
den_joint_07.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c7b243e55c69b6cdd9c5e7bbef313f6cc6be0c0f7ad32e428187e22b2183a67
3
+ size 56134100
den_joint_08.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0eb17bea8ae04ea0d88d79b327515e1a106bb5339d72440a8ec63712865e3f27
3
+ size 56134100
den_joint_09.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:353ece07e6b4888d80ab08e2bdd6dc2dd456f213ca87694f6d77f987d5d4debc
3
+ size 56134100
den_joint_10.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c7e15340ea4212369ae85f3f9db625f21ccd1747038e0725675c5a7ca32ac07
3
+ size 56134100