Rewrite the model card

README.md

@@ -1,90 +1,136 @@
 ---
 license: apache-2.0
-base_model: depth-anything/DA3-SMALL
 library_name: coreml
 pipeline_tag: depth-estimation
+base_model: depth-anything/DA3-SMALL
+base_model_relation: quantized
 tags:
   - coreml
   - core-ml
-  - depth-estimation
+  - depth-anything
   - depth-anything-3
-  - DA3
+  - depth-anything-3-small
+  - da3
+  - da3-small
+  - depth-estimation
   - multi-view
+  - multi-view-depth
   - multi-view-geometry
   - pose-conditioned
+  - camera-pose
+  - pose-estimation
+  - 3d-reconstruction
+  - 3d-scanning
   - arkit
   - ios
   - iphone
+  - iphone-12-pro
+  - iphone-17-pro
+  - apple-silicon
   - apple-neural-engine
+  - ane
   - on-device
+  - swift
 ---
 
-# Depth Anything 3 (DA3) Small — Multi-View, Pose-Conditioned Core ML for iPhone
+# Depth Anything 3 Small — Multi-View Core ML for iPhone
 
-Core ML packages of [Depth Anything 3 Small](https://huggingface.co/depth-anything/DA3-SMALL) (`DA3-SMALL`,
-revision `e08cab65ca0ec38e7826075418411ab90cab4da3`) that run the **multi-view, pose-conditioned** path of the
-network on iPhone. Each model takes 2 or 3 images together with their camera poses (extrinsics and intrinsics, for
-example from ARKit) and returns, for every image, a depth map, a confidence map and the camera pose the network
-estimates. The variants are rewritten to run faster and use less memory on the iPhone Neural Engine than a plain
-conversion.
+**Multi-view depth with camera poses, running on the iPhone Neural Engine. `DA3-Small-3views-392-fast` is
+recommended: 2.5x (iPhone 17 Pro) to 3.2x (iPhone 12 Pro) faster than the full-size 504 file.**
 
-Existing Core ML conversions of DA3 Small take a single image. These packages keep the camera encoder and decoder,
-so depth from several views is consistent across the views and can be brought to metres with the input poses.
+[Depth Anything 3](https://huggingface.co/depth-anything/DA3-SMALL) (DA3) Small, converted to Core ML with the
+multi-view, pose-conditioned path of the network kept. Give it 2 or 3 photos and their camera poses (for example from
+ARKit), and it returns a depth map and a confidence map for every photo, plus the camera pose it estimates. The depth
+of all views comes out in one consistent frame, so you can fuse it into a mesh or a point cloud and bring it to metres
+with the poses you passed. Other Core ML conversions of DA3 Small take one image at a time.
 
-## Variants
+The graph is rewritten for the iPhone: the attention across views is computed in slices, and the depth head and the
+input size are changed for speed. The weights are the original ones in FP16.
 
-| Package | Views | Input | What is changed | Minimum iOS | Size |
+## Which file should I choose?
+
+| File | Views | Input | What is changed | Min iOS | Size |
 |---|---|---|---|---|---|
-| `models/DA3-Small-3views-392-fast.mlpackage` (recommended) | 3 | 392 × 392 | 392 input, global attention in 8 slices, depth head rewritten | 18.0 | 72.0 MB |
-| `models/DA3-Small-2views-392-fast.mlpackage` | 2 | 392 × 392 | 392 input, global attention in 16 slices, depth head rewritten | 18.0 | 71.8 MB |
-| `models/DA3-Small-3views-504-sliced.mlpackage` | 3 | 504 × 504 | global attention in 16 slices | 18.0 | 86.8 MB |
-| `models/DA3-Small-3views-504.mlpackage` | 3 | 504 × 504 | none (plain conversion of the pose-conditioned path) | 17.0 | 86.7 MB |
-
-All packages are FP16.
-
-- **Sliced global attention** (`sliced`, `sliced8`): the attention across all views is computed in slices of the
-  query, so the intermediate attention matrices stay small. It only changes the order of the operations: the
-  weights are identical (`DA3-Small-3views-504` and `DA3-Small-3views-504-sliced` have the same `weight.bin`), and the output matches the
-  original PyTorch model (mean depth difference 0.03–0.05% on a Mac).
-- **Head rewrite** (`headcombo`): the DPT depth head uses transposed-convolution upsampling and upsamples later in
-  the head. Depth and confidence change slightly, mostly at object edges. The pose output does not change (the
-  camera decoder reads the backbone, not the head).
-- **392 input**: fewer tokens per image. With the poses from ARKit, the metric scale estimated from the 392
-  variants differs from the 504 sliced variant by a median of 2.6% (3 views) and 4.5% (2 views).
-
-## Measured on iPhone
-
-Two devices only: iPhone 12 Pro (A14, iOS 27.0.1) and iPhone 17 Pro (A19 Pro, iOS 27.0.1), default compute units
-(`.all`). Times are the median inference time of one forward pass (all views). The conditions differ between rows,
-so compare numbers within a row.
-
-| Comparison | Device | Condition | Result |
+| [`DA3-Small-3views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-392-fast.mlpackage) | 3 | 392 × 392 | 392 input, attention in 8 slices, depth head rewritten. **Recommended.** | 18 | 72 MB |
+| [`DA3-Small-2views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-2views-392-fast.mlpackage) | 2 | 392 × 392 | The same speed-ups with 2 views. Fastest. | 18 | 72 MB |
+| [`DA3-Small-3views-504-sliced`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504-sliced.mlpackage) | 3 | 504 × 504 | Attention in 16 slices only. Same output as the original model. | 18 | 87 MB |
+| [`DA3-Small-3views-504`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504.mlpackage) | 3 | 504 × 504 | Nothing. The plain conversion, for comparison and for iOS 17. | 17 | 87 MB |
+
+- Start with **`DA3-Small-3views-392-fast`**. It is the one to use for scanning with ARKit.
+- Use **`DA3-Small-2views-392-fast`** if you only have two photos or need the shortest time per inference.
+- Use **`DA3-Small-3views-504-sliced`** if you want the output of the original model at its native 504 input: the
+  slicing only changes the order of the operations (it has the same `weight.bin` as `DA3-Small-3views-504`), and the
+  mean depth difference from the original PyTorch model is 0.03–0.05% on a Mac.
+- The `fast` files trade a little accuracy for speed. The rewritten head changes depth and confidence slightly, mostly
+  at object edges (the pose output does not change). With ARKit poses, the metric scale from the 392 files differs
+  from the 504 sliced file by a median of 2.6% (3 views) and 4.5% (2 views).
+
+## Speed on iPhone
+
+Median time of one inference (all views at once), default compute units (`.all`), iOS 27.0.1. Measured on two
+devices only. The conditions differ between rows, so compare the numbers within a row.
+
+| Device | Comparison | Condition | Result |
 |---|---|---|---|
-| `DA3-Small-3views-504` → `DA3-Small-3views-504-sliced` | 17 Pro | Release, ARKit poses, handheld, 120 s each from nominal thermal state | 464.3 → 310.5 ms, app memory 1,121 → 762 MB |
-| `DA3-Small-3views-504` → `DA3-Small-3views-504-sliced` | 12 Pro | Release, ARKit poses, handheld, 1 min each, not cooled in between | 774 → 566 ms, Neural Engine memory 760 → 367 MB |
-| `DA3-Small-3views-504-sliced` → `DA3-Small-3views-392-fast` | 17 Pro | Debug build test, fixed images | 224 → 91 ms |
-| `DA3-Small-3views-504-sliced` → `DA3-Small-3views-392-fast` | 12 Pro | Debug build test, fixed images | 579.5 → 183.4 ms |
-| `DA3-Small-2views-392-fast` | 17 Pro | Debug build test, fixed images | 53.6–54.6 ms |
-
-Notes from these devices:
-
-- On the iPhone 17 Pro Neural Engine, the 2-view 504 sliced package does not return (the first inference stalls
-  and is cancelled). The 2-view 392 package above returns.
-- A 3-view 392 package with 16 slices also does not return on the iPhone 17 Pro, which is why the recommended
-  3-view variant uses 8 slices.
+| iPhone 17 Pro (A19 Pro) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 224 → **91 ms** |
+| iPhone 12 Pro (A14) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 579.5 → **183.4 ms** |
+| iPhone 17 Pro (A19 Pro) | `2views-392-fast` | Fixed images, Debug build | **53.6–54.6 ms** |
+| iPhone 17 Pro (A19 Pro) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 120 s each from the nominal thermal state | 464.3 → **310.5 ms**, app memory 1,121 → 762 MB |
+| iPhone 12 Pro (A14) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 1 min each | 774 → **566 ms**, Neural Engine memory 760 → 367 MB |
+
+Things to know on these devices:
+
 - The first load compiles the model for the Neural Engine and can take one to several minutes on an iPhone 12 Pro.
+  Later loads are quick.
+- A 2-view 504 file sliced in 16 does not return on the iPhone 17 Pro Neural Engine (the first inference stalls and is
+  cancelled), and neither does a 3-view 392 file sliced in 16. That is why the files here are 2 views at 392, and 3
+  views at 392 sliced in 8.
+
+## Use it
+
+Download one file (a Core ML package is a folder):
+
+```bash
+hf download shu223/Depth-Anything-3-Small-MultiView-CoreML \
+  --include "models/DA3-Small-3views-392-fast.mlpackage/*" --local-dir .
+```
+
+Then add the `.mlpackage` to your Xcode project, or download it at run time and compile it on the device with
+`MLModel.compileModel(at:)`. A minimal prediction in Swift:
+
+```swift
+import CoreML
+
+let compiledURL = try await MLModel.compileModel(at: packageURL)   // once; keep the .mlmodelc
+let model = try MLModel(contentsOf: compiledURL)
+
+// 3 views, row-major: element (0, view, row, column)
+let extrinsics = try MLMultiArray(shape: [1, 3, 4, 4], dataType: .float32)  // normalised world-to-camera
+let intrinsics = try MLMultiArray(shape: [1, 3, 3, 3], dataType: .float32)  // of the 392 x 392 images
+// ... fill both (see Inputs below)
+
+let input = try MLDictionaryFeatureProvider(dictionary: [
+    "image_0": MLFeatureValue(pixelBuffer: image0),   // 392 x 392 BGRA CVPixelBuffer
+    "image_1": MLFeatureValue(pixelBuffer: image1),
+    "image_2": MLFeatureValue(pixelBuffer: image2),
+    "extrinsics": MLFeatureValue(multiArray: extrinsics),
+    "intrinsics": MLFeatureValue(multiArray: intrinsics),
+])
+let output = try model.prediction(from: input)
+let depth = output.featureValue(for: "depth")!.multiArrayValue!                 // [1, 3, 392, 392]
+let poseEncoding = output.featureValue(for: "pose_encoding")!.multiArrayValue!  // [1, 3, 9]
+```
 
 ## Inputs
 
 | Name | Type | Shape | Content |
 |---|---|---|---|
 | `image_0` … `image_{S-1}` | Image (RGB, 8-bit) | 392 × 392 or 504 × 504 | The views. View 0 is the reference view. ImageNet normalisation is inside the model. |
-| `extrinsics` | MultiArray Float32 | (1, S, 4, 4) | World-to-camera matrices in the OpenCV camera convention (x right, y down, z forward), **already normalised** (see below). |
-| `intrinsics` | MultiArray Float32 | (1, S, 3, 3) | Pinhole intrinsics of each model input image in pixels (after cropping and resizing to the input size). Only `fx` and `fy` reach the network. |
+| `extrinsics` | Float32 | (1, S, 4, 4) | World-to-camera matrices in the OpenCV camera axes (x right, y down, z forward), **already normalised** as below. |
+| `intrinsics` | Float32 | (1, S, 3, 3) | Pinhole intrinsics of each input image in pixels, after your crop and resize. Only `fx` and `fy` reach the network. |
 
-The normalisation of the extrinsics is the one the official `DepthAnything3._normalize_extrinsics` (`api.py`) does
-before the forward pass. It is left to the caller so that it runs in Float32 / Float64 instead of the FP16 of the
-model:
+Normalise the extrinsics the way the official `DepthAnything3._normalize_extrinsics` (`api.py`) does before the
+forward pass. It is left out of the model so that it runs in Float32 / Float64 instead of the model's FP16:
 
 ```python
 import numpy as np
@@ -100,41 +146,48 @@ def normalize_extrinsics(w2c):            # w2c: (S, 4, 4) world-to-camera, Open
 
 With 2 views the median is the distance of view 0 to itself, so the divisor is always 0.1.
 
-ARKit's camera axes (`ARCamera.transform`) are x right, y up and z toward the viewer in the sensor's landscape
-orientation, so the camera-to-world from ARKit has to be rotated into the OpenCV axes of the image you pass before it
-is inverted to world-to-camera. For an image rotated 90 degrees clockwise to portrait (the usual way to show it
-upright when the phone is held in portrait), the model's x is ARKit's +y, the model's y is ARKit's +x and the
-model's z is ARKit's −z: `c2w_model = c2w_arkit @ R`, with the columns of `R` being (0, 1, 0), (1, 0, 0) and
-(0, 0, −1). The intrinsics have to follow the same rotation, centre crop and resize.
+From ARKit: `ARCamera.transform` is camera-to-world with x right, y up and z toward the viewer in the sensor's
+landscape orientation. For an image rotated 90 degrees clockwise to portrait, the model's x is ARKit's +y, the model's
+y is ARKit's +x and the model's z is ARKit's −z: `c2w_model = c2w_arkit @ R`, where the columns of `R` are
+(0, 1, 0), (1, 0, 0) and (0, 0, −1). Invert it to get world-to-camera. The intrinsics follow the same rotation, crop
+and resize.
 
 ## Outputs
 
 | Name | Type | Shape | Content |
 |---|---|---|---|
-| `depth` | MultiArray Float16 | (1, S, H, W) | Depth of every view in the normalised frame of the input poses. |
-| `confidence` | MultiArray Float16 | (1, S, H, W) | Confidence of every depth value. |
-| `pose_encoding` | MultiArray Float16 | (1, S, 9) | The pose the network estimates for every view: translation (3), quaternion xyzw (4), horizontal and vertical field of view. The translation is the camera position (camera-to-world) in the normalised frame. |
+| `depth` | Float16 | (1, S, H, W) | Depth of every view, in the normalised frame of the input poses. |
+| `confidence` | Float16 | (1, S, H, W) | Confidence of every depth value. |
+| `pose_encoding` | Float16 | (1, S, 9) | The camera pose the network estimates for every view: translation (3), quaternion xyzw (4), horizontal and vertical field of view. The translation is the camera position (camera-to-world) in the normalised frame. |
 
-To bring the depth to metres, do what the official `api.py` does: align the estimated camera positions from
-`pose_encoding` to the input camera positions with Umeyama's method (similarity transform) and divide the depth by
-the resulting scale.
+To get metres, do what the official `api.py` does: align the camera positions in `pose_encoding` to the camera
+positions you passed with Umeyama's method (a similarity transform), and divide the depth by its scale.
 
-## Model list
+## Downloading from an app
 
-`models.json` lists every package with the size and SHA-256 of each file inside it, the input size, the number of
-views and the minimum iOS version. Apps can read it to download the files of a package one by one from
-`https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/resolve/main/<path>`, check them and compile
-the package on the device with `MLModel.compileModel(at:)`.
+`models.json` at the top of this repository lists every file with the size and SHA-256 of each file inside its
+package, the input size, the number of views and the minimum iOS version. An app can read it, download the files of
+one package from `https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/resolve/main/<path>`, check
+them and compile the package on the device. Compiling the `.mlpackage` on the device works on iOS 27 (checked on an
+iPhone 12 Pro and the simulator).
 
-## Conversion
+## How it is built
 
 Converted with coremltools from the original PyTorch implementation
-([ByteDance-Seed/Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)), with the changes listed in
-`NOTICE`. Ray decoding, sky handling and the Gaussian branch of the original model are not included.
+([ByteDance-Seed/Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)), `DA3-SMALL` revision
+`e08cab65ca0ec38e7826075418411ab90cab4da3`:
+
+- The pose-conditioned path is kept: the camera encoder turns the extrinsics and intrinsics into the camera tokens,
+  and the camera decoder returns the estimated poses. Ray decoding, sky handling and the Gaussian branch are not
+  included.
+- The number of views and the input size are fixed per file (Core ML needs fixed shapes).
+- **Sliced attention**: the global attention across all views is computed in slices of the query, so the attention
+  matrices stay small on the Neural Engine.
+- **Rewritten depth head** (`fast`): the DPT head upsamples with transposed convolutions, later in the head.
 
 ## License
 
-Apache License 2.0, the same as the original weights. See `LICENSE` and `NOTICE`.
+Apache License 2.0, the same as the original `DA3-SMALL` weights. See `LICENSE` and `NOTICE`.
 
 ## Citation