--- license: apache-2.0 library_name: coreml pipeline_tag: depth-estimation base_model: depth-anything/DA3-SMALL base_model_relation: quantized tags: - coreml - core-ml - depth-anything - depth-anything-3 - depth-anything-3-small - da3 - da3-small - depth-estimation - multi-view - multi-view-depth - multi-view-geometry - pose-conditioned - camera-pose - pose-estimation - 3d-reconstruction - 3d-scanning - arkit - ios - iphone - iphone-12-pro - iphone-17-pro - apple-silicon - apple-neural-engine - ane - on-device - swift --- # Depth Anything 3 Small — Multi-View Core ML for iPhone **Multi-view depth with camera poses, on iPhone. `DA3-Small-3views-392-fast` is recommended: 2.5x (iPhone 17 Pro) to 3.2x (iPhone 12 Pro) faster than the same model at 504 × 504 with sliced attention (`DA3-Small-3views-504-sliced`).** [Depth Anything 3](https://huggingface.co/depth-anything/DA3-SMALL) (DA3) Small, converted to Core ML with the multi-view, pose-conditioned path of the network kept. Give it 2 or 3 photos and their camera poses (for example from ARKit), and it returns a depth map and a confidence map for every photo, plus the camera pose it estimates. The depth of all views comes out in one consistent frame, so you can fuse it into a mesh or a point cloud and bring it to metres with the poses you passed. The other Core ML conversion of DA3 Small on the Hub ([mlboydaisuke/Depth-Anything-3-Small-CoreML](https://huggingface.co/mlboydaisuke/Depth-Anything-3-Small-CoreML), as of November 2026) takes one image at a time. The graph is rewritten for the iPhone: the attention across views is computed in slices, and the depth head and the input size are changed for speed. The weights are the original ones in FP16. ## Which file should I choose? | File | Views | Input | What is changed | Min iOS | Size | |---|---|---|---|---|---| | [`DA3-Small-3views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-392-fast.mlpackage) | 3 | 392 × 392 | 392 input, attention in 8 slices, depth head rewritten. **Recommended.** | 18 | 72 MB | | [`DA3-Small-2views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-2views-392-fast.mlpackage) | 2 | 392 × 392 | The same speed-ups with 2 views. Fastest. | 18 | 72 MB | | [`DA3-Small-3views-504-sliced`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504-sliced.mlpackage) | 3 | 504 × 504 | Attention in 16 slices only. Same output as the original model. | 18 | 87 MB | | [`DA3-Small-3views-504`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504.mlpackage) | 3 | 504 × 504 | Nothing. The plain conversion, for comparison and for iOS 17. | 17 | 87 MB | - Start with **`DA3-Small-3views-392-fast`**. It is the one to use for scanning with ARKit. - Use **`DA3-Small-2views-392-fast`** if you only have two photos or need the shortest time per inference. - Use **`DA3-Small-3views-504-sliced`** if you want the output of the original model at its native 504 input: the slicing only changes the order of the operations (it has the same `weight.bin` as `DA3-Small-3views-504`), and the mean depth difference from the original PyTorch model is 0.03–0.05% on a Mac. - The `fast` files trade a little accuracy for speed. The rewritten head changes depth and confidence slightly, mostly at object edges (the pose output does not change). With ARKit poses, the metric scale from the 392 files differs from the 504 sliced file by a median of 2.6% (3 views) and 4.5% (2 views). ## Speed on iPhone Median time of one inference (all views at once). Measured on two devices only. The conditions differ between rows, so compare the numbers within a row. | Device | Comparison | Condition | Result | |---|---|---|---| | iPhone 17 Pro (A19 Pro) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 224 → **91 ms** | | iPhone 12 Pro (A14) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 579.5 → **183.4 ms** | | iPhone 17 Pro (A19 Pro) | `2views-392-fast` | Fixed images, Debug build | **53.6–54.6 ms** | | iPhone 17 Pro (A19 Pro) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 120 s each from the nominal thermal state | 464.3 → **310.5 ms**, app memory 1,121 → 762 MB | | iPhone 12 Pro (A14) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 1 min each | 774 → **566 ms**, Neural Engine memory 760 → 367 MB | | iPhone 12 Pro (A14) | `3views-392-fast` | Inside a scanning app that also builds a TSDF mesh on the GPU, Debug build, handheld 1–2 min, already in the serious thermal state | 270–293 ms per inference, app memory 330–356 MB | Things to know on these devices: - The first load compiles the model for the Neural Engine and can take one to several minutes on an iPhone 12 Pro. Later loads are quick. - A 2-view 504 file sliced in 16 does not return on the iPhone 17 Pro Neural Engine (the first inference stalls and is cancelled), and neither does a 3-view 392 file sliced in 16. That is why the files here are 2 views at 392, and 3 views at 392 sliced in 8. ## Use it Download one file (a Core ML package is a folder): ```bash hf download shu223/Depth-Anything-3-Small-MultiView-CoreML \ --include "models/DA3-Small-3views-392-fast.mlpackage/*" --local-dir . ``` Then add the `.mlpackage` to your Xcode project, or download it at run time and compile it on the device with `MLModel.compileModel(at:)`. A minimal prediction in Swift: ```swift import CoreML let compiledURL = try await MLModel.compileModel(at: packageURL) // once; keep the .mlmodelc let model = try MLModel(contentsOf: compiledURL) // 3 views, row-major: element (0, view, row, column) let extrinsics = try MLMultiArray(shape: [1, 3, 4, 4], dataType: .float32) // normalised world-to-camera let intrinsics = try MLMultiArray(shape: [1, 3, 3, 3], dataType: .float32) // of the 392 x 392 images // ... fill both (see Inputs below) let input = try MLDictionaryFeatureProvider(dictionary: [ "image_0": MLFeatureValue(pixelBuffer: image0), // 392 x 392 RGB image input: a 32BGRA or 32ARGB CVPixelBuffer "image_1": MLFeatureValue(pixelBuffer: image1), "image_2": MLFeatureValue(pixelBuffer: image2), "extrinsics": MLFeatureValue(multiArray: extrinsics), "intrinsics": MLFeatureValue(multiArray: intrinsics), ]) let output = try model.prediction(from: input) let depth = output.featureValue(for: "depth")!.multiArrayValue! // [1, 3, 392, 392] let poseEncoding = output.featureValue(for: "pose_encoding")!.multiArrayValue! // [1, 3, 9] ``` ## Inputs | Name | Type | Shape | Content | |---|---|---|---| | `image_0` … `image_{S-1}` | Image (RGB, 8-bit) | 392 × 392 or 504 × 504 | The views. View 0 is the reference view. ImageNet normalisation is inside the model. | | `extrinsics` | Float32 | (1, S, 4, 4) | World-to-camera matrices in the OpenCV camera axes (x right, y down, z forward), **already normalised** as below. | | `intrinsics` | Float32 | (1, S, 3, 3) | Pinhole intrinsics of each input image in pixels, after your crop and resize. Only `fx` and `fy` reach the network. | Normalise the extrinsics the way the official `DepthAnything3._normalize_extrinsics` (`api.py`) does before the forward pass. It is left out of the model so that it runs in Float32 / Float64 instead of the model's FP16: ```python import numpy as np def normalize_extrinsics(w2c): # w2c: (S, 4, 4) world-to-camera, OpenCV axes, metres rel = w2c @ np.linalg.inv(w2c[0]) # view 0 becomes the identity centres = np.linalg.inv(rel)[:, :3, 3] dist = np.sort(np.linalg.norm(centres, axis=1)) median = max(dist[(len(dist) - 1) // 2], 0.1) # torch.median: the lower middle value rel[:, :3, 3] /= median return rel.astype(np.float32), median ``` With 2 views the median is the distance of view 0 to itself, so the divisor is always 0.1. From ARKit: `ARCamera.transform` is camera-to-world with x right, y up and z toward the viewer in the sensor's landscape orientation. For an image rotated 90 degrees clockwise to portrait, the model's x is ARKit's +y, the model's y is ARKit's +x and the model's z is ARKit's −z: `c2w_model = c2w_arkit @ R`, where the columns of `R` are (0, 1, 0), (1, 0, 0) and (0, 0, −1). Invert it to get world-to-camera. The intrinsics follow the same rotation, crop and resize. ## Outputs | Name | Type | Shape | Content | |---|---|---|---| | `depth` | Float16 | (1, S, H, W) | Depth of every view, in the normalised frame of the input poses. | | `confidence` | Float16 | (1, S, H, W) | Confidence of every depth value. | | `pose_encoding` | Float16 | (1, S, 9) | The camera pose the network estimates for every view: translation (3), quaternion xyzw (4), horizontal and vertical field of view. The translation is the camera position (camera-to-world) in the normalised frame. | To get metres, do what the official `api.py` does: align the camera positions in `pose_encoding` to the camera positions you passed with Umeyama's method (a similarity transform), and divide the depth by its scale. ## Downloading from an app `models.json` at the top of this repository lists every file with the size and SHA-256 of each file inside its package, the input size, the number of views and the minimum iOS version. An app can read it, download the files of one package from `https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/resolve/main/`, check them and compile the package on the device. Compiling the `.mlpackage` on the device works on iOS 27 (checked on an iPhone 12 Pro and the simulator). ## How it is built Converted with coremltools from the original PyTorch implementation ([ByteDance-Seed/Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)), `DA3-SMALL` revision `e08cab65ca0ec38e7826075418411ab90cab4da3`: - The pose-conditioned path is kept: the camera encoder turns the extrinsics and intrinsics into the camera tokens, and the camera decoder returns the estimated poses. Ray decoding, sky handling and the Gaussian branch are not included. - The number of views and the input size are fixed per file (Core ML needs fixed shapes). - **Sliced attention**: the global attention across all views is computed in slices of the query, so the attention matrices stay small on the Neural Engine. - **Rewritten depth head** (`fast`): the DPT head upsamples with transposed convolutions, and its last convolution runs before the final upsample, so only the two output channels (depth and confidence) are upsampled to full size. ## License Apache License 2.0, the same as the original `DA3-SMALL` weights. See `LICENSE` and `NOTICE`. ## Citation ```bibtex @article{depthanything3, title = {Depth Anything 3: Recovering the Visual Space from Any Views}, author = {Lin, Haotong and Chen, Sili and Liew, Jun Hao and Chen, Donny Y. and Li, Zhenyu and Shi, Guang and Feng, Jiashi and Kang, Bingyi}, journal = {arXiv preprint arXiv:2511.10647}, year = {2025} } ```