README.md · shu223/Depth-Anything-3-Small-MultiView-CoreML at main

Depth-Anything-3-Small-MultiView-CoreML / README.md

Fix the model card

3a9e9fe verified 2 days ago

11.3 kB

---
license: apache-2.0
library_name: coreml
pipeline_tag: depth-estimation
base_model: depth-anything/DA3-SMALL
base_model_relation: quantized
tags:
  - coreml
  - core-ml
  - depth-anything
  - depth-anything-3
  - depth-anything-3-small
  - da3
  - da3-small
  - depth-estimation
  - multi-view
  - multi-view-depth
  - multi-view-geometry
  - pose-conditioned
  - camera-pose
  - pose-estimation
  - 3d-reconstruction
  - 3d-scanning
  - arkit
  - ios
  - iphone
  - iphone-12-pro
  - iphone-17-pro
  - apple-silicon
  - apple-neural-engine
  - ane
  - on-device
  - swift
---
# Depth Anything 3 Small — Multi-View Core ML for iPhone

**Multi-view depth with camera poses, on iPhone. `DA3-Small-3views-392-fast` is
recommended: 2.5x (iPhone 17 Pro) to 3.2x (iPhone 12 Pro) faster than the same model at 504 × 504 with sliced attention (`DA3-Small-3views-504-sliced`).**

[Depth Anything 3](https://huggingface.co/depth-anything/DA3-SMALL) (DA3) Small, converted to Core ML with the
multi-view, pose-conditioned path of the network kept. Give it 2 or 3 photos and their camera poses (for example from
ARKit), and it returns a depth map and a confidence map for every photo, plus the camera pose it estimates. The depth
of all views comes out in one consistent frame, so you can fuse it into a mesh or a point cloud and bring it to metres
with the poses you passed. The other Core ML conversion of DA3 Small on the Hub
([mlboydaisuke/Depth-Anything-3-Small-CoreML](https://huggingface.co/mlboydaisuke/Depth-Anything-3-Small-CoreML), as of
November 2026) takes one image at a time.

The graph is rewritten for the iPhone: the attention across views is computed in slices, and the depth head and the
input size are changed for speed. The weights are the original ones in FP16.

## Which file should I choose?

| File | Views | Input | What is changed | Min iOS | Size |
|---|---|---|---|---|---|
| [`DA3-Small-3views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-392-fast.mlpackage) | 3 | 392 × 392 | 392 input, attention in 8 slices, depth head rewritten. **Recommended.** | 18 | 72 MB |
| [`DA3-Small-2views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-2views-392-fast.mlpackage) | 2 | 392 × 392 | The same speed-ups with 2 views. Fastest. | 18 | 72 MB |
| [`DA3-Small-3views-504-sliced`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504-sliced.mlpackage) | 3 | 504 × 504 | Attention in 16 slices only. Same output as the original model. | 18 | 87 MB |
| [`DA3-Small-3views-504`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504.mlpackage) | 3 | 504 × 504 | Nothing. The plain conversion, for comparison and for iOS 17. | 17 | 87 MB |

- Start with **`DA3-Small-3views-392-fast`**. It is the one to use for scanning with ARKit.
- Use **`DA3-Small-2views-392-fast`** if you only have two photos or need the shortest time per inference.
- Use **`DA3-Small-3views-504-sliced`** if you want the output of the original model at its native 504 input: the
  slicing only changes the order of the operations (it has the same `weight.bin` as `DA3-Small-3views-504`), and the
  mean depth difference from the original PyTorch model is 0.03–0.05% on a Mac.
- The `fast` files trade a little accuracy for speed. The rewritten head changes depth and confidence slightly, mostly
  at object edges (the pose output does not change). With ARKit poses, the metric scale from the 392 files differs
  from the 504 sliced file by a median of 2.6% (3 views) and 4.5% (2 views).

## Speed on iPhone

Median time of one inference (all views at once). Measured on two devices only. The conditions differ between rows,
so compare the numbers within a row.

| Device | Comparison | Condition | Result |
|---|---|---|---|
| iPhone 17 Pro (A19 Pro) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 224 → **91 ms** |
| iPhone 12 Pro (A14) | `3views-504-sliced` → `3views-392-fast` | Fixed images, Debug build | 579.5 → **183.4 ms** |
| iPhone 17 Pro (A19 Pro) | `2views-392-fast` | Fixed images, Debug build | **53.6–54.6 ms** |
| iPhone 17 Pro (A19 Pro) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 120 s each from the nominal thermal state | 464.3 → **310.5 ms**, app memory 1,121 → 762 MB |
| iPhone 12 Pro (A14) | `3views-504` → `3views-504-sliced` | Release, ARKit, handheld, 1 min each | 774 → **566 ms**, Neural Engine memory 760 → 367 MB |
| iPhone 12 Pro (A14) | `3views-392-fast` | Inside a scanning app that also builds a TSDF mesh on the GPU, Debug build, handheld 1–2 min, already in the serious thermal state | 270–293 ms per inference, app memory 330–356 MB |

Things to know on these devices:

- The first load compiles the model for the Neural Engine and can take one to several minutes on an iPhone 12 Pro.
  Later loads are quick.
- A 2-view 504 file sliced in 16 does not return on the iPhone 17 Pro Neural Engine (the first inference stalls and is
  cancelled), and neither does a 3-view 392 file sliced in 16. That is why the files here are 2 views at 392, and 3
  views at 392 sliced in 8.

## Use it

Download one file (a Core ML package is a folder):

```bash
hf download shu223/Depth-Anything-3-Small-MultiView-CoreML \
  --include "models/DA3-Small-3views-392-fast.mlpackage/*" --local-dir .
```

Then add the `.mlpackage` to your Xcode project, or download it at run time and compile it on the device with
`MLModel.compileModel(at:)`. A minimal prediction in Swift:

```swift
import CoreML
let compiledURL = try await MLModel.compileModel(at: packageURL)   // once; keep the .mlmodelc
let model = try MLModel(contentsOf: compiledURL)
// 3 views, row-major: element (0, view, row, column)
let extrinsics = try MLMultiArray(shape: [1, 3, 4, 4], dataType: .float32)  // normalised world-to-camera
let intrinsics = try MLMultiArray(shape: [1, 3, 3, 3], dataType: .float32)  // of the 392 x 392 images
// ... fill both (see Inputs below)
let input = try MLDictionaryFeatureProvider(dictionary: [
    "image_0": MLFeatureValue(pixelBuffer: image0),   // 392 x 392 RGB image input: a 32BGRA or 32ARGB CVPixelBuffer
    "image_1": MLFeatureValue(pixelBuffer: image1),
    "image_2": MLFeatureValue(pixelBuffer: image2),
    "extrinsics": MLFeatureValue(multiArray: extrinsics),
    "intrinsics": MLFeatureValue(multiArray: intrinsics),
])
let output = try model.prediction(from: input)
let depth = output.featureValue(for: "depth")!.multiArrayValue!                 // [1, 3, 392, 392]
let poseEncoding = output.featureValue(for: "pose_encoding")!.multiArrayValue!  // [1, 3, 9]
```

## Inputs

| Name | Type | Shape | Content |
|---|---|---|---|
| `image_0` … `image_{S-1}` | Image (RGB, 8-bit) | 392 × 392 or 504 × 504 | The views. View 0 is the reference view. ImageNet normalisation is inside the model. |
| `extrinsics` | Float32 | (1, S, 4, 4) | World-to-camera matrices in the OpenCV camera axes (x right, y down, z forward), **already normalised** as below. |
| `intrinsics` | Float32 | (1, S, 3, 3) | Pinhole intrinsics of each input image in pixels, after your crop and resize. Only `fx` and `fy` reach the network. |

Normalise the extrinsics the way the official `DepthAnything3._normalize_extrinsics` (`api.py`) does before the
forward pass. It is left out of the model so that it runs in Float32 / Float64 instead of the model's FP16:

```python
import numpy as np
def normalize_extrinsics(w2c):            # w2c: (S, 4, 4) world-to-camera, OpenCV axes, metres
    rel = w2c @ np.linalg.inv(w2c[0])     # view 0 becomes the identity
    centres = np.linalg.inv(rel)[:, :3, 3]
    dist = np.sort(np.linalg.norm(centres, axis=1))
    median = max(dist[(len(dist) - 1) // 2], 0.1)   # torch.median: the lower middle value
    rel[:, :3, 3] /= median
    return rel.astype(np.float32), median
```

With 2 views the median is the distance of view 0 to itself, so the divisor is always 0.1.

From ARKit: `ARCamera.transform` is camera-to-world with x right, y up and z toward the viewer in the sensor's
landscape orientation. For an image rotated 90 degrees clockwise to portrait, the model's x is ARKit's +y, the model's
y is ARKit's +x and the model's z is ARKit's −z: `c2w_model = c2w_arkit @ R`, where the columns of `R` are
(0, 1, 0), (1, 0, 0) and (0, 0, −1). Invert it to get world-to-camera. The intrinsics follow the same rotation, crop
and resize.

## Outputs

| Name | Type | Shape | Content |
|---|---|---|---|
| `depth` | Float16 | (1, S, H, W) | Depth of every view, in the normalised frame of the input poses. |
| `confidence` | Float16 | (1, S, H, W) | Confidence of every depth value. |
| `pose_encoding` | Float16 | (1, S, 9) | The camera pose the network estimates for every view: translation (3), quaternion xyzw (4), horizontal and vertical field of view. The translation is the camera position (camera-to-world) in the normalised frame. |

To get metres, do what the official `api.py` does: align the camera positions in `pose_encoding` to the camera
positions you passed with Umeyama's method (a similarity transform), and divide the depth by its scale.

## Downloading from an app

`models.json` at the top of this repository lists every file with the size and SHA-256 of each file inside its
package, the input size, the number of views and the minimum iOS version. An app can read it, download the files of
one package from `https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/resolve/main/<path>`, check
them and compile the package on the device. Compiling the `.mlpackage` on the device works on iOS 27 (checked on an
iPhone 12 Pro and the simulator).

## How it is built

Converted with coremltools from the original PyTorch implementation
([ByteDance-Seed/Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)), `DA3-SMALL` revision
`e08cab65ca0ec38e7826075418411ab90cab4da3`:

- The pose-conditioned path is kept: the camera encoder turns the extrinsics and intrinsics into the camera tokens,
  and the camera decoder returns the estimated poses. Ray decoding, sky handling and the Gaussian branch are not
  included.
- The number of views and the input size are fixed per file (Core ML needs fixed shapes).
- **Sliced attention**: the global attention across all views is computed in slices of the query, so the attention
  matrices stay small on the Neural Engine.
- **Rewritten depth head** (`fast`): the DPT head upsamples with transposed convolutions, and its last convolution
  runs before the final upsample, so only the two output channels (depth and confidence) are upsampled to full size.

## License

Apache License 2.0, the same as the original `DA3-SMALL` weights. See `LICENSE` and `NOTICE`.

## Citation

```bibtex
@article{depthanything3,
  title   = {Depth Anything 3: Recovering the Visual Space from Any Views},
  author  = {Lin, Haotong and Chen, Sili and Liew, Jun Hao and Chen, Donny Y. and Li, Zhenyu and Shi, Guang and Feng, Jiashi and Kang, Bingyi},
  journal = {arXiv preprint arXiv:2511.10647},
  year    = {2025}
}
```

	---
	license: apache-2.0
	library_name: coreml
	pipeline_tag: depth-estimation
	base_model: depth-anything/DA3-SMALL
	base_model_relation: quantized
	tags:
	- coreml
	- core-ml
	- depth-anything
	- depth-anything-3
	- depth-anything-3-small
	- da3
	- da3-small
	- depth-estimation
	- multi-view
	- multi-view-depth
	- multi-view-geometry
	- pose-conditioned
	- camera-pose
	- pose-estimation
	- 3d-reconstruction
	- 3d-scanning
	- arkit
	- ios
	- iphone
	- iphone-12-pro
	- iphone-17-pro
	- apple-silicon
	- apple-neural-engine
	- ane
	- on-device
	- swift
	---

	# Depth Anything 3 Small — Multi-View Core ML for iPhone

	**Multi-view depth with camera poses, on iPhone. `DA3-Small-3views-392-fast` is
	recommended: 2.5x (iPhone 17 Pro) to 3.2x (iPhone 12 Pro) faster than the same model at 504 × 504 with sliced attention (`DA3-Small-3views-504-sliced`).**

	[Depth Anything 3](https://huggingface.co/depth-anything/DA3-SMALL) (DA3) Small, converted to Core ML with the
	multi-view, pose-conditioned path of the network kept. Give it 2 or 3 photos and their camera poses (for example from
	ARKit), and it returns a depth map and a confidence map for every photo, plus the camera pose it estimates. The depth
	of all views comes out in one consistent frame, so you can fuse it into a mesh or a point cloud and bring it to metres
	with the poses you passed. The other Core ML conversion of DA3 Small on the Hub
	([mlboydaisuke/Depth-Anything-3-Small-CoreML](https://huggingface.co/mlboydaisuke/Depth-Anything-3-Small-CoreML), as of
	November 2026) takes one image at a time.

	The graph is rewritten for the iPhone: the attention across views is computed in slices, and the depth head and the
	input size are changed for speed. The weights are the original ones in FP16.

	## Which file should I choose?

	\| File \| Views \| Input \| What is changed \| Min iOS \| Size \|
	\|---\|---\|---\|---\|---\|---\|
	\| [`DA3-Small-3views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-392-fast.mlpackage) \| 3 \| 392 × 392 \| 392 input, attention in 8 slices, depth head rewritten. Recommended. \| 18 \| 72 MB \|
	\| [`DA3-Small-2views-392-fast`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-2views-392-fast.mlpackage) \| 2 \| 392 × 392 \| The same speed-ups with 2 views. Fastest. \| 18 \| 72 MB \|
	\| [`DA3-Small-3views-504-sliced`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504-sliced.mlpackage) \| 3 \| 504 × 504 \| Attention in 16 slices only. Same output as the original model. \| 18 \| 87 MB \|
	\| [`DA3-Small-3views-504`](https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/tree/main/models/DA3-Small-3views-504.mlpackage) \| 3 \| 504 × 504 \| Nothing. The plain conversion, for comparison and for iOS 17. \| 17 \| 87 MB \|

	- Start with `DA3-Small-3views-392-fast`. It is the one to use for scanning with ARKit.
	- Use `DA3-Small-2views-392-fast` if you only have two photos or need the shortest time per inference.
	- Use `DA3-Small-3views-504-sliced` if you want the output of the original model at its native 504 input: the
	slicing only changes the order of the operations (it has the same `weight.bin` as `DA3-Small-3views-504`), and the
	mean depth difference from the original PyTorch model is 0.03–0.05% on a Mac.
	- The `fast` files trade a little accuracy for speed. The rewritten head changes depth and confidence slightly, mostly
	at object edges (the pose output does not change). With ARKit poses, the metric scale from the 392 files differs
	from the 504 sliced file by a median of 2.6% (3 views) and 4.5% (2 views).

	## Speed on iPhone

	Median time of one inference (all views at once). Measured on two devices only. The conditions differ between rows,
	so compare the numbers within a row.

	\| Device \| Comparison \| Condition \| Result \|
	\|---\|---\|---\|---\|
	\| iPhone 17 Pro (A19 Pro) \| `3views-504-sliced` → `3views-392-fast` \| Fixed images, Debug build \| 224 → 91 ms \|
	\| iPhone 12 Pro (A14) \| `3views-504-sliced` → `3views-392-fast` \| Fixed images, Debug build \| 579.5 → 183.4 ms \|
	\| iPhone 17 Pro (A19 Pro) \| `2views-392-fast` \| Fixed images, Debug build \| 53.6–54.6 ms \|
	\| iPhone 17 Pro (A19 Pro) \| `3views-504` → `3views-504-sliced` \| Release, ARKit, handheld, 120 s each from the nominal thermal state \| 464.3 → 310.5 ms, app memory 1,121 → 762 MB \|
	\| iPhone 12 Pro (A14) \| `3views-504` → `3views-504-sliced` \| Release, ARKit, handheld, 1 min each \| 774 → 566 ms, Neural Engine memory 760 → 367 MB \|
	\| iPhone 12 Pro (A14) \| `3views-392-fast` \| Inside a scanning app that also builds a TSDF mesh on the GPU, Debug build, handheld 1–2 min, already in the serious thermal state \| 270–293 ms per inference, app memory 330–356 MB \|

	Things to know on these devices:

	- The first load compiles the model for the Neural Engine and can take one to several minutes on an iPhone 12 Pro.
	Later loads are quick.
	- A 2-view 504 file sliced in 16 does not return on the iPhone 17 Pro Neural Engine (the first inference stalls and is
	cancelled), and neither does a 3-view 392 file sliced in 16. That is why the files here are 2 views at 392, and 3
	views at 392 sliced in 8.

	## Use it

	Download one file (a Core ML package is a folder):

	```bash
	hf download shu223/Depth-Anything-3-Small-MultiView-CoreML \
	--include "models/DA3-Small-3views-392-fast.mlpackage/*" --local-dir .
	```

	Then add the `.mlpackage` to your Xcode project, or download it at run time and compile it on the device with
	`MLModel.compileModel(at:)`. A minimal prediction in Swift:

	```swift
	import CoreML

	let compiledURL = try await MLModel.compileModel(at: packageURL) // once; keep the .mlmodelc
	let model = try MLModel(contentsOf: compiledURL)

	// 3 views, row-major: element (0, view, row, column)
	let extrinsics = try MLMultiArray(shape: [1, 3, 4, 4], dataType: .float32) // normalised world-to-camera
	let intrinsics = try MLMultiArray(shape: [1, 3, 3, 3], dataType: .float32) // of the 392 x 392 images
	// ... fill both (see Inputs below)

	let input = try MLDictionaryFeatureProvider(dictionary: [
	"image_0": MLFeatureValue(pixelBuffer: image0), // 392 x 392 RGB image input: a 32BGRA or 32ARGB CVPixelBuffer
	"image_1": MLFeatureValue(pixelBuffer: image1),
	"image_2": MLFeatureValue(pixelBuffer: image2),
	"extrinsics": MLFeatureValue(multiArray: extrinsics),
	"intrinsics": MLFeatureValue(multiArray: intrinsics),
	])
	let output = try model.prediction(from: input)
	let depth = output.featureValue(for: "depth")!.multiArrayValue! // [1, 3, 392, 392]
	let poseEncoding = output.featureValue(for: "pose_encoding")!.multiArrayValue! // [1, 3, 9]
	```

	## Inputs

	\| Name \| Type \| Shape \| Content \|
	\|---\|---\|---\|---\|
	\| `image_0` … `image_{S-1}` \| Image (RGB, 8-bit) \| 392 × 392 or 504 × 504 \| The views. View 0 is the reference view. ImageNet normalisation is inside the model. \|
	\| `extrinsics` \| Float32 \| (1, S, 4, 4) \| World-to-camera matrices in the OpenCV camera axes (x right, y down, z forward), already normalised as below. \|
	\| `intrinsics` \| Float32 \| (1, S, 3, 3) \| Pinhole intrinsics of each input image in pixels, after your crop and resize. Only `fx` and `fy` reach the network. \|

	Normalise the extrinsics the way the official `DepthAnything3._normalize_extrinsics` (`api.py`) does before the
	forward pass. It is left out of the model so that it runs in Float32 / Float64 instead of the model's FP16:

	```python
	import numpy as np

	def normalize_extrinsics(w2c): # w2c: (S, 4, 4) world-to-camera, OpenCV axes, metres
	rel = w2c @ np.linalg.inv(w2c[0]) # view 0 becomes the identity
	centres = np.linalg.inv(rel)[:, :3, 3]
	dist = np.sort(np.linalg.norm(centres, axis=1))
	median = max(dist[(len(dist) - 1) // 2], 0.1) # torch.median: the lower middle value
	rel[:, :3, 3] /= median
	return rel.astype(np.float32), median
	```

	With 2 views the median is the distance of view 0 to itself, so the divisor is always 0.1.

	From ARKit: `ARCamera.transform` is camera-to-world with x right, y up and z toward the viewer in the sensor's
	landscape orientation. For an image rotated 90 degrees clockwise to portrait, the model's x is ARKit's +y, the model's
	y is ARKit's +x and the model's z is ARKit's −z: `c2w_model = c2w_arkit @ R`, where the columns of `R` are
	(0, 1, 0), (1, 0, 0) and (0, 0, −1). Invert it to get world-to-camera. The intrinsics follow the same rotation, crop
	and resize.

	## Outputs

	\| Name \| Type \| Shape \| Content \|
	\|---\|---\|---\|---\|
	\| `depth` \| Float16 \| (1, S, H, W) \| Depth of every view, in the normalised frame of the input poses. \|
	\| `confidence` \| Float16 \| (1, S, H, W) \| Confidence of every depth value. \|
	\| `pose_encoding` \| Float16 \| (1, S, 9) \| The camera pose the network estimates for every view: translation (3), quaternion xyzw (4), horizontal and vertical field of view. The translation is the camera position (camera-to-world) in the normalised frame. \|

	To get metres, do what the official `api.py` does: align the camera positions in `pose_encoding` to the camera
	positions you passed with Umeyama's method (a similarity transform), and divide the depth by its scale.

	## Downloading from an app

	`models.json` at the top of this repository lists every file with the size and SHA-256 of each file inside its
	package, the input size, the number of views and the minimum iOS version. An app can read it, download the files of
	one package from `https://huggingface.co/shu223/Depth-Anything-3-Small-MultiView-CoreML/resolve/main/<path>`, check
	them and compile the package on the device. Compiling the `.mlpackage` on the device works on iOS 27 (checked on an
	iPhone 12 Pro and the simulator).

	## How it is built

	Converted with coremltools from the original PyTorch implementation
	([ByteDance-Seed/Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)), `DA3-SMALL` revision
	`e08cab65ca0ec38e7826075418411ab90cab4da3`:

	- The pose-conditioned path is kept: the camera encoder turns the extrinsics and intrinsics into the camera tokens,
	and the camera decoder returns the estimated poses. Ray decoding, sky handling and the Gaussian branch are not
	included.
	- The number of views and the input size are fixed per file (Core ML needs fixed shapes).
	- Sliced attention: the global attention across all views is computed in slices of the query, so the attention
	matrices stay small on the Neural Engine.
	- Rewritten depth head (`fast`): the DPT head upsamples with transposed convolutions, and its last convolution
	runs before the final upsample, so only the two output channels (depth and confidence) are upsampled to full size.

	## License

	Apache License 2.0, the same as the original `DA3-SMALL` weights. See `LICENSE` and `NOTICE`.

	## Citation

	```bibtex
	@article{depthanything3,
	title = {Depth Anything 3: Recovering the Visual Space from Any Views},
	author = {Lin, Haotong and Chen, Sili and Liew, Jun Hao and Chen, Donny Y. and Li, Zhenyu and Shi, Guang and Feng, Jiashi and Kang, Bingyi},
	journal = {arXiv preprint arXiv:2511.10647},
	year = {2025}
	}
	```