1of2/omniparser-v2-coreml-w8

OmniParser v2 — Apple Silicon Core ML (W8A16, ANE + Metal GPU)

Production Core ML (macOS 15+) release of Microsoft's OmniParser v2, combining the MIT-licensed icon_detect_v3 (YOLOv9-E) interactive-element detector with the Florence-2 icon captioner, quantized to W8A16 (Int8 symmetric per-channel weights + Float16 activations) with multifunction static batching (B = 1, 4, 8).

• Minimum OS: macOS 15.0 (Apple Silicon)
• Hardware Residency:
  • OmniParserDetector: 795 / 795 ops (100.0%) on MLNeuralEngine (0 CPU fallback ops)
  • OmniParserCaptioner (ANE profile): 1,056 / 1,056 encode_image ops (100.0%) on MLNeuralEngine across B = 1, 4, 8 (0 CPU fallback ops)
  • OmniParserCaptionerGPU (Metal GPU profile): 1,532 / 1,532 encode_image ops (100.0%) on MLGPU (0 CPU fallback ops)

Bundle Contents

Both source .mlpackage bundles (for Xcode inspection / custom compilation) and pre-compiled .mlmodelc bundles (for instant zero-compile startup) are included:

Path	Description	Entrypoints & Shapes	Size
runtime_config.json	Schema v4 runtime contract (batch_sizes: [1, 4, 8], detect_640, detect_1280)	schema_version: 4	4 KB
detector/OmniParserDetector.mlpackage & .mlmodelc	Multi-function YOLOv9-E (icon_detect_v3, MIT) W8A16 detector (100% deduplicated weights across resolutions)	detect_640: [1, 3, 640, 640] fp16 -> [1, 5, 420, 20] fp16 detect_1280: [1, 3, 1280, 1280] fp16 -> [1, 5, 840, 40] fp16	56.7 MB / 56.8 MB
detector/LICENSE & detector/provenance.json	MIT license & checkpoint provenance (sha256: 11c6cbb7...6e3fb00c)	Verified at load time	2 KB
caption/OmniParserCaptioner.mlpackage & .mlmodelc	ANE-optimized W8A16 Florence-2 captioner ([B, C, 1, S] 4D layout, native 4D ANE einsum SDPA, 9 entrypoints, 100% weight-deduplicated)	encode_image, project_cross_kv, decode_step for B = 1, 4, 8	224.3 MB / 224.4 MB
caption/OmniParserCaptionerGPU.mlpackage & .mlmodelc	GPU-optimized W8A16 Florence-2 captioner ([B, S, C] 3D layout, SDPA across 9 entrypoints, 100% weight-deduplicated)	encode_image, project_cross_kv, decode_step for B = 1, 4, 8	223.3 MB / 223.5 MB
caption/shared_embedding.bin	Raw Float16 token embedding table ([51289, 768]), memory-mapped in Swift	78,779,904 bytes	75.1 MB
caption/decoder_embed_positions.bin	Raw Float16 learned positional embeddings ([20, 768], offset 2 baked in)	30,720 bytes	30 KB
caption/tokenizer.json	GPT-2 / BART byte-level BPE tokenizer	51,289 vocab	1.4 MB

Performance on Apple M4 Max

Workload (640 Detector)	Official PyTorch (mps FP16)	Core ML .gpu (W8A16, p50)	Core ML .neuralEngine (W8A16, p50)	Speedup (Core ML .gpu vs. PyTorch mps)
Detector (detect_640)	309.5 ms (CPU fallback)	31.3 ms	24.1 ms	12.8× faster (on ANE)
Caption B = 1 (1 icon)	279.4 ms (279.4 ms/icon)	129.7 ms (129.7 ms/icon)	149.4 ms (149.4 ms/icon)	2.15× faster
Caption B = 4 (4 icons)	561.0 ms (140.3 ms/icon)	293.0 ms (73.3 ms/icon)	468.2 ms (117.1 ms/icon)	1.91× faster
Caption B = 8 (8 icons)	1,166.3 ms (145.8 ms/icon)	567.6 ms (71.0 ms/icon)	922.4 ms (115.3 ms/icon)	2.05× faster
Parse B = 1 (detect + 1 icon)	451.8 ms	163.8 ms	172.3 ms	2.76× faster
Parse B = 4 (detect + 4 icons)	870.5 ms	324.3 ms (81.1 ms/icon)	491.3 ms (122.8 ms/icon)	2.68× faster
Parse B = 8 (detect + 8 icons)	1,475.8 ms	598.4 ms (74.8 ms/icon)	946.4 ms (118.3 ms/icon)	2.47× faster
Parse N = 20 (detect + 20 icons)	3,210.8 ms	1,448.4 ms (72.4 ms/icon)	2,340.5 ms (117.0 ms/icon)	2.22× faster
Peak Process RSS	2,244 MB	720 – 1,401 MB	339 – 894 MB	up to 6.6× lower RAM on ANE

---

Minimal Swift Usage (CoreML)

The pre-compiled .mlmodelc (or .mlpackage) bundles can be driven directly with Apple's built-in CoreML framework (macOS 15+, zero third-party dependencies).

1. Multifunction Icon Detector (detect_640 / detect_1280)

import CoreGraphics
import CoreML
import Foundation

let detectorURL = URL(fileURLWithPath: "omniparser-v2-coreml/detector/OmniParserDetector.mlmodelc")
let imageSize = 640 // or 1280 for "detect_1280"

let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine
config.functionName = "detect_\(imageSize)"

let detector = try MLModel(contentsOf: detectorURL, configuration: config)

// 1. Input: NCHW Float16 tensor in [0, 1], square-letterboxed to 640x640 with gray (114/255) padding
let imageTensor = try MLMultiArray(shape: [1, 3, NSNumber(value: imageSize), NSNumber(value: imageSize)], dataType: .float16)
let provider = try MLDictionaryFeatureProvider(dictionary: [
    "image": MLFeatureValue(multiArray: imageTensor)
])

// 2. Run inference -> "predictions" shaped [1, 5, 420, 20] (contiguous [1, 5, 8400] in memory)
//    Channel 0 = raw confidence logit; Channels 1..4 = (left, top, right, bottom) in stride units
let out = try detector.prediction(from: provider)
let predictions = out.featureValue(for: "predictions")!.multiArrayValue!
let rawPtr = predictions.dataPointer.assumingMemoryBound(to: Float16.self)

let strides = [8, 16, 32]
let anchors = strides.reduce(0) { $0 + (imageSize / $1) * (imageSize / $1) } // 8,400 at 640; 33,600 at 1280
let confidenceThreshold: Float = 0.05

var boxes: [(rect: CGRect, score: Float)] = []
var anchorIdx = 0
for stride in strides {
    let grid = imageSize / stride
    let s = Float(stride)
    for gy in 0..<grid {
        for gx in 0..<grid {
            defer { anchorIdx += 1 }
            let logit = Float(rawPtr[anchorIdx])
            let score = 1.0 / (1.0 + exp(-logit))
            guard score > confidenceThreshold else { continue }

            let l = Float(rawPtr[1 * anchors + anchorIdx]) * s
            let t = Float(rawPtr[2 * anchors + anchorIdx]) * s
            let r = Float(rawPtr[3 * anchors + anchorIdx]) * s
            let b = Float(rawPtr[4 * anchors + anchorIdx]) * s
            let cx = (Float(gx) + 0.5) * s
            let cy = (Float(gy) + 0.5) * s

            // Unletterbox from 640x640 detector space back to original image coordinates:
            // x = (detectorX - padLeft) / scale, y = (detectorY - padTop) / scale
            let rect = CGRect(x: CGFloat(cx - l), y: CGFloat(cy - t), width: CGFloat(l + r), height: CGFloat(t + b))
            boxes.append((rect, score))
        }
    }
}

2. 3-Stage Florence-2 Captioner (encode_image -> project_cross_kv -> decode_step)

Both OmniParserCaptioner.mlmodelc (4D ANE profile, .cpuAndNeuralEngine) and OmniParserCaptionerGPU.mlmodelc (3D Metal GPU profile, .cpuAndGPU) expose 9 functions across static batch sizes B = 1, 4, 8:

• encode_image, encode_image_b4, encode_image_b8 (the prompt "What does the image describe?" is baked into the graph)
• project_cross_kv, project_cross_kv_b4, project_cross_kv_b8
• decode_step, decode_step_b4, decode_step_b8

Passing MLFeatureValue outputs directly between stages keeps KV-cache tensors on device without intermediate copies:

import CoreML
import Foundation

let bundleDir = URL(fileURLWithPath: "omniparser-v2-coreml/caption")
let captionerURL = bundleDir.appendingPathComponent("OmniParserCaptioner.mlmodelc")

func loadFunction(_ name: String) throws -> MLModel {
    let cfg = MLModelConfiguration()
    cfg.computeUnits = .cpuAndNeuralEngine // use .cpuAndGPU with OmniParserCaptionerGPU.mlmodelc
    cfg.functionName = name
    return try MLModel(contentsOf: captionerURL, configuration: cfg)
}

let encodeImage = try loadFunction("encode_image")         // or "encode_image_b4", "encode_image_b8"
let projectCrossKV = try loadFunction("project_cross_kv")  // or "project_cross_kv_b4", "project_cross_kv_b8"
let decodeStep = try loadFunction("decode_step")           // or "decode_step_b4", "decode_step_b8"

// Memory-map the raw Float16 embedding sidecars: [51289, 768] token table & [20, 768] position table
let tokenTable = try Data(contentsOf: bundleDir.appendingPathComponent("shared_embedding.bin"), options: .mappedIfSafe)
let posTable = try Data(contentsOf: bundleDir.appendingPathComponent("decoder_embed_positions.bin"), options: .mappedIfSafe)

func embeddingSlice(from table: Data, row: Int, shape: [NSNumber]) throws -> MLMultiArray {
    let arr = try MLMultiArray(shape: shape, dataType: .float16)
    table.withUnsafeBytes { raw in
        let src = raw.bindMemory(to: UInt16.self).baseAddress! + row * 768
        arr.dataPointer.assumingMemoryBound(to: UInt16.self).update(from: src, count: 768)
    }
    return arr
}

// 1. Encode normalized [1, 3, 768, 768] Float16 icon crop (ImageNet mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
let pixelValues = try MLMultiArray(shape: [1, 3, 768, 768], dataType: .float16)
let encOut = try encodeImage.prediction(from: MLDictionaryFeatureProvider(dictionary: [
    "pixel_values": MLFeatureValue(multiArray: pixelValues)
]))

// 2. Project encoder hidden states once into 6-layer cross-attention K/V tensors
let crossKVOut = try projectCrossKV.prediction(from: MLDictionaryFeatureProvider(dictionary: [
    "encoder_hidden_states": encOut.featureValue(for: "encoder_hidden_states")!
]))

// 3. Seed decoder self-attention KV cache ([1, 12, 64, 20] on ANE; [1, 12, 20, 64] on GPU)
var stepInputs: [String: Any] = [:]
for layer in 0..<6 {
    stepInputs["cross_k_\(layer)"] = crossKVOut.featureValue(for: "cross_k_\(layer)")!
    stepInputs["cross_v_\(layer)"] = crossKVOut.featureValue(for: "cross_v_\(layer)")!
    stepInputs["self_k_\(layer)"] = MLFeatureValue(multiArray: try MLMultiArray(shape: [1, 12, 64, 20], dataType: .float16))
    stepInputs["self_v_\(layer)"] = MLFeatureValue(multiArray: try MLMultiArray(shape: [1, 12, 64, 20], dataType: .float16))
}

// 4. Autoregressive greedy decode loop (max 20 steps; start token = 2, forced BOS at step 0 = 0, EOS = 2)
var generatedTokens: [Int] = [2]
for step in 0..<20 {
    let writeMask = try MLMultiArray(shape: [1, 1, 1, 20], dataType: .float16)
    let attnMask = try MLMultiArray(shape: [1, 1, 1, 20], dataType: .float16)
    for i in 0..<20 {
        writeMask[i] = NSNumber(value: i == step ? 1.0 : 0.0)
        attnMask[i] = NSNumber(value: i <= step ? 0.0 : -30000.0)
    }

    // ANE token/pos shape is [B, 768, 1, 1]; GPU shape is [B, 1, 768]
    stepInputs["token_embed"] = MLFeatureValue(multiArray: try embeddingSlice(from: tokenTable, row: generatedTokens.last!, shape: [1, 768, 1, 1]))
    stepInputs["pos_embed"] = MLFeatureValue(multiArray: try embeddingSlice(from: posTable, row: step, shape: [1, 768, 1, 1]))
    stepInputs["write_mask"] = MLFeatureValue(multiArray: writeMask)
    stepInputs["attn_mask"] = MLFeatureValue(multiArray: attnMask)

    let stepOut = try decodeStep.prediction(from: MLDictionaryFeatureProvider(dictionary: stepInputs))

    // Feed updated self-attention KV cache directly into next step
    for layer in 0..<6 {
        stepInputs["self_k_\(layer)"] = stepOut.featureValue(for: "new_self_k_\(layer)")!
        stepInputs["self_v_\(layer)"] = stepOut.featureValue(for: "new_self_v_\(layer)")!
    }

    if step == 0 {
        generatedTokens.append(0) // forced_bos_token_id = 0
        continue
    }

    // Argmax across logits_0 (25,644) + logits_1 (25,645) = 51,289 vocab
    let logits0 = stepOut.featureValue(for: "logits_0")!.multiArrayValue!
    let logits1 = stepOut.featureValue(for: "logits_1")!.multiArrayValue!
    let p0 = logits0.dataPointer.assumingMemoryBound(to: Float16.self)
    let p1 = logits1.dataPointer.assumingMemoryBound(to: Float16.self)

    var bestToken = 0
    var bestScore = -Float.infinity
    for i in 0..<25_644 {
        let v = Float(p0[i])
        if v > bestScore { bestScore = v; bestToken = i }
    }
    for i in 0..<25_645 {
        let v = Float(p1[i])
        if v > bestScore { bestScore = v; bestToken = 25_644 + i }
    }

    generatedTokens.append(bestToken)
    if bestToken == 2 { break } // eos_token_id = 2
}

License

MIT License (both icon_detect_v3 and icon_caption weights are MIT-licensed by Microsoft Corporation).