smdesai/granite-docling-258M-coreml

granite-docling-258M — Core ML

IBM granite-docling-258M converted to Core ML for iPhone, iPad and Mac. It turns an image of a document page into DocTags (text, layout boxes, tables, formulas, code, lists), entirely on device.

• FP16 weights, no quantization. Greedy output on the GPU is token-identical to the Hugging Face model, including all 1,759 tokens of a full arXiv sample page.
• Stateful decoder split into three 10-layer chunks so each KV cache stays small enough to load on iPhone.
• Requires iOS 18 / iPadOS 18 / macOS 15 or later (MLState, multifunction models).

Device	Compute units	Decode	Memory
iPhone 17 Pro	CPU + GPU	42–49 tok/s	~280 MB
M-series Mac	CPU + GPU	48–53 tok/s	—

Use CPU + GPU (.cpuAndGPU). The Neural Engine is slower for this decoder because the KV-state updates stay on the CPU, and the vision encoder is less accurate there (min cosine 0.983 vs 0.9987 on the GPU), which can flip near-tie tokens.

Files

File	Size	Role
GraniteDoclingVisionEncoder.mlpackage	179 MB	SigLIP2 vision tower + pixel-shuffle connector
GraniteDoclingDecoder_chunk0.mlpackage	68 MB	decoder layers 0–9
GraniteDoclingDecoder_chunk1.mlpackage	68 MB	decoder layers 10–19
GraniteDoclingDecoder_chunk2.mlpackage	68 MB	decoder layers 20–29
GraniteDoclingDecoder_head.mlpackage	110 MB	final norm + vocabulary projection
embed_tokens_fp16.bin	110 MB	token embedding table, [100352, 576] float16, row-major
tokenizer.json	6.8 MB	tokenizer (identical to the base model's)
granite_docling_coreml.json	<1 KB	export configuration
*.mlmodelc (five folders)	493 MB	the same five models, already compiled

Either use the precompiled .mlmodelc folders directly (MLModel(contentsOf:); compiled with Xcode 27's Core ML compiler for iOS 18+, and identical for macOS 15+), or add the .mlpackage files to an Xcode target (Xcode compiles them to .mlmodelc) or compile them at runtime with MLModel.compileModel(at:).

Model interfaces

Model	Inputs	Outputs
Vision encoder	image: 512×512 RGB image (normalization is inside the model)	image_embeds [1, 64, 576] fp16
Each decoder chunk, function prefill	hidden_in [1, 256, 576], causal_mask [1, 1, 256, 4096], current_pos [1] int32, rope_cos / rope_sin [1, 1, 256, 64]	hidden_out [1, 256, 576]
Each decoder chunk, function decode	same with length 1	hidden_out [1, 1, 576]
Head	hidden [1, 1, 576]	logits [1, 1, 100352] fp16

All decoder tensors are fp16. Each chunk has one MLState, kv_cache [10, 3, 4096, 128] (K and V packed on the last axis), shared by its prefill and decode functions: load both functions of a chunk (MLModelConfiguration.functionName) and pass the same state object to every call. Maximum context is 4096.

Running it

1. Tile the page with Lanczos resampling: resize so the longest edge is 2048 (also when upscaling), keeping aspect ratio and rounding the short edge up to even; resize again so both edges are multiples of 512; cut 512×512 tiles row-major, then append the whole page resized to 512×512 as the global view. A 1540×1190 page gives 4×4 tiles + 1 global = 17 tiles.
2. Encode each tile with the vision encoder: 64 embeddings per tile.
3. Build the prompt (no BOS) and tokenize it with tokenizer.json:

   <|start_of_role|>user<|end_of_role|>
   for each tile row r, column c:  <fake_token_around_image><row_{r}_col_{c}> + <image>×64, then "\n" after each row
   \n<fake_token_around_image><global-img> + <image>×64 + <fake_token_around_image>
   Convert this page to docling.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>
   

   <image> is token 100270. 1-based row/column indices, as in the base model's processor.
4. Embed: look up each token's row in embed_tokens_fp16.bin; for every <image> token use the next vision embedding instead (tile order, 64 rows per tile).
5. Prefill in calls of 256 positions through chunk 0 → 1 → 2 (hidden_out of one chunk is hidden_in of the next), zero-padding the last call. current_pos is the absolute position of the call's first row.
   • causal_mask[0, 0, r, c] = 0 if c <= current_pos + r, else −10000.
   • rope_cos / rope_sin row r = cos / sin of (current_pos + r) · θ_i with θ_i = 100000^(−2i/64), i = 0…31, and the 32 values repeated twice (rotate-half layout).
6. Head: feed the hidden row of the last real prompt token to the head; argmax of logits is the next token.
7. Decode one token at a time with the decode functions, same mask and RoPE rules. Stop at <|end_of_text|> (100257), <end_of_utterance> or </doctag>.

Greedy decoding can loop on some pages; stopping when a short token cycle repeats several times is a cheap safeguard.

Conversion notes

The decoder's first prompt token develops residual activations of ~1.2×10⁵ (an attention-sink "massive activation"), which overflows FP16. The conversion re-expresses RMSNorm so it stays finite in FP16 and folds the norm and scale weights into the adjacent projections; grouped-query attention is computed without repeating K/V. Per-tensor INT8 quantization breaks both models, so these weights are FP16.

License and attribution

Apache 2.0, the same as the base model. All model weights are IBM's granite-docling-258M; this repository only changes their format. Please cite and credit IBM's model when you use it.