Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)

de90e9c verified 2 months ago

8.28 kB

---
license: mit
tags:
- executorch
- xnnpack
- pte
- on-device
- image-text-to-text
- image-to-text
- object-detection
base_model:
- microsoft/Florence-2-large
base_model_relation: quantized
---
# Florence-2-large — ExecuTorch (vision + text encoder + text decoder)

One set of weights that captions, detects, reads text and grounds phrases, with the
task chosen by the prompt you send. 0.77 B parameters — the same interface as
[Florence-2-base](https://huggingface.co/mlboydaisuke/Florence-2-base-ExecuTorch)
at roughly three times the size, and it notices more: on the same photograph base
says *"A black and white photo of a person playing a piano"* where large reads the
brand off the instrument.

Three `.pte` files, split the way Whisper is on this shelf and for the same reason:
the vision tower and the text encoder run once per image, the decoder runs once per
generated token.

```
vision : pixel_values (1,3,768,768)                        -> image_features (1,577,1024)
encoder: (image_features, input_ids (1,32), mask (1,32))   -> hidden (1,609,1024)
decoder: (hidden, mask (1,32), decoder_input_ids (1,128))  -> logits (1,128,51328)
```

Every file takes and returns fp32 tensors, so a precision is a file swap and the three
parts can be mixed.

| part | build | file | MB | corr vs fp32 eager | Mac ms* |
|---|---|---|---|---|---|
| vision | XNNPACK fp32 | `florence2_large_vision_xnnpack_fp32.pte` | 1452.6 | 1.000000 | 803.9 |
| vision | Core ML (iOS) | `florence2_large_vision_coreml_all.pte` | 729.5 | 0.999953 | 238.2 |
| encoder | XNNPACK fp32 | `florence2_large_encoder_xnnpack_fp32.pte` | 831.8 | 1.000000 | 144.4 |
| encoder | Core ML (iOS) | `florence2_large_encoder_coreml_all.pte` | 409.2 | 0.999944 | 43.1 |
| decoder | XNNPACK fp32 | `florence2_large_decoder_xnnpack_fp32.pte` | 1243.8 | 1.000000 | 85.8 |
| decoder | Core ML (iOS) | `florence2_large_decoder_coreml_all.pte` | 509.5 | 0.999980 | 13.8 |

Two sets: **3528 MB** all-fp32, **1648 MB** Core ML. Torch eager fp32 on the same
machine: vision 1288.2 ms, encoder 96.8 ms, decoder 60.6 ms.

\*Mac arm64, single process, median of 5 — a reference point for relative cost, not a
device number. A caption of *n* tokens costs one vision pass, one encoder pass and *n*
decoder passes: about 2.2 s for a 15-token caption on the fp32 set and 0.49 s on Core
ML, on this Mac. The decoder graph is a fixed 128-token window, so every step costs the
same whether it is the first token or the fiftieth.

## Running it

Identical to the base model's contract except for the width (1024 instead of 768).

**1. The image.** RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std
.229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`.

**2. The prompt.** Florence-2's task tokens are shorthand the processor expands into a
sentence before tokenising — the model never sees `<CAPTION>`:

| task | the sentence that is actually tokenised |
|---|---|
| `<CAPTION>` | What does the image describe? |
| `<DETAILED_CAPTION>` | Describe in detail what is shown in the image. |
| `<MORE_DETAILED_CAPTION>` | Describe with a paragraph what is shown in the image. |
| `<OD>` | Locate the objects with category name in the image. |
| `<DENSE_REGION_CAPTION>` | Locate the objects in the image, with their descriptions. |
| `<REGION_PROPOSAL>` | Locate the region proposals in the image. |
| `<OCR>` | What is the text in the image? |
| `<OCR_WITH_REGION>` | What is the text in the image, with regions? |
| `<CAPTION_TO_PHRASE_GROUNDING>` | Locate the phrases in the caption: {your caption} |
| `<OPEN_VOCABULARY_DETECTION>` | Locate {your phrase} in the image. |

Tokenise that sentence as `<s> sentence </s>` with the repo's tokenizer, right-pad to 32
with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and 0 on
the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`.

The 577 image tokens are handled inside the graphs. The prompt sequence the original
model sees is `<image>×577 + <s> prompt </s>`, and because the image tokens are a
contiguous prefix, the encoder here concatenates the vision features in front of the text
embeddings instead of scattering them into placeholder positions.

**3. Greedy decoding, and the rule you cannot skip.** Fill a `(1,128)` int64 window with
the pad id, write the decoder start token (2) at position 0, then for step `t`:

```
logits = decoder(hidden, mask, window)      # mask is the same one the encoder took
ban every token that would repeat a 3-gram already in the output   # <- see below
next = argmax(logits[0, t])
if next == 2: stop                          # </s>
window[0, t + 1] = next
```

**`no_repeat_ngram_size: 3` from the model's `generation_config.json` is load-bearing on
this size.** Large's decoder returns `<s>` as its argmax three times in a row on most
images. The ban on repeating that 3-gram is the only thing that moves it on to the
caption — a plain argmax loop emits `<s>` forever and returns an empty string. On five
test photographs it did so every time, before and after conversion, in eager PyTorch as
well as through the `.pte`. Base tolerates the omission and large does not, so implement
the rule.

**4. Reading a detection answer.** Grounded tasks answer with `<loc_N>` tokens, `N` in
0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side / 1000` in the
original image's pixels — `side` being the image's width for x and its height for y, not
768.

## Verification

The three wrappers reproduce `Florence2ForConditionalGeneration` **exactly**: composition
`max_abs_diff` 0.000e+00 against the full model's logits on the same image and prompt.

End to end through the three `.pte` files, greedy `<CAPTION>` on five photographs against
the same decoding in eager PyTorch: **fp32 5/5 and Core ML 5/5 captions identical**,
character for character.

```
A person's hands playing a piano with the words Lauberger and Gloss written on it.
A long wooden pier stretching out into the ocean on a sunny day.
A forest of dead trees in the middle of a forest.
A road in the middle of a pine forest lined with tall trees.
A couple of wooden benches sitting on top of a park bench covered in leaves.
```

```bash
python convert/check_florence2.py large fp32     # or coreml_all
```

## Not shipped, and why

**int8 converts and is not published.** Dynamic int8 brings the set from 3528 MB to
1304 MB at correlations of 0.999, 0.9995 and 0.9893 — numbers that would pass any gate
on this shelf. The captions do not: **2 of 5 match eager**, and the other three are
plausible but different sentences, one of them reworded from the first word. The same
recipe on Florence-2-base keeps all five, so this is a property of the larger decoder
rather than of the recipe. On iOS the Core ML set is the small one (1648 MB, 5/5); on
Android it is fp32 or nothing until a better recipe is found.

**fp16 for the encoder and the decoder does not export.** BART clamps its activations
when, and only when, they are half precision:

```python
if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():
```

In fp32 that line short-circuits and never reaches the graph. Halve the model and it
becomes a question about values `torch.export` cannot answer, and export stops with
`GuardOnDataDependentSymNode`.

## Conversion notes

Converted from
[florence-community/Florence-2-large](https://huggingface.co/florence-community/Florence-2-large),
the transformers-format mirror of
[microsoft/Florence-2-large](https://huggingface.co/microsoft/Florence-2-large) — same MIT
weights. The original repo predates the in-tree implementation and its weight names do not
match it: loading it into `Florence2ForConditionalGeneration` prints a load report where
every key is unexpected and hands back a randomly initialised model without raising.

The checkpoint declares `torch_dtype: float16` and transformers honours it, so
`from_pretrained` must be given `dtype=torch.float32` explicitly.

- **Source**: microsoft/Florence-2-large (via florence-community mirror)
- **License**: MIT

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))

	---
	license: mit
	tags:
	- executorch
	- xnnpack
	- pte
	- on-device
	- image-text-to-text
	- image-to-text
	- object-detection
	base_model:
	- microsoft/Florence-2-large
	base_model_relation: quantized
	---
	# Florence-2-large — ExecuTorch (vision + text encoder + text decoder)

	One set of weights that captions, detects, reads text and grounds phrases, with the
	task chosen by the prompt you send. 0.77 B parameters — the same interface as
	[Florence-2-base](https://huggingface.co/mlboydaisuke/Florence-2-base-ExecuTorch)
	at roughly three times the size, and it notices more: on the same photograph base
	says "A black and white photo of a person playing a piano" where large reads the
	brand off the instrument.

	Three `.pte` files, split the way Whisper is on this shelf and for the same reason:
	the vision tower and the text encoder run once per image, the decoder runs once per
	generated token.

	```
	vision : pixel_values (1,3,768,768) -> image_features (1,577,1024)
	encoder: (image_features, input_ids (1,32), mask (1,32)) -> hidden (1,609,1024)
	decoder: (hidden, mask (1,32), decoder_input_ids (1,128)) -> logits (1,128,51328)
	```

	Every file takes and returns fp32 tensors, so a precision is a file swap and the three
	parts can be mixed.

	\| part \| build \| file \| MB \| corr vs fp32 eager \| Mac ms* \|
	\|---\|---\|---\|---\|---\|---\|
	\| vision \| XNNPACK fp32 \| `florence2_large_vision_xnnpack_fp32.pte` \| 1452.6 \| 1.000000 \| 803.9 \|
	\| vision \| Core ML (iOS) \| `florence2_large_vision_coreml_all.pte` \| 729.5 \| 0.999953 \| 238.2 \|
	\| encoder \| XNNPACK fp32 \| `florence2_large_encoder_xnnpack_fp32.pte` \| 831.8 \| 1.000000 \| 144.4 \|
	\| encoder \| Core ML (iOS) \| `florence2_large_encoder_coreml_all.pte` \| 409.2 \| 0.999944 \| 43.1 \|
	\| decoder \| XNNPACK fp32 \| `florence2_large_decoder_xnnpack_fp32.pte` \| 1243.8 \| 1.000000 \| 85.8 \|
	\| decoder \| Core ML (iOS) \| `florence2_large_decoder_coreml_all.pte` \| 509.5 \| 0.999980 \| 13.8 \|

	Two sets: 3528 MB all-fp32, 1648 MB Core ML. Torch eager fp32 on the same
	machine: vision 1288.2 ms, encoder 96.8 ms, decoder 60.6 ms.

	\*Mac arm64, single process, median of 5 — a reference point for relative cost, not a
	device number. A caption of n tokens costs one vision pass, one encoder pass and n
	decoder passes: about 2.2 s for a 15-token caption on the fp32 set and 0.49 s on Core
	ML, on this Mac. The decoder graph is a fixed 128-token window, so every step costs the
	same whether it is the first token or the fiftieth.

	## Running it

	Identical to the base model's contract except for the width (1024 instead of 768).

	1. The image. RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std
	.229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`.

	2. The prompt. Florence-2's task tokens are shorthand the processor expands into a
	sentence before tokenising — the model never sees `<CAPTION>`:

	\| task \| the sentence that is actually tokenised \|
	\|---\|---\|
	\| `<CAPTION>` \| What does the image describe? \|
	\| `<DETAILED_CAPTION>` \| Describe in detail what is shown in the image. \|
	\| `<MORE_DETAILED_CAPTION>` \| Describe with a paragraph what is shown in the image. \|
	\| `<OD>` \| Locate the objects with category name in the image. \|
	\| `<DENSE_REGION_CAPTION>` \| Locate the objects in the image, with their descriptions. \|
	\| `<REGION_PROPOSAL>` \| Locate the region proposals in the image. \|
	\| `<OCR>` \| What is the text in the image? \|
	\| `<OCR_WITH_REGION>` \| What is the text in the image, with regions? \|
	\| `<CAPTION_TO_PHRASE_GROUNDING>` \| Locate the phrases in the caption: {your caption} \|
	\| `<OPEN_VOCABULARY_DETECTION>` \| Locate {your phrase} in the image. \|

	Tokenise that sentence as `<s> sentence </s>` with the repo's tokenizer, right-pad to 32
	with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and 0 on
	the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`.

	The 577 image tokens are handled inside the graphs. The prompt sequence the original
	model sees is `<image>×577 + <s> prompt </s>`, and because the image tokens are a
	contiguous prefix, the encoder here concatenates the vision features in front of the text
	embeddings instead of scattering them into placeholder positions.

	3. Greedy decoding, and the rule you cannot skip. Fill a `(1,128)` int64 window with
	the pad id, write the decoder start token (2) at position 0, then for step `t`:

	```
	logits = decoder(hidden, mask, window) # mask is the same one the encoder took
	ban every token that would repeat a 3-gram already in the output # <- see below
	next = argmax(logits[0, t])
	if next == 2: stop # </s>
	window[0, t + 1] = next
	```

	**`no_repeat_ngram_size: 3` from the model's `generation_config.json` is load-bearing on
	this size.** Large's decoder returns `<s>` as its argmax three times in a row on most
	images. The ban on repeating that 3-gram is the only thing that moves it on to the
	caption — a plain argmax loop emits `<s>` forever and returns an empty string. On five
	test photographs it did so every time, before and after conversion, in eager PyTorch as
	well as through the `.pte`. Base tolerates the omission and large does not, so implement
	the rule.

	4. Reading a detection answer. Grounded tasks answer with `<loc_N>` tokens, `N` in
	0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side / 1000` in the
	original image's pixels — `side` being the image's width for x and its height for y, not
	768.

	## Verification

	The three wrappers reproduce `Florence2ForConditionalGeneration` exactly: composition
	`max_abs_diff` 0.000e+00 against the full model's logits on the same image and prompt.

	End to end through the three `.pte` files, greedy `<CAPTION>` on five photographs against
	the same decoding in eager PyTorch: fp32 5/5 and Core ML 5/5 captions identical,
	character for character.

	```
	A person's hands playing a piano with the words Lauberger and Gloss written on it.
	A long wooden pier stretching out into the ocean on a sunny day.
	A forest of dead trees in the middle of a forest.
	A road in the middle of a pine forest lined with tall trees.
	A couple of wooden benches sitting on top of a park bench covered in leaves.
	```

	```bash
	python convert/check_florence2.py large fp32 # or coreml_all
	```

	## Not shipped, and why

	int8 converts and is not published. Dynamic int8 brings the set from 3528 MB to
	1304 MB at correlations of 0.999, 0.9995 and 0.9893 — numbers that would pass any gate
	on this shelf. The captions do not: 2 of 5 match eager, and the other three are
	plausible but different sentences, one of them reworded from the first word. The same
	recipe on Florence-2-base keeps all five, so this is a property of the larger decoder
	rather than of the recipe. On iOS the Core ML set is the small one (1648 MB, 5/5); on
	Android it is fp32 or nothing until a better recipe is found.

	fp16 for the encoder and the decoder does not export. BART clamps its activations
	when, and only when, they are half precision:

	```python
	if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():
	```

	In fp32 that line short-circuits and never reaches the graph. Halve the model and it
	becomes a question about values `torch.export` cannot answer, and export stops with
	`GuardOnDataDependentSymNode`.

	## Conversion notes

	Converted from
	[florence-community/Florence-2-large](https://huggingface.co/florence-community/Florence-2-large),
	the transformers-format mirror of
	[microsoft/Florence-2-large](https://huggingface.co/microsoft/Florence-2-large) — same MIT
	weights. The original repo predates the in-tree implementation and its weight names do not
	match it: loading it into `Florence2ForConditionalGeneration` prints a load report where
	every key is unexpected and hands back a randomly initialised model without raising.

	The checkpoint declares `torch_dtype: float16` and transformers honours it, so
	`from_pretrained` must be given `dtype=torch.float32` explicitly.

	- Source: microsoft/Florence-2-large (via florence-community mirror)
	- License: MIT

	torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
	(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))