Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)

56ac506 verified about 2 months ago

9.51 kB

---
license: mit
tags:
- executorch
- xnnpack
- pte
- on-device
- image-text-to-text
- image-to-text
- object-detection
base_model:
- microsoft/Florence-2-base
base_model_relation: quantized
---
# Florence-2-base — ExecuTorch (vision + text encoder + text decoder)

One set of weights that captions, detects, reads text and grounds phrases, with the
task chosen by the prompt you send. 0.23 B parameters. Three `.pte` files, split the
way Whisper is on this shelf and for the same reason: the vision tower and the text
encoder run once per image, the decoder runs once per generated token.

```
vision : pixel_values (1,3,768,768)                        -> image_features (1,577,768)
encoder: (image_features, input_ids (1,32), mask (1,32))   -> hidden (1,609,768)
decoder: (hidden, mask (1,32), decoder_input_ids (1,128))  -> logits (1,128,51328)
```

Every file takes and returns fp32 tensors, so a precision is a file swap, and the
three parts can be mixed — an fp16 vision tower pairs with an fp32 encoder.

| part | build | file | MB | corr vs fp32 eager | Mac ms* |
|---|---|---|---|---|---|
| vision | XNNPACK fp32 | `florence2_base_vision_xnnpack_fp32.pte` | 365.7 | 1.000000 | 360.0 |
| vision | XNNPACK int8 | `florence2_base_vision_xnnpack_int8.pte` | **112.5** | 0.998901 | 334.7 |
| vision | XNNPACK fp16 | `florence2_base_vision_xnnpack_fp16.pte` | 196.2 | 0.999946 | 1034.5 |
| vision | Core ML (iOS) | `florence2_base_vision_coreml_all.pte` | 185.1 | 0.998304 | 59.7 |
| encoder | XNNPACK fp32 | `florence2_base_encoder_xnnpack_fp32.pte` | 331.0 | 1.000000 | 51.0 |
| encoder | XNNPACK int8 | `florence2_base_encoder_xnnpack_int8.pte` | **203.8** | 0.999534 | 44.3 |
| encoder | Core ML (iOS) | `florence2_base_encoder_coreml_all.pte` | 165.1 | 0.999993 | 13.9 |
| decoder | XNNPACK fp32 | `florence2_base_decoder_xnnpack_fp32.pte` | 545.5 | 1.000000 | 35.7 |
| decoder | XNNPACK int8 | `florence2_base_decoder_xnnpack_int8.pte` | **257.9** | 0.999470 | 33.1 |
| decoder | Core ML (iOS) | `florence2_base_decoder_coreml_all.pte` | 193.0 | 0.999995 | 4.7 |

Three sets: **1242 MB** all-fp32, **574 MB** all-int8, **543 MB** Core ML. The int8
set is the one to reach for on Android — it is smaller than fp32 by more than half
and no slower, and it returns the same captions (below). The int8 recipe is dynamic
quantisation, which reaches the linear layers and leaves the 51328×768 token
embedding table in fp32; that table is 158 MB of the decoder's file, which is why the
decoder halves rather than quarters.

\*Mac arm64, single process, median of 10 — a reference point for relative cost, not
a device number. Torch eager fp32 on the same machine: vision 466.0 ms, encoder
30.8 ms, decoder 23.5 ms.
A caption of *n* tokens costs one vision pass, one encoder pass and *n* decoder
passes: about 0.95 s for a 15-token caption on the fp32 XNNPACK set, 0.88 s on the
int8 set, 0.14 s on Core ML, on this Mac. The decoder graph is a fixed 128-token
window, so every step costs the same whether it is the first token or the fiftieth.

## Running it

**1. The image.** RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std
.229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`.

**2. The prompt.** Florence-2's task tokens are shorthand the processor expands into
a sentence before tokenising — the model never sees `<CAPTION>`:

| task | the sentence that is actually tokenised |
|---|---|
| `<CAPTION>` | What does the image describe? |
| `<DETAILED_CAPTION>` | Describe in detail what is shown in the image. |
| `<MORE_DETAILED_CAPTION>` | Describe with a paragraph what is shown in the image. |
| `<OD>` | Locate the objects with category name in the image. |
| `<DENSE_REGION_CAPTION>` | Locate the objects in the image, with their descriptions. |
| `<REGION_PROPOSAL>` | Locate the region proposals in the image. |
| `<OCR>` | What is the text in the image? |
| `<OCR_WITH_REGION>` | What is the text in the image, with regions? |
| `<CAPTION_TO_PHRASE_GROUNDING>` | Locate the phrases in the caption: {your caption} |
| `<OPEN_VOCABULARY_DETECTION>` | Locate {your phrase} in the image. |

Tokenise that sentence as `<s> sentence </s>` with the repo's tokenizer, right-pad to
32 with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and
0 on the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`.

The 577 image tokens are handled inside the graphs. The prompt sequence the original
model sees is `<image>×577 + <s> prompt </s>`, and because the image tokens are a
contiguous prefix, the encoder here concatenates the vision features in front of the
text embeddings instead of scattering them into placeholder positions. Same
arithmetic, no data-dependent mask, and nothing for the caller to line up.

**3. Greedy decoding.** Fill a `(1,128)` int64 window with the pad id, write the
decoder start token (2) at position 0, then for step `t = 0, 1, 2, ...`:

```
logits = decoder(hidden, mask, window)      # mask is the same one the encoder took
ban every token that would repeat a 3-gram already in the output   # <- see below
next   = argmax(logits[0, t])
if next == 2: stop                          # </s>
window[0, t + 1] = next
```

Detokenise the collected ids with the repo's tokenizer. There is no KV cache: the
decoder is a plain static graph over the window, which is what makes it a single
`.pte` with no state to carry between calls.

**Implement the 3-gram ban.** `no_repeat_ngram_size: 3` is in the model's own
`generation_config.json`, and it is not decoration: Florence-2-**large**'s decoder
returns `<s>` as its argmax three times in a row and only that ban moves it on to the
caption — without it the loop emits `<s>` forever and returns an empty string. This
base model happens to move on by itself on every image tested here, so a plain argmax
loop gives the same five captions. That is luck, not a contract.

**4. Reading a detection answer.** Grounded tasks answer with `<loc_N>` tokens,
`N` in 0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side /
1000` in the original image's pixels — `side` being the image's width for x and its
height for y, not 768. That is the same arithmetic the processor's own parser does.

## Verification

The three wrappers reproduce `Florence2ForConditionalGeneration` **exactly**:
composition `max_abs_diff` 0.000e+00 against the full model's logits on the same
image and prompt.

End to end through the three `.pte` files, greedy `<CAPTION>` on five photographs
against the same decoding in eager PyTorch: **5/5 captions identical**, character for
character. Correlation is measured per part in the table above; the caption test is
what says the split, the prompt layout and the decode agree.

```
A black and white photo of a person playing a piano.
A wooden walkway leading to the ocean on a sunny day.
A group of dead trees in a forest under a cloudy sky.
A long empty road in the middle of a forest.
A couple of wooden benches sitting on top of a field of leaves.
```

The **int8 set answers with the same five sentences**, character for character, with
all three parts quantised at once — which is how they would be used together.

```bash
python convert/check_florence2.py fp32     # or int8
```

## Not shipped, and why

**fp16 for the encoder and the decoder does not export.** BART clamps its activations
when, and only when, they are half precision:

```python
if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():
```

In fp32 that line short-circuits and never reaches the graph. Halve the model and it
becomes a question about values `torch.export` cannot answer, and export stops with
`GuardOnDataDependentSymNode`. The clamp could be shimmed away, but then the shipped
graph would lack the overflow guard the original has, which is not a trade worth
making silently. Nothing is lost by it: int8 is both smaller and faster than fp16
here, and the Core ML builds compute in fp16 internally anyway, holding at
correlation 0.999993 and 0.999995.

**The fp16 vision tower ships and is probably not what you want.** It passes both
gates — 196.2 MB at correlation 0.999946 — but the int8 tower is 112.5 MB and three
times faster (334.7 ms against 1034.5 ms), because XNNPACK has no fp16 kernels here
and inserts casts instead. It is in the repo for anyone who needs more fidelity than
int8 without carrying the fp32 file.

## Conversion notes

Converted from
[florence-community/Florence-2-base](https://huggingface.co/florence-community/Florence-2-base),
the transformers-format mirror of
[microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) — same
MIT weights. The original repo predates the in-tree implementation and its weight
names do not match it: loading it into `Florence2ForConditionalGeneration` prints a
load report where every key is unexpected and hands back a randomly initialised model
without raising.

The checkpoint declares `torch_dtype: float16` and transformers honours it, so
`from_pretrained` must be given `dtype=torch.float32` explicitly. Without it the
"fp32" build serialises fp16 weights — 196 MB for a 90 M parameter tower — and the
lowered vision graph returns NaN.

- **Source**: microsoft/Florence-2-base (via florence-community mirror)
- **License**: MIT

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))

	---
	license: mit
	tags:
	- executorch
	- xnnpack
	- pte
	- on-device
	- image-text-to-text
	- image-to-text
	- object-detection
	base_model:
	- microsoft/Florence-2-base
	base_model_relation: quantized
	---
	# Florence-2-base — ExecuTorch (vision + text encoder + text decoder)

	One set of weights that captions, detects, reads text and grounds phrases, with the
	task chosen by the prompt you send. 0.23 B parameters. Three `.pte` files, split the
	way Whisper is on this shelf and for the same reason: the vision tower and the text
	encoder run once per image, the decoder runs once per generated token.

	```
	vision : pixel_values (1,3,768,768) -> image_features (1,577,768)
	encoder: (image_features, input_ids (1,32), mask (1,32)) -> hidden (1,609,768)
	decoder: (hidden, mask (1,32), decoder_input_ids (1,128)) -> logits (1,128,51328)
	```

	Every file takes and returns fp32 tensors, so a precision is a file swap, and the
	three parts can be mixed — an fp16 vision tower pairs with an fp32 encoder.

	\| part \| build \| file \| MB \| corr vs fp32 eager \| Mac ms* \|
	\|---\|---\|---\|---\|---\|---\|
	\| vision \| XNNPACK fp32 \| `florence2_base_vision_xnnpack_fp32.pte` \| 365.7 \| 1.000000 \| 360.0 \|
	\| vision \| XNNPACK int8 \| `florence2_base_vision_xnnpack_int8.pte` \| 112.5 \| 0.998901 \| 334.7 \|
	\| vision \| XNNPACK fp16 \| `florence2_base_vision_xnnpack_fp16.pte` \| 196.2 \| 0.999946 \| 1034.5 \|
	\| vision \| Core ML (iOS) \| `florence2_base_vision_coreml_all.pte` \| 185.1 \| 0.998304 \| 59.7 \|
	\| encoder \| XNNPACK fp32 \| `florence2_base_encoder_xnnpack_fp32.pte` \| 331.0 \| 1.000000 \| 51.0 \|
	\| encoder \| XNNPACK int8 \| `florence2_base_encoder_xnnpack_int8.pte` \| 203.8 \| 0.999534 \| 44.3 \|
	\| encoder \| Core ML (iOS) \| `florence2_base_encoder_coreml_all.pte` \| 165.1 \| 0.999993 \| 13.9 \|
	\| decoder \| XNNPACK fp32 \| `florence2_base_decoder_xnnpack_fp32.pte` \| 545.5 \| 1.000000 \| 35.7 \|
	\| decoder \| XNNPACK int8 \| `florence2_base_decoder_xnnpack_int8.pte` \| 257.9 \| 0.999470 \| 33.1 \|
	\| decoder \| Core ML (iOS) \| `florence2_base_decoder_coreml_all.pte` \| 193.0 \| 0.999995 \| 4.7 \|

	Three sets: 1242 MB all-fp32, 574 MB all-int8, 543 MB Core ML. The int8
	set is the one to reach for on Android — it is smaller than fp32 by more than half
	and no slower, and it returns the same captions (below). The int8 recipe is dynamic
	quantisation, which reaches the linear layers and leaves the 51328×768 token
	embedding table in fp32; that table is 158 MB of the decoder's file, which is why the
	decoder halves rather than quarters.

	\*Mac arm64, single process, median of 10 — a reference point for relative cost, not
	a device number. Torch eager fp32 on the same machine: vision 466.0 ms, encoder
	30.8 ms, decoder 23.5 ms.

	A caption of n tokens costs one vision pass, one encoder pass and n decoder
	passes: about 0.95 s for a 15-token caption on the fp32 XNNPACK set, 0.88 s on the
	int8 set, 0.14 s on Core ML, on this Mac. The decoder graph is a fixed 128-token
	window, so every step costs the same whether it is the first token or the fiftieth.

	## Running it

	1. The image. RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std
	.229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`.

	2. The prompt. Florence-2's task tokens are shorthand the processor expands into
	a sentence before tokenising — the model never sees `<CAPTION>`:

	\| task \| the sentence that is actually tokenised \|
	\|---\|---\|
	\| `<CAPTION>` \| What does the image describe? \|
	\| `<DETAILED_CAPTION>` \| Describe in detail what is shown in the image. \|
	\| `<MORE_DETAILED_CAPTION>` \| Describe with a paragraph what is shown in the image. \|
	\| `<OD>` \| Locate the objects with category name in the image. \|
	\| `<DENSE_REGION_CAPTION>` \| Locate the objects in the image, with their descriptions. \|
	\| `<REGION_PROPOSAL>` \| Locate the region proposals in the image. \|
	\| `<OCR>` \| What is the text in the image? \|
	\| `<OCR_WITH_REGION>` \| What is the text in the image, with regions? \|
	\| `<CAPTION_TO_PHRASE_GROUNDING>` \| Locate the phrases in the caption: {your caption} \|
	\| `<OPEN_VOCABULARY_DETECTION>` \| Locate {your phrase} in the image. \|

	Tokenise that sentence as `<s> sentence </s>` with the repo's tokenizer, right-pad to
	32 with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and
	0 on the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`.

	The 577 image tokens are handled inside the graphs. The prompt sequence the original
	model sees is `<image>×577 + <s> prompt </s>`, and because the image tokens are a
	contiguous prefix, the encoder here concatenates the vision features in front of the
	text embeddings instead of scattering them into placeholder positions. Same
	arithmetic, no data-dependent mask, and nothing for the caller to line up.

	3. Greedy decoding. Fill a `(1,128)` int64 window with the pad id, write the
	decoder start token (2) at position 0, then for step `t = 0, 1, 2, ...`:

	```
	logits = decoder(hidden, mask, window) # mask is the same one the encoder took
	ban every token that would repeat a 3-gram already in the output # <- see below
	next = argmax(logits[0, t])
	if next == 2: stop # </s>
	window[0, t + 1] = next
	```

	Detokenise the collected ids with the repo's tokenizer. There is no KV cache: the
	decoder is a plain static graph over the window, which is what makes it a single
	`.pte` with no state to carry between calls.

	Implement the 3-gram ban. `no_repeat_ngram_size: 3` is in the model's own
	`generation_config.json`, and it is not decoration: Florence-2-large's decoder
	returns `<s>` as its argmax three times in a row and only that ban moves it on to the
	caption — without it the loop emits `<s>` forever and returns an empty string. This
	base model happens to move on by itself on every image tested here, so a plain argmax
	loop gives the same five captions. That is luck, not a contract.

	4. Reading a detection answer. Grounded tasks answer with `<loc_N>` tokens,
	`N` in 0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side /
	1000` in the original image's pixels — `side` being the image's width for x and its
	height for y, not 768. That is the same arithmetic the processor's own parser does.

	## Verification

	The three wrappers reproduce `Florence2ForConditionalGeneration` exactly:
	composition `max_abs_diff` 0.000e+00 against the full model's logits on the same
	image and prompt.

	End to end through the three `.pte` files, greedy `<CAPTION>` on five photographs
	against the same decoding in eager PyTorch: 5/5 captions identical, character for
	character. Correlation is measured per part in the table above; the caption test is
	what says the split, the prompt layout and the decode agree.

	```
	A black and white photo of a person playing a piano.
	A wooden walkway leading to the ocean on a sunny day.
	A group of dead trees in a forest under a cloudy sky.
	A long empty road in the middle of a forest.
	A couple of wooden benches sitting on top of a field of leaves.
	```

	The int8 set answers with the same five sentences, character for character, with
	all three parts quantised at once — which is how they would be used together.

	```bash
	python convert/check_florence2.py fp32 # or int8
	```

	## Not shipped, and why

	fp16 for the encoder and the decoder does not export. BART clamps its activations
	when, and only when, they are half precision:

	```python
	if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():
	```

	In fp32 that line short-circuits and never reaches the graph. Halve the model and it
	becomes a question about values `torch.export` cannot answer, and export stops with
	`GuardOnDataDependentSymNode`. The clamp could be shimmed away, but then the shipped
	graph would lack the overflow guard the original has, which is not a trade worth
	making silently. Nothing is lost by it: int8 is both smaller and faster than fp16
	here, and the Core ML builds compute in fp16 internally anyway, holding at
	correlation 0.999993 and 0.999995.

	The fp16 vision tower ships and is probably not what you want. It passes both
	gates — 196.2 MB at correlation 0.999946 — but the int8 tower is 112.5 MB and three
	times faster (334.7 ms against 1034.5 ms), because XNNPACK has no fp16 kernels here
	and inserts casts instead. It is in the repo for anyone who needs more fidelity than
	int8 without carrying the fp32 file.

	## Conversion notes

	Converted from
	[florence-community/Florence-2-base](https://huggingface.co/florence-community/Florence-2-base),
	the transformers-format mirror of
	[microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) — same
	MIT weights. The original repo predates the in-tree implementation and its weight
	names do not match it: loading it into `Florence2ForConditionalGeneration` prints a
	load report where every key is unexpected and hands back a randomly initialised model
	without raising.

	The checkpoint declares `torch_dtype: float16` and transformers honours it, so
	`from_pretrained` must be given `dtype=torch.float32` explicitly. Without it the
	"fp32" build serialises fp16 weights — 196 MB for a 90 M parameter tower — and the
	lowered vision graph returns NaN.

	- Source: microsoft/Florence-2-base (via florence-community mirror)
	- License: MIT

	torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
	(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))