Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)

e1e46e7 verified about 2 months ago

5.81 kB

---
license: mit
tags:
- executorch
- xnnpack
- pte
- on-device
- feature-extraction
- sentence-similarity
base_model:
- BAAI/bge-m3
base_model_relation: quantized
---
# bge-m3 — ExecuTorch (dense + sparse + multi-vector, one pass)

Three retrieval signals out of one forward pass. Every other embedding model on this
shelf returns a vector; this one returns a vector, a set of per-token lexical weights,
and a per-token matrix — and they are meant to be combined.

```
input_ids, attention_mask  [1, 512] int64
    -> dense    [1, 1024]        the CLS row, L2-normalised
    -> sparse   [1, 512]         one weight per token, masked
    -> colbert  [1, 511, 1024]   one vector per token, L2-normalised, CLS excluded
```

- **Source**: [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) — 568M parameters,
  XLM-RoBERTa large, 24 layers, 100+ languages
- **License**: MIT
- **No prefix.** Unlike E5 and Qwen3-Embedding on this shelf, bge-m3 wants the text as
  it is, on both sides.

## Using the three heads

**Dense** is ordinary vector search: cosine against other dense vectors.

**Sparse** is lexical matching, BM25-shaped. The graph gives one weight per token
position; the vocabulary-space vector is one line of indexing in the caller:

```python
weights = {}
for w, t in zip(sparse[0], input_ids[0]):
    if t in (0, 1, 2, 3):            # <s>, <pad>, </s>, <unk> — never scored
        continue
    weights[int(t)] = max(weights.get(int(t), 0.0), float(w))   # max over repeats
score = sum(w * other[t] for t, w in weights.items() if t in other)
```

That scatter stays outside the graph on purpose: emitting `[1, 250002]` on every call
would be a megabyte of almost entirely zeros to save the caller those six lines.

**ColBERT** is late interaction: for each query token take its best match among the
document tokens, and sum. Row `i` of the output is token `i+1` of the input — the CLS
row is dropped before projection, which is what the reference implementation does and
is easy to get wrong by one.

## Verification

| build | file | size (MB) | Mac ms* | dense | colbert | sparse weight shift |
|---|---|---|---|---|---|---|
| XNNPACK fp32 | `bge_m3_xnnpack_fp32.pte` | 2271.5 | 233.3 | 1.000000 | 1.000000 | 0.0000 |
| Core ML | `bge_m3_coreml_all.pte` | 1137.2 | **64.8** | 0.999990 | 0.999976 | 0.0008 |
| XNNPACK fp16 | `bge_m3_xnnpack_fp16.pte` | 1136.3 | 484.3 | 0.999999 | 0.999998 | 0.0004 |

\*Mac arm64, median of 10, one 512-token sequence — a reference point for relative
cost, not a device number. Torch eager fp32 on the same machine is 182.8 ms, so the
Core ML build is **2.8x eager**, 100% delegated in a single subgraph. XNNPACK fp32 is
63.7% delegated across 100 subgraphs; its fp16 build is slower than fp32 because
XNNPACK has no fp16 kernels for this graph and inserts casts.
Dense and colbert are worst-case cosine against the eager model over six sentences;
sparse is the largest change to any single token's weight.
**The recipe was checked against the authors' implementation before anything was
exported.** All three heads have a detail that does not throw when wrong — dense is
CLS and not mean, colbert drops the CLS row, and `sparse_linear` is a
`Linear(1024, 1)` giving a scalar per token rather than a projection into vocabulary
space. Against `FlagEmbedding`'s `BGEM3FlagModel` on six sentences:
```
dense    max_abs_diff  2.645e-07
sparse   max_abs_diff  3.427e-07
colbert  max_abs_diff  4.061e-07
```
**And the published number reproduces.** The model card computes a lexical matching
score of `0.19554901123046875` between its two example sentences. Running those same
sentences through the fp32 `.pte` and the scatter above gives **0.1955** — which is the
only independent check there is on a step that happens outside the graph.
Both retrieval heads separate an answer from an unrelated sentence:
```
dense    0.6259 answer  vs  0.3625 unrelated
sparse   0.1955 answer  vs  0.0115 unrelated
```
```bash
python convert/check_bge_m3.py fp32 --reference   # against FlagEmbedding
python convert/check_bge_m3.py fp32               # or fp16, int8, coreml
```
## Two decisions worth knowing about
**The window is 512, not 8192.** bge-m3 accepts 8192 tokens, and the colbert head
returns one 1024-vector per token — so an 8192 window would be a 32 MB output on every
call for a passage that is almost always shorter. 512 covers an ordinary passage;
longer input is the caller's chunking problem.
**The sparse head is masked in the graph, which upstream does not do.** Upstream
returns the raw relu and relies on the caller dropping special tokens at scatter time.
Measured on one 31-token sentence padded to 512, the fp32 model puts weights of up to
**0.196** on padding positions. A caller who forgets to drop them scatters that onto
the pad token's vocabulary slot. Zeroing them here changes no score — the scatter
discards them either way — and removes a silent trap.
That masking also fixed the measurement. Before it, this build's sparse head read
**correlation -0.162** against fp32 eager, which looks like a broken head; on the 31
real token positions it was **+0.998**, and the other 481 were padding neither arm's
caller ever reads.
## Not shipped
**int8** converts and holds — worst head 0.985 — but it comes out at **1363.3 MB
against fp16's 1136.3 MB**. Dynamic int8 quantises the linear weights and leaves the
token embedding table in fp32, and with a 250k vocabulary at 1024 dimensions that
table is **1024 MB of the 2271 MB model, 45%**. This shelf's rule of thumb: int8 beats
fp16 only when the embedding table is under about a third of the weights.
torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))

	---
	license: mit
	tags:
	- executorch
	- xnnpack
	- pte
	- on-device
	- feature-extraction
	- sentence-similarity
	base_model:
	- BAAI/bge-m3
	base_model_relation: quantized
	---
	# bge-m3 — ExecuTorch (dense + sparse + multi-vector, one pass)

	Three retrieval signals out of one forward pass. Every other embedding model on this
	shelf returns a vector; this one returns a vector, a set of per-token lexical weights,
	and a per-token matrix — and they are meant to be combined.

	```
	input_ids, attention_mask [1, 512] int64
	-> dense [1, 1024] the CLS row, L2-normalised
	-> sparse [1, 512] one weight per token, masked
	-> colbert [1, 511, 1024] one vector per token, L2-normalised, CLS excluded
	```

	- Source: [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) — 568M parameters,
	XLM-RoBERTa large, 24 layers, 100+ languages
	- License: MIT
	- No prefix. Unlike E5 and Qwen3-Embedding on this shelf, bge-m3 wants the text as
	it is, on both sides.

	## Using the three heads

	Dense is ordinary vector search: cosine against other dense vectors.

	Sparse is lexical matching, BM25-shaped. The graph gives one weight per token
	position; the vocabulary-space vector is one line of indexing in the caller:

	```python
	weights = {}
	for w, t in zip(sparse[0], input_ids[0]):
	if t in (0, 1, 2, 3): # <s>, <pad>, </s>, <unk> — never scored
	continue
	weights[int(t)] = max(weights.get(int(t), 0.0), float(w)) # max over repeats
	score = sum(w * other[t] for t, w in weights.items() if t in other)
	```

	That scatter stays outside the graph on purpose: emitting `[1, 250002]` on every call
	would be a megabyte of almost entirely zeros to save the caller those six lines.

	ColBERT is late interaction: for each query token take its best match among the
	document tokens, and sum. Row `i` of the output is token `i+1` of the input — the CLS
	row is dropped before projection, which is what the reference implementation does and
	is easy to get wrong by one.

	## Verification

	\| build \| file \| size (MB) \| Mac ms* \| dense \| colbert \| sparse weight shift \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| XNNPACK fp32 \| `bge_m3_xnnpack_fp32.pte` \| 2271.5 \| 233.3 \| 1.000000 \| 1.000000 \| 0.0000 \|
	\| Core ML \| `bge_m3_coreml_all.pte` \| 1137.2 \| 64.8 \| 0.999990 \| 0.999976 \| 0.0008 \|
	\| XNNPACK fp16 \| `bge_m3_xnnpack_fp16.pte` \| 1136.3 \| 484.3 \| 0.999999 \| 0.999998 \| 0.0004 \|

	\*Mac arm64, median of 10, one 512-token sequence — a reference point for relative
	cost, not a device number. Torch eager fp32 on the same machine is 182.8 ms, so the
	Core ML build is 2.8x eager, 100% delegated in a single subgraph. XNNPACK fp32 is
	63.7% delegated across 100 subgraphs; its fp16 build is slower than fp32 because
	XNNPACK has no fp16 kernels for this graph and inserts casts.

	Dense and colbert are worst-case cosine against the eager model over six sentences;
	sparse is the largest change to any single token's weight.

	**The recipe was checked against the authors' implementation before anything was
	exported.** All three heads have a detail that does not throw when wrong — dense is
	CLS and not mean, colbert drops the CLS row, and `sparse_linear` is a
	`Linear(1024, 1)` giving a scalar per token rather than a projection into vocabulary
	space. Against `FlagEmbedding`'s `BGEM3FlagModel` on six sentences:

	```
	dense max_abs_diff 2.645e-07
	sparse max_abs_diff 3.427e-07
	colbert max_abs_diff 4.061e-07
	```

	And the published number reproduces. The model card computes a lexical matching
	score of `0.19554901123046875` between its two example sentences. Running those same
	sentences through the fp32 `.pte` and the scatter above gives 0.1955 — which is the
	only independent check there is on a step that happens outside the graph.

	Both retrieval heads separate an answer from an unrelated sentence:

	```
	dense 0.6259 answer vs 0.3625 unrelated
	sparse 0.1955 answer vs 0.0115 unrelated
	```

	```bash
	python convert/check_bge_m3.py fp32 --reference # against FlagEmbedding
	python convert/check_bge_m3.py fp32 # or fp16, int8, coreml
	```

	## Two decisions worth knowing about

	The window is 512, not 8192. bge-m3 accepts 8192 tokens, and the colbert head
	returns one 1024-vector per token — so an 8192 window would be a 32 MB output on every
	call for a passage that is almost always shorter. 512 covers an ordinary passage;
	longer input is the caller's chunking problem.

	The sparse head is masked in the graph, which upstream does not do. Upstream
	returns the raw relu and relies on the caller dropping special tokens at scatter time.
	Measured on one 31-token sentence padded to 512, the fp32 model puts weights of up to
	0.196 on padding positions. A caller who forgets to drop them scatters that onto
	the pad token's vocabulary slot. Zeroing them here changes no score — the scatter
	discards them either way — and removes a silent trap.

	That masking also fixed the measurement. Before it, this build's sparse head read
	correlation -0.162 against fp32 eager, which looks like a broken head; on the 31
	real token positions it was +0.998, and the other 481 were padding neither arm's
	caller ever reads.

	## Not shipped

	int8 converts and holds — worst head 0.985 — but it comes out at **1363.3 MB
	against fp16's 1136.3 MB**. Dynamic int8 quantises the linear weights and leaves the
	token embedding table in fp32, and with a 250k vocabulary at 1024 dimensions that
	table is 1024 MB of the 2271 MB model, 45%. This shelf's rule of thumb: int8 beats
	fp16 only when the embedding table is under about a third of the weights.

	torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
	(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))