Card: base_model_relation: quantized (so the conversion lists under the base model's Quantizations, not Finetunes)

d2b60a9 verified about 2 months ago

8.09 kB

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- text-ranking
base_model:
- Qwen/Qwen3-Reranker-0.6B
base_model_relation: quantized
---
# Qwen3-Reranker-0.6B — ExecuTorch

A reranker that is not a cross-encoder. The other five on this shelf are BERTs with a
regression head; this one is a **causal language model asked a yes/no question**, and
the score is how much more it would answer "yes" than "no". Same job — read a query and
one document together and score the pair — with a different machine underneath.

```
input_ids      (1, 512) int64   the prompt, LEFT-padded
attention_mask (1, 512) int64
    -> score   (1, 1)   fp32    log-odds: logit("yes") - logit("no")
```

- **Source**: [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) — 595.8M parameters, 28 Qwen3 layers, hidden 1024, 151,669 vocabulary
- **License**: apache-2.0
- **Output**: one number per pair. Higher is more relevant; `sigmoid(x)` maps it to 0..1
  and does not change the ordering.

## Variants

| build | file | size (MB) | worst score error vs eager | Mac ms* | backend takes |
|---|---|---|---|---|---|
| fp32 | `rerank_qwen3_0_6b_xnnpack_fp32.pte` | 2383.7 | **0.0001 logits** | 320.5 | 80.8% |
| fp16 | `rerank_qwen3_0_6b_xnnpack_fp16.pte` | 1192.4 | 0.0268 logits | 1188.8 | 70.4% |
| **Core ML (fp16, iOS)** | `rerank_qwen3_0_6b_coreml_all.pte` | 1196.2 | 0.0886 logits | **83.4** | 100% |

\*Mac arm64, one 512-token pair, **fastest of five medians of ten** — a reference point for
relative cost, not a device number. The host shares its cores with other work and a single
median does not survive that; contention only ever adds time, so the fastest repetition is
the one that means something. Torch eager fp32, measured the same way, is 320.1 ms.
**Core ML is the one to use where it exists**: 3.8x faster than XNNPACK fp32, the whole
graph in one delegated subgraph where XNNPACK takes 80.8% across 172. A reranker earns its
keep over a list of fifty candidates, so that factor is the whole story.
Correlation cannot judge this model — the output is a single number, and the correlation
of a one-element vector is undefined. The gate is the score error in the model's own
units together with whether the ranking survives. Over six real pairs, **every shipped
build reproduces eager's order**, and the narrowest adjacent gap in that ranking is
0.7433 logits.
## Running it
**1. Build the prompt.** The model was trained to read one specific frame, and it is not
a chat model you can prompt freely:
```python
PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
          'based on the Query and the Instruct provided. Note that the answer can '
          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"
text = (f"{PREFIX}<Instruct>: {INSTRUCT}\n<Query>: {query}\n"
        f"<Document>: {document}{SUFFIX}")
```
`INSTRUCT` is a real input, not decoration: it names the retrieval task, and Qwen
reports it is worth a few points to write your own instead of the generic default.
**2. Tokenise with LEFT padding to exactly 512.**
```python
tokenizer = AutoTokenizer.from_pretrained(repo, padding_side="left")
batch = tokenizer([text], padding="max_length", truncation=True, max_length=512,
                  return_tensors="pt")
assert batch["attention_mask"][0, -1] == 1
```
**This is the one that will bite.** The graph reads position −1 unconditionally, which
is the last *real* token only when the padding is on the left. Right-padded input scores
a pad token and returns a confident-looking number that means nothing — nothing raises.
The assertion above is one line and catches it.

**3. Run it once per candidate** and sort by the score, descending.

## The output projection is folded into one vector

The score needs two of the 151,669 logits, and the difference of two dot products is one
dot product against the difference of two rows:

```
logit_yes - logit_no = h . W[yes] - h . W[no] = h . (W[yes] - W[no])
```

So `lm_head` never runs. A 151,669-wide matmul per pair does not happen, and the delegate
does not take its own copy of a 621 MB table — which is exactly why this shelf's Whisper
decoders come out bigger than their weights.

The two vocabulary ids are read from the repo's own `1_LogitScore/config.json`
(**9693** for "yes", **2152** for "no") rather than looked up through the tokenizer,
where a leading-space variant is a different token.

**Checked against the reference, at matching precision.** sentence-transformers'
`CrossEncoder` scores these pairs at +6.6875 / −6.0625 / −7.2812 and the folded head at
+6.7271 / −6.1107 / −7.3159 — a gap of 4.8e-02 that has nothing to do with the fold.
The reference loads the checkpoint in its own **bfloat16**; run both arms in fp32 and the
folded head matches it to **4.673e-05**, while the reference's bf16 and fp32 arms differ
from each other by the full 4.820e-02.

## It ranks across languages

The six test pairs include a Japanese passage that says what the query asks. It comes out
**first of six** at +7.470, above the English document carrying the actual number
(+6.727). The shelf's `ms-marco-MiniLM` rerankers are English-only and put it far lower —
which is the right answer for them, not a defect.

## The attention is eager, and the left padding is why that needed checking

`F.scaled_dot_product_attention` does not survive export as one operation. The edge
dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key
at all leaves **eleven operations XNNPACK cannot take, in every attention block** —
`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Over
28 layers that is 308 operations, each one cutting the subgraph in two. Exporting with
`attn_implementation="eager"` removes them: **72.0% to 80.8% delegated**, fp32 from 381.5
to 320.5 ms.

**This is the one model on the shelf where that guard could plausibly have been doing
work.** Its prompt is left-padded, so the leading rows of a causal mask have no unmasked
key at all — exactly the case the guard exists for. Every other model here is
right-padded, where even an all-padding row still sees the real tokens and the question
never arises.

So it was measured rather than assumed. With 109 real tokens of 512, sdpa against eager:
no `NaN` appears on either arm, and the scored last position reads **54.7696 against
54.7695**. The reason is that the guard fires only on `-inf`, and eager masks with
`torch.finfo(dtype).min`, a large finite number — a fully-masked row comes back uniform
instead of zeroed. Both answers are defined; they differ only about padding rows, and
this graph reads position −1, which is a real token by construction.

## Not shipped: int8

`rerank_qwen3_0_6b_xnnpack_int8.pte` is **1063.9 MB** and runs at 265.4 ms — smaller and
faster than the fp32 build, and 85.3% of it delegates, the most of any build here. It is
withheld on the number that decides.

Dynamic int8 moves a pair score by **0.6336 logits**, against a narrowest adjacent gap of
**0.7433** in the ranking it has to preserve. It happens to keep the order on these six
pairs, but an error that is 85% of the distance between two neighbours is not a build
that ranks reliably — a slightly different candidate list would reorder.

The size was predictable before it was built: the token embedding table is
151,669 x 1024 x 4 = 621 MB of the 2383.7 MB model, a **26.1%** share, and this shelf's
rule `int8/fp16 = 0.5 + 1.5 x (table share)` puts the file at 0.891 of fp16. It came out
at **0.892** — the same figure as Qwen3-Embedding-0.6B, which shares this backbone.

Going eager (below) made this build 13% faster and took it from 78.1% to 85.3%
delegated. It did not move the verdict: speed and quality are separate questions,
and what withholds this file is the score error, which barely changed.

	---
	license: apache-2.0
	tags:
	- executorch
	- xnnpack
	- pte
	- on-device
	- text-ranking
	base_model:
	- Qwen/Qwen3-Reranker-0.6B
	base_model_relation: quantized
	---
	# Qwen3-Reranker-0.6B — ExecuTorch

	A reranker that is not a cross-encoder. The other five on this shelf are BERTs with a
	regression head; this one is a causal language model asked a yes/no question, and
	the score is how much more it would answer "yes" than "no". Same job — read a query and
	one document together and score the pair — with a different machine underneath.

	```
	input_ids (1, 512) int64 the prompt, LEFT-padded
	attention_mask (1, 512) int64
	-> score (1, 1) fp32 log-odds: logit("yes") - logit("no")
	```

	- Source: [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) — 595.8M parameters, 28 Qwen3 layers, hidden 1024, 151,669 vocabulary
	- License: apache-2.0
	- Output: one number per pair. Higher is more relevant; `sigmoid(x)` maps it to 0..1
	and does not change the ordering.

	## Variants

	\| build \| file \| size (MB) \| worst score error vs eager \| Mac ms* \| backend takes \|
	\|---\|---\|---\|---\|---\|---\|
	\| fp32 \| `rerank_qwen3_0_6b_xnnpack_fp32.pte` \| 2383.7 \| 0.0001 logits \| 320.5 \| 80.8% \|
	\| fp16 \| `rerank_qwen3_0_6b_xnnpack_fp16.pte` \| 1192.4 \| 0.0268 logits \| 1188.8 \| 70.4% \|
	\| Core ML (fp16, iOS) \| `rerank_qwen3_0_6b_coreml_all.pte` \| 1196.2 \| 0.0886 logits \| 83.4 \| 100% \|

	\Mac arm64, one 512-token pair, fastest of five medians of ten* — a reference point for
	relative cost, not a device number. The host shares its cores with other work and a single
	median does not survive that; contention only ever adds time, so the fastest repetition is
	the one that means something. Torch eager fp32, measured the same way, is 320.1 ms.
	Core ML is the one to use where it exists: 3.8x faster than XNNPACK fp32, the whole
	graph in one delegated subgraph where XNNPACK takes 80.8% across 172. A reranker earns its
	keep over a list of fifty candidates, so that factor is the whole story.

	Correlation cannot judge this model — the output is a single number, and the correlation
	of a one-element vector is undefined. The gate is the score error in the model's own
	units together with whether the ranking survives. Over six real pairs, **every shipped
	build reproduces eager's order**, and the narrowest adjacent gap in that ranking is
	0.7433 logits.

	## Running it

	1. Build the prompt. The model was trained to read one specific frame, and it is not
	a chat model you can prompt freely:

	```python
	PREFIX = ('<\|im_start\|>system\nJudge whether the Document meets the requirements '
	'based on the Query and the Instruct provided. Note that the answer can '
	'only be "yes" or "no".<\|im_end\|>\n<\|im_start\|>user\n')
	SUFFIX = "<\|im_end\|>\n<\|im_start\|>assistant\n<think>\n\n</think>\n\n"
	INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"

	text = (f"{PREFIX}<Instruct>: {INSTRUCT}\n<Query>: {query}\n"
	f"<Document>: {document}{SUFFIX}")
	```

	`INSTRUCT` is a real input, not decoration: it names the retrieval task, and Qwen
	reports it is worth a few points to write your own instead of the generic default.

	2. Tokenise with LEFT padding to exactly 512.

	```python
	tokenizer = AutoTokenizer.from_pretrained(repo, padding_side="left")
	batch = tokenizer([text], padding="max_length", truncation=True, max_length=512,
	return_tensors="pt")
	assert batch["attention_mask"][0, -1] == 1
	```

	This is the one that will bite. The graph reads position −1 unconditionally, which
	is the last real token only when the padding is on the left. Right-padded input scores
	a pad token and returns a confident-looking number that means nothing — nothing raises.
	The assertion above is one line and catches it.

	3. Run it once per candidate and sort by the score, descending.

	## The output projection is folded into one vector

	The score needs two of the 151,669 logits, and the difference of two dot products is one
	dot product against the difference of two rows:

	```
	logit_yes - logit_no = h . W[yes] - h . W[no] = h . (W[yes] - W[no])
	```

	So `lm_head` never runs. A 151,669-wide matmul per pair does not happen, and the delegate
	does not take its own copy of a 621 MB table — which is exactly why this shelf's Whisper
	decoders come out bigger than their weights.

	The two vocabulary ids are read from the repo's own `1_LogitScore/config.json`
	(9693 for "yes", 2152 for "no") rather than looked up through the tokenizer,
	where a leading-space variant is a different token.

	Checked against the reference, at matching precision. sentence-transformers'
	`CrossEncoder` scores these pairs at +6.6875 / −6.0625 / −7.2812 and the folded head at
	+6.7271 / −6.1107 / −7.3159 — a gap of 4.8e-02 that has nothing to do with the fold.
	The reference loads the checkpoint in its own bfloat16; run both arms in fp32 and the
	folded head matches it to 4.673e-05, while the reference's bf16 and fp32 arms differ
	from each other by the full 4.820e-02.

	## It ranks across languages

	The six test pairs include a Japanese passage that says what the query asks. It comes out
	first of six at +7.470, above the English document carrying the actual number
	(+6.727). The shelf's `ms-marco-MiniLM` rerankers are English-only and put it far lower —
	which is the right answer for them, not a defect.

	## The attention is eager, and the left padding is why that needed checking

	`F.scaled_dot_product_attention` does not survive export as one operation. The edge
	dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key
	at all leaves eleven operations XNNPACK cannot take, in every attention block —
	`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Over
	28 layers that is 308 operations, each one cutting the subgraph in two. Exporting with
	`attn_implementation="eager"` removes them: 72.0% to 80.8% delegated, fp32 from 381.5
	to 320.5 ms.

	**This is the one model on the shelf where that guard could plausibly have been doing
	work.** Its prompt is left-padded, so the leading rows of a causal mask have no unmasked
	key at all — exactly the case the guard exists for. Every other model here is
	right-padded, where even an all-padding row still sees the real tokens and the question
	never arises.

	So it was measured rather than assumed. With 109 real tokens of 512, sdpa against eager:
	no `NaN` appears on either arm, and the scored last position reads **54.7696 against
	54.7695**. The reason is that the guard fires only on `-inf`, and eager masks with
	`torch.finfo(dtype).min`, a large finite number — a fully-masked row comes back uniform
	instead of zeroed. Both answers are defined; they differ only about padding rows, and
	this graph reads position −1, which is a real token by construction.

	## Not shipped: int8

	`rerank_qwen3_0_6b_xnnpack_int8.pte` is 1063.9 MB and runs at 265.4 ms — smaller and
	faster than the fp32 build, and 85.3% of it delegates, the most of any build here. It is
	withheld on the number that decides.

	Dynamic int8 moves a pair score by 0.6336 logits, against a narrowest adjacent gap of
	0.7433 in the ranking it has to preserve. It happens to keep the order on these six
	pairs, but an error that is 85% of the distance between two neighbours is not a build
	that ranks reliably — a slightly different candidate list would reorder.

	The size was predictable before it was built: the token embedding table is
	151,669 x 1024 x 4 = 621 MB of the 2383.7 MB model, a 26.1% share, and this shelf's
	rule `int8/fp16 = 0.5 + 1.5 x (table share)` puts the file at 0.891 of fp16. It came out
	at 0.892 — the same figure as Qwen3-Embedding-0.6B, which shares this backbone.

	Going eager (below) made this build 13% faster and took it from 78.1% to 85.3%
	delegated. It did not move the verdict: speed and quality are separate questions,
	and what withholds this file is the score error, which barely changed.