Upload skillrep-reranker-0.6b

a32e6e0 verified 13 days ago

5.16 kB

---
license: apache-2.0
base_model: Qwen/Qwen3-Reranker-0.6B
library_name: transformers
pipeline_tag: text-ranking
tags:
  - reranker
  - cross-encoder
  - retrieval
  - agent-skills
  - skill-routing
  - skillcorpus
language:
  - en
---
# skillrep-reranker-0.6b

A cross-encoder for **agent-skill retrieval**: given a task description and a
candidate skill document, judge whether the skill helps complete the task.
Fine-tuned from
[Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B).

It reorders the candidates recalled by
[skillrep-embedding-0.6b](https://huggingface.co/EverMind-AI/skillrep-embedding-0.6b).
Scoring is one forward pass per (task, skill) pair, so run it on a shortlist,
the encoder's top 10, not on the whole registry. Skill documents follow the
schema of
[skillcorpus-demo-1k](https://huggingface.co/datasets/EverMind-AI/skillcorpus-demo-1k).

| Property | Value |
|---|---|
| Parameters | 596M |
| Layers | 28 |
| Precision | bfloat16 |
| Output | `logit("yes") - logit("no")` at the final position |

Requires `transformers>=4.56` (the `dtype=` argument was named `torch_dtype=`
before that).

## Usage

The model answers a yes/no question, and the relevance score is the difference
between the `yes` and `no` logits at the final position. `sigmoid` of this score
is `P("yes")` and gives the same ordering.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "EverMind-AI/skillrep-reranker-0.6b"
tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16).cuda().eval()
PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
          'based on the Query and the Instruct provided. Note that the answer can '
          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
INSTRUCT = ("Given a coding task description, judge whether the skill document "
            "is relevant and useful for completing the task")
YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")
PRE = tok.encode(PREFIX, add_special_tokens=False)
SUF = tok.encode(SUFFIX, add_special_tokens=False)
def pair(task, skill):
    doc = f"{skill['name']} | {skill['description'][:500]} | {skill['body'][:2000]}"
    return f"<Instruct>: {INSTRUCT}\n\n<Query>: {task}\n\n<Document>: {doc}"
def score(task, skills, max_length=4096):
    """One score per skill; higher means more relevant."""
    budget = max_length - len(PRE) - len(SUF)       # truncate the pair, never the template
    ids = [PRE + tok.encode(pair(task, s), add_special_tokens=False,
                            truncation=True, max_length=budget) + SUF for s in skills]
    batch = tok.pad({"input_ids": ids}, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**batch).logits[:, -1, :]
    return (logits[:, YES] - logits[:, NO]).float().tolist()
task = "resolve conflicts after a git merge"
skills = [
    {"name": "resolve-conflicts", "description": "Resolve git merge conflicts.",
     "body": "# Resolving merge conflicts\n1. Run `git status` to list conflicted files ..."},
    {"name": "sourdough", "description": "Bake sourdough bread.",
     "body": "# Sourdough\nMix flour, water and starter ..."},
]
print(score(task, skills))
# -> [3.0, -4.06]
```

Exact scores shift in the last decimal with dtype and hardware; the ordering is
what matters.

Keep the `PREFIX` / `SUFFIX` template and the `<Instruct>` / `<Query>` /
`<Document>` layout, with blank lines between the parts, exactly as above. The
score is read off the final-position logits, so truncate the pair text, not the
template.

### Reranking a shortlist

```python
# candidates: the top 10 skills that skillrep-embedding-0.6b returns for `task`
scores = score(task, candidates)
reranked = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])]
```

### Via the chat template

`chat_template.jinja` reproduces the prompt string above, so the tokenizer can
assemble it instead. It reads three roles: `system` carries the instruction
(omit it to get the default shown above), `query` the task, and `document` the
skill:

```python
prompt = tok.apply_chat_template([
    {"role": "query",    "content": task},
    {"role": "document", "content": "resolve-conflicts | Resolve git merge conflicts. | ..."},
], tokenize=False)
```

### Input lengths

| input | limit |
|---|---|
| task description | not cut by characters |
| skill `description` | first 500 characters |
| skill `body` | first 2,000 characters |
| whole prompt | 4,096 tokens, cutting the pair text and keeping the template |

## Intended use

Second-stage reranking of the top 10 candidates that
[skillrep-embedding-0.6b](https://huggingface.co/EverMind-AI/skillrep-embedding-0.6b)
recalls for a task. Treat the output as a ranking signal rather than a
calibrated probability: compare scores within one candidate list, not across
different tasks.

## License

Apache-2.0, inherited from the base model. Skills keep their own upstream
licenses.

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-Reranker-0.6B
	library_name: transformers
	pipeline_tag: text-ranking
	tags:
	- reranker
	- cross-encoder
	- retrieval
	- agent-skills
	- skill-routing
	- skillcorpus
	language:
	- en
	---

	# skillrep-reranker-0.6b

	A cross-encoder for agent-skill retrieval: given a task description and a
	candidate skill document, judge whether the skill helps complete the task.
	Fine-tuned from
	[Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B).

	It reorders the candidates recalled by
	[skillrep-embedding-0.6b](https://huggingface.co/EverMind-AI/skillrep-embedding-0.6b).
	Scoring is one forward pass per (task, skill) pair, so run it on a shortlist,
	the encoder's top 10, not on the whole registry. Skill documents follow the
	schema of
	[skillcorpus-demo-1k](https://huggingface.co/datasets/EverMind-AI/skillcorpus-demo-1k).

	\| Property \| Value \|
	\|---\|---\|
	\| Parameters \| 596M \|
	\| Layers \| 28 \|
	\| Precision \| bfloat16 \|
	\| Output \| `logit("yes") - logit("no")` at the final position \|

	Requires `transformers>=4.56` (the `dtype=` argument was named `torch_dtype=`
	before that).

	## Usage

	The model answers a yes/no question, and the relevance score is the difference
	between the `yes` and `no` logits at the final position. `sigmoid` of this score
	is `P("yes")` and gives the same ordering.

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	MODEL = "EverMind-AI/skillrep-reranker-0.6b"
	tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
	model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16).cuda().eval()

	PREFIX = ('<\|im_start\|>system\nJudge whether the Document meets the requirements '
	'based on the Query and the Instruct provided. Note that the answer can '
	'only be "yes" or "no".<\|im_end\|>\n<\|im_start\|>user\n')
	SUFFIX = '<\|im_end\|>\n<\|im_start\|>assistant\n<think>\n\n</think>\n\n'
	INSTRUCT = ("Given a coding task description, judge whether the skill document "
	"is relevant and useful for completing the task")

	YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")
	PRE = tok.encode(PREFIX, add_special_tokens=False)
	SUF = tok.encode(SUFFIX, add_special_tokens=False)


	def pair(task, skill):
	doc = f"{skill['name']} \| {skill['description'][:500]} \| {skill['body'][:2000]}"
	return f"<Instruct>: {INSTRUCT}\n\n<Query>: {task}\n\n<Document>: {doc}"


	def score(task, skills, max_length=4096):
	"""One score per skill; higher means more relevant."""
	budget = max_length - len(PRE) - len(SUF) # truncate the pair, never the template
	ids = [PRE + tok.encode(pair(task, s), add_special_tokens=False,
	truncation=True, max_length=budget) + SUF for s in skills]
	batch = tok.pad({"input_ids": ids}, return_tensors="pt").to(model.device)
	with torch.no_grad():
	logits = model(**batch).logits[:, -1, :]
	return (logits[:, YES] - logits[:, NO]).float().tolist()


	task = "resolve conflicts after a git merge"
	skills = [
	{"name": "resolve-conflicts", "description": "Resolve git merge conflicts.",
	"body": "# Resolving merge conflicts\n1. Run `git status` to list conflicted files ..."},
	{"name": "sourdough", "description": "Bake sourdough bread.",
	"body": "# Sourdough\nMix flour, water and starter ..."},
	]
	print(score(task, skills))
	# -> [3.0, -4.06]
	```

	Exact scores shift in the last decimal with dtype and hardware; the ordering is
	what matters.

	Keep the `PREFIX` / `SUFFIX` template and the `<Instruct>` / `<Query>` /
	`<Document>` layout, with blank lines between the parts, exactly as above. The
	score is read off the final-position logits, so truncate the pair text, not the
	template.

	### Reranking a shortlist

	```python
	# candidates: the top 10 skills that skillrep-embedding-0.6b returns for `task`
	scores = score(task, candidates)
	reranked = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])]
	```

	### Via the chat template

	`chat_template.jinja` reproduces the prompt string above, so the tokenizer can
	assemble it instead. It reads three roles: `system` carries the instruction
	(omit it to get the default shown above), `query` the task, and `document` the
	skill:

	```python
	prompt = tok.apply_chat_template([
	{"role": "query", "content": task},
	{"role": "document", "content": "resolve-conflicts \| Resolve git merge conflicts. \| ..."},
	], tokenize=False)
	```

	### Input lengths

	\| input \| limit \|
	\|---\|---\|
	\| task description \| not cut by characters \|
	\| skill `description` \| first 500 characters \|
	\| skill `body` \| first 2,000 characters \|
	\| whole prompt \| 4,096 tokens, cutting the pair text and keeping the template \|

	## Intended use

	Second-stage reranking of the top 10 candidates that
	[skillrep-embedding-0.6b](https://huggingface.co/EverMind-AI/skillrep-embedding-0.6b)
	recalls for a task. Treat the output as a ranking signal rather than a
	calibrated probability: compare scores within one candidate list, not across
	different tasks.

	## License

	Apache-2.0, inherited from the base model. Skills keep their own upstream
	licenses.