File size: 4,210 Bytes
89be259 33e4171 89be259 af10c3b 89be259 fcbb966 89be259 fcbb966 89be259 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
license: apache-2.0
base_model: Qwen/Qwen3-Embedding-0.6B
library_name: sentence-transformers
pipeline_tag: feature-extraction
tags:
- transformers
- sentence-similarity
- retrieval
- agent-skills
- skill-routing
- skillcorpus
- contrastive-learning
- text-embeddings-inference
language:
- en
---
# skillrep-embedding-0.6b
A bi-encoder for **agent-skill retrieval**: given a task description, embed it and
the skill documents into one space so the relevant skills come back by nearest
neighbour. Fine-tuned from
[Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
It indexes a skill by its **body** only, unlike
[skillcorpus-embedding-0.6b](https://huggingface.co/EverMind-AI/skillcorpus-embedding-0.6b),
which indexes `name | description | body`. Skill documents follow the schema of
[skillcorpus-demo-1k](https://huggingface.co/datasets/EverMind-AI/skillcorpus-demo-1k).
| Property | Value |
|---|---|
| Parameters | 596M |
| Hidden size | 1024 (embedding dimension) |
| Layers | 28 |
| Precision | bfloat16 |
| Pooling | last token |
| Normalization | L2 |
Requires `transformers>=4.56` (the `dtype=` argument was named `torch_dtype=`
before that) or `sentence-transformers>=3.0`.
## Usage
The two sides are encoded asymmetrically: a task description carries an
instruction prefix followed by one space, and a skill is its bare body. Encode
them this way or retrieval quality drops. Embeddings come back L2-normalized, so
cosine similarity is a plain dot product.
```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
MODEL = "EverMind-AI/skillrep-embedding-0.6b"
tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
model = AutoModel.from_pretrained(MODEL, dtype=torch.bfloat16).cuda().eval()
QUERY_INSTRUCTION = (
"Instruct: Given a coding task description, retrieve the most relevant "
"skill document that would help an agent complete the task\nQuery: "
)
def last_token_pool(hidden, attention_mask):
if attention_mask[:, -1].sum() == attention_mask.shape[0]: # left padding
return hidden[:, -1]
idx = attention_mask.sum(dim=1) - 1
return hidden[torch.arange(hidden.shape[0], device=hidden.device), idx]
def embed(texts, max_length=4096):
enc = tok(texts, padding=True, truncation=True,
max_length=max_length, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model(**enc).last_hidden_state
return F.normalize(last_token_pool(out, enc["attention_mask"]), p=2, dim=1)
query = embed([QUERY_INSTRUCTION + "resolve conflicts after a git merge"])
docs = embed(["# Resolving merge conflicts\n1. Run `git status` to list conflicted files ...",
"# Sourdough\nMix flour, water and starter ..."])
print((query @ docs.T).tolist())
```
Exact scores shift in the last decimal with dtype, hardware and batch
composition; the ordering is what matters.
### With sentence-transformers
The repo ships a sentence-transformers configuration (last-token pooling + L2
normalization, and the query instruction registered as the `query` prompt), so
this path is equivalent to the code above, up to bf16 noise:
```python
from sentence_transformers import SentenceTransformer
st = SentenceTransformer("EverMind-AI/skillrep-embedding-0.6b")
q = st.encode(["resolve conflicts after a git merge"], prompt_name="query")
d = st.encode(["# Resolving merge conflicts\n1. Run `git status` to list conflicted files ...",
"# Sourdough\nMix flour, water and starter ..."])
print(st.similarity(q, d))
```
Pass `prompt_name="query"` for tasks and nothing for skill documents.
### Input lengths
| input | limit |
|---|---|
| task description (after the instruction prefix) | not cut by characters; 4,096 tokens |
| skill `body` | first 8,000 characters; 4,096 tokens |
## Intended use
First-stage retrieval over a large skill registry: encode the registry offline,
encode each incoming task online and take the top K by cosine similarity. Not a
generative model: it produces embeddings, not answers.
## License
Apache-2.0, inherited from the base model. Skills keep their own upstream
licenses.
|