SkillRet-Embedding-8B

This is a sentence-transformers model fine-tuned for AI agent skill retrieval. Given a natural-language user request, the model retrieves relevant agent skills from a large skill library.

The model is fine-tuned from Qwen/Qwen3-Embedding-8B on the SkillRet benchmark training split using contrastive learning (MultipleNegativesRankingLoss).

Usage

Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("anonymous-ed-benchmark/SkillRet-Embedding-8B", trust_remote_code=True)

query_prompt = "Instruct: Given a skill search query, retrieve relevant skills that match the query\nQuery: "

queries = [
    query_prompt + "Help me set up a CI/CD pipeline for my Python project"
]
skills = [
    "ci-cd-setup | Configure continuous integration and deployment pipelines ...",
    "python-debugging | Debug Python applications using pdb and logging ...",
]

q_emb = model.encode(queries, normalize_embeddings=True)
s_emb = model.encode(skills, normalize_embeddings=True)

similarities = q_emb @ s_emb.T
print(similarities)

Training Details

• Base model: Qwen3-Embedding-8B (8B parameters)
• Training data: SkillRet benchmark training split (127,190 query–skill pairs from 63,259 queries and 10,123 skills)
• Loss: MultipleNegativesRankingLoss (InfoNCE) with cross-GPU negative sharing
• Hardware: 4× NVIDIA B200 GPUs (DDP)
• Effective batch size: 80 (20 per device × 4 GPUs)
• Max sequence length: 8,192 tokens
• Learning rate: 2e-5
• Epochs: 1
• Precision: BF16

Evaluation Results

Evaluated on the SkillRet benchmark test split (4,392 queries, 6,006 skills).

Metric	@5	@10	@15
NDCG	0.8458	0.8644	0.8695
Recall	0.8855	0.9325	0.9484
Completeness	0.8035	0.8811	0.9080
MAP	0.8036	0.8155	0.8178
Precision	0.2828	0.1506	0.1025

Intended Use

This model is designed for retrieving agent skills given natural-language user requests. It is part of the SkillRet benchmark submission for evaluating skill retrieval systems for AI agents.

Limitations

• Optimized for English-language queries and agent skills.
• Performance may vary on domains outside the SkillRet benchmark distribution.
• The model retrieves skills but does not execute them.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.4.1
• Transformers: 5.5.4
• PyTorch: 2.7.1+cu128

Citation

Citation information will be added in the de-anonymized release.