Text Ranking
sentence-transformers
Safetensors
Spanish
English
modernbert
cross-encoder
reranker
retrieval
spanish
cross-lingual
kervyn
preview
text-embeddings-inference
Instructions to use Kervyn-AI/Kervyn-Rerank-100M-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use Kervyn-AI/Kervyn-Rerank-100M-Preview with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores) - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Kervyn-AI/Kervyn-Rerank-100M-Preview: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/README.md
- Command line
-
hf download hf://Kervyn-AI/Kervyn-Rerank-100M-Preview/README.md
-
curl -L -o README.md https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/README.md
17.8 kB
| language: | |
| - es | |
| - en | |
| license: apache-2.0 | |
| library_name: sentence-transformers | |
| pipeline_tag: text-ranking | |
| tags: | |
| - sentence-transformers | |
| - cross-encoder | |
| - reranker | |
| - text-ranking | |
| - retrieval | |
| - spanish | |
| - cross-lingual | |
| - kervyn | |
| - preview | |
| datasets: | |
| - HuggingFaceFW/fineweb-2 | |
| - HuggingFaceFW/fineweb-edu | |
| - wikimedia/wikipedia | |
| - sentence-transformers/msmarco-bm25 | |
| - dariolopez/ms-marco-es-500k | |
| - sentence-transformers/natural-questions | |
| - sentence-transformers/gooaq | |
| - sentence-transformers/squad | |
| - sentence-transformers/eli5 | |
| - sentence-transformers/all-nli | |
| - sentence-transformers/parallel-sentences-talks | |
| - sentence-transformers/codesearchnet | |
| - PlanTL-GOB-ES/SQAC | |
| - flax-sentence-embeddings/stackexchange_title_body_jsonl | |
| - flax-sentence-embeddings/stackexchange_titlebody_best_voted_answer_jsonl | |
| # Kervyn Rerank 100M · Preview | |
| <p align="center" style="display:flex; justify-content:center; align-items:center; gap:12px;"> | |
| <a href="#español"><img src="https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/assets/flag-es.svg" width="24" height="16" alt="" style="display:inline-block; vertical-align:middle; margin:0 4px 0 0;"> <b>Español</b></a> | |
| · | |
| <a href="#english"><img src="https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/assets/flag-gb.svg" width="24" height="16" alt="" style="display:inline-block; vertical-align:middle; margin:0 4px 0 0;"> <b>English</b></a> | |
| </p> | |
| --- | |
| ## Español | |
| > **🔬 Preview.** Es la primera versión pública del reranker de Kervyn y todavía está en | |
| > desarrollo. | |
| > | |
| > - **Funciona y está medido:** las cifras de abajo son reales y reproducibles. | |
| > - **Los pesos pueden cambiar** en próximas versiones. Para fijar esta, usa | |
| > `revision="v1"`. | |
| > - Tu opinión nos ayuda a mejorarlo. | |
| Kervyn Rerank es un **cross-encoder**: lee la pregunta y el pasaje juntos y les da una | |
| puntuación de relevancia. | |
| - **Para qué sirve:** reordenar los primeros resultados de cualquier buscador (denso, BM25 | |
| o híbrido) para que el pasaje correcto quede primero. | |
| - **Idiomas:** funciona en **español, inglés y entre los dos**. | |
| - **Modelo propio, de cero:** arquitectura, tokenizer y pesos de Kervyn, entrenados desde | |
| cero. | |
| - **Licencia Apache-2.0.** | |
| - **Pareja:** hace equipo con | |
| [Kervyn Embeddings 100M · Preview](https://huggingface.co/Kervyn-AI/Kervyn-Embeddings-100M-Preview). | |
| ### Ventajas frente a otros modelos del mercado | |
| - **⚡ 5 veces más pequeño y 2,8 veces más rápido en CPU que bge-reranker-v2-m3.** | |
| - Son 104M parámetros frente a 568M. | |
| - Reordena un top-20 en 2,5 s con 8 vCPU, frente a 7,1 s. | |
| - **🥇 Mejor media de búsqueda que bge-reranker-v2-m3**, sobre el top-20 de los dos | |
| buscadores probados: | |
| - **94,45** frente a 94,23 sobre Kervyn Embeddings; | |
| - **94,67** frente a 94,18 sobre bge-m3. | |
| - **💻 No rompe la búsqueda de código.** | |
| - bge-reranker-v2-m3 baja CodeSearchNet de 90,1 a 82,8, peor que no reordenar. | |
| - Kervyn Rerank lo sube a **91,7**. | |
| - **📈 Gran mejora sobre cualquier buscador:** de +2,9 a +3,7 puntos de nDCG@10 de media. | |
| - **🔓 100 % propio y Apache-2.0**, sin licencias heredadas. | |
| - **🛠️ Ideal para ajustarlo con los datos de tu empresa.** Ver | |
| [Ajuste fino](#ajuste-fino-con-los-datos-de-tu-empresa). | |
| ### Uso | |
| ```python | |
| from sentence_transformers import SentenceTransformer, CrossEncoder | |
| embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview") | |
| reranker = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512) | |
| pasajes = [...] # tus documentos, en trozos de ~200 palabras | |
| query = "¿Qué plazo de preaviso tiene el contrato?" | |
| # 1) Recuperación densa: los 20 mejores | |
| scores = embedder.similarity(embedder.encode(query), embedder.encode(pasajes))[0] | |
| top = scores.argsort(descending=True)[:20].tolist() | |
| # 2) Reordenación con Kervyn Rerank | |
| relevancia = reranker.predict([(query, pasajes[i]) for i in top]) | |
| orden = [top[i] for i in relevancia.argsort()[::-1]] | |
| ``` | |
| - **Salida:** una puntuación por par. Más alta es más relevante. | |
| - **Límites de entrada:** hasta 512 tokens entre pregunta y pasaje. La pregunta se recorta a | |
| 400 caracteres. | |
| ### Ajuste fino con los datos de tu empresa | |
| Un reranker ajustado a tu dominio suele ser la mejora más rentable de un buscador: | |
| - **Pequeño:** 104M parámetros, así que el ajuste completo cabe en una GPU de consumo. | |
| - **Estándar:** el ajuste usa el entrenamiento normal de sentence-transformers. | |
| - **Privado:** tus datos no salen de tu infraestructura. | |
| - **Apache-2.0:** el modelo ajustado es tuyo. | |
| Con los mismos pares *(pregunta, pasaje correcto)* que uses para Kervyn Embeddings, el | |
| propio buscador mina los negativos difíciles: pasajes parecidos que no responden a la | |
| pregunta. | |
| ```python | |
| from datasets import Dataset | |
| from sentence_transformers import SentenceTransformer | |
| from sentence_transformers.cross_encoder import (CrossEncoder, CrossEncoderTrainer, | |
| CrossEncoderTrainingArguments) | |
| from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss | |
| from sentence_transformers.util import mine_hard_negatives | |
| embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview") | |
| model = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512) | |
| pairs = Dataset.from_dict({ | |
| "query": ["¿Cuál es el plazo de devolución?", "¿Cómo descargo una factura?"], | |
| "answer": ["Las devoluciones se aceptan durante 30 días desde la entrega.", | |
| "Las facturas se descargan desde Mi cuenta > Pedidos > Factura."], | |
| }) | |
| # 3 negativos por pregunta entre los puestos 5 y 30 del buscador (los primeros pueden ser | |
| # respuestas válidas) | |
| train = mine_hard_negatives(pairs, embedder, num_negatives=3, range_min=5, range_max=30, | |
| output_format="labeled-pair") | |
| args = CrossEncoderTrainingArguments( | |
| output_dir="kervyn-rerank-mi-empresa", | |
| num_train_epochs=1, | |
| per_device_train_batch_size=32, | |
| learning_rate=1e-5, | |
| warmup_ratio=0.1, | |
| ) | |
| trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train, | |
| loss=BinaryCrossEntropyLoss(model)) | |
| trainer.train() | |
| model.save_pretrained("kervyn-rerank-mi-empresa/final") | |
| ``` | |
| ### Benchmarks | |
| Las métricas son estas: | |
| - **nDCG@10** reordenando el top-20 de un buscador denso. | |
| - **Tareas:** Belebele y XQuAD en español, en inglés y entre idiomas (ES→EN: pregunta en | |
| español, pasaje en inglés), documentación técnica en español (kervyn-docs) y | |
| CodeSearchNet Python. | |
| **Sobre el top-20 de Kervyn Embeddings 100M:** | |
| | Tarea | Sin reranker | **+ Kervyn Rerank (104M)** | + bge-reranker-v2-m3 (568M) | | |
| |---|---:|---:|---:| | |
| | Belebele ES | 92,19 | 95,63 | 96,09 | | |
| | Belebele EN | 93,91 | 97,11 | 97,56 | | |
| | Belebele ES→EN | 90,13 | 93,56 | 95,37 | | |
| | XQuAD ES | 95,32 | 98,78 | 99,11 | | |
| | XQuAD EN | 96,25 | 98,77 | 99,23 | | |
| | XQuAD ES→EN | 94,53 | 98,09 | 98,98 | | |
| | kervyn-docs (ES) | 75,78 | 81,89 | 84,67 | | |
| | CodeSearchNet Python | 90,08 | **91,74** | 82,81 | | |
| | **Media (8)** | 91,02 | **94,45** | 94,23 | | |
| | Media sin código (7) | 91,16 | 94,83 | **95,86** | | |
| **Sobre el top-20 de bge-m3:** | |
| | Sistema | Media (8) | Media sin código (7) | | |
| |---|---:|---:| | |
| | bge-m3 sin reranker | 91,41 | 92,28 | | |
| | bge-m3 + **Kervyn Rerank** | **94,67** | 95,16 | | |
| | bge-m3 + bge-reranker-v2-m3 | 94,18 | **95,86** | | |
| - **Media total:** Kervyn Rerank tiene la mejor media en los dos buscadores. | |
| - **Solo texto:** bge-reranker-v2-m3, 5 veces mayor, queda entre 0,7 y 1 punto por delante. | |
| ### Velocidad en CPU | |
| Medido en una CPU x86 de servidor con 8 vCPU, PyTorch en fp32. Cada pregunta puntúa 20 | |
| pares (pregunta, pasaje de ~140 tokens). | |
| | Reranker | Pares/s | Reordenar un top-20 (p50) | | |
| |---|---:|---:| | |
| | **Kervyn Rerank 100M** | 8,1 | **2,5 s** | | |
| | bge-reranker-v2-m3 | 2,9 | 7,1 s | | |
| - **En CPU** el reranker es la parte cara de la búsqueda. Para respuestas interactivas, | |
| reordena el top-5 o el top-10 (~0,6–1,3 s) o usa GPU. | |
| - **Cuantización int8 dinámica:** no la recomendamos en esta Preview, porque cambia el orden | |
| de los resultados. Usa fp32. | |
| ### Detalles del modelo | |
| | Campo | Valor | | |
| |---|---| | |
| | Arquitectura | Encoder bidireccional: 12 capas × 768, atención completa y RoPE, más una cabeza de relevancia (un logit) | | |
| | Parámetros | 104M | | |
| | Entrada | `<|bos|> pregunta <|eos|> pasaje <|eos|>`, hasta 512 tokens | | |
| | Tokenizer | Propio, ~24.600 tokens (el mismo que Kervyn Embeddings) | | |
| | Formato | `ModernBertForSequenceClassification` (safetensors), compatible con `CrossEncoder` | | |
| **Cómo se entrenó:** | |
| 1. **Base:** el mismo preentrenamiento propio que Kervyn Embeddings, con 48.000 millones de | |
| tokens en español, inglés y código. | |
| 2. **Ordenación por listas:** ~680.000 preguntas, cada una con su pasaje correcto y 7 | |
| negativos difíciles minados por Kervyn Embeddings. La pérdida es una entropía cruzada | |
| sobre el grupo de 8. | |
| - **Pares públicos:** MS MARCO en inglés y en español, Natural Questions, GooAQ, SQuAD, | |
| SQAC, ELI5, Stack Exchange, charlas TED ES↔EN y CodeSearchNet. | |
| - **Queries sintéticas**, escritas a partir de pasajes públicos con modelos abiertos de | |
| licencia MIT. | |
| - Se descartan los falsos negativos. | |
| - No se destila ningún reranker de terceros. | |
| ### Limitaciones | |
| - **Preview:** los pesos pueden cambiar. Fija `revision="v1"` si necesitas | |
| reproducibilidad. | |
| - **Texto:** queda algo por detrás de bge-reranker-v2-m3, sobre todo en documentación (−2,8) | |
| y en Belebele ES→EN (−1,8). | |
| - **Velocidad en CPU:** 2,5 s por pregunta para un top-20. Reordena menos pasajes o usa GPU. | |
| - **Idiomas:** solo español e inglés. | |
| - **Datos de entrenamiento:** los datasets públicos listados en `datasets`. | |
| --- | |
| ## English | |
| > **🔬 Preview.** This is the first public release of Kervyn's reranker, and it is still | |
| > under development. | |
| > | |
| > - **It works and it is measured:** the numbers below are real and reproducible. | |
| > - **Weights may change** in future versions. To pin this one, use `revision="v1"`. | |
| > - Your feedback helps us improve it. | |
| Kervyn Rerank is a **cross-encoder**: it reads the question and the passage together and | |
| gives them a relevance score. | |
| - **What it is for:** reranking the top results of any retriever (dense, BM25 or hybrid) | |
| so the right passage comes first. | |
| - **Languages:** it works in **Spanish, English and across the two**. | |
| - **Own model, from scratch:** Kervyn's own architecture, tokenizer and weights, trained | |
| from scratch. | |
| - **Apache-2.0 license.** | |
| - **Pair:** it teams up with | |
| [Kervyn Embeddings 100M · Preview](https://huggingface.co/Kervyn-AI/Kervyn-Embeddings-100M-Preview). | |
| ### Advantages over other models | |
| - **⚡ 5× smaller and 2.8× faster on CPU than bge-reranker-v2-m3.** | |
| - It has 104M parameters vs 568M. | |
| - It reranks a top-20 in 2.5 s on 8 vCPU, vs 7.1 s. | |
| - **🥇 A higher average than bge-reranker-v2-m3**, over the top-20 of both retrievers | |
| tested: | |
| - **94.45** vs 94.23 over Kervyn Embeddings; | |
| - **94.67** vs 94.18 over bge-m3. | |
| - **💻 It does not break code search.** | |
| - bge-reranker-v2-m3 drops CodeSearchNet from 90.1 to 82.8, worse than not reranking. | |
| - Kervyn Rerank lifts it to **91.7**. | |
| - **📈 A big gain over any retriever:** +2.9 to +3.7 nDCG@10 points on average. | |
| - **🔓 100% own and Apache-2.0**, with no inherited licenses. | |
| - **🛠️ Ideal for fine-tuning on your company data.** See | |
| [Fine-tuning](#fine-tuning-on-your-company-data). | |
| ### Usage | |
| ```python | |
| from sentence_transformers import SentenceTransformer, CrossEncoder | |
| embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview") | |
| reranker = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512) | |
| passages = [...] # your documents, in chunks of ~200 words | |
| query = "What notice period does the contract have?" | |
| # 1) Dense retrieval: the top 20 | |
| scores = embedder.similarity(embedder.encode(query), embedder.encode(passages))[0] | |
| top = scores.argsort(descending=True)[:20].tolist() | |
| # 2) Reranking with Kervyn Rerank | |
| relevance = reranker.predict([(query, passages[i]) for i in top]) | |
| order = [top[i] for i in relevance.argsort()[::-1]] | |
| ``` | |
| - **Output:** one score per pair. Higher means more relevant. | |
| - **Input limits:** up to 512 tokens for question and passage together. The question is | |
| cut to 400 characters. | |
| ### Fine-tuning on your company data | |
| A reranker tuned to your domain is often the most cost-effective upgrade to a search | |
| engine: | |
| - **Small:** 104M parameters, so full fine-tuning fits on a consumer GPU. | |
| - **Standard:** fine-tuning uses regular sentence-transformers training. | |
| - **Private:** your data never leaves your infrastructure. | |
| - **Apache-2.0:** the fine-tuned model is yours. | |
| With the same *(question, correct passage)* pairs you use for Kervyn Embeddings, the | |
| retriever itself mines the hard negatives: similar passages that do not answer the | |
| question. | |
| ```python | |
| from datasets import Dataset | |
| from sentence_transformers import SentenceTransformer | |
| from sentence_transformers.cross_encoder import (CrossEncoder, CrossEncoderTrainer, | |
| CrossEncoderTrainingArguments) | |
| from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss | |
| from sentence_transformers.util import mine_hard_negatives | |
| embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview") | |
| model = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512) | |
| pairs = Dataset.from_dict({ | |
| "query": ["What is the return period?", "How do I download an invoice?"], | |
| "answer": ["Returns are accepted for 30 days after delivery.", | |
| "Invoices can be downloaded from My account > Orders > Invoice."], | |
| }) | |
| # 3 negatives per question from ranks 5 to 30 of the retriever (the very top ones may be | |
| # valid answers) | |
| train = mine_hard_negatives(pairs, embedder, num_negatives=3, range_min=5, range_max=30, | |
| output_format="labeled-pair") | |
| args = CrossEncoderTrainingArguments( | |
| output_dir="kervyn-rerank-my-company", | |
| num_train_epochs=1, | |
| per_device_train_batch_size=32, | |
| learning_rate=1e-5, | |
| warmup_ratio=0.1, | |
| ) | |
| trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train, | |
| loss=BinaryCrossEntropyLoss(model)) | |
| trainer.train() | |
| model.save_pretrained("kervyn-rerank-my-company/final") | |
| ``` | |
| ### Benchmark results | |
| These are the metrics: | |
| - **nDCG@10** when reranking the top-20 of a dense retriever. | |
| - **Tasks:** Belebele and XQuAD in Spanish, in English and across languages (ES→EN: Spanish | |
| question, English passage), technical documentation in Spanish (kervyn-docs) and | |
| CodeSearchNet Python. | |
| **Over the top-20 of Kervyn Embeddings 100M:** | |
| | Task | No reranker | **+ Kervyn Rerank (104M)** | + bge-reranker-v2-m3 (568M) | | |
| |---|---:|---:|---:| | |
| | Belebele ES | 92.19 | 95.63 | 96.09 | | |
| | Belebele EN | 93.91 | 97.11 | 97.56 | | |
| | Belebele ES→EN | 90.13 | 93.56 | 95.37 | | |
| | XQuAD ES | 95.32 | 98.78 | 99.11 | | |
| | XQuAD EN | 96.25 | 98.77 | 99.23 | | |
| | XQuAD ES→EN | 94.53 | 98.09 | 98.98 | | |
| | kervyn-docs (ES) | 75.78 | 81.89 | 84.67 | | |
| | CodeSearchNet Python | 90.08 | **91.74** | 82.81 | | |
| | **Average (8)** | 91.02 | **94.45** | 94.23 | | |
| | Average without code (7) | 91.16 | 94.83 | **95.86** | | |
| **Over the top-20 of bge-m3:** | |
| | System | Average (8) | Average without code (7) | | |
| |---|---:|---:| | |
| | bge-m3, no reranker | 91.41 | 92.28 | | |
| | bge-m3 + **Kervyn Rerank** | **94.67** | 95.16 | | |
| | bge-m3 + bge-reranker-v2-m3 | 94.18 | **95.86** | | |
| - **Overall average:** Kervyn Rerank has the best average over both retrievers. | |
| - **Text only:** bge-reranker-v2-m3, 5× larger, is 0.7 to 1 point ahead. | |
| ### CPU speed | |
| Measured on an x86 server CPU with 8 vCPU, PyTorch in fp32. Each question scores 20 pairs | |
| (question, passage of ~140 tokens). | |
| | Reranker | Pairs/s | Rerank a top-20 (p50) | | |
| |---|---:|---:| | |
| | **Kervyn Rerank 100M** | 8.1 | **2.5 s** | | |
| | bge-reranker-v2-m3 | 2.9 | 7.1 s | | |
| - **On CPU** the reranker is the expensive part of search. For interactive answers, rerank | |
| the top-5 or top-10 (~0.6–1.3 s) or use a GPU. | |
| - **Dynamic int8 quantization:** we do not recommend it in this Preview, because it | |
| changes the order of the results. Use fp32. | |
| ### Model details | |
| | Field | Value | | |
| |---|---| | |
| | Architecture | Bidirectional encoder: 12 layers × 768, full attention and RoPE, plus a relevance head (one logit) | | |
| | Parameters | 104M | | |
| | Input | `<|bos|> question <|eos|> passage <|eos|>`, up to 512 tokens | | |
| | Tokenizer | Own, ~24,600 tokens (the same as Kervyn Embeddings) | | |
| | Format | `ModernBertForSequenceClassification` (safetensors), compatible with `CrossEncoder` | | |
| **How it was trained:** | |
| 1. **Base:** the same own pre-training as Kervyn Embeddings, with 48 billion tokens in | |
| Spanish, English and code. | |
| 2. **Listwise ranking:** ~680,000 questions, each with its correct passage and 7 hard | |
| negatives mined by Kervyn Embeddings. The loss is a cross-entropy over the group of 8. | |
| - **Public pairs:** MS MARCO in English and Spanish, Natural Questions, GooAQ, SQuAD, | |
| SQAC, ELI5, Stack Exchange, ES↔EN TED talks and CodeSearchNet. | |
| - **Synthetic queries**, written from public passages with open MIT-licensed models. | |
| - False negatives are discarded. | |
| - No third-party reranker is distilled. | |
| ### Limitations | |
| - **Preview:** weights may change. Pin `revision="v1"` if you need reproducibility. | |
| - **Text:** slightly behind bge-reranker-v2-m3, mostly on documentation (−2.8) and | |
| Belebele ES→EN (−1.8). | |
| - **CPU speed:** 2.5 s per question for a top-20. Rerank fewer passages or use a GPU. | |
| - **Languages:** Spanish and English only. | |
| - **Training data:** the public datasets listed in `datasets`. | |