Luisbh commited on
Commit
e13b8a7
·
0 Parent(s):

Kervyn Rerank 100M Preview v1

Browse files
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,431 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - es
4
+ - en
5
+ license: apache-2.0
6
+ library_name: sentence-transformers
7
+ pipeline_tag: text-ranking
8
+ tags:
9
+ - sentence-transformers
10
+ - cross-encoder
11
+ - reranker
12
+ - text-ranking
13
+ - retrieval
14
+ - spanish
15
+ - cross-lingual
16
+ - kervyn
17
+ - preview
18
+ datasets:
19
+ - HuggingFaceFW/fineweb-2
20
+ - HuggingFaceFW/fineweb-edu
21
+ - wikimedia/wikipedia
22
+ - sentence-transformers/msmarco-bm25
23
+ - dariolopez/ms-marco-es-500k
24
+ - sentence-transformers/natural-questions
25
+ - sentence-transformers/gooaq
26
+ - sentence-transformers/squad
27
+ - sentence-transformers/eli5
28
+ - sentence-transformers/all-nli
29
+ - sentence-transformers/parallel-sentences-talks
30
+ - sentence-transformers/codesearchnet
31
+ - PlanTL-GOB-ES/SQAC
32
+ - flax-sentence-embeddings/stackexchange_title_body_jsonl
33
+ - flax-sentence-embeddings/stackexchange_titlebody_best_voted_answer_jsonl
34
+ ---
35
+
36
+ # Kervyn Rerank 100M · Preview
37
+
38
+ <p align="center" style="display:flex; justify-content:center; align-items:center; gap:12px;">
39
+ <a href="#español"><img src="https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/assets/flag-es.svg" width="24" height="16" alt="" style="display:inline-block; vertical-align:middle; margin:0 4px 0 0;"> <b>Español</b></a>
40
+ &nbsp;&nbsp;·&nbsp;&nbsp;
41
+ <a href="#english"><img src="https://huggingface.co/Kervyn-AI/Kervyn-Rerank-100M-Preview/resolve/main/assets/flag-gb.svg" width="24" height="16" alt="" style="display:inline-block; vertical-align:middle; margin:0 4px 0 0;"> <b>English</b></a>
42
+ </p>
43
+
44
+ ---
45
+
46
+ ## Español
47
+
48
+ > **🔬 Preview.** Es la primera versión pública del reranker de Kervyn y todavía está en
49
+ > desarrollo.
50
+ >
51
+ > - **Funciona y está medido:** las cifras de abajo son reales y reproducibles.
52
+ > - **Los pesos pueden cambiar** en próximas versiones. Para fijar esta, usa
53
+ > `revision="v1"`.
54
+ > - Tu opinión nos ayuda a mejorarlo.
55
+
56
+ Kervyn Rerank es un **cross-encoder**: lee la pregunta y el pasaje juntos y les da una
57
+ puntuación de relevancia.
58
+
59
+ - **Para qué sirve:** reordenar los primeros resultados de cualquier buscador (denso, BM25
60
+ o híbrido) para que el pasaje correcto quede primero.
61
+ - **Idiomas:** funciona en **español, inglés y entre los dos**.
62
+ - **Modelo propio, de cero:** arquitectura, tokenizer y pesos de Kervyn, entrenados desde
63
+ cero.
64
+ - **Licencia Apache-2.0.**
65
+ - **Pareja:** hace equipo con
66
+ [Kervyn Embeddings 100M · Preview](https://huggingface.co/Kervyn-AI/Kervyn-Embeddings-100M-Preview).
67
+
68
+ ### Ventajas frente a otros modelos del mercado
69
+
70
+ - **⚡ 5 veces más pequeño y 2,8 veces más rápido en CPU que bge-reranker-v2-m3.**
71
+ - Son 104M parámetros frente a 568M.
72
+ - Reordena un top-20 en 2,5 s con 8 vCPU, frente a 7,1 s.
73
+ - **🥇 Mejor media de búsqueda que bge-reranker-v2-m3**, sobre el top-20 de los dos
74
+ buscadores probados:
75
+ - **94,45** frente a 94,23 sobre Kervyn Embeddings;
76
+ - **94,67** frente a 94,18 sobre bge-m3.
77
+ - **💻 No rompe la búsqueda de código.**
78
+ - bge-reranker-v2-m3 baja CodeSearchNet de 90,1 a 82,8, peor que no reordenar.
79
+ - Kervyn Rerank lo sube a **91,7**.
80
+ - **📈 Gran mejora sobre cualquier buscador:** de +2,9 a +3,7 puntos de nDCG@10 de media.
81
+ - **🔓 100 % propio y Apache-2.0**, sin licencias heredadas.
82
+ - **🛠️ Ideal para ajustarlo con los datos de tu empresa.** Ver
83
+ [Ajuste fino](#ajuste-fino-con-los-datos-de-tu-empresa).
84
+
85
+ ### Uso
86
+
87
+ ```python
88
+ from sentence_transformers import SentenceTransformer, CrossEncoder
89
+
90
+ embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview")
91
+ reranker = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512)
92
+
93
+ pasajes = [...] # tus documentos, en trozos de ~200 palabras
94
+ query = "¿Qué plazo de preaviso tiene el contrato?"
95
+
96
+ # 1) Recuperación densa: los 20 mejores
97
+ scores = embedder.similarity(embedder.encode(query), embedder.encode(pasajes))[0]
98
+ top = scores.argsort(descending=True)[:20].tolist()
99
+
100
+ # 2) Reordenación con Kervyn Rerank
101
+ relevancia = reranker.predict([(query, pasajes[i]) for i in top])
102
+ orden = [top[i] for i in relevancia.argsort()[::-1]]
103
+ ```
104
+
105
+ - **Salida:** una puntuación por par. Más alta es más relevante.
106
+ - **Límites de entrada:** hasta 512 tokens entre pregunta y pasaje. La pregunta se recorta a
107
+ 400 caracteres.
108
+
109
+ ### Ajuste fino con los datos de tu empresa
110
+
111
+ Un reranker ajustado a tu dominio suele ser la mejora más rentable de un buscador:
112
+
113
+ - **Pequeño:** 104M parámetros, así que el ajuste completo cabe en una GPU de consumo.
114
+ - **Estándar:** el ajuste usa el entrenamiento normal de sentence-transformers.
115
+ - **Privado:** tus datos no salen de tu infraestructura.
116
+ - **Apache-2.0:** el modelo ajustado es tuyo.
117
+
118
+ Con los mismos pares *(pregunta, pasaje correcto)* que uses para Kervyn Embeddings, el
119
+ propio buscador mina los negativos difíciles: pasajes parecidos que no responden a la
120
+ pregunta.
121
+
122
+ ```python
123
+ from datasets import Dataset
124
+ from sentence_transformers import SentenceTransformer
125
+ from sentence_transformers.cross_encoder import (CrossEncoder, CrossEncoderTrainer,
126
+ CrossEncoderTrainingArguments)
127
+ from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
128
+ from sentence_transformers.util import mine_hard_negatives
129
+
130
+ embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview")
131
+ model = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512)
132
+
133
+ pairs = Dataset.from_dict({
134
+ "query": ["¿Cuál es el plazo de devolución?", "¿Cómo descargo una factura?"],
135
+ "answer": ["Las devoluciones se aceptan durante 30 días desde la entrega.",
136
+ "Las facturas se descargan desde Mi cuenta > Pedidos > Factura."],
137
+ })
138
+ # 3 negativos por pregunta entre los puestos 5 y 30 del buscador (los primeros pueden ser
139
+ # respuestas válidas)
140
+ train = mine_hard_negatives(pairs, embedder, num_negatives=3, range_min=5, range_max=30,
141
+ output_format="labeled-pair")
142
+
143
+ args = CrossEncoderTrainingArguments(
144
+ output_dir="kervyn-rerank-mi-empresa",
145
+ num_train_epochs=1,
146
+ per_device_train_batch_size=32,
147
+ learning_rate=1e-5,
148
+ warmup_ratio=0.1,
149
+ )
150
+ trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train,
151
+ loss=BinaryCrossEntropyLoss(model))
152
+ trainer.train()
153
+ model.save_pretrained("kervyn-rerank-mi-empresa/final")
154
+ ```
155
+
156
+ ### Benchmarks
157
+
158
+ Las métricas son estas:
159
+
160
+ - **nDCG@10** reordenando el top-20 de un buscador denso.
161
+ - **Tareas:** Belebele y XQuAD en español, en inglés y entre idiomas (ES→EN: pregunta en
162
+ español, pasaje en inglés), documentación técnica en español (kervyn-docs) y
163
+ CodeSearchNet Python.
164
+
165
+ **Sobre el top-20 de Kervyn Embeddings 100M:**
166
+
167
+ | Tarea | Sin reranker | **+ Kervyn Rerank (104M)** | + bge-reranker-v2-m3 (568M) |
168
+ |---|---:|---:|---:|
169
+ | Belebele ES | 92,19 | 95,63 | 96,09 |
170
+ | Belebele EN | 93,91 | 97,11 | 97,56 |
171
+ | Belebele ES→EN | 90,13 | 93,56 | 95,37 |
172
+ | XQuAD ES | 95,32 | 98,78 | 99,11 |
173
+ | XQuAD EN | 96,25 | 98,77 | 99,23 |
174
+ | XQuAD ES→EN | 94,53 | 98,09 | 98,98 |
175
+ | kervyn-docs (ES) | 75,78 | 81,89 | 84,67 |
176
+ | CodeSearchNet Python | 90,08 | **91,74** | 82,81 |
177
+ | **Media (8)** | 91,02 | **94,45** | 94,23 |
178
+ | Media sin código (7) | 91,16 | 94,83 | **95,86** |
179
+
180
+ **Sobre el top-20 de bge-m3:**
181
+
182
+ | Sistema | Media (8) | Media sin código (7) |
183
+ |---|---:|---:|
184
+ | bge-m3 sin reranker | 91,41 | 92,28 |
185
+ | bge-m3 + **Kervyn Rerank** | **94,67** | 95,16 |
186
+ | bge-m3 + bge-reranker-v2-m3 | 94,18 | **95,86** |
187
+
188
+ - **Media total:** Kervyn Rerank tiene la mejor media en los dos buscadores.
189
+ - **Solo texto:** bge-reranker-v2-m3, 5 veces mayor, queda entre 0,7 y 1 punto por delante.
190
+
191
+ ### Velocidad en CPU
192
+
193
+ Medido en una CPU x86 de servidor con 8 vCPU, PyTorch en fp32. Cada pregunta puntúa 20
194
+ pares (pregunta, pasaje de ~140 tokens).
195
+
196
+ | Reranker | Pares/s | Reordenar un top-20 (p50) |
197
+ |---|---:|---:|
198
+ | **Kervyn Rerank 100M** | 8,1 | **2,5 s** |
199
+ | bge-reranker-v2-m3 | 2,9 | 7,1 s |
200
+
201
+ - **En CPU** el reranker es la parte cara de la búsqueda. Para respuestas interactivas,
202
+ reordena el top-5 o el top-10 (~0,6–1,3 s) o usa GPU.
203
+ - **Cuantización int8 dinámica:** no la recomendamos en esta Preview, porque cambia el orden
204
+ de los resultados. Usa fp32.
205
+
206
+ ### Detalles del modelo
207
+
208
+ | Campo | Valor |
209
+ |---|---|
210
+ | Arquitectura | Encoder bidireccional: 12 capas × 768, atención completa y RoPE, más una cabeza de relevancia (un logit) |
211
+ | Parámetros | 104M |
212
+ | Entrada | `<|bos|> pregunta <|eos|> pasaje <|eos|>`, hasta 512 tokens |
213
+ | Tokenizer | Propio, ~24.600 tokens (el mismo que Kervyn Embeddings) |
214
+ | Formato | `ModernBertForSequenceClassification` (safetensors), compatible con `CrossEncoder` |
215
+
216
+ **Cómo se entrenó:**
217
+
218
+ 1. **Base:** el mismo preentrenamiento propio que Kervyn Embeddings, con 48.000 millones de
219
+ tokens en español, inglés y código.
220
+ 2. **Ordenación por listas:** ~680.000 preguntas, cada una con su pasaje correcto y 7
221
+ negativos difíciles minados por Kervyn Embeddings. La pérdida es una entropía cruzada
222
+ sobre el grupo de 8.
223
+ - **Pares públicos:** MS MARCO en inglés y en español, Natural Questions, GooAQ, SQuAD,
224
+ SQAC, ELI5, Stack Exchange, charlas TED ES↔EN y CodeSearchNet.
225
+ - **Queries sintéticas**, escritas a partir de pasajes públicos con modelos abiertos de
226
+ licencia MIT.
227
+ - Se descartan los falsos negativos.
228
+ - No se destila ningún reranker de terceros.
229
+
230
+ ### Limitaciones
231
+
232
+ - **Preview:** los pesos pueden cambiar. Fija `revision="v1"` si necesitas
233
+ reproducibilidad.
234
+ - **Texto:** queda algo por detrás de bge-reranker-v2-m3, sobre todo en documentación (−2,8)
235
+ y en Belebele ES→EN (−1,8).
236
+ - **Velocidad en CPU:** 2,5 s por pregunta para un top-20. Reordena menos pasajes o usa GPU.
237
+ - **Idiomas:** solo español e inglés.
238
+ - **Datos de entrenamiento:** los datasets públicos listados en `datasets`.
239
+
240
+ ---
241
+
242
+ ## English
243
+
244
+ > **🔬 Preview.** This is the first public release of Kervyn's reranker, and it is still
245
+ > under development.
246
+ >
247
+ > - **It works and it is measured:** the numbers below are real and reproducible.
248
+ > - **Weights may change** in future versions. To pin this one, use `revision="v1"`.
249
+ > - Your feedback helps us improve it.
250
+
251
+ Kervyn Rerank is a **cross-encoder**: it reads the question and the passage together and
252
+ gives them a relevance score.
253
+
254
+ - **What it is for:** reranking the top results of any retriever (dense, BM25 or hybrid)
255
+ so the right passage comes first.
256
+ - **Languages:** it works in **Spanish, English and across the two**.
257
+ - **Own model, from scratch:** Kervyn's own architecture, tokenizer and weights, trained
258
+ from scratch.
259
+ - **Apache-2.0 license.**
260
+ - **Pair:** it teams up with
261
+ [Kervyn Embeddings 100M · Preview](https://huggingface.co/Kervyn-AI/Kervyn-Embeddings-100M-Preview).
262
+
263
+ ### Advantages over other models
264
+
265
+ - **⚡ 5× smaller and 2.8× faster on CPU than bge-reranker-v2-m3.**
266
+ - It has 104M parameters vs 568M.
267
+ - It reranks a top-20 in 2.5 s on 8 vCPU, vs 7.1 s.
268
+ - **🥇 A higher average than bge-reranker-v2-m3**, over the top-20 of both retrievers
269
+ tested:
270
+ - **94.45** vs 94.23 over Kervyn Embeddings;
271
+ - **94.67** vs 94.18 over bge-m3.
272
+ - **💻 It does not break code search.**
273
+ - bge-reranker-v2-m3 drops CodeSearchNet from 90.1 to 82.8, worse than not reranking.
274
+ - Kervyn Rerank lifts it to **91.7**.
275
+ - **📈 A big gain over any retriever:** +2.9 to +3.7 nDCG@10 points on average.
276
+ - **🔓 100% own and Apache-2.0**, with no inherited licenses.
277
+ - **🛠️ Ideal for fine-tuning on your company data.** See
278
+ [Fine-tuning](#fine-tuning-on-your-company-data).
279
+
280
+ ### Usage
281
+
282
+ ```python
283
+ from sentence_transformers import SentenceTransformer, CrossEncoder
284
+
285
+ embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview")
286
+ reranker = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512)
287
+
288
+ passages = [...] # your documents, in chunks of ~200 words
289
+ query = "What notice period does the contract have?"
290
+
291
+ # 1) Dense retrieval: the top 20
292
+ scores = embedder.similarity(embedder.encode(query), embedder.encode(passages))[0]
293
+ top = scores.argsort(descending=True)[:20].tolist()
294
+
295
+ # 2) Reranking with Kervyn Rerank
296
+ relevance = reranker.predict([(query, passages[i]) for i in top])
297
+ order = [top[i] for i in relevance.argsort()[::-1]]
298
+ ```
299
+
300
+ - **Output:** one score per pair. Higher means more relevant.
301
+ - **Input limits:** up to 512 tokens for question and passage together. The question is
302
+ cut to 400 characters.
303
+
304
+ ### Fine-tuning on your company data
305
+
306
+ A reranker tuned to your domain is often the most cost-effective upgrade to a search
307
+ engine:
308
+
309
+ - **Small:** 104M parameters, so full fine-tuning fits on a consumer GPU.
310
+ - **Standard:** fine-tuning uses regular sentence-transformers training.
311
+ - **Private:** your data never leaves your infrastructure.
312
+ - **Apache-2.0:** the fine-tuned model is yours.
313
+
314
+ With the same *(question, correct passage)* pairs you use for Kervyn Embeddings, the
315
+ retriever itself mines the hard negatives: similar passages that do not answer the
316
+ question.
317
+
318
+ ```python
319
+ from datasets import Dataset
320
+ from sentence_transformers import SentenceTransformer
321
+ from sentence_transformers.cross_encoder import (CrossEncoder, CrossEncoderTrainer,
322
+ CrossEncoderTrainingArguments)
323
+ from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
324
+ from sentence_transformers.util import mine_hard_negatives
325
+
326
+ embedder = SentenceTransformer("Kervyn-AI/Kervyn-Embeddings-100M-Preview")
327
+ model = CrossEncoder("Kervyn-AI/Kervyn-Rerank-100M-Preview", max_length=512)
328
+
329
+ pairs = Dataset.from_dict({
330
+ "query": ["What is the return period?", "How do I download an invoice?"],
331
+ "answer": ["Returns are accepted for 30 days after delivery.",
332
+ "Invoices can be downloaded from My account > Orders > Invoice."],
333
+ })
334
+ # 3 negatives per question from ranks 5 to 30 of the retriever (the very top ones may be
335
+ # valid answers)
336
+ train = mine_hard_negatives(pairs, embedder, num_negatives=3, range_min=5, range_max=30,
337
+ output_format="labeled-pair")
338
+
339
+ args = CrossEncoderTrainingArguments(
340
+ output_dir="kervyn-rerank-my-company",
341
+ num_train_epochs=1,
342
+ per_device_train_batch_size=32,
343
+ learning_rate=1e-5,
344
+ warmup_ratio=0.1,
345
+ )
346
+ trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train,
347
+ loss=BinaryCrossEntropyLoss(model))
348
+ trainer.train()
349
+ model.save_pretrained("kervyn-rerank-my-company/final")
350
+ ```
351
+
352
+ ### Benchmark results
353
+
354
+ These are the metrics:
355
+
356
+ - **nDCG@10** when reranking the top-20 of a dense retriever.
357
+ - **Tasks:** Belebele and XQuAD in Spanish, in English and across languages (ES→EN: Spanish
358
+ question, English passage), technical documentation in Spanish (kervyn-docs) and
359
+ CodeSearchNet Python.
360
+
361
+ **Over the top-20 of Kervyn Embeddings 100M:**
362
+
363
+ | Task | No reranker | **+ Kervyn Rerank (104M)** | + bge-reranker-v2-m3 (568M) |
364
+ |---|---:|---:|---:|
365
+ | Belebele ES | 92.19 | 95.63 | 96.09 |
366
+ | Belebele EN | 93.91 | 97.11 | 97.56 |
367
+ | Belebele ES→EN | 90.13 | 93.56 | 95.37 |
368
+ | XQuAD ES | 95.32 | 98.78 | 99.11 |
369
+ | XQuAD EN | 96.25 | 98.77 | 99.23 |
370
+ | XQuAD ES→EN | 94.53 | 98.09 | 98.98 |
371
+ | kervyn-docs (ES) | 75.78 | 81.89 | 84.67 |
372
+ | CodeSearchNet Python | 90.08 | **91.74** | 82.81 |
373
+ | **Average (8)** | 91.02 | **94.45** | 94.23 |
374
+ | Average without code (7) | 91.16 | 94.83 | **95.86** |
375
+
376
+ **Over the top-20 of bge-m3:**
377
+
378
+ | System | Average (8) | Average without code (7) |
379
+ |---|---:|---:|
380
+ | bge-m3, no reranker | 91.41 | 92.28 |
381
+ | bge-m3 + **Kervyn Rerank** | **94.67** | 95.16 |
382
+ | bge-m3 + bge-reranker-v2-m3 | 94.18 | **95.86** |
383
+
384
+ - **Overall average:** Kervyn Rerank has the best average over both retrievers.
385
+ - **Text only:** bge-reranker-v2-m3, 5× larger, is 0.7 to 1 point ahead.
386
+
387
+ ### CPU speed
388
+
389
+ Measured on an x86 server CPU with 8 vCPU, PyTorch in fp32. Each question scores 20 pairs
390
+ (question, passage of ~140 tokens).
391
+
392
+ | Reranker | Pairs/s | Rerank a top-20 (p50) |
393
+ |---|---:|---:|
394
+ | **Kervyn Rerank 100M** | 8.1 | **2.5 s** |
395
+ | bge-reranker-v2-m3 | 2.9 | 7.1 s |
396
+
397
+ - **On CPU** the reranker is the expensive part of search. For interactive answers, rerank
398
+ the top-5 or top-10 (~0.6–1.3 s) or use a GPU.
399
+ - **Dynamic int8 quantization:** we do not recommend it in this Preview, because it
400
+ changes the order of the results. Use fp32.
401
+
402
+ ### Model details
403
+
404
+ | Field | Value |
405
+ |---|---|
406
+ | Architecture | Bidirectional encoder: 12 layers × 768, full attention and RoPE, plus a relevance head (one logit) |
407
+ | Parameters | 104M |
408
+ | Input | `<|bos|> question <|eos|> passage <|eos|>`, up to 512 tokens |
409
+ | Tokenizer | Own, ~24,600 tokens (the same as Kervyn Embeddings) |
410
+ | Format | `ModernBertForSequenceClassification` (safetensors), compatible with `CrossEncoder` |
411
+
412
+ **How it was trained:**
413
+
414
+ 1. **Base:** the same own pre-training as Kervyn Embeddings, with 48 billion tokens in
415
+ Spanish, English and code.
416
+ 2. **Listwise ranking:** ~680,000 questions, each with its correct passage and 7 hard
417
+ negatives mined by Kervyn Embeddings. The loss is a cross-entropy over the group of 8.
418
+ - **Public pairs:** MS MARCO in English and Spanish, Natural Questions, GooAQ, SQuAD,
419
+ SQAC, ELI5, Stack Exchange, ES↔EN TED talks and CodeSearchNet.
420
+ - **Synthetic queries**, written from public passages with open MIT-licensed models.
421
+ - False negatives are discarded.
422
+ - No third-party reranker is distilled.
423
+
424
+ ### Limitations
425
+
426
+ - **Preview:** weights may change. Pin `revision="v1"` if you need reproducibility.
427
+ - **Text:** slightly behind bge-reranker-v2-m3, mostly on documentation (−2.8) and
428
+ Belebele ES→EN (−1.8).
429
+ - **CPU speed:** 2.5 s per question for a top-20. Rerank fewer passages or use a GPU.
430
+ - **Languages:** Spanish and English only.
431
+ - **Training data:** the public datasets listed in `datasets`.
assets/flag-es.svg ADDED
assets/flag-gb.svg ADDED
config.json ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ModernBertForSequenceClassification"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "classifier_activation": "gelu",
9
+ "classifier_bias": false,
10
+ "classifier_dropout": 0.0,
11
+ "classifier_pooling": "mean",
12
+ "cls_token_id": 1,
13
+ "decoder_bias": true,
14
+ "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 2,
18
+ "hidden_activation": "gelu",
19
+ "hidden_size": 768,
20
+ "id2label": {
21
+ "0": "LABEL_0"
22
+ },
23
+ "initializer_cutoff_factor": 2.0,
24
+ "initializer_range": 0.02,
25
+ "intermediate_size": 2048,
26
+ "label2id": {
27
+ "LABEL_0": 0
28
+ },
29
+ "layer_types": [
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention"
42
+ ],
43
+ "local_attention": 128,
44
+ "max_position_embeddings": 1024,
45
+ "mlp_bias": false,
46
+ "mlp_dropout": 0.0,
47
+ "model_type": "modernbert",
48
+ "norm_bias": false,
49
+ "norm_eps": 1e-05,
50
+ "num_attention_heads": 12,
51
+ "num_hidden_layers": 12,
52
+ "pad_token_id": 0,
53
+ "rope_parameters": {
54
+ "full_attention": {
55
+ "rope_theta": 160000.0,
56
+ "rope_type": "default"
57
+ },
58
+ "sliding_attention": {
59
+ "rope_theta": 10000.0,
60
+ "rope_type": "default"
61
+ }
62
+ },
63
+ "sep_token_id": 2,
64
+ "sparse_pred_ignore_index": -100,
65
+ "sparse_prediction": false,
66
+ "tie_word_embeddings": true,
67
+ "transformers_version": "5.22.2",
68
+ "vocab_size": 24640
69
+ }
kervyn_rerank.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "rerank_config": {
3
+ "max_length": 512,
4
+ "query_max_chars": 400,
5
+ "group_size": 8,
6
+ "queries_per_step": 32,
7
+ "micro_queries": 8,
8
+ "max_queries": 720000,
9
+ "per_source_cap": 40000,
10
+ "source_caps": {
11
+ "msmarco-en": 50000,
12
+ "mmarco-es": 60000,
13
+ "synth-en2es-question": 70000,
14
+ "synth2-es-keyword": 40000,
15
+ "synth2-es-question": 50000,
16
+ "synth-es2en-question": 60000,
17
+ "synth-es-question": 30000,
18
+ "synth-es-keyword": 30000,
19
+ "synth-en-question": 20000,
20
+ "es-so-title-answer": 40000,
21
+ "se-question-answer": 25000,
22
+ "gooaq-en": 25000,
23
+ "nq-en": 30000,
24
+ "squad-en": 25000,
25
+ "eli5-en": 20000,
26
+ "stackexchange-titles": 15000,
27
+ "talks-en-es": 20000,
28
+ "talks-es-en": 20000,
29
+ "kervyn-sft": 25000,
30
+ "sqac-es": 20000,
31
+ "codesearchnet": 12000,
32
+ "synth-es2code": 8000,
33
+ "synth-en2code": 8000,
34
+ "nli-en": 8000
35
+ },
36
+ "val_queries": 3000,
37
+ "lr": 4e-05,
38
+ "min_lr_ratio": 0.1,
39
+ "warmup_steps": 500,
40
+ "weight_decay": 0.01,
41
+ "grad_clip": 1.0,
42
+ "classifier_pooling": "mean",
43
+ "eval_interval": 2000,
44
+ "log_interval": 50,
45
+ "seed": 1337
46
+ },
47
+ "input": "<|bos|> query <|eos|> pasaje <|eos|>",
48
+ "output": "logit de relevancia",
49
+ "step": 18000,
50
+ "val_loss": 0.16773750543083102,
51
+ "val_acc1": 0.9476666666666667,
52
+ "val_mrr": 0.9703087302595377
53
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:70550327d9526de0cb952ba8d0b450d88e4675d435d433a63acf2053db66e333
3
+ size 417882892
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<|bos|>",
4
+ "cls_token": "<|bos|>",
5
+ "eos_token": "<|eos|>",
6
+ "is_local": true,
7
+ "local_files_only": false,
8
+ "mask_token": "<|mask|>",
9
+ "model_max_length": 1024,
10
+ "pad_token": "<|pad|>",
11
+ "sep_token": "<|eos|>",
12
+ "tokenizer_class": "TokenizersBackend"
13
+ }