kenga-embed-prophet5

44M-parameter Russian/English sentence encoder from the Kenga project. Bidirectional Z-factored transformer, SentencePiece-16k tokenizer, mean pooling, 768-d L2-normalised output, 512-token context. Prefix protocol is the FRIDA / BERTA one, so it drops into any pipeline that already uses them.

kenga-embed-prophet2 starts from kenga-embed-z2 and adds a supervised stage: InfoNCE with mined hard negatives on 80k retrieval pairs (RuBQ / MIRACL-style QA, wiki title-passage), CoSENT on 20k STS pairs, plus an anchor loss that keeps the embedding close to the teacher so the distilled knowledge is not overwritten. This is the Kenga Prophet stage: learn from verified pairs, penalise confident mistakes (hard negatives).

Official MTEB(rus, v1.1) numbers

Run with mteb==2.20.5, all splits and subsets as defined by the benchmark, no task was skipped or re-weighted. Reference columns are the models' own leaderboard submissions (embeddings-benchmark/results). 23/23 tasks done.

task	kenga-embed-prophet5	Giga-Embeddings-instruct-480M	BERTA-128M	USER2-small-34M	rubert-tiny-turbo-29M
GeoreviewClassification	46.8	55.4	54.8	41.1	41.4
HeadlineClassification	84.6	89.0	89.0	74.3	68.9
InappropriatenessClassification	61.3	86.1	74.8	60.7	59.1
KinopoiskClassification	62.2	73.0	67.8	52.2	50.5
MassiveIntentClassification	61.9	85.3	74.0	66.1	58.0
MassiveScenarioClassification	72.8	90.9	84.5	70.3	62.9
RuReviewsClassification	69.9	76.3	72.3	60.8	60.7
RuSciBenchGRNTIClassification	62.9	74.0	69.0	63.1	52.9
RuSciBenchOECDClassification	49.4	59.9	54.8	49.2	40.8
CEDRClassification	54.6	69.8	73.0	39.4	39.0
SensitiveTopicsClassification	28.6	44.3	39.9	27.5	25.2
GeoreviewClusteringP2P	46.8	73.8	73.8	66.2	59.7
RuSciBenchGRNTIClusteringP2P	59.6	70.5	65.0	56.4	48.1
RuSciBenchOECDClusteringP2P	51.9	58.1	55.6	48.6	41.1
TERRa	61.7	79.6	65.7	54.0	56.3
RuBQReranking	66.0	80.5	75.2	66.0	62.2
MIRACLReranking	47.7	67.5	64.3	50.5	47.7
RiaNewsRetrievalHardNegatives.v2	47.1	88.9	84.5	74.5	52.3
RuBQRetrieval	54.3	80.6	71.0	61.1	51.7
MIRACLRetrievalHardNegatives.v2	43.2	74.7	65.9	46.1	42.4
RUParaPhraserSTS	66.6	78.3	77.8	69.6	72.1
RuSTSBenchmarkSTS	72.6	83.6	82.2	81.0	78.5
STS22	52.1	65.3	61.1	66.1	64.6
---	---	---	---	---	---
Classification (mean)	63.6	76.7	71.2	59.8	55.0
MultilabelClassification (mean)	41.6	57.1	56.5	33.5	32.1
Clustering (mean)	52.8	67.5	64.8	57.1	49.6
PairClassification (mean)	61.7	79.6	65.7	54.0	56.3
Reranking (mean)	56.9	74.0	69.7	58.3	54.9
Retrieval (mean)	48.2	81.4	73.8	60.6	48.8
STS (mean)	63.8	75.7	73.7	72.2	71.7
mean over tasks	57.6	74.2	69.4	58.5	53.7
mean over task types (leaderboard)	55.5	73.1	67.9	56.5	52.6
tasks done	23	23	23	23	23

Leaderboard-style mean (average of task-type means): 55.5.

What this is and is not: a 44M model, roughly 10x smaller than Giga-Embeddings-instruct-480M, meant to be compared with the 30-40M Russian encoders (USER2-small, rubert-tiny-turbo). On the 23 finished tasks (plain mean) it scores 57.6 vs 74.2 for Giga-Embeddings-instruct-480M (+16.6 gap) and 58.5 for USER2-small-34M (-0.9). It does not beat Giga. The numbers above are the whole story; the raw result files are in mteb_results/ of the training tree.

Usage

import sys; sys.path.insert(0, "<this folder>")     # or trust_remote_code-style import after download
from modeling_kenga_embed_v2 import KengaEmbedV2HF

m = KengaEmbedV2HF.from_pretrained("<this folder>", device="cuda")   # cpu works too

q = m.encode(["??? ??????? ????? ? ????"], prefix="search_query")
d = m.encode(["????? ???? ???????? ...", "?????? ?????"], prefix="search_document")
print(q @ d.T)                       # cosine, embeddings are L2-normalised

a = m.encode(["??? ???? ?? ??????."], prefix="paraphrase")
b = m.encode(["?? ?????? ???? ?????."], prefix="paraphrase")

Prefixes ("<prefix>: <text>" is prepended for you):

use	prefix
retrieval query	search_query
retrieval document	search_document
STS / paraphrase (both sides)	paraphrase
classification / clustering	categorize, categorize_sentiment, categorize_topic
NLI / entailment (TERRa)	categorize_entailment

Download with huggingface_hub.snapshot_download("GermannM/kenga-embed-prophet5"); the folder contains pytorch_model.bin, config.json, kenga_spm.model and the self-contained modeling_kenga_embed_v2.py (torch + sentencepiece only).

Architecture

d=768, layers=8, heads=12, dff=3072, factorised token embedding (16385 x 128 -> 768), Z-factored attention/FF projections with rank 192/512, learned positions up to 512. 44.2M parameters, fp32 checkpoint 177 MB. Checkpoint step 250.

Training code

PyTorch trainers live in the z-system lab tree (embed_v2/: build_segments.py, teacher.py, distill.py, build_ft_data.py, mine_hard.py, finetune_prophet.py, run_mteb.py), not in the public kenga-lang repo; the recipe and the Prophet contract are documented in docs/PROPHETS.md. Trained on one GTX 1660 (6 GB).

License

MIT.