junaid008/hindko-embeddings

Hindko Embeddings

License: CC BY-NC 4.0 (non-commercial) for this model's weights and files. It is open for research, teaching and other non-commercial use; commercial use is not permitted without written permission from the author. The base model, intfloat/multilingual-e5-large, is MIT-licensed; its notice is kept in LICENSE. See License.

A sentence-embedding model for Hindko (Northern hno and Southern hnd), the Indo-Aryan language of Peshawar, Hazara and Kohat in Khyber Pakhtunkhwa, Pakistan, written in Urdu (Perso-Arabic) script. It turns a Hindko headline, sentence, verse or passage into a vector, so that texts about the same thing end up close together: for search, retrieval for RAG (retrieval-augmented generation), similarity, clustering and classification. It is intfloat/multilingual-e5-large fine-tuned on 41,533 training pairs from a Hindko corpus (39,517 of them from Hindko-variety records), and to our knowledge it is the first embedding model trained for Hindko.

On the held-out test split of HindkoEmbedBench v1.0, the five-task benchmark built for this project (first used in round 1, for our earlier model), it scores 0.7391 overall. A character n-gram TF-IDF baseline scores 0.6864, and the best public embedding model tested, bge-m3, scores 0.6332 at the same 512-token length. The model is ahead of TF-IDF on four of the five tasks and behind it on one: finding a book passage from its table-of-contents title (see Limitations).

This card documents the model, the benchmark, every result and the training run in detail, with 25 benchmark charts. Every score on this page can be checked against the files in eval/.

At a glance

Type	Sentence-transformers bi-encoder: transformer encoder, mean pooling, L2-normalized output
Base model	intfloat/multilingual-e5-large (MIT), 560M parameters; its XLM-RoBERTa tokenizer is unchanged
Output	1,024-dimensional vectors; compare with cosine similarity (dot product of normalized vectors)
Input length	max_seq_length 512 tokens; longer input is truncated
Prefixes	query: for queries and for similarity, clustering and classification; passage: for documents to be retrieved (details)
Library	sentence-transformers 6.0 or later with transformers 5.x (saved with sentence-transformers 6.1.0 and transformers 5.3.0); older versions may not load the saved configuration
Language	Hindko in Urdu script (hno, hnd). Not evaluated on Romanized Hindko or other languages
Training data	41,533 pairs (87,176 distinct texts) from the train split of a Hindko corpus of newspapers, books and web pages, with 23,061 mined hard negatives
Training compute	322 optimizer steps (2 epochs), 27.0 minutes on one NVIDIA A100-SXM4-80GB, 25.4 GB peak GPU memory
Benchmark	HindkoEmbedBench v1.0: five tasks (headline retrieval, book-title retrieval, passage continuation, verse/prose classification, book clustering), each with a dev and a test split
Test score	0.7391 overall; TF-IDF 0.6864, bge-m3 0.6332, our round-1 model 0.6923
Dev score	0.7117 overall, the highest of the 15 models scored on dev
Selection	Chosen on dev by a rule fixed in advance; scored on test once, after the choice
Known weak spot	Book-title retrieval: 0.3460 nDCG@10 on test, below TF-IDF (0.3756) and bge-m3 (0.3778)
Weights	float16 safetensors, about 1.12 GB (1,068 MiB): the same weights that were benchmarked
License	CC BY-NC 4.0: research and other non-commercial use only; commercial use needs written permission (details)
Version	1.0.0, 2026-10-01 (card updated 2026-10-30; licence changed to CC BY-NC 4.0 on 2026-10-31)
Sibling project	Hindko Tokenizer (GitHub). This model does not use it; it keeps its base model's tokenizer
Code	The benchmark and the training code will be released

Contents

• Highlights
• Quick start
• Usage recipes
• HindkoEmbedBench
• Results
• Detailed analysis
• Training
• Model selection protocol
• Ablation: mined negatives and same-book batches
• Comparison to round 1
• Intended uses
• Out-of-scope uses
• Limitations and bias
• Ethical considerations and privacy
• Reproducibility
• License
• Citation
• Acknowledgements
• Contact
• Changelog

Highlights

• Highest overall test score of all 12 models scored on test: 0.7391, against 0.6864 for TF-IDF (+0.0527), 0.6332 for bge-m3 (+0.1059), 0.5977 for its own base model without fine-tuning (+0.1414) and 0.6923 for our round-1 model (+0.0468).
• First on four of the six test columns (overall, headline retrieval, passage continuation, book clustering), third on verse/prose classification and fourth on book-title retrieval.
• Large gains on passage tasks: continuation retrieval 0.8756 nDCG@10 (TF-IDF 0.7899, bge-m3 0.7174), with the right passage in the top 10 for 356 of 360 test queries; book clustering 0.8566 V-measure (TF-IDF 0.6895, bge-m3 0.5168).
• Highest dev score as well: 0.7117, first of the 15 models scored on dev.
• Better than our round-1 model (0.6923 on test) on four of five tasks; slightly lower on verse/prose classification (0.9801 vs 0.9827).
• Chosen on dev only, by a rule fixed in advance; this model was scored on test once, after the choice.
• Weak spot: book-title retrieval, where TF-IDF and bge-m3 score higher on test (28 queries) and TF-IDF scores higher on dev (262 queries); on dev the model is also narrowly behind TF-IDF on verse/prose (0.9406 vs 0.9463).

Chart 00. Scorecard on the test split. The released model against our other fine-tunes, the lexical TF-IDF reference, the best public model and the untuned base model. It has the best overall (0.7391), headline (0.6372), continuation (0.8756) and clustering (0.8566) scores in the table. On verse/prose the round-1 e5-base fine-tune (0.9827) and the round-2 runner-up (0.9818) are slightly higher (this model 0.9801); on book titles bge-m3 (0.3778), TF-IDF (0.3756) and the round-1 e5-small fine-tune (0.3664) are higher (this model 0.3460).

Quick start

Install

pip install -U "sentence-transformers>=6.0"

Encode and search

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("junaid008/hindko-embeddings")
if model.device.type == "cpu":
    model = model.float()          # weights are stored in float16; use float32 on a CPU

query = "query: ڈرامے دی پیدائش"
passages = [
    "passage: ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔",
    "passage: اج اساں اگر آپڑیں بزرگاں دی انہاں گلاں تے غور نہ کیتا تے پچھتاوے دے سوا کج ہتھ نہ آسی۔",
    "passage: اسی طراں خیبر پختونخوا اچ بولی جانڑیں ولی دیگر زباناں دے ادب و ثقافت دی بی نمائندگی کیتی گئی اَئی۔",
]

q = model.encode([query], normalize_embeddings=True)
p = model.encode(passages, normalize_embeddings=True)
scores = model.similarity(q, p)    # cosine similarity, shape (1, 3); higher = more relevant
ranking = scores[0].argsort(descending=True)

For similarity, clustering and classification, use query: on every text when the texts play the same role:

sentences = [
    "query: ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔",
    "query: اج اساں اگر آپڑیں بزرگاں دی انہاں گلاں تے غور نہ کیتا تے پچھتاوے دے سوا کج ہتھ نہ آسی۔",
    "query: اسی طراں خیبر پختونخوا اچ بولی جانڑیں ولی دیگر زباناں دے ادب و ثقافت دی بی نمائندگی کیتی گئی اَئی۔",
]
emb = model.encode(sentences, normalize_embeddings=True)
sim = model.similarity(emb, emb)   # 3 x 3 cosine-similarity matrix

The example sentences are from the held-out test split of the Hindko Tokenizer, and they also occur in HindkoEmbedBench test documents; they were already published in the tokenizer card. Approximate meanings: (1) "Very few people know how and where drama was born." (2) "If we do not reflect on these words of our elders today, we will be left with nothing but regret." (3) "In the same way, the literature and culture of the other languages spoken in Khyber Pakhtunkhwa were also represented." The query means "the birth of drama".

Use the scores to rank or group texts; they are not calibrated probabilities, so set any threshold on your own data. Before encoding, normalize the input the way the training data was normalized.

Prefixes

The model was trained and evaluated with the e5 prefixes below; results without them were not measured, so keep them.

Text	Prefix
Search query, headline, title, first line of a poem: anything used to look something up	query:
Document, article, passage or poem to be found	passage:
Both sides of a similarity comparison; texts to cluster or classify	query:

In training, title, verse, prose-sentence, headline and continuation pairs used query: for the anchor and passage: for the positive and the negatives; pairs of neighbouring chunks used query: on both sides. The benchmark uses the same scheme.

Write the prefix into the text and call model.encode, as in the examples. encode_query() and encode_document() do not add these prefixes for this model.

Usage recipes

The recipes below are short and runnable once you supply your own texts. They use the same settings as the benchmark (prefixes, normalize_embeddings=True, cosine scoring). Where a recipe goes beyond what the benchmark measured, it says so.

Every recipe assumes the model is loaded as in the Quick start:

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("junaid008/hindko-embeddings")
if model.device.type == "cpu":
    model = model.float()

Semantic search and RAG retrieval

Embed the collection once with passage: , then embed each query with query: and take the top-k passages. For RAG, pass the top-k passages to your generator as context.

docs = [   # your Hindko passages, normalized (see "Input normalization")
    "ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔",
    "اج اساں اگر آپڑیں بزرگاں دی انہاں گلاں تے غور نہ کیتا تے پچھتاوے دے سوا کج ہتھ نہ آسی۔",
    "اسی طراں خیبر پختونخوا اچ بولی جانڑیں ولی دیگر زباناں دے ادب و ثقافت دی بی نمائندگی کیتی گئی اَئی۔",
]
doc_emb = model.encode(["passage: " + d for d in docs], normalize_embeddings=True, convert_to_tensor=True)

def search(query, top_k=3):
    q_emb = model.encode(["query: " + query], normalize_embeddings=True, convert_to_tensor=True)
    hits = util.semantic_search(q_emb, doc_emb, top_k=top_k)[0]
    return [(docs[h["corpus_id"]], h["score"]) for h in hits]

for text, score in search("ڈرامے دی پیدائش"):
    print(f"{score:.3f}  {text}")

context = "\n\n".join(text for text, _ in search("ڈرامے دی پیدائش", top_k=2))   # RAG context

For large collections, store doc_emb in a vector index; with normalized vectors, inner-product search equals cosine search. On the benchmark, the right document was in the top 10 for 57 of 80 test headline queries and 356 of 360 continuation queries, so a top-k of 10 or more is a sensible start for RAG.

Titles and first lines: add lexical search. Book-title retrieval is this model's weak spot, and a character n-gram TF-IDF is stronger there (see Book-title retrieval). A simple hybrid is reciprocal rank fusion of the two rankings. This combination was not scored on HindkoEmbedBench; check it on your own data.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4), sublinear_tf=True)
doc_tfidf = tfidf.fit_transform(docs)          # same docs and doc_emb as above

def hybrid_search(query, top_k=10, k=60):
    q_emb = model.encode(["query: " + query], normalize_embeddings=True, convert_to_tensor=True)
    dense = model.similarity(q_emb, doc_emb)[0].tolist()
    lexical = (doc_tfidf @ tfidf.transform([query]).T).toarray().ravel().tolist()
    fused = {}
    for scores in (dense, lexical):
        order = sorted(range(len(docs)), key=lambda i: -scores[i])
        for rank, i in enumerate(order):
            fused[i] = fused.get(i, 0.0) + 1.0 / (k + rank + 1)
    return sorted(fused, key=fused.get, reverse=True)[:top_k]   # document indices, best first

Sentence similarity

Score pairs of texts that play the same role, with query: on both sides:

left = ["...", "..."]    # your Hindko sentences
right = ["...", "..."]
a = model.encode(["query: " + s for s in left], normalize_embeddings=True)
b = model.encode(["query: " + s for s in right], normalize_embeddings=True)
scores = model.similarity_pairwise(a, b)   # one cosine score per pair

HindkoEmbedBench has no graded similarity task, so these scores are useful for ranking; they were not checked against human similarity judgments.

Clustering with k-means

from sklearn.cluster import KMeans

texts = ["...", "..."]   # your Hindko passages
emb = model.encode(["query: " + t for t in texts], normalize_embeddings=True, batch_size=64)
labels = KMeans(n_clusters=8, n_init=10, random_state=0).fit_predict(emb)

The benchmark task (BookClustering) uses exactly this input: query: prefixes and normalized vectors, with k equal to the number of books. It averages 10 k-means runs (one initialization each, seeds 0 to 9) and scores them with V-measure.

Classification with logistic regression

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

X_train = model.encode(["query: " + t for t in train_texts], normalize_embeddings=True)
X_test = model.encode(["query: " + t for t in test_texts], normalize_embeddings=True)
clf = LogisticRegression(max_iter=1000).fit(X_train, train_labels)
print(f1_score(test_labels, clf.predict(X_test), average="macro"))

This is the benchmark protocol for VerseProseClassification: frozen embeddings, LogisticRegression(max_iter=1000), macro-F1. With 256 training examples per label it reaches 0.9801 macro-F1 on the test split (mean of 10 runs).

Near-duplicate detection

emb = model.encode(["query: " + t for t in texts], normalize_embeddings=True, convert_to_tensor=True)
pairs = util.paraphrase_mining_embeddings(emb, top_k=10)       # [score, i, j], highest first
THRESHOLD = 0.95   # placeholder: choose it on pairs you have checked by hand
near_dups = [(i, j, s) for s, i, j in pairs if s >= THRESHOLD]

Near-duplicate detection was not evaluated. Normalize the text first; for reprints with identical wording, exact or character-shingle matching is cheaper and more precise.

Batching, GPU and float16

import torch

model = SentenceTransformer("junaid008/hindko-embeddings", device="cuda" if torch.cuda.is_available() else "cpu")
model = model.half() if model.device.type == "cuda" else model.float()

emb = model.encode(
    ["passage: " + d for d in docs],
    batch_size=64,                 # lower it if the GPU runs out of memory
    normalize_embeddings=True,
    convert_to_numpy=True,
    show_progress_bar=True,
)
emb = emb.astype("float16")        # 2 KB per text instead of 4 KB for 1,024 dimensions

• GPU in float16. All benchmark numbers for this model were measured from the float16 weights on a GPU in float16, so this is the tested setting.
• CPU in float32. Convert to float32 on a CPU, as above. CPU and float32 results were not benchmarked; expect small numerical differences.
• Batch size. sentence-transformers sorts each call's inputs by length, so one large call is faster than many small ones.
• Shorter max_seq_length. Lowering model.max_seq_length speeds up encoding but cuts more text; this model was only evaluated at 512 (see Truncation).

Long documents: chunking at 512 tokens

The model reads at most 512 tokens, including the prefix and special tokens. Split long documents into passages at sentence boundaries, embed each passage, and score a document by its best passage:

import re

def chunks(text, model, max_tokens=500, prefix="passage: "):
    sents = [s for s in re.split(r"(?<=[۔؟!?\n])\s*", text) if s.strip()]
    out, cur = [], ""
    for s in sents:
        cand = f"{cur} {s}".strip() if cur else s
        if cur and len(model.tokenizer(prefix + cand)["input_ids"]) > max_tokens:
            out.append(cur)
            cur = s
        else:
            cur = cand
    if cur:
        out.append(cur)
    return out          # a single sentence longer than the limit stays one (truncated) chunk

doc_chunks = [(doc_id, c) for doc_id, d in enumerate(long_docs) for c in chunks(d, model)]
chunk_emb = model.encode(["passage: " + c for _, c in doc_chunks], normalize_embeddings=True, convert_to_tensor=True)

def search_long(query):
    q = model.encode(["query: " + query], normalize_embeddings=True, convert_to_tensor=True)
    best = {}
    for (doc_id, _), s in zip(doc_chunks, model.similarity(q, chunk_emb)[0].tolist()):
        best[doc_id] = max(best.get(doc_id, -1.0), s)
    return sorted(best, key=best.get, reverse=True)

In training, continuation pairs were the two halves of a window of up to 1,200 characters (median 531 and at most 837 characters each), neighbour pairs were chunks of at most 500 characters, and title and headline positives were up to 2,000 characters. Hindko text averages about 2.7 characters per token with this tokenizer, so max_tokens=500 allows chunks of about 1,300 characters, longer than most training passages; chunk sizes were not evaluated. The benchmark itself does not chunk: it cuts each document at 3,000 characters and lets the model truncate at 512 tokens.

Input normalization

Training and benchmark text was put in one canonical form: Unicode NFC, invisible and control characters removed, spaces collapsed, Arabic presentation forms mapped to letters, and Arabic-keyboard YEH/KAF folded where the word is provably Urdu or Hindko spelling. The Hindko Tokenizer repository ships a self-contained normalize() that produces this form:

import os, sys
from huggingface_hub import hf_hub_download

qs = hf_hub_download("junaid008/hindko-tokenizer", "examples/quickstart.py")
sys.path.insert(0, os.path.dirname(qs))
from quickstart import normalize

raw_text = "ایہہ  بہت کہٹ لوک جانڑدین"   # any Hindko input
text = normalize(raw_text)            # here: the double space is collapsed

Apply it to queries and documents alike, before adding the prefix.

HindkoEmbedBench

Why a new benchmark

To our knowledge there was no benchmark for Hindko text embeddings, so one was built for this project (first used in round 1). HindkoEmbedBench v1.0 has five tasks in the style of MTEB (the Massive Text Embedding Benchmark), each with a dev version (used for model selection) and a test version (scored once per model and setting, after selection; the public models were scored at 256 and again at 512 tokens). It covers the main ways an embedding model is used: short-query retrieval (headlines and titles), passage-to-passage retrieval, classification and clustering.

How it was built

The benchmark was built automatically from corpus metadata: headlines and their articles, table-of-contents titles and their passages, the verse/prose label of book passages, and the book each passage comes from. No human relevance judgments were collected. The build is deterministic (seed 20260929; a rebuild is byte-identical), and every dropped query is logged with its reason.

• Records. Only Hindko-variety records are used; Urdu-variety and mixed records are excluded everywhere in the benchmark.
• Normalization. Every stored text is in the normalized form described above (normalization version 1.0.1); the harness applies it again before encoding, which changes nothing.
• Shortcut filters. A record's own title is stripped from its text; generic or repeated titles, titles that appear verbatim in the target, and near-duplicate documents are removed. Documents are cut at a sentence boundary at or below 3,000 characters.
• Frozen. Every task file is hashed, and the harness refuses to run if a file changes.

Splits and leak control

• Group-disjoint, leak-checked splits. Whole newspaper editions, books and web sites sit on one side of the split. The benchmark uses the same record-level splits as the Hindko Tokenizer, published as split_manifest.jsonl (sha256 76582d3a1e0afefe…).
• Classification and clustering follow the MTEB protocol (10 logistic-regression runs, 10 k-means runs), and the classifier's training pool comes from the train split, from books disjoint from the evaluation books.
• One known artifact remains. Book-title relevant passages rarely start with a short genre label line (a line reading only "ghazal" or "naat", for example), while about 31 % of test book documents (5 % on dev) do, so a model could gain by ranking such documents low. Our training data strips these lines from both positives and mined negatives, so training does not reward this cue.

Task 1: HeadlineRetrieval

• Input and target. A newspaper headline or a web page title is the query; its own article or page body is the one relevant document, in a corpus of all documents of the split.
• Corpus. All Hindko records of the split that are articles, book passages, web pages, transcripts or sentence sets; contents pages, front matter, captions, fragments and lexicon entries are excluded. Each record's own title is removed from the start of its text (for distractors too). Documents under 50 characters are dropped, and near-duplicates (5-character-shingle Jaccard of 0.5 or more on the first 1,500 characters) are removed: 14 on dev, 8 on test. The title was stripped from 268 dev and 95 test documents.
• Query filters. A headline is dropped if it has fewer than 4 words or 15 characters (web titles: 2 words or 10 characters), is less than 80 % Urdu-script letters, is shared by another record of the split or by more than one record anywhere in the corpus (generic and recurring headlines), still appears verbatim in the body, or has a body under 100 characters. Web page titles lose their site suffix.
• Dropped queries. 27 on dev (9 duplicated in the split, 5 generic, 6 whose document was a near-duplicate, 4 verbatim in the body, 3 too short) and 33 on test (8 verbatim, 7 generic, 6 near-duplicate documents, 5 too short, 3 duplicated, 3 with a short body, 1 not in Urdu script).
• Size. Dev: 85 queries (73 newspaper, 12 web) over 784 documents (279 newspaper, 469 book, 36 web). Test: 80 queries (72 newspaper, 8 web) over 517 documents (289 newspaper, 179 book, 49 web).
• Metric. nDCG@10 (main), plus Recall@1, Recall@10, MRR@10 and hit@1, and a breakdown by query source.

Task 2: BookTitleRetrieval

• Input and target. A table-of-contents title, often the first line of a poem, is the query; the passage or passages it heads are relevant. It shares the HeadlineRetrieval corpus.
• Chapters. A title carried by several contiguous passages of one book is one query (a chapter), and all of those passages are relevant. For poems whose title is the first line, the line is stripped, so the query is the first line and the target is the rest of the poem.
• Query filters. A title is dropped if it has fewer than 2 words or 8 characters, is held by more than 2 books in the corpus (generic titles such as "ghazal"), is carried by another book of the split, sits on non-contiguous passages, or appears verbatim in any relevant passage.
• Dropped queries. 47 on dev (25 verbatim in the passage, 20 too short, 2 near-duplicate documents) and 23 on test (13 too short, 7 verbatim, 2 on non-contiguous passages, 1 generic).
• Size. Dev: 262 queries from 6 books with 264 relevant passages (2 queries are multi-passage chapters). Test: 28 queries from 5 books with 57 relevant passages (19 queries are multi-passage chapters).
• Metric. nDCG@10 (main), plus Recall@1, Recall@10, MRR@10 and hit@1. When a query has several relevant passages, Recall@1 is below hit@1.

Task 3: PassageContinuation

• Input and target. The first half of a passage is the query; its second half is the one relevant document, among the second halves of all passages of the split.
• Construction. Each record's text (headline kept) is cut to at most 1,200 characters at a sentence or line boundary; the window must be at least 400 characters. It is split at the boundary closest to the middle, with both halves between 30 % and 70 % of the window and at least 150 characters each. Near-duplicates are removed on both halves.
• Size. Dev: 416 queries and 416 documents (203 newspaper, 187 book, 26 web; no near-duplicates removed). Test: 360 queries and 360 documents (193 newspaper, 127 book, 40 web; 1 near-duplicate removed from 361 candidates).
• Metric. nDCG@10 (main), plus Recall@1, Recall@10, MRR@10 and a breakdown by source.

Task 4: VerseProseClassification

• Input and target. A book passage (at least 100 characters, cut to 1,500) is labelled verse or prose, using the book pipeline's per-passage form field; passages marked mixed, list or lexicon are not used.
• Training pool (train split). At most 30 passages per book and class, then 800 per class: 800 verse passages from 55 books and 800 prose passages from 63 books. Train records whose split group also holds dev or test records are excluded, so training and evaluation books are disjoint. The pool is the same for dev and test.
• Evaluation set. At most 40 passages per book and class. Dev: 297 passages (166 verse from 7 books, 131 prose from 9 books). Test: 157 passages (42 verse from 3 books, 40 of them from one book; 115 prose from 5 books).
• Protocol. 10 experiments; each draws 256 examples per label from the pool, fits LogisticRegression(max_iter=1000) on the frozen embeddings and scores the evaluation set.
• Metric. Mean macro-F1 over the 10 runs (main; the evaluation set is imbalanced), with its standard deviation, plus mean accuracy.

Task 5: BookClustering

• Input and target. Book passages (at least 150 characters, cut to 1,500) from books with at least 15 such passages in the split, at most 60 per book; the label is the book. Book identity is a proxy for author, topic and style.
• Size. Dev: 355 passages from 8 books (5 poetry, 1 film script, 1 criticism and reviews, 1 unlabelled; 212 verse, 140 prose and 3 list passages). Test: 176 passages from 5 books (2 short-story collections, 1 essays, 1 poetry, 1 research and criticism; 115 prose, 60 verse and 1 list passage).
• Protocol. k = number of books; 10 k-means runs (one initialization each, seeds 0 to 9).
• Metric. Mean V-measure over the 10 runs (main), with its standard deviation, plus the adjusted Rand index (ARI).

Tasks considered and dropped

• Genre classification. Across dev and test only poetry has held-out books in both splits (5 and 1); every other genre has at most 2 held-out books in one split and none in the other.
• Newspaper section clustering. The section labels are page positions (inner page, back page, editorial), not topics, and are missing for 67 % of newspaper records.

Scoring and harness

• Overall score = unweighted mean of the five main metrics.
• Encoding. sentence-transformers with normalize_embeddings=True and cosine (dot-product) scoring. Each text is encoded once per run: the headline and book-title tasks share their corpus, and the clustering passages overlap with the classification texts.
• Lengths. This model, the round-2 fine-tunes and the public models were run at max_seq_length 512 in float16; the two round-1 models at 256 tokens in float32 (their round-1 setting). The public models were also run at 256 tokens in float32 in round 1 (see Context length).
• Lexical reference. A character 2-4-gram TF-IDF model goes through the same metric code.
• Prefixes. Each model is run with its own recommended input format:

Model family	Query	Document	Classification and clustering
e5 (multilingual-e5-small, -base, -large and every e5 fine-tune, including this model)	query:	passage:	query:
BAAI/bge-m3 and the Round-2 bge-m3 fine-tune, LaBSE, paraphrase-multilingual-mpnet-base-v2	none	none	none
Qwen/Qwen3-Embedding-0.6B	Instruct: Given a Hindko query, retrieve relevant Hindko passages\nQuery:	none	task instructions: "Classify a given Hindko passage as verse or prose"; "Identify the book that a given Hindko passage comes from"

Metrics in brief

Metric	Meaning
nDCG@10	Ranking quality of the top 10, with more credit for relevant documents ranked higher; 1 is perfect
Recall@k	Share of a query's relevant documents found in the top k, averaged over queries
MRR@10	Mean of 1 / rank of the first relevant document (0 if it is not in the top 10)
hit@1	Share of queries whose first result is relevant
macro-F1	F1 averaged over the two classes, so the smaller class counts equally
V-measure	Agreement between clusters and books (harmonic mean of homogeneity and completeness); 1 is perfect
ARI	Adjusted Rand index: pairwise agreement between clusters and books, corrected for chance

Benchmark sizes

Chart 20. How big each benchmark task is. Query, document and item counts for the dev split (model selection) and the test split (scored once), on a log scale. Most tasks have a few hundred items per split, but the test book side is small: 28 book-title queries with 57 relevant passages, and 5 books in the clustering task (against 262 queries, 264 relevant passages and 8 books on dev). The headline task has 85 dev and 80 test queries, and the classifier's training pool of 1,600 passages is shared by both splits. These sizes are why dev and test are both reported and why differences of a few points on the small tasks are within noise.

Release status

The benchmark data contains newspaper and book text and is not released yet. This card describes it; the benchmark code will be released with the training code.

Results

Model names used in this card

Name in this card	Run ID in eval/ files	Base model	Tokens	Role
Hindko Embeddings (this model)	r2-e5l	intfloat/multilingual-e5-large	512	Round 2, selected and released
Round-2 e5-base	r2-e5b	intfloat/multilingual-e5-base	512	Round 2, runner-up on dev; also scored on test
Round-2 e5-base, no hard negatives	r2-e5b-plain	intfloat/multilingual-e5-base	512	Round 2 ablation (no mined negatives, no same-book blocks); dev only
Round-2 e5-small	r2-e5s	intfloat/multilingual-e5-small	512	Round 2 candidate; dev only
Round-2 bge-m3	r2-bgem3	BAAI/bge-m3	512	Round 2 candidate, trained and run without prefixes; dev only
Round-1 e5-base fine-tune (the round-1 model)	r1-ft-e5-base (ft-e5-base in selection.json)	intfloat/multilingual-e5-base	256	Selected in round 1
Round-1 e5-small fine-tune	r1-ft-e5-small (ft-e5-small)	intfloat/multilingual-e5-small	256	Round 1 candidate
TF-IDF char 2-4-gram	tfidf-char2-4	none	n/a	Lexical reference

Public models are listed by their Hugging Face ids and were run untuned. Only this model is released in this repository.

Test split (scored once per model, after selection)

Model	Tokens	Overall	Headline nDCG@10	BookTitle nDCG@10	Continuation nDCG@10	Verse/prose macro-F1	Clustering V-measure
Hindko Embeddings (this model)	512	0.7391	0.6372	0.3460	0.8756	0.9801	0.8566
Round-1 e5-small fine-tune (ours)	256	0.7197	0.6091	0.3664	0.8314	0.9736	0.8177
Round-1 model, e5-base fine-tune (ours)	256	0.6923	0.6097	0.2740	0.8340	0.9827	0.7612
Round-2 e5-base fine-tune (ours, runner-up on dev)	512	0.6919	0.6027	0.2422	0.8350	0.9818	0.7979
TF-IDF char 2-4-gram (lexical reference)	n/a	0.6864	0.6173	0.3756	0.7899	0.9597	0.6895
BAAI/bge-m3	512	0.6332	0.5960	0.3778	0.7174	0.9578	0.5168
intfloat/multilingual-e5-large (base model)	512	0.5977	0.5886	0.3122	0.6706	0.9413	0.4758
intfloat/multilingual-e5-small	512	0.5619	0.5381	0.2934	0.5664	0.9197	0.4918
Qwen/Qwen3-Embedding-0.6B	512	0.5576	0.5634	0.2784	0.6186	0.9051	0.4227
intfloat/multilingual-e5-base	512	0.5532	0.5626	0.2776	0.6139	0.8540	0.4580
sentence-transformers/LaBSE	512	0.5314	0.5064	0.1608	0.5383	0.9616	0.4899
sentence-transformers/paraphrase-multilingual-mpnet-base-v2	512	0.4428	0.3399	0.0815	0.5030	0.9018	0.3879

Best value per column in bold. The public models are untuned. They were first run at 256 tokens (float32) in round 1 and re-run at 512 tokens (float16) for this comparison, so length and precision changed together. On test this raised six of the seven (bge-m3: 0.6185 to 0.6332; multilingual-e5-large: 0.5605 to 0.5977) and left paraphrase-multilingual-mpnet unchanged (0.4429 to 0.4428), so the 512-token numbers are the stronger comparison overall. On dev, LaBSE (0.5200 to 0.5139) and multilingual-e5-small (0.5371 to 0.5353) fell slightly.

Of our own models, only this model and the round-2 runner-up were scored on test in round 2; the two round-1 models were scored on test in round 1. The round-1 e5-small model scores second on test (0.7197) but was not chosen in round 1, because its dev score was lower.

Chart 01. Overall test score of every model. The mean of the five task scores for all 12 models scored on test. This model leads with 0.7391, followed by our round-1 e5-small fine-tune (0.7197), the round-1 model (0.6923), the round-2 runner-up (0.6919) and TF-IDF (0.6864). Every fine-tuned model and the lexical TF-IDF reference score above every untuned public model; the best of those, bge-m3, reaches 0.6332, and the weakest, paraphrase-multilingual-mpnet-base-v2, 0.4428.

Chart 02. Per-task test scores against TF-IDF and bge-m3. This model is ahead of both references on four tasks: headline (0.6372 against 0.6173 and 0.5960), continuation (0.8756 against 0.7899 and 0.7174), verse/prose (0.9801 against 0.9597 and 0.9578) and clustering (0.8566 against 0.6895 and 0.5168). On book-title retrieval it is behind both (0.3460 against 0.3756 and 0.3778), on only 28 test queries.

Dev split (used for model selection)

Model	Tokens	Overall	Headline	BookTitle	Continuation	Verse/prose	Clustering
Hindko Embeddings (this model)	512	0.7117	0.6357	0.4257	0.8308	0.9406	0.7258
Round-2 e5-base fine-tune	512	0.7079	0.6006	0.4464	0.8176	0.9411	0.7339
Round-2 e5-base, no hard negatives	512	0.7066	0.5941	0.4408	0.8295	0.9453	0.7234
Round-2 e5-small fine-tune	512	0.7039	0.6191	0.4358	0.8094	0.9480	0.7070
Round-2 bge-m3 fine-tune	512	0.7008	0.6459	0.4435	0.8246	0.9429	0.6470
Round-1 model, e5-base fine-tune	256	0.6851	0.6299	0.3288	0.8081	0.9432	0.7157
Round-1 e5-small fine-tune	256	0.6700	0.6193	0.2778	0.7943	0.9382	0.7204
TF-IDF char 2-4-gram	n/a	0.6667	0.6049	0.4594	0.7655	0.9463	0.5573
BAAI/bge-m3	512	0.5923	0.6183	0.2443	0.6617	0.9363	0.5011
intfloat/multilingual-e5-large	512	0.5872	0.6028	0.2004	0.6524	0.9425	0.5379
intfloat/multilingual-e5-base	512	0.5737	0.5728	0.1827	0.6293	0.9381	0.5457
Qwen/Qwen3-Embedding-0.6B	512	0.5449	0.5327	0.1735	0.6297	0.8985	0.4902
intfloat/multilingual-e5-small	512	0.5353	0.5467	0.1186	0.5698	0.9311	0.5101
sentence-transformers/LaBSE	512	0.5139	0.4165	0.2089	0.5036	0.9455	0.4948
sentence-transformers/paraphrase-multilingual-mpnet-base-v2	512	0.4280	0.2822	0.1138	0.4360	0.8869	0.4210

Same metrics as the test table. On dev this model is ahead of TF-IDF on three of five tasks (headline, continuation, clustering) and behind it on book titles (0.4257 vs 0.4594) and, narrowly, on verse/prose (0.9406 vs 0.9463).

Every model on every task

Chart 07. Every model on every task, test split. Each cell is shaded from the lowest to the highest value in its column, and the best value per column is outlined. This model is best on overall (0.7391), headline (0.6372), continuation (0.8756) and clustering (0.8566). The two exceptions are book title, where bge-m3 is best (0.3778), and verse/prose, where the round-1 e5-base fine-tune is best (0.9827). The verse/prose column is compressed: 11 of the 12 models score between 0.9018 and 0.9827.

Chart 08. Every model on every task, dev split. All 15 models scored on dev, including the three round-2 candidates that were never scored on test. This model is best on overall (0.7117) and continuation (0.8308). The other columns have different winners: Round-2 bge-m3 on headline (0.6459), TF-IDF on book title (0.4594), Round-2 e5-small on verse/prose (0.9480) and Round-2 e5-base on clustering (0.7339). On dev this model ranks 2nd on headline and clustering, 6th on book title and 9th on verse/prose.

Task profile against the references

Chart 09. Task profile against the references (test). One lane per task, with this model, TF-IDF, bge-m3 and the untuned multilingual-e5-large; the right column gives this model's lead over the best of the other three. The lead is largest on clustering (+0.1671 over TF-IDF) and continuation (+0.0857 over TF-IDF), smaller on verse/prose (+0.0204) and headline (+0.0199), and +0.0527 overall. On book title the model is behind, by 0.0318 against bge-m3.

Task	This model	Best of TF-IDF, bge-m3 and untuned e5-large	Lead
Overall	0.7391	TF-IDF 0.6864	+0.0527
Headline	0.6372	TF-IDF 0.6173	+0.0199
Book title	0.3460	bge-m3 0.3778	-0.0318
Continuation	0.8756	TF-IDF 0.7899	+0.0857
Verse/prose	0.9801	TF-IDF 0.9597	+0.0204
Clustering	0.8566	TF-IDF 0.6895	+0.1671

Rank summary

Chart 24. Rank of every model on every task (test, 12 models). Lines connect each model's rank across the columns. This model ranks 1st on overall, headline, continuation and clustering, 3rd on verse/prose and 4th on book title. Ranks move a lot between tasks: bge-m3 is 1st on book title but 6th overall, and the round-2 runner-up is 2nd on continuation but 10th on book title.

Split	Overall	Headline	Book title	Continuation	Verse/prose	Clustering
Test rank of this model (of 12)	1	1	4	1	3	1
Dev rank of this model (of 15)	1	2	6	1	9	2

Retrieval details (test)

Model	Headline R@1	Headline R@10	Headline nDCG@10, newspaper (72 q)	Headline nDCG@10, web (8 q)	BookTitle R@1	BookTitle R@10	BookTitle MRR@10
This model	0.5625	0.7125	0.6072	0.9077	0.1548	0.4494	0.3608
TF-IDF char 2-4-gram	0.5500	0.6875	0.5879	0.8827	0.1875	0.5030	0.3579
BAAI/bge-m3 (512)	0.5500	0.6500	0.5581	0.9375	0.2262	0.4286	0.4113
Round-1 model (256)	0.5375	0.6750	0.5715	0.9539	0.1012	0.4315	0.2731

R@k is recall at k; a book-title query can have several relevant passages (57 for 28 queries).

Detailed analysis

Headline retrieval

Chart 12. Headline retrieval by query source. nDCG@10 for newspaper headlines and web page titles, test (left) and dev (right), for five key models. On test, the 72 newspaper headlines carry the score: this model reaches 0.6072, ahead of TF-IDF (0.5879), the round-1 model (0.5715), Round-2 e5-base (0.5706) and bge-m3 (0.5581). On the 8 web titles it scores 0.9077, below the round-1 model (0.9539) and bge-m3 (0.9375); with 8 queries one query can move the score by up to 0.125. On dev the newspaper score (0.5963) is just below the round-1 model's 0.6024 and the web score (0.8750, 12 queries) below bge-m3's 0.8859; Round-2 bge-m3, not shown, has the best dev newspaper score (0.6064).

Of the four tasks where the model beats TF-IDF, headline retrieval has the smallest lead: +0.0199 on test. The first result is the right article for 45 of the 80 test queries (Recall@1 0.5625) and the article is in the top 10 for 57 (Recall@10 0.7125); TF-IDF has 44 and 55, bge-m3 44 and 52. A few headlines in the corpus are attached to the wrong article, which lowers every model's score.

Book-title retrieval: the weak spot

Chart 06. Book-title retrieval in detail. Recall@1, Recall@10, MRR@10 and nDCG@10 on dev (262 queries) and test (28 queries, 57 relevant passages), for this model, TF-IDF and bge-m3. On dev TF-IDF is ahead on every metric: nDCG@10 0.4594 against 0.4257, Recall@1 0.3282 against 0.2977, Recall@10 0.6069 against 0.5630 and MRR@10 0.4132 against 0.3826. On test TF-IDF and bge-m3 are ahead on nDCG@10 (0.3756 and 0.3778 against 0.3460) and Recall@1 (0.1875 and 0.2262 against 0.1548); this model is narrowly ahead of TF-IDF on MRR@10 (0.3608 against 0.3579) and ahead of bge-m3 on Recall@10 (0.4494 against 0.4286).

What else the data shows:

• It is the task that improved most from round 1 on dev: +0.0969 (0.3288 to 0.4257). On test it gained +0.0720 (0.2740 to 0.3460), second only to clustering (+0.0954). Against the untuned base model the dev gain is +0.2253 (0.2004 to 0.4257), the test gain +0.0338 (0.3122 to 0.3460).
• Top-1 hits. The first result is a relevant passage for 7 of the 28 test queries (hit@1 0.2500), the same as TF-IDF; bge-m3 and the untuned e5-large get 9 (0.3214).
• Dev and test are built differently. On test, 19 of the 28 queries are chapters with several relevant passages; on dev only 2 of 262 are. Recall@1 cannot reach 1 for a multi-passage query, which is one reason test Recall@1 is far below dev.
• Long targets. In the round-2 plan, 89.5 % of the relevant test passages were longer than 512 tokens, so the model sees only their beginning (see Truncation).
• Among the round-2 candidates, this model had the lowest dev book-title score (0.4257, against 0.4358 to 0.4464). It was selected on the overall score.
• Residual overlap and the genre-label artifact both touch this task; see Limitations.

For title or first-line lookup, combine the model with a lexical search (hybrid retrieval).

Passage continuation

Chart 13. Passage continuation by source (test). nDCG@10 for book, newspaper and web passages and for all 360 queries. This model leads on book passages (0.8189, 127 queries; next best TF-IDF 0.7804) and newspaper passages (0.9048, 193 queries; next best among the models shown the round-1 model, 0.8651). On the 40 web queries Round-2 e5-base (0.9437) and the round-1 model (0.9364) score higher than this model (0.9145). Over all 360 queries it scores 0.8756, against 0.8350 for Round-2 e5-base and 0.6706 for the untuned base model.

On dev (416 queries) this model is again first overall (0.8308, just ahead of Round-2 e5-base, no hard negatives, at 0.8295). By source it leads on newspaper passages (0.8802, 203 queries), is second on book passages (0.7721 against 0.7771 for the no-hard-negatives run, 187 queries) and is fourth on the 26 web passages (0.8681; the round-1 model has 0.8910). Continuation is also the retrieval task where fine-tuning helped most: +0.2050 nDCG@10 over the untuned base model on test (only clustering gained more, +0.3808 V-measure).

Retrieval depth

Chart 14. Retrieval at different depths (test). Recall@1, MRR@10, nDCG@10 and Recall@10 for the three retrieval tasks, for this model, TF-IDF and bge-m3. On continuation, the right second half is first for 272 of 360 queries (Recall@1 0.7556; TF-IDF 0.6472, bge-m3 0.5583) and in the top 10 for 356 (Recall@10 0.9889; TF-IDF 0.9194, bge-m3 0.8861). On headlines this model is ahead at every depth (Recall@1 0.5625, MRR@10 0.6136, Recall@10 0.7125). On book titles it is last of the three on Recall@1 (0.1548 against 0.1875 and 0.2262) and nDCG@10, and below TF-IDF on Recall@10 (0.4494 against 0.5030).

Classification and clustering

Chart 15. Classification and clustering in detail (test). Main and second metric with plus or minus one standard deviation over the 10 runs. Verse/prose: this model reaches 0.9801 ± 0.0041 macro-F1 and 0.9847 ± 0.0031 accuracy, within one standard deviation of the round-1 model (0.9827 ± 0.0024) and Round-2 e5-base (0.9818 ± 0.0063). Book clustering: 0.8566 ± 0.0614 V-measure and 0.8176 ARI, against 0.6895 ± 0.0853 and 0.5701 for TF-IDF and 0.5168 ± 0.0307 and 0.4557 for bge-m3.

Notes on these two tasks:

• Verse/prose is near the ceiling and compressed. On test, 11 of 12 models score between 0.9018 and 0.9827. On dev, 13 of the 15 models score between 0.9311 and 0.9480, and this model is 9th with 0.9406, below the untuned e5-large (0.9425) and TF-IDF (0.9463). The test verse class is 42 passages from 3 books, 40 of them from one book.
• Clustering separates the models most. It is the task with the largest gain over the untuned base model (+0.3808 V-measure on test, from 0.4758) and over round 1 (+0.0954, from 0.7612). On dev it scores 0.7258 ± 0.0375 (ARI 0.5994), 2nd behind Round-2 e5-base (0.7339). The test task has only 5 books, so its standard deviation over k-means runs is large (0.0614).
• Recorded spreads. TF-IDF's classification score has a standard deviation of 0 on test but 0.0024 on dev, and Round-2 e5-small's dev standard deviation is also 0; ARI has no spread recorded.

Dev versus test

Chart 10. Dev score against test score. Overall dev score (x) against overall test score (y) for the 12 models scored on both; points above the dashed line score higher on test. 10 of the 12 do; the exceptions are Round-2 e5-base (0.7079 to 0.6919, -0.0160) and the untuned multilingual-e5-base (0.5737 to 0.5532, -0.0205). This model moves from 0.7117 to 0.7391 (+0.0274), the round-1 e5-small fine-tune from 0.6700 to 0.7197 (+0.0497) and TF-IDF from 0.6667 to 0.6864 (+0.0197).

The test split is not simply easier: the shift comes from individual tasks. For this model, verse/prose rises from 0.9406 to 0.9801 and clustering from 0.7258 (8 books) to 0.8566 (5 books), while book title falls from 0.4257 to 0.3460. The two e5 fine-tunes that were 0.0038 apart on dev are 0.0472 apart on test (0.7391 and 0.6919), and the round-1 e5-small model, 7th of 15 on dev, is 2nd of 12 on test. Small dev differences between fine-tunes are therefore not reliable predictions of test order; see Model selection protocol.

Context length: 256 versus 512 tokens

Chart 11. Public models at 256 and at 512 tokens. Overall score of the seven untuned public models in round 1 (256 tokens, float32) and in the round-2 re-run (512 tokens, float16), test (left) and dev (right). On test 6 of 7 rise, most for multilingual-e5-large (0.5605 to 0.5977, +0.0372) and Qwen3-Embedding-0.6B (0.5275 to 0.5576, +0.0301); paraphrase-multilingual-mpnet-base-v2 is flat (0.4429 to 0.4428). On dev 5 of 7 rise; multilingual-e5-small (-0.0018) and LaBSE (-0.0061) fall. Length and precision changed together, so their effects cannot be separated.

Model	Test 256	Test 512	Change	Dev 256	Dev 512	Change
BAAI/bge-m3	0.6185	0.6332	+0.0147	0.5836	0.5923	+0.0087
intfloat/multilingual-e5-large	0.5605	0.5977	+0.0372	0.5821	0.5872	+0.0051
intfloat/multilingual-e5-small	0.5388	0.5619	+0.0231	0.5371	0.5353	-0.0018
Qwen/Qwen3-Embedding-0.6B	0.5275	0.5576	+0.0301	0.5126	0.5449	+0.0323
intfloat/multilingual-e5-base	0.5465	0.5532	+0.0067	0.5720	0.5737	+0.0017
sentence-transformers/LaBSE	0.5292	0.5314	+0.0022	0.5200	0.5139	-0.0061
sentence-transformers/paraphrase-multilingual-mpnet-base-v2	0.4429	0.4428	-0.0001	0.4116	0.4280	+0.0164

Length explains only a small part of this model's lead: the untuned base model at 512 tokens scores 0.5977 on test, 0.1414 below this model at the same length.

Truncation

Chart 23. Why the length went from 256 to 512 tokens. The share of texts longer than the model's limit, for the round-1 model at 256 tokens and this model at 512 (same XLM-RoBERTa tokenizer and prefixes, so only max_seq_length differs). Over all texts of a run, 32.9 % (1,309 of 3,977) are cut at 256 tokens on dev against 18.1 % (719) at 512, and 40.1 % (1,252 of 3,122) against 24.1 % (753) on test. At 512 no continuation text is cut (at 256: 43 of 832 on dev, 31 of 720 on test), but verse/prose passages are still cut 29.9 % of the time on dev (566 of 1,896) and 34.2 % on test (601 of 1,757).

How to read the counts: each text is encoded once per run and counted under the first task that needs it. The book-title count therefore covers the queries only (none is cut), the shared retrieval corpus sits under headline (that row is 16.6 % cut at 512 on dev and 25.5 % on test; the headline queries are short and none is cut, so 18.4 % of the 784 dev documents and 29.4 % of the 517 test documents are cut), and clustering counts only passages not already encoded for classification. Retrieval mostly sees the start of each document: benchmark documents are cut at 3,000 characters, and the model reads at most 512 tokens.

Model size versus score

Chart 05. Fine-tuning matters more than size (dev). Dev overall score against parameter count, for the round-2 fine-tunes and the untuned public models, with TF-IDF (0.6667) as a dashed line. Every fine-tuned model scores above every untuned one: the lowest round-2 fine-tune (Round-2 bge-m3, 0.7008) is well above the best untuned model (bge-m3, 0.5923). Fine-tuning added 0.1686 to e5-small (118M parameters), 0.1342 to e5-base (278M), 0.1245 to e5-large (560M) and 0.1085 to bge-m3 (568M).

After fine-tuning the four round-2 models lie within 0.0109 of each other on dev (0.7008 to 0.7117), against a 0.0570 spread for the same four models untuned (0.5353 to 0.5923). Among the fine-tuned e5 models, the larger ones score slightly higher (0.7039, 0.7079, 0.7117), but these steps are smaller than the dev standard error of about 0.01.

From base model to this model

Chart 22. From the base model to this model (test). The untuned multilingual-e5-large at 512 tokens, the round-1 model (e5-base, 256 tokens) and this model. The round-1 model has a different base, so the middle bar is a reference, not an ancestor. Against the untuned base model this model gains +0.1414 overall, with the largest gains on clustering (+0.3808) and continuation (+0.2050) and the smallest on book title (+0.0338). Against the round-1 model it gains +0.0468 overall and is lower only on verse/prose (-0.0026).

Task	Untuned e5-large	Round-1 model	This model	vs untuned	vs round 1
Overall	0.5977	0.6923	0.7391	+0.1414	+0.0468
Headline	0.5886	0.6097	0.6372	+0.0486	+0.0275
Book title	0.3122	0.2740	0.3460	+0.0338	+0.0720
Continuation	0.6706	0.8340	0.8756	+0.2050	+0.0416
Verse/prose	0.9413	0.9827	0.9801	+0.0388	-0.0026
Clustering	0.4758	0.7612	0.8566	+0.3808	+0.0954

Training

Data

Pairs were built only from the train split of a Hindko corpus: issues of the Weekly Hindkowan (Peshawar), the Gandhara Hindko Academy book series (much of it poetry) and Hindko web pages. Each pair is an anchor and a positive; some rows also carry mined hard negatives.

Dataset	Pair (anchor → positive)	Rows	Mined negatives per row
title	table-of-contents title → its passage (1,886); first line of a poem → the rest of the poem (2,505)	4,391	3 (2 from the same book, 1 from another book)
verse_ict	one line of a poem → the poem with that line removed (the line is kept in about 10 % of rows)	2,949	1 (same book)
prose_q	headline or web page title → article body (1,585: 1,447 news, 138 web); one sentence → the surrounding text (up to 900 characters) with that sentence removed, though it is kept in about 10 % of rows (5,354)	6,939	1
continuation	first half of a passage of up to 1,200 characters → second half	12,904	0
neighbour	a chunk of about 500 characters → the next chunk of the same text	14,350	0
Total	87,176 distinct texts	41,533	23,061

• Sources: 24,745 rows from books, 14,073 from newspapers and 2,715 from web pages.
• Variety: 39,517 rows come from Hindko-variety records. The title, verse and prose-sentence datasets also use 1,268 Urdu-variety and 748 mixed-variety rows from the same corpus (title 641 Urdu and 361 mixed, verse 227 and 319, prose sentences 400 and 68). Continuation, neighbour and headline rows are Hindko only.
• Hard negatives: 23,061 in all (3 for each of the 4,391 title rows, 1 for each verse and prose row).

Chart 19. What the model was trained on. The 41,533 training rows by pair type (left) and by dataset, source and record variety (right); under each bar, rows by source and by variety. Neighbour (14,350, 34.6 %) and continuation (12,904, 31.1 %) pairs make up two thirds of the rows; the short-query datasets are prose_q (6,939), title (4,391) and verse_ict (2,949). Books supply 24,745 rows (59.6 %), newspapers 14,073 (33.9 %) and web pages 2,715 (6.5 %); 95.1 % of rows come from Hindko-variety records.

Pair types in detail

Pair type	Rows	Sources	How it was built
title_any	1,886	book 1,886	A table-of-contents title of a train book and the passages of its chapter (first contiguous run). Titles must have 2 or more words and 8 or more characters, be at least 80 % Urdu-script letters and be held by at most 2 books. The title is stripped from the passage; on the first passage, up to 2 short label lines are dropped too. Rows whose title still appears verbatim in the passage are dropped. Passages are cut at a boundary at or below 2,000 characters and must keep at least 100.
verse_first	2,505	book 2,505	The first real line of a regular poem (3 or more lines, median 4 to 16 words per line, at least 60 % of lines near the median) and the rest of the poem. Up to 3 leading lines that look like a poet's name, a label or a heading are skipped. The anchor has 3 to 20 words; the positive has 2 or more lines and 100 or more characters, cut at 1,500. Refrains (the anchor repeated in the poem) are dropped. At most 200 rows per book.
verse_line	2,949	book 2,949	Inverse cloze: one line (3 to 20 words) of a poem with 6 or more content lines, and the poem with that line removed; the line was kept in 10.4 % of rows. At most 150 rows per book.
headline	1,585	newspaper 1,447, web 138	Round-1 headline pairs (headline or web page title and the title-stripped body), minus rows whose anchor appears verbatim in the body, rows caught by the new leak guards and duplicates of other datasets' anchors.
prose_sent	5,354	newspaper 3,306, book 1,500, web 548	Inverse cloze on prose: a window of 400 to 900 characters with 3 or more sentences; one sentence of 6 to 30 words is removed (kept in 9.6 % of rows), and the rest (250 or more characters) is the positive. At most 40 rows per split group.
continuation	12,904	book 7,063, newspaper 4,888, web 953	From round 1: consecutive windows of up to 1,200 characters of a record, each split at the sentence boundary nearest the middle.
neighbour	14,350	book 8,842, newspaper 4,432, web 1,076	From round 1: adjacent, non-overlapping chunks of about 500 characters packed from whole sentences, within an article or page or along a book's passages.

Deduplication removed exact duplicate pairs, repeated anchors within a dataset, and anchors repeated across datasets (keeping the first in the order title, verse, prose).

Leak prevention

• Only train-split records were used, and none from a split group that has dev or test records (230 such train records were excluded).
• One train book (204 records) was excluded because it is a re-edition of a dev book.
• Any text with word 8-gram containment of 0.3 or more in any dev or test text was dropped, under two text normalizations. Anchors equal to or close to a dev or test query (character 3-gram Jaccard 0.5 or more), and anchors of 12 or more characters contained in any dev or test text, were dropped as well.
• These guards removed 262 candidate rows: 92 for 8-gram containment, 94 for anchors found inside dev or test text, 70 for anchors close to a dev or test query and 6 for anchors equal to one.
• A separately implemented audit script (it re-implements every check and does not import or read the pair builder; the two share only small generic helpers for text normalization, title stripping and word and character n-grams) re-checked every row against the split manifest and the frozen benchmark files: 0 violations in all six checks. It traced all 41,533 anchors, 41,533 positives and 23,061 negatives to 12,867 train records, and compared all 87,176 texts with the 3,626 distinct dev and test texts and their 1,231 queries. The highest remaining 8-gram containment is 0.2973, and the closest anchor to any dev or test query has a 3-gram Jaccard of 0.4948.
• The tokenizer is the base model's, unchanged; no vocabulary was added.

Hard-negative mining

• Index. A character 2-4-gram TF-IDF index (char_wb, sublinear term frequency, minimum document frequency 2) was fit only on training documents: 13,519 train records outside every dev and test split group, each cut at a boundary at or below 1,600 characters. Its vocabulary has 121,791 n-grams. No neural model was used for mining, so mining does not depend on any dev-selected model.
• Per dataset. Title rows get 2 negatives sampled from the 10 most similar passages of the same book and 1 from the 10 most similar passages of other books. Verse rows get 1 negative from the same book. Prose rows get 1 negative: from newspaper and web documents for news and web anchors, from prose passages of the same book for book anchors; a candidate scoring 0.95 or more of the positive's own score is skipped as a likely reprint.
• What a negative may not be. The same record, a member of the same near-duplicate cluster, a passage of the same chapter, the adjacent passage of the same book (unless it has its own title), a text containing the anchor (also in variant spellings), or a text that reuses a large part of the positive (5-character-shingle Jaccard of 0.3 or more, or 30 % of its lines). The three checks for hidden false negatives (anchor inside the candidate, a variant of the anchor line, reused positive lines) skipped 622 candidates.
• Near-duplicate clusters. Records with the same opening or with near-identical text were grouped: 238 clusters hold 674 records (the largest has 31).
• Genre label lines. Short label lines at the start of a passage (up to 2 lines of at most 2 words) were stripped from the mining documents (1,674 train records had such lines) and from title positives, so neither side of a row carries the cue described under Splits and leak control.

Batching

• One dataset per batch. Every batch holds 256 rows from one dataset. Each epoch has 161 batches (17 title, 11 verse, 27 prose, 50 continuation and 56 neighbour), in shuffled order.
• Same-book blocks. Title and verse rows are drawn in blocks of 32 rows from the same book, so many in-batch negatives are other poems and chapters of the same book. The round-2 plan noted that round-1 batches gave a title anchor only 0.62 passages from its own book on average.
• Conflict rule. A row may join a batch only if its own record, near-duplicate cluster, chapter and texts are not already used by a positive or negative in the batch, and its negatives are not another row's positive. This keeps duplicate texts, two rows from one record or duplicate cluster, two passages of one chapter, and a row whose mined negative is another row's positive out of the same batch. Rows that conflict wait for the next batch.

Loss and hyperparameters

• Loss: CachedMultipleNegativesRankingLoss with scale 20, i.e. temperature 0.05. Every other positive and negative in the batch also serves as a negative: an anchor in a title batch must pick its own passage out of the batch's 256 positives and 768 mined negatives.
• Gradient cache. The loss splits each batch of 256 into mini-batches (32 for this model) for the forward and backward passes, so large batches fit in memory without changing the loss.

Setting	Value
Base model	intfloat/multilingual-e5-large
Epochs, optimizer steps	2 epochs, 322 steps (161 per epoch)
Batch size	256 rows, one dataset per batch
Gradient-cache mini-batch	32
Learning rate	2e-5, linear warmup over 32 steps (10 %), then linear decay
Optimizer	AdamW (fused), weight decay 0.01
Precision	bf16 mixed precision, TF32 matmuls; saved in float16
max_seq_length	512
Prefixes	query: anchors, passage: positives and negatives; query: on both sides of neighbour pairs
Same-book block	32 rows (title and verse datasets)
Seed	20261001
Gradient checkpointing	off
Hardware	one NVIDIA A100-SXM4-80GB

Output. Saved in float16. The dev and test scores of this model were measured from the float16 weights that are published here.

All five round-2 runs used the same pairs, 2 epochs, batch 256, 512 tokens, warmup 10 % and seed 20261001:

Run	Base model	Params	Embedding dim	LR	Mini-batch	Mined negatives	Book blocks	Prefixes	Dev overall
Hindko Embeddings (this model)	multilingual-e5-large	560M	1,024	2e-5	32	yes	32	e5	0.7117
Round-2 e5-base	multilingual-e5-base	278M	768	3e-5	64	yes	32	e5	0.7079
Round-2 e5-base, no hard negatives	multilingual-e5-base	278M	768	3e-5	64	no	none	e5	0.7066
Round-2 e5-small	multilingual-e5-small	118M	384	5e-5	128	yes	32	e5	0.7039
Round-2 bge-m3	BAAI/bge-m3	568M	1,024	1.5e-5	32	yes	32	none	0.7008

Parameter counts are those of the base models; fine-tuning does not change them.

Training curves

Chart 17. Training loss of the five round-2 runs. Loss at every optimizer step (faint) and its 15-step moving average, with the warm-up and the start of epoch 2 (step 161) marked. This model's loss falls from 5.52 at the first step to a mean of 1.353 over the last 15 steps (mean over all steps 1.864; epoch 1 mean 2.293, epoch 2 mean 1.435). Because every step is one dataset, the raw loss jumps with the dataset. The run without mined negatives ends lowest (0.994 over the last 15 steps), but its loss is not comparable: its batch order differs (13 of its last 15 steps are continuation or neighbour batches, against 10 in the other runs), and its title, verse and prose anchors face fewer and easier candidates (no mined negatives, no same-book blocks). Its dev score (0.7066) is close to, and slightly below, that of the same base model trained with them (Round-2 e5-base, 0.7079).

Run	Mean loss, last 15 steps	Mean loss, all steps
Hindko Embeddings (this model)	1.353	1.864
Round-2 e5-base	1.543	2.067
Round-2 e5-small	1.561	1.992
Round-2 bge-m3	1.467	1.796
Round-2 e5-base, no hard negatives	0.994	1.774

The curves are not fully comparable across runs: the no-hard-negatives run has a different batch order and fewer candidates per anchor, and Round-2 bge-m3 was trained without prefixes.

Chart 18. This model's loss by training dataset. Left: the loss of each step, split by the dataset the batch came from, with a 5-point moving average. Right: the mean loss per dataset for all five round-2 runs. Over the last 5 steps of each dataset this model's loss is 3.82 on verse, 3.44 on title, 1.96 on prose, 0.83 on neighbour and 0.45 on continuation pairs. Title and verse rows carry mined negatives from the same book and stay hardest; without mined negatives the mean title loss is 2.97, against 4.42 with them (Round-2 e5-base).

Run	title	verse_ict	prose_q	continuation	neighbour
This model	4.13	4.41	2.44	0.92	1.25
Round-2 e5-base	4.42	4.59	2.69	1.06	1.45
Round-2 e5-small	4.38	4.52	2.63	0.99	1.36
Round-2 bge-m3	4.20	4.30	2.36	0.82	1.17
Round-2 e5-base, no hard negatives	2.97	3.72	2.12	1.18	1.39

Mean loss per dataset over all steps. Over the two epochs this model saw 34 title, 22 verse, 54 prose, 100 continuation and 112 neighbour batches. Per-step losses of all five runs are in eval/training_loss.csv.

Compute cost and hardware

Chart 21. What each round-2 run cost. Training time, seconds per step and peak allocated GPU memory of the five runs on one NVIDIA A100-SXM4-80GB, next to their dev scores. This model took 1,620 s (27.0 minutes) at 5.03 s per step with 25.4 GB peak memory (26.0 GB reserved). Round-2 bge-m3, of similar size, took 26.8 minutes and 25.6 GB; Round-2 e5-base 9.1 minutes, the no-hard-negatives run 6.2 minutes, and Round-2 e5-small 5.2 minutes with 15.0 GB.

Run	Training time	Seconds per step	Peak memory allocated	Peak memory reserved
This model	1,620.1 s (27.0 min)	5.031	25.4 GB	26.0 GB
Round-2 bge-m3	1,608.7 s (26.8 min)	4.996	25.6 GB	26.1 GB
Round-2 e5-base	548.6 s (9.1 min)	1.704	17.1 GB	17.4 GB
Round-2 e5-base, no hard negatives	369.8 s (6.2 min)	1.148	17.1 GB	17.5 GB
Round-2 e5-small	309.6 s (5.2 min)	0.961	15.0 GB	15.4 GB

All five runs together took 4,456.8 s (74.3 minutes) of training. Peak memory depends on the gradient-cache mini-batch as well as on the model size.

Design choices not taken

The round-2 plan considered and rejected these options before training:

• Extending the vocabulary (for example with the Hindko Tokenizer's pieces): new embeddings cannot be learned well from the roughly 5.75M tokens available.
• Masked-language-model domain adaptation of e5 before contrastive training: its expected gain was below the dev standard error, and it risks the base model's embedding geometry.
• TSDAE or unsupervised SimCSE: e5 is already contrastively trained, and supervised and inverse-cloze pairs exist.
• Mining negatives with the round-1 neural model: it would add a dependence on a dev-selected model; TF-IDF mining on the train split is dev-free.
• A guide-model loss (GISTEmbed): the guide would be weak on book titles; the explicit record, cluster and chapter conflict rule is exact.
• A positive-score filter for title and verse negatives: TF-IDF ranks the true positive first for only 16 % to 39 % of those anchors, so the filter would delete true sibling poems. It is used for prose rows only, as a reprint guard.
• Other bases (Qwen3-Embedding-0.6B, LaBSE, mpnet): weak untuned baselines, and Qwen3 needs about 1.4 times more tokens on this script.
• Matryoshka loss, larger batches or a different temperature: no expected benchmark gain and untested settings.

Model selection protocol

1. The benchmark was frozen first. Task files are hashed, and the harness refuses to run on changed files.
2. Recipe gate. After the first round-2 run (r2-e5b), a gate fixed in advance decided the recipe for the remaining runs: book-title nDCG@10 at least 0.36, overall at least 0.685 and clustering at least 0.68 on dev. It passed (0.4464, 0.7079, 0.7339; decided 2026-10-01 15:38 UTC), so the remaining runs (e5-large, e5-small and bge-m3) were trained with mined negatives and same-book blocks.
3. Candidates. Five round-2 models (e5-small, e5-base, e5-large, bge-m3, and e5-base without mined negatives) were trained on the same pairs. Each was evaluated on dev from its saved float16 weights at 512 tokens. The pool also held the two round-1 models, with their round-1 dev scores (0.6851 and 0.6700).
4. Rule, written before any round-2 result: take the highest dev overall score; among models within 0.003 of it, take the smallest (size ranks: e5-small 1, e5-base 2, e5-large 3, bge-m3 4). r2-e5l scored 0.7117. The next best, r2-e5b, scored 0.7079 (0.0038 lower) and was outside the band, so r2-e5l was selected and r2-e5b became the runner-up. The selection file was written (2026-10-01 17:48 UTC) before any round-2 test result existed.
5. Test, once. The selected model and the runner-up were scored on the test split once. The untuned public models were re-run at 512 tokens after the selection and were never in the pool.

Chart 03. Selected on dev by a fixed rule. The overall dev score of every model in the selection pool, with the pre-registered tie band (within 0.003 of the best) shaded and TF-IDF on dev (0.6667) dashed. Only the best model, r2-e5l at 0.7117, falls in the band, so it was selected; the runner-up r2-e5b is 0.0038 below at 0.7079. All five round-2 candidates (0.7008 to 0.7117) score above both round-1 incumbents (0.6851 and 0.6700).

The round-2 choice used dev only, and test was scored once per model, after it. Test is not untouched across rounds, though: the round-1 test scores of the public models and of our round-1 models were known when round 2 was planned, and the plan cited them, plus a test-split document-length statistic (that 89.5 % of the relevant test book-title passages are longer than 512 tokens), when it chose the candidate base models and adopted 512 tokens. No checkpoint, hyperparameter or threshold was chosen on test. The dev scores of the round-2 candidates (0.7008 to 0.7117) differ by about the dev standard error estimated in the training plan (about 0.01), so the choice of e5-large over e5-base is not statistically established on dev. On test the two models moved in different directions: 0.7391 for e5-large and 0.6919 for e5-base.

Ablation: mined negatives and same-book batches

Chart 16. Ablation: mined hard negatives and same-book batches (dev). Round-2 e5-base trained twice on the same pairs: with mined hard negatives and same-book blocks of 32, and without both (in-batch negatives only); the bars show with minus without. With them, dev overall is 0.7079 against 0.7066 (+0.0013), with gains on headline (+0.0065), book title (+0.0056) and clustering (+0.0105) and losses on continuation (-0.0119) and verse/prose (-0.0042). Every difference is within plus or minus 0.012; the shaded band (plus or minus 0.01) is roughly one standard error of the dev overall score, and single tasks are noisier than that.

It is one seed on dev, so these differences are within noise. The two runs also differ in batch order, and the run without negatives trained faster (369.8 s against 548.6 s). The recipe gate, fixed in advance, kept mined negatives and same-book blocks for the larger runs; this ablation does not show that they help, nor that they hurt.

Comparison to round 1

	Round 1 (round-1 model)	Round 2 (this model)
Base model	multilingual-e5-base	multilingual-e5-large
max_seq_length	256	512
Training pairs	30,431: headline 1,743, book title 1,390, continuation 12,932, neighbour 14,366	41,533: adds first-line, verse-line and prose-sentence pairs; 4,391 title rows
Negatives	in-batch only	in-batch plus 23,061 mined hard negatives
Same-book blocks	no	blocks of 32 for title and verse rows
Learning rate, gradient-cache mini-batch	3e-5, 64	2e-5, 32
Epochs, batch size	2, 256	2, 256
Evaluated in	float32	float16
Selection rule	highest dev score; within 0.003, the smaller model	same, with a pre-registered recipe gate
Dev overall	0.6851	0.7117
Test overall	0.6923	0.7391

Chart 04. Round 2 against round 1 (test). Test scores of the round-1 model and this model, with the change under each task. Round 2 changed the base model (e5-base to e5-large), the data (new short-query pairs, mined hard negatives, same-book batches) and the length (256 to 512 tokens) at the same time. The overall score rises by 0.0468 (0.6923 to 0.7391), with the largest gains on clustering (+0.0954) and book title (+0.0720); verse/prose is 0.0026 lower (0.9827 to 0.9801).

Because several things changed at once, the gain cannot be split between them, and the other runs point in different directions. On dev, the round-2 data, recipe and 512-token length on the same e5-base model add 0.0228 over the round-1 model (0.7079 against 0.6851), while e5-large adds only 0.0038 more. On test, the same round-2 e5-base scores 0.6919, about the same as the round-1 model (0.6923), and e5-large is 0.0472 higher.

Intended uses

The model is released for research, teaching and other non-commercial use (see License). Within that scope it is intended for:

• Semantic search and retrieval over Hindko text in Urdu script, including retrieval for RAG systems.
• Similarity, clustering and classification features for Hindko documents, verses and sentences.
• A baseline for research on embeddings for low-resource, Perso-Arabic-script languages, and a reference point for future versions of HindkoEmbedBench.

Out-of-scope uses

• Commercial use. The licence (CC BY-NC 4.0) does not permit use primarily intended for or directed towards commercial advantage or monetary compensation, such as selling the model or using it in a paid product, service or API, without written permission from the author (see Contact).
• Romanized Hindko, speech transcripts, cross-lingual search (for example an English query over Hindko documents) and other languages were not evaluated; the base model's multilingual ability may or may not survive fine-tuning.
• Decisions about people. Do not use similarity scores alone for decisions about people, such as hiring, grading, moderation or surveillance.
• Authorship or plagiarism claims. Book clustering measures book identity on 5 to 8 books; it was not validated for attributing authors or detecting plagiarism.
• Calibrated probabilities. Cosine scores are not probabilities, and no threshold was validated.
• Generation or translation. This is an encoder; it produces vectors, not text.

Limitations and bias

• Book-title retrieval is a weak spot. On test, this model scores 0.3460 nDCG@10 against 0.3756 for TF-IDF and 0.3778 for bge-m3; our round-1 e5-small model also scored higher (0.3664). On dev it is also behind TF-IDF (0.4257 vs 0.4594). It also has the lowest dev book-title score of the five round-2 candidates (0.4257, against 0.4358 to 0.4464); it was selected on the overall score. For title or first-line lookup, combine it with a lexical search (hybrid retrieval).
• Residual overlap. The leak filters drop training texts with word 8-gram containment of 0.3 or more in dev or test text. A separate line-level fuzzy check still finds one test book-title query whose relevant passage has at least half of its lines closely matched in train-split newspaper reprints, and 1 to 3 dev book-title queries per book with at least 30 %. These may slightly favour models trained on this corpus on book-title retrieval; this model still scores below TF-IDF there.
• Small test sets. The test split has 80 headline queries, only 28 book-title queries (from 5 books), 157 verse/prose items (the verse class is essentially one book) and 176 clustering passages from 5 books. Differences of a few points on these tasks are within noise. Dev and test are both reported for this reason.
• One seed. Every round-2 run was trained once; run-to-run variation was not measured, and the ablation is a single comparison.
• Automatic benchmark. Relevance comes from metadata (headline to article, contents title to passage, book identity), not from human judgments. A few headlines are attached to the wrong article, and book identity is only a proxy for topic and style.
• Domain. Training and evaluation text is newspaper prose (mostly from one Peshawar weekly), literary books (much of it poetry) and a small amount of web text. Expect weaker results on conversational, social-media, technical or legal text.
• Dialect and spelling. Peshawari and literary spellings dominate; Hazara and other varieties are less represented. Hindko spelling varies (retroflex nasal, harakat, Arabic-keyboard letters). The model was trained on normalized text, so normalize input the same way.
• Script. Only Urdu (Perso-Arabic) script was evaluated; Romanized Hindko was not.
• Truncation. Retrieval mostly sees the start of each document: benchmark documents are cut at 3,000 characters and the model reads at most 512 tokens. At 512 tokens, 24.1 % of all test texts are still cut.
• Precision. All scores were measured in float16 on a GPU; float32 on a CPU was not benchmarked.
• Selection noise. See Model selection protocol: the dev margin over the runner-up is smaller than the dev standard error.
• Bias. The model reflects the viewpoints and vocabulary of the published newspapers and books it was trained on, and any biases of its multilingual base model. No bias evaluation was done.

Ethical considerations and privacy

• No training text is included. The repository holds model weights, configuration, evaluation outputs, benchmark charts and the script that draws them.
• Provenance. The training and benchmark texts belong to their publishers and authors (Weekly Hindkowan, the Gandhara Hindko Academy and web sources). They are not distributed here (apart from the three example sentences in the Quick start, which are also published in the Hindko Tokenizer card), and the licence below does not cover them.
• Use with care. Embeddings of personal text can reveal what the text is about. Treat stored embeddings with the same care as the text they come from.

Reproducibility

• Versions. Saved with sentence-transformers 6.1.0 and transformers 5.3.0; load with sentence-transformers 6.0 or later and transformers 5.x.
• Seeds. Training seed 20261001 (all five round-2 runs); benchmark build seed 20260929; k-means seeds 0 to 9.
• Fixed inputs. The training pairs are identified by sha256 in eval/selection.json (rows.jsonl 9ddde8f6ef27…, texts.jsonl f0c31d84f79e…), and the record-level splits by the published split manifest (76582d3a1e0afefe…). The benchmark task files are frozen under a manifest with sha256 6f82ec11e0f4e934….
• Release status. The benchmark data is not released yet; the benchmark and the training code will be released. Until then the files below let you check every number and chart on this page.

Files in eval/:

File	Contents
results.csv	One row per model and split: overall and task scores, headline Recall@1/@10 and per-source scores, book-title Recall@1/@10 and MRR@10
selection.json	The selection rule, the dev scores of the pool, the gate result, the selected model and runner-up, and the sha256 of the training pairs
training.json	Settings, training time and peak memory of the five round-2 runs; training rows per dataset
full_metrics.json	Every metric of every result file (the 27 result entries behind results.csv, plus the 14 round-1 public-model runs at 256 tokens): all retrieval metrics with per-source breakdowns and query counts, classification and clustering spreads, texts encoded and truncated, plus training metadata, benchmark task sizes and drop counts, and the training-data mix
training_loss.csv	Loss at every optimizer step of the five round-2 runs (1,610 rows: model, step, dataset type, loss)

Charts. The 25 images in benchmarks/ (3200 x 1800 PNG, free to reuse with credit for non-commercial purposes under the licence) are drawn from the files in eval/ by make_charts.py:

python make_charts.py            # all charts
python make_charts.py 00,03      # only these

It needs Python 3 and Google Chrome or Chromium (it renders HTML pages headlessly; if the browser is not found, set the CHROME environment variable to its path) and downloads its fonts from Google Fonts once per run. Apart from the parameter counts of the public models, the dev standard error of about 0.01 from the training plan (the shaded band in chart 16), and a few fixed counts typed into chart subtitles and notes (task sizes in charts 00 to 06, 14, 15 and 20; run counts, steps, epochs, batch and block sizes, GPU and seed in charts 15 to 18 and 21, and the max_seq_length and precision settings (512 or 256 tokens, float16 or float32) named in subtitles and legends, all matching eval/), every number in the charts is read from eval/.

License

Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) for this model; full text in LICENSE.

• Open for research. You may use, share and adapt the model, for example by fine-tuning it, for research, teaching, personal and other non-commercial purposes.
• Not for commercial use. You may not use it for purposes primarily intended for or directed towards commercial advantage or monetary compensation, such as selling it or using it in a paid product, service or API. What counts is the purpose, not who you are: research done for a commercial product or service is commercial use. For commercial use, ask the author for written permission first (see Contact).
• Give credit: "Hindko Embeddings" by junaid008, with a link to this repository; in a paper, the citation below is enough. If you share the model or a version you changed (for example a fine-tune), also keep the notices in LICENSE, link to the licence and say what you changed.
• The base model, intfloat/multilingual-e5-large, is released under the MIT License (Copyright (c) Microsoft Corporation). Its notice is reproduced in LICENSE and must be kept with copies of the weights, tokenizer and configuration files.
• The licence covers the files in this repository. It grants no rights in the texts the model was trained or evaluated on. Those texts are not included, apart from the three example sentences in the Quick start, which belong to their authors and are not covered by the licence.

Citation

@misc{hindko_embeddings_2026,
  title        = {Hindko Embeddings: a sentence-embedding model for Hindko},
  author       = {junaid008},
  year         = {2026},
  version      = {1.0.0},
  howpublished = {\url{https://huggingface.co/junaid008/hindko-embeddings}},
  note         = {CC BY-NC 4.0. Fine-tuned from intfloat/multilingual-e5-large; selected on the dev split of HindkoEmbedBench v1.0 by a pre-registered rule; test overall 0.7391}
}

Please also cite the base model:

@article{wang2024multilingual,
  title   = {Multilingual E5 Text Embeddings: A Technical Report},
  author  = {Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal = {arXiv preprint arXiv:2402.05672},
  year    = {2024}
}

Acknowledgements

Heartfelt thanks to the Chief Minister's Policy Office, the Weekly Hindkowan and the Gandhara Hindko Board for supporting this work and sharing the texts that made it possible.

This project was built together with Muhammad Uzair.

Contact

junaid008 on Hugging Face. Please open a discussion on this repository for questions, errors, clearance matters or requests for commercial use.

Changelog

Version	Date	Changes
1.0.0	2026-10-01	First release: multilingual-e5-large fine-tuned on 41,533 pairs from a Hindko corpus (round 2, r2-e5l); HindkoEmbedBench v1.0 results, selection record and benchmark images.
1.0.0 (card update)	2026-10-30	Model card rewritten and expanded: usage recipes, a full description of each benchmark task, detailed analysis, and training details. 18 new benchmark charts (07 to 24); charts 00 to 03, 05 and 06 unchanged, and chart 04 now rounds each change from the unrounded scores (clustering +0.095 instead of +0.096). New evaluation files eval/full_metrics.json and eval/training_loss.csv; make_charts.py now draws all 25 charts. Weights and all scores unchanged.
1.0.0 (licence)	2026-10-31	Licence changed from CC BY 4.0 to CC BY-NC 4.0 (research and other non-commercial use only). Weights, scores and charts unchanged.