sagepond/embedding-small

SAGE POND Luganda Sentence Embedding Model

This is a sentence-embedding model developed by SAGE POND for experimentation with semantic representations in Luganda. It maps an input sentence to a 3,072-dimensional, L2-normalized embedding that can be compared with other sentence embeddings using cosine similarity or used as a representation in downstream semantic applications.

Intended use

The model was trained on sentences, not question-and-answer pairs, instruction-response examples, or conversational QA data. It should therefore be treated as a sentence-level embedding model.

The release is intended as a research example and an accessible starting point for people experimenting with Luganda language representations. It can also be used to build prototypes and semantic applications, including:

• Sentence similarity and semantic matching.
• Semantic search over Luganda text.
• Sentence clustering and grouping.
• Duplicate or near-duplicate sentence exploration.
• Embedding-based retrieval and other downstream experiments.

These are potential application areas, not claims that the model has been evaluated or optimized for every task listed above. Retrieval quality will depend on the corpus, preprocessing, similarity strategy, and evaluation data used by the application.

STS benchmark results

The model was evaluated on the SAGE POND Semantic Textual Similarity benchmark, sagepond/sts. The evaluation set contains approximately 282 samples.

Metric	Score
Pearson correlation score	34.06
Spearman correlation score	47.91
STS score	40.98

The reported STS score is the arithmetic mean of the reported Pearson and Spearman scores. The values above are shown on the same 0–100 scale as the evaluation output.

Benchmark characteristics: although the evaluation set is compact (approximately 282 examples), it contains notably challenging sentence pairs. Its size should not be mistaken for an indication that the examples are easy or that the score is low simply because the benchmark is small. The results reflect the model's ability to capture semantic similarity across the particular difficult examples included in sagepond/sts; a similarly sized but easier test set could produce substantially different scores.

These results are an initial evaluation on the SAGE POND benchmark, not evidence of broad performance across every Luganda domain. They should not be directly compared with scores from other STS datasets or evaluation protocols without checking differences in data selection, difficulty, scoring, and preprocessing. A larger and independently curated evaluation would provide a stronger estimate of generalization.

Model details

Property	Value
Model type	Transformer sentence encoder
Primary language	Luganda
Training data format	Sentences (not QA-style examples)
Tokenizer	SentencePiece
Vocabulary size	40,000 tokens
Transformer layers	12
Hidden size	768
Attention heads	16
Feed-forward dimension	3,072
Maximum sequence length	600 tokens
Sentence embedding dimension	3,072

The model adds learned positional embeddings to token embeddings, encodes the sequence with a Transformer encoder, applies masked mean pooling, and projects the pooled representation to the final 3,072-dimensional embedding space. The output embeddings are L2-normalized.

Usage

The repository uses custom Transformers code. Load it with trust_remote_code=True:

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "sagepond/embedding-small"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()

sentences = [
    "Ekigambo eky'okulabirako eky'Oluganda.",
    "Sentensi endala ey'okugezesa.",
]

inputs = tokenizer(
    sentences,
    padding=True,
    truncation=True,
    max_length=600,
    return_tensors="pt",
)

with torch.inference_mode():
    embeddings = model(**inputs).pooler_output

print(embeddings.shape)  # torch.Size([2, 3072])

Because the model returns normalized embeddings, cosine similarity can be calculated with a dot product:

similarity = embeddings[0] @ embeddings[1]
print(similarity.item())

For a larger collection of sentences, encode texts in batches and store the resulting vectors in a vector index or another retrieval system. Use the same model and tokenizer version for both indexed documents and incoming queries.

Limitations

• The benchmark contains approximately 282 samples, so its scores have limited statistical coverage.
• The model was trained on sentence-level data, not question-answer pairs. Do not assume that it is optimized for question-to-passage retrieval or QA matching without task-specific evaluation.
• The 600-token maximum sequence length is a model input limit; long documents should be split into suitable chunks and embedded separately.
• Similarity scores are relative signals for ranking or comparison. They are not calibrated probabilities or factual confidence scores.

Intended audience

Researchers, developers, students, and language-technology practitioners interested in experimenting with Luganda sentence embeddings, semantic similarity, and semantic retrieval.

Dataset

The STS evaluation dataset is available at sagepond/sts.

Citation and attribution

This model was developed by SAGE POND. When using it in research or derivative work, cite the model repository and the sagepond/sts dataset, and report the evaluation setup used in your experiments.