Adata-Embeddings

A Russian embedding model for finding semantically similar short texts: product, work and service names, catalog categories, user queries. It is ai-sage/Giga-Embeddings-instruct (GigaChat-3B backbone, latent-attention pooling) fine-tuned with LoRA on the category tree of the National Catalog of Goods of Kazakhstan (НКТ, nct.gov.kz); the adapter is merged into the weights.

Base	ai-sage/Giga-Embeddings-instruct, commit 2cf0fdc97194aaedf10ac0e6bf798834acd31042
Parameters	3.48 B (BF16 weights, ~6.9 GB)
Embedding dimension	2048
Max input	4096 tokens (trained on texts up to 64 tokens)
Language	Russian
License	MIT (same as the base model)

Typical use: embed the catalog category names once, embed an incoming product or service name, and take the nearest categories by cosine similarity. The same vectors work for finding duplicate names and grouping similar texts.

Quality

Category retrieval on 1,000 held-out catalog categories that were excluded from training. Query: one alternative name of a held-out category. Corpus: 261,716 short texts (every category's name plus its alternative names; for the held-out categories, the name and two other alternative names). A hit means the correct category is among the top-k distinct categories.

Model	R@1	R@5	R@10
BAAI/bge-m3 (zero-shot)	0.283	0.527	0.633
ai-sage/Giga-Embeddings-instruct (zero-shot)	0.324	0.588	0.706
Adata-Embeddings	0.384	0.682	0.796

Evaluation protocol: BF16 weights, max length 64 tokens, queries prefixed with Найди описание категории каталога товаров, соответствующей запросу: ("Find the catalog category description that matches the query: "), corpus texts without a prefix, L2-normalized vectors, exact inner-product search (FAISS IndexFlatIP).

Usage

Requirements

pip install torch "transformers==4.57.3" "accelerate==1.12.0" "einops==0.8.2"
pip install "bitsandbytes==0.49.2"   # only for 4-bit loading

The repository contains custom model code (modeling_gigarembed.py), so trust_remote_code=True is required.

	BF16	4-bit NF4
Memory	weights ~6.9 GB	~3.7 GB of GPU memory per process (measured on an RTX 5060 Ti 16 GB)

Tested with torch 2.11 and transformers 4.57.3. BF16 also runs on CPU: drop .cuda() below.

Transformers

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

MODEL = "NAUMOV/Adata-Embeddings"

tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda().eval()

def instruct(task: str, text: str) -> str:
    return f"Instruct: {task}\nQuery: {text}"

task = "Given a product/work/service description, map it to a concise category title"
queries = [instruct(task, "home speakers with a built-in amplifier")]
categories = ["Active speaker systems", "Passive speaker systems", "Projector lamps"]

with torch.no_grad():
    q = model(**tokenizer(queries, padding=True, truncation=True, max_length=4096,
                          return_tensors="pt").to(model.device), return_embeddings=True)
    c = model(**tokenizer(categories, padding=True, truncation=True, max_length=4096,
                          return_tensors="pt").to(model.device), return_embeddings=True)

scores = F.normalize(q.float(), dim=-1) @ F.normalize(c.float(), dim=-1).T
print(categories[scores.argmax().item()], [[round(x, 4) for x in row] for row in scores.tolist()])

Output (BF16; the last digits may differ across hardware):

Active speaker systems [[0.4593, 0.2729, 0.1481]]

For comparison, the base Giga-Embeddings-instruct scores the same three categories 0.6010 / 0.4580 / 0.4798: the fine-tuned model separates close categories much more clearly. The model was trained and evaluated on Russian (the same example in Russian gives 0.3891 / 0.2843 / 0.1272); English inputs work, but no metrics were measured for them.

4-bit loading

from transformers import BitsAndBytesConfig

model = AutoModel.from_pretrained(
    MODEL, trust_remote_code=True,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16),
).eval()

4-bit vectors differ slightly from BF16 ones (cosine similarity ≈ 0.98 between the two on our checks), so build the index and encode queries with the same precision.

Training data

Source. The category tree of the National Catalog of Goods of Kazakhstan (НКТ), collected from the public API of nct.gov.kz. Training uses the leaf categories: 65,679 categories in 69 top-level sections, each with its name and full path in the hierarchy (mostly 4 levels). Category names are short: 4 words at the median.

Top-level section	Categories	Share
Medicines and medical devices	28,128	42.8%
Vehicles and transport equipment	11,847	18.0%
Machinery, equipment and parts	10,920	16.6%
Measuring instruments	1,754	2.7%
Food products and beverages	900	1.4%
Building materials	878	1.3%
Laboratory	771	1.2%
Electrical equipment	690	1.1%
61 other sections	9,791	14.9%

Alternative names. For every category an LLM generated exactly 3 alternative Russian names: short (2–6 words, 4 at the median), recognizable names under which the item is searched for or listed in price lists, each keeping the key attribute (type, purpose, characteristic) that makes it unambiguous within its section. Examples (English glosses: Active speaker systems → "active sound speakers", "speaker systems with a built-in amplifier", "active audio system"; Hydraulic motor tee of the feeder drive → "feeder hydraulic motor tee of a harvester", "hydraulic tee of the feeder drive", "forage harvester hydraulic system tee"):

{"name": "Активные акустические системы",
 "full_path": "Аудио-видео товары и фото-техника - Звуковое оборудование - Акустические системы - Активные акустические системы",
 "alt_names": ["активные колонки для звука", "акустические системы со встроенным усилителем", "активная аудиосистема"]}
{"name": "Тройник гидромотора привода питающего аппарата",
 "full_path": "Машины, оборудование и их части - Части сельскохозяйственных, садовых или лесохозяйственных машин и оборудования - Части комбайна самоходного кормоуборочного - Тройник гидромотора привода питающего аппарата",
 "alt_names": ["тройник гидромотора питателя комбайна", "гидравлический тройник привода питающего аппарата", "тройник гидросистемы кормоуборочного комбайна"]}

Training pairs. For each category, all unordered pairs among its 4 texts (name + 3 alternative names): 6 positive pairs per category, 388,074 pairs in total. Texts are used without instructions. 1,000 randomly chosen categories (seed 42) were held out entirely for evaluation.

The dataset is not included in this repository.

Training

Method	LoRA r = 16, alpha = 32, dropout 0.05 on q_proj, dkv_proj, kr_proj, o_proj, gate_proj, up_proj, down_proj; merged into BF16 weights after training
Trainable parameters	31.3 M (0.90%)
Loss	InfoNCE with in-batch negatives, temperature 0.05
Optimization	1 epoch, 12,128 steps, batch 8 × accumulation 4, lr 1·10⁻⁴ with cosine decay, 8-bit AdamW, seed 42
Input	max 64 tokens
Hardware	NVIDIA RTX 4080 SUPER 16 GB, ~4.5 hours

Limitations

• Trained and evaluated only on short Russian names from the НКТ catalog. Quality on long descriptions, other catalogs, other languages or general-purpose retrieval was not measured and may be worse than the base model.
• The catalog is skewed: medicines, vehicles and machinery make up 77% of the categories.
• The alternative names used for training were generated by an LLM, not collected from real queries.
• The metrics above were measured with BF16 weights, a plain query prefix and a 64-token limit. The Instruct: … Query: … format and 4-bit loading shown in Usage were not evaluated separately.
• Inputs longer than 4096 tokens are not supported.

License

MIT, same as ai-sage/Giga-Embeddings-instruct.