--- license: mit language: - hi - en library_name: sentence-transformers pipeline_tag: sentence-similarity base_model: intfloat/multilingual-e5-small datasets: - rahul27474/hinglish-retrieval-train - rahul27474/hinglish-retrieval-bench tags: - sentence-transformers - sentence-similarity - feature-extraction - embeddings - retrieval - rag - hinglish - code-mixed - romanized-hindi - hindi - onnx --- # hinglish-embed-small A **37M-parameter** text embedding model for **Romanized Hinglish**, Hindi (Devanagari) and English, built for RAG and semantic search. It is fast on a plain CPU. Indian users write `bhai upi se paise kat gaye par receiver ko nhi mile` far more often than proper Hindi or English. General multilingual embedders handle this mixed, misspelled, SMS-style text poorly. This model is built for it, and it matches queries in any of the three forms against documents in any of the three. - **384-dim** embeddings, cosine similarity - **37M params**, 3.2× smaller than `multilingual-e5-small` and 15× smaller than `bge-m3` - ONNX + **int8** weights included (`onnx/model_quint8_avx2.onnx`, 38 MB) - MIT license, so commercial use is OK - **[Try it in your browser →](https://huggingface.co/spaces/rahul27474/hinglish-embed-small-demo)** (runs locally through transformers.js, no server) Fine-tuned and vocab-trimmed from [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small). It is not trained from scratch. ## Results The [hinglish-retrieval-bench](https://huggingface.co/datasets/rahul27474/hinglish-retrieval-bench) has 3,626 mixed-language passages and 268 held-out topics across 40 Indian everyday and help-desk domains (UPI, IRCTC, Aadhaar/PAN, telecom, insurance, food delivery, …). Scores are nDCG@10 with graded relevance. | model | params | Hinglish queries | Hindi queries | English queries | |---|---|---|---|---| | **hinglish-embed-small** | **37M** | **0.788** | **0.802** | **0.817** | | hinglish-embed-small, ONNX int8 | 37M | 0.785 | 0.800 | 0.815 | | BAAI/bge-m3 | 568M | 0.612 | 0.717 | 0.743 | | intfloat/multilingual-e5-small | 118M | 0.482 | 0.570 | 0.704 | | intfloat/multilingual-e5-base | 278M | 0.444 | 0.549 | 0.679 | Recall@10 on Hinglish queries is 0.932, against 0.733 for bge-m3 and 0.573 for e5-small. > **Read this caveat.** The benchmark is synthetic and was produced by the same LLM pipeline as the training data. The topics are disjoint, but the style is in-distribution for this model, so its lead is likely smaller on your real data. Evaluate on a sample of your own queries before you switch. ### CPU speed Measured on a 4-vCPU Kaggle CPU box with sentence-transformers. | | 1 query latency | batch throughput | |---|---|---| | hinglish-embed-small, ONNX int8 | **12.8** ms | 112 sent/s | | hinglish-embed-small, ONNX fp32 | 15.7 ms | 101 sent/s | | hinglish-embed-small, PyTorch fp32 | 34.3 ms | 110 sent/s | | multilingual-e5-small, PyTorch fp32 | 29.2 ms | 115 sent/s | | bge-m3, PyTorch fp32 ¹ | ~248 ms | ~16 sent/s | Single-query latency, the number that matters for RAG search, is **2.3× lower than e5-small** with int8 ONNX and about 19× lower than bge-m3. Batch throughput on 4 vCPUs is about the same as e5-small, because the compute per token is the same; the vocab trim saves memory, not FLOPs. The int8 weights keep 99.7% of fp32 quality (table above). ¹ bge-m3 was measured in a separate run on the same Kaggle CPU tier. ## Usage Prefix queries with `query: ` and documents with `passage: `, as in e5. The prefixes are saved as prompts, so `encode_query` / `encode_document` add them for you. ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("rahul27474/hinglish-embed-small") docs = [ "UPI transaction fail hone par paisa 3-5 working days mein automatically refund ho jata hai.", "ट्रेन टिकट कैंसिल करने पर चार्ज यात्रा से पहले के समय पर निर्भर करता है।", "To port your mobile number, send an SMS 'PORT ' to 1900 to get a UPC code.", ] q = "bhai upi se paise kat gaye par receiver ko nhi mile" q_emb = model.encode_query([q]) # sentence-transformers >= 5 d_emb = model.encode_document(docs) # older versions: model.encode(["query: " + q]), model.encode(["passage: " + d for d in docs]) print(model.similarity(q_emb, d_emb)) ``` ### Faster on CPU: ONNX int8 ```python # pip install "sentence-transformers[onnx]" model = SentenceTransformer("rahul27474/hinglish-embed-small", backend="onnx", model_kwargs={"file_name": "onnx/model_quint8_avx2.onnx"}) ``` ### Plain transformers ```python import torch, torch.nn.functional as F from transformers import AutoTokenizer, AutoModel tok = AutoTokenizer.from_pretrained("rahul27474/hinglish-embed-small") model = AutoModel.from_pretrained("rahul27474/hinglish-embed-small") texts = ["query: refund kab aayega", "passage: Refund 5-7 din mein aapke account mein aa jayega."] b = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt") with torch.no_grad(): h = model(**b).last_hidden_state emb = F.normalize((h * b["attention_mask"][..., None]).sum(1) / b["attention_mask"].sum(1, keepdim=True), dim=-1) print(emb[0] @ emb[1]) ``` ### JavaScript (browser or Node) with transformers.js ```js import { pipeline } from '@huggingface/transformers'; const embed = await pipeline('feature-extraction', 'rahul27474/hinglish-embed-small', { dtype: 'q8' }); // 38 MB int8 const out = await embed(['query: refund kab aayega', 'passage: Refund 5-7 din mein account mein aa jayega.'], { pooling: 'mean', normalize: true }); ``` It works with LangChain / LlamaIndex through their sentence-transformers / HuggingFace embedding wrappers. Remember the prefixes. ## Use cases ### 1. Support chatbot: answer from FAQ ```python answers = [ "Refund 5-7 working days mein aapke original payment method mein aa jata hai.", "Order cancel karne ke liye My Orders > Cancel par jaayein. Ship hone ke baad cancel nahi hota.", "Damaged product ke liye 48 ghante ke andar photo ke saath return request daalein.", ] a_emb = model.encode_document(answers) def reply(msg, min_score=0.40): # see "Score ranges" below; tune on your own data s = model.similarity(model.encode_query([msg]), a_emb)[0] i = int(s.argmax()) return answers[i] if s[i] >= min_score else "Connecting you to an agent..." print(reply("mera refund kab tak aayega bhai")) ``` ### 2. Help-center search ```python from sentence_transformers import util art_emb = model.encode_document(articles, batch_size=64, convert_to_tensor=True) # do once, cache it hits = util.semantic_search(model.encode_query(["pan aadhar link nahi hua to kya hoga"], convert_to_tensor=True), art_emb, top_k=5)[0] ``` ### 3. RAG with LangChain + FAISS ```python from langchain_huggingface import HuggingFaceEmbeddings from langchain_community.vectorstores import FAISS emb = HuggingFaceEmbeddings( model_name="rahul27474/hinglish-embed-small", encode_kwargs={"normalize_embeddings": True, "prompt": "passage: "}, query_encode_kwargs={"normalize_embeddings": True, "prompt": "query: "}, ) db = FAISS.from_texts(chunks, emb) # chunks: Hindi / English / Hinglish text, ~100-150 words each docs = db.similarity_search("leave kitne din ki milti h saal me", k=4) ``` ### 4. Duplicate questions / similar tickets ```python from sentence_transformers import util tickets = ["upi payment fail hua par paisa kat gaya", "UPI se paise cut ho gaye lekin transaction failed dikha raha", "यूपीआई से पैसे कट गए पर पेमेंट फेल हो गया", "order wrong address par deliver ho gaya"] emb = model.encode_query(tickets, convert_to_tensor=True) # same prefix on both sides for symmetric tasks for group in util.community_detection(emb, threshold=0.60, min_community_size=2): print([tickets[i] for i in group]) ``` ### Score ranges Unlike the original e5, whose cosine scores all sit around 0.7–0.9, this model **spreads its scores out**, so a fixed threshold is meaningful. These ranges come from 300 Hinglish benchmark queries against all 3,626 passages, using int8 ONNX: | pair | p10 | median | p90 | |---|---|---|---| | query → its own passage | 0.54 | 0.73 | 0.83 | | query → another passage on the same topic | 0.52 | 0.73 | 0.83 | | query → unrelated passage | −0.14 | −0.04 | 0.09 | At a threshold of **0.40**, 98% of correct passages pass and 0.3% of unrelated ones do; at **0.30** the figures are 99.7% and 0.9%. For duplicate detection, start around 0.60. Check any threshold on a few dozen examples of your own data. ## How it was built 1. **Vocab trim:** the 250k-token XLM-R vocabulary was cut to the 40k tokens used by English, Hindi and Hinglish text (99.93% token coverage on a Hinglish/Hindi/English corpus). This takes params from 118M to 37M. Before any training, embeddings matched the original model at mean cosine 0.99994. 2. **Synthetic data:** `Qwen3-14B-AWQ` generated 18.9k passages across 40 domains, each with 3 Hinglish queries in different styles, a Hindi query, an English query and a hard negative. `bge-m3` filtered out false negatives. Details are in [hinglish-retrieval-train](https://huggingface.co/datasets/rahul27474/hinglish-retrieval-train). 3. **Training** (1 epoch, one T4, ~40 min): - `CachedMultipleNegativesRankingLoss` (batch 256, hard negatives) on 112k query-passage triplets. These cover Hinglish, Hindi, English and auto-misspelled queries (`hai→h`, `nahi→nhi`, `kaise→kese`, `please→plz`, …). - `MSELoss` alignment on 406k English↔Hinglish/Hindi pairs against the frozen original model. This maps Hinglish into the space the model already had for English, so general English and Hindi quality is kept. Max sequence length 512 (trained at 256). The training code is in the repo's `training/` folder. ## Limitations - Benchmarks are synthetic and in-distribution (see the caveat above). There is no MTEB score yet. - The model is tuned for short, help-desk-style questions and passages of 50–150 words. Very long documents should be chunked. - It only handles Hindi-English. Other Indic languages (Tamil, Bengali, …) and their romanized forms were not trained, and the trimmed vocab may tokenize them poorly. - Like any embedder, it captures similarity, not truth. The training passages contain invented facts, and the model does not learn facts from them. ## Citation / credit - Base model: [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) (MIT). Wang et al., *Multilingual E5 Text Embeddings*, 2024. - Data generator: [Qwen/Qwen3-14B-AWQ](https://huggingface.co/Qwen/Qwen3-14B-AWQ) (Apache-2.0). Filter and baseline: [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) (MIT). - Alignment pairs: [findnitai/english-to-hinglish](https://huggingface.co/datasets/findnitai/english-to-hinglish) (Apache-2.0) and [sk-community/romanized_hindi](https://huggingface.co/datasets/sk-community/romanized_hindi) (MIT).