These checkpoints are the unofficial one to one mirror of the original checkpoints uploaded on AIKosh by the original author ZenteiQ on AIKosh
Please contact the original author for more information on the model. It was uploaded to HF by me for more easy access to the model
Dhฤซr 5B MoE
A sparse Mixture-of-Experts foundation model, designed and pretrained from scratch in India, with an in-house tokenizer built for Indian languages.
ZenteiQ ยท BrahmAI family ยท Apache 2.0
Overview
Dhฤซr 5B MoE is the higher-capacity model in ZenteiQ's BrahmAI family. It was designed and pretrained from scratch by ZenteiQ's team in India, rather than adapted from an existing foundation model. It understands English and ten Indian languages. Its sparse Mixture-of-Experts design holds 5.09B parameters, but each token activates only 1.79B of them: two of twelve experts in every layer. This gives it more capacity than a dense model of the same per-token cost, for enterprise workloads on a single GPU or inside a private infrastructure boundary.
| Developer | ZenteiQ AiTech Innovations Pvt. Ltd., Bengaluru, India |
| Model type | Sparse Mixture-of-Experts decoder-only Transformer, instruction-tuned (SFT and GRPO reinforcement learning) |
| Parameters | 5.09B total, 1.79B active per token |
| Experts | 12 routed experts per layer, top-2 routing, no shared experts |
| Languages | English, Hindi, Gujarati, Tamil, Bengali, Kannada, Malayalam, Marathi, Odia, Punjabi, Telugu |
| Context length | 16,384 tokens |
| Licence | Apache 2.0 |
| Support | IndiaAI Mission, MeitY, under the Foundation Model initiative |
| Companion model | Dhฤซr 2B (dense, for lightweight and on-device deployment) |
Why Dhฤซr 5B MoE
- Built in India, end to end. Model design, tokenizer, data pipeline, pretraining and post-training were carried out by ZenteiQ's team in India.
- Efficient by design. 5.09B parameters of capacity with 1.79B active per token, so inference cost tracks a ~1.8B model rather than a 5B one.
- Strong Indian-language fit. The best Indic โ English translation and Indic cross-lingual summarisation among the 17 compared open models, including 4B-class dense models, and first of 18 on IndicSentiment and IndicCOPA.
- Answers from your documents. Leads the compared models of up to 3.2B parameters on question answering over supplied Indian-language passages (IndicGenBench XQuAD-IN and XorQA-IN), the core skill of RAG applications.
- Instruction following and code. IFEval 71.3 and code pass@1 57.2, both the best among models of up to 3.2B parameters.
- Native tool and function calling. Calls tools and APIs in a native tool format (BFCL v3 accuracy 76.3), so it can power assistants that act on documents, apps and services.
Architecture
| Component | Configuration |
|---|---|
| Parameters | 5.09B total, 1.79B active per token |
| Layers | 28 |
| Hidden dimension | 2560 |
| Attention | Grouped-Query Attention: 20 query heads, 5 key/value heads, head dim 128, QK normalisation |
| Feed-forward | Mixture of Experts: 12 routed experts, top-2 routing, expert intermediate dimension 1536 |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| Normalisation | RMSNorm |
| Attention kernel | Flash Attention |
| Vocabulary | 131,072 (in-house tokenizer for Indian languages) |
| Context | 16k tokens |
| Precision | bfloat16 |
Training stack: MaxText (JAX) on Google TPU v6e. Checkpoints are trained in Orbax format and converted to Hugging Face format for release.
Quick start
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "zenteiq/Dhir-5B-MoE" # confirm final repository name
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
messages = [{"role": "user", "content": "เคญเคพเคฐเคค เคเฅ เคฐเคพเคเคงเคพเคจเฅ เคเฅ เคฌเคพเคฐเฅ เคฎเฅเค เคฆเฅ เคตเคพเคเฅเคฏ เคฒเคฟเคเคฟเคเฅค"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
output = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
trust_remote_code=True is required. Dhฤซr ships its architecture as custom code, included in this repository, so Transformers must be allowed to run it when loading the model. The same applies to other tools that load the model through Transformers: for example, pass --trust-remote-code to vllm serve.
Intended use
| Use | Notes |
|---|---|
| Indic โ English translation | Drafting and everyday communication. Human review is recommended for published or legal content. |
| Question answering over supplied documents (RAG reader) | Answers from a passage you provide, in English and Indian languages. |
| Indian-language text understanding | Sentiment, reading comprehension and summarisation. |
| Base model for Indic fine-tuning | A strong starting point for organisations building their own Indian-language model with MoE efficiency. |
| Basic Python assistance | Short, self-contained functions. |
| Tool and function calling | Native function calling for assistants that call tools and APIs. |
| Private and on-premises AI | Enterprise assistants and privacy-sensitive applications on local infrastructure. |
Evaluation
All figures are for the Dhฤซr 5B MoE release checkpoint, from ZenteiQ's internal benchmark suite (2 November 2026). Ranks are against open models run on the same benchmark and harness: 17 instruction-tuned models for generative tasks, 18 models (base and instruct) for log-likelihood tasks, Dhฤซr 5B MoE included. "โค3.2B class" means the compared models of up to 3.2B parameters.
| Capability | Benchmark | Dhฤซr 5B MoE | Next best | Rank |
|---|---|---|---|---|
| Translation, 24 Indic โ English directions (chrF) | IndicGenBench FLORES | 50.0 | 46.0 | #1 of 17 |
| Cross-lingual summarisation, 13 languages | IndicGenBench CrossSum | 18.6 | 16.9 | #1 of 17 |
| Sentiment, 14 languages (accuracy) | IndicSentiment | 96.2 | 94.7 | #1 of 18 |
| Causal reasoning, 19 languages (accuracy) | IndicCOPA | 62.9 | 62.5 | #1 of 18 |
| Document QA, 13 languages (token F1) | IndicGenBench XQuAD-IN | 45.4 | 41.8 (best โค3.2B peer) | Leads models up to 3.2B |
| Cross-lingual QA, 13 languages (token F1) | IndicGenBench XorQA-IN | 26.7 | 20.5 (best โค3.2B peer) | #2 of 17; leads models up to 3.2B |
| Reading comprehension, 20 languages (accuracy) | Belebele | 60.1 | 49.5 (best โค3.2B peer) | #3 of 18; leads models up to 3.2B |
| Story completion (accuracy) | XStoryCloze | 68.5 | 67.0 (best โค3.2B peer) | #2 of 18; leads models up to 3.2B |
| Indian knowledge (accuracy) | MILU / INCLUDE | 41.6 / 40.3 | 41.5 / 37.6 (best โค3.2B peer) | Leads models up to 3.2B |
| Instruction following, English (strict) | IFEval | 71.3 | 70.4 (best โค3.2B peer) | Leads models up to 3.2B |
| Python (pass@1) | MBPP + HumanEval | 57.2 | 44.4 (best โค3.2B peer) | Leads models up to 3.2B; ahead of Gemma-3-4B-it (43.4) |
| Grade-school maths, English | GSM8K | 71.3 | Basic maths reasoning | |
| Grade-school maths, Indian languages | MGSM | 44.2 | ||
| Function calling, native tool format (accuracy) | BFCL v3, AST checker | 76.3 |
Translation, summarisation, sentiment and IndicCOPA are first across the whole comparison, which includes 4B-class dense models. The other rows lead within the โค3.2B class; larger 4B-class dense models score higher on those benchmarks and on English knowledge. Document QA is measured when the answer is in a supplied passage. Function calling is scored in the model's native tool format (BFCL v3, AST checker), averaged over 15 categories.
Deployment. Weights take 10.19 GB in BF16. Quantised variants, hardware validation and serving throughput will be published with the Dhฤซr Technical Report.
Evaluation notes
- Results come from ZenteiQ's internal evaluation suite on the release checkpoint. Independent reproduction is part of the release process, and the full methodology will appear in the Dhฤซr Technical Report.
- Every model in a comparison was scored with the same harness, prompt and metric.
- Each claim is tied to the benchmark that supports it. We do not extend a result beyond what the benchmark measures.
Precautions
Dhฤซr 5B MoE is a compact model, and we recommend the following for responsible deployment:
- Expert validation. Legal, medical and financial uses should be reviewed by a qualified professional. The model is not intended to give such advice on its own.
- Grounding. Answers are most reliable when the relevant text is supplied in the prompt, as in RAG applications.
- Language coverage. Supported languages are English and the ten Indian languages listed above.
- Reasoning mode. Thinking mode is not supported in this release.
- Human oversight. Like all language models, Dhฤซr 5B MoE can produce incorrect or biased output. Keep a human in the loop for consequential decisions.
Safety
Dhฤซr 5B MoE has been evaluated for safety behaviour, including probing with the NVIDIA Garak vulnerability scanner across jailbreak, prompt-injection and harmful-request categories. Detailed results will be published with the technical report. Deployers should add application-level safeguards suited to their use case.
Training
- Pretraining: from scratch on Google TPU v6e using MaxText (JAX).
- Tokenizer: in-house, designed for Indian languages (vocabulary 131,072), shared with Dhฤซr 2B.
- Post-training: two rounds of supervised fine-tuning, each followed by GRPO reinforcement learning.
Data composition, training compute and methodology will be provided in the Dhฤซr Technical Report.
Availability
Released under the Apache 2.0 licence through AIKosh, Hugging Face and ZenteiQ.
Acknowledgement
ZenteiQ acknowledges the support of the IndiaAI Mission, Ministry of Electronics and Information Technology (MeitY), under the Foundation Model initiative.
Citation
@misc{zenteiq2026dhir,
title = {Dh\={\i}r : Sovereign Compact Foundation Models },
author = {{ZenteiQ AiTech Innovations Pvt. Ltd.}},
year = {2026},
howpublished = {Technical report forthcoming}
}
Contact
ZenteiQ AiTech Innovations Pvt. Ltd. ยท Vista Pixel, Jakkuru, Bangalore 560092, India research@zenteiq.ai ยท www.zenteiq.ai
- Downloads last month
- 49