Yusiko/khazri-2-mini

Khazri 2 Mini — 100M

Khazri 2 Mini is a compact, open-weight decoder-only language model in the Khazri family. It combines a modern LLaMA-style Transformer with a custom Byte-Level BPE tokenizer and a 2B-token English training corpus.

Hugging Face · Khazri · Contact

At a glance

Item	Detail
Model	Khazri 2 Mini
Parameters	100.68M
Status	Open weights on Hugging Face
Architecture	LLaMA-style, decoder-only Transformer
Training precision	bf16
Context configured for training	1,024 tokens
Vocabulary	32,768 tokens
Tokenizer	Custom Byte-Level BPE
Attention	Grouped-Query Attention: 12 query heads / 4 KV heads

Architecture

Component	Configuration
Hidden size	768
Transformer layers	12
Attention heads	12
Key/value heads	4
MLP intermediate size	2,048
Positional encoding	RoPE
Normalization	RMSNorm
MLP activation	SwiGLU / SiLU
Attention backend	FlashAttention-2 where available; PyTorch SDPA fallback
Embeddings	Tied input/output embeddings

Training data

Khazri 2 Mini is trained on a custom, pretokenized English-only corpus with a target size of 2,000,000,000 tokens. The corpus is packed into 1,953,125 sequences of 1,024 tokens and stored in Arrow shards with source identifiers.

The documented token budget is:

Source	Token budget	Share	Role
Cosmopedia	850M	42.5%	General English educational and synthetic-text coverage
The Stack v2 Dedup	450M	22.5%	Code from Python, JavaScript, TypeScript, Java, C++, C, Go and Rust
TinyStories	300M	15.0%	Simple narrative language
LIMA	50M	2.5%	Instruction and conversation examples
SYNAPSE synthetic instruction data	350M	17.5%	Arithmetic, context, abstention, web-needed, identity, symbolic-math and general-assistant routes

The listed values are the documented source-token budget. The release manifest should be used for the final source counts of a particular weight revision.

Data processing and safeguards

• Only English text is retained for this corpus. Short records are excluded, very long records are capped, and language/character checks are applied before tokenization.
• The code portion is limited to the eight languages listed above.
• The training mix combines general English, narrative, code, instruction and route-aware synthetic material. This preserves general capabilities while teaching specialized SYNAPSE behaviours.
• Checkpoint evaluation should cover general Q&A, code, arithmetic, context extraction, abstention and current-information requests. A route-specific gain should not be accepted if it degrades general behaviour.
• Original source datasets remain subject to their own terms and licences. Consult their source pages and the model repository licence before use.

Tokenizer

Khazri 2 Mini uses a 32,768-token custom Byte-Level BPE tokenizer. It reserves structural whitespace, chat and SYNAPSE route tokens as single tokens, preserves indentation for code, and uses single-digit splitting to make arithmetic strings more explicit to the model.

Installation

pip install -U torch transformers accelerate safetensors

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Yusiko/khazri-2-mini"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Write a concise explanation of a small language model."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Khazri 2 Preview comparison

The following table reports the project-provided compact-model comparison for Khazri 2 Preview, not Khazri 2 Mini. Higher is better for every listed task.

Model	Parameters	Context extraction	Mixed speed/proxy	Arithmetic	Word problems	Abstention
Khazri 2 Preview	~250M	100%	62%	99%	99%	97.4%
Gemma 3	270M	100%	36%	0%	0%	18%
Qwen 2.5	0.5B	89%	45%	14%	28%	46%
Pythia	160M	22%	10%	0%	2%	1%

These are internal preview results on selected compact-model tests. They are not independently audited and should not be used to make claims about Khazri 2 Mini. Publish prompts, model revisions, scoring rules, hardware and complete evaluation assets with any future benchmark announcement.

Responsible use

Khazri 2 Mini can produce incorrect, incomplete or biased outputs. Evaluate it on your own task, verify material claims and keep a human in the loop for consequential decisions. Do not rely on it as the sole basis for legal, medical, financial, hiring, safety or other high-impact decisions.

Roadmap

Khazri 2 Mini is part of the second Khazri generation. The next planned stage is Khazri 3: a larger parameter scale and stronger results.

Contact

For research, integration or partnership inquiries, visit khazri.dev or email contact@khazri.dev.