1111AI-Tamil1B-Instruct

A 1.24B-parameter Tamil-first language model trained from scratch — not a multilingual model adapted to Tamil, and not an English model with Tamil tokens bolted on. Tokenizer, corpus and weights are all Tamil-native from the start.

This is the instruction-tuned variant. The base model is mvbalaji/1111AI-Tamil1B.

Read the Limitations section before using this. This model has two well-characterised failure modes and will produce visibly broken output if you call generate() without the settings below. They are baked into generation_config.json, so the defaults are correct — but if you override them, override them deliberately.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("mvbalaji/1111AI-Tamil1B-Instruct")
model = AutoModelForCausalLM.from_pretrained("mvbalaji/1111AI-Tamil1B-Instruct")

msgs = [{"role": "user", "content": "தமிழ்நாட்டின் தலைநகரம் எது?"}]
text = tok.apply_chat_template(msgs, tokenize=False)
ids = tok(text, return_tensors="pt")

out = model.generate(**ids, max_new_tokens=128, tokenizer=tok)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

The tokenizer must not prepend BOS (add_bos_token: false, already set). The model never saw <s> during pretraining or SFT; prepending it pushes the model out of distribution and output degenerates into multilingual noise. This is the single most likely way to get garbage out of this model.

Model details

Parameters	1,236,076,544 (1.12B non-embedding)
Architecture	Llama-style decoder-only
Layers / d_model	16 / 2048
Attention	16 heads, 8 KV heads (GQA), head_dim 128
FFN	SwiGLU, 8192 intermediate
Norm	RMSNorm (fp32 reduction), eps 1e-5
Positional	RoPE, θ = 500,000
Context length	2048
Vocabulary	56,000 (SentencePiece, Tamil-native)
Embeddings	tied
Precision	bfloat16

Weights are a bit-exact conversion of the training checkpoint: the exporter verifies max |logit delta| = 0 against the original implementation in fp32.

Training

Pretraining. ~57.6B tokens seen (3 epochs over a 19.05B-token corpus), ending at step 27,455 with a learning-rate decay anneal. Global batch 2,097,152 tokens, seq_len 2048, lr 3e-4, 2k warmup, AdamW (0.9, 0.95), weight decay 0.1, grad clip 1.0, bfloat16.

Corpus composition (19.05B train tokens, 99.0M validation):

bucket	tokens	share
Tamil (native script)	9.03B	47.4%
Tanglish (romanised)	6.69B	35.1%
English	3.33B	17.5%

Instruction tuning. 11,261 QA pairs — 8,867 filtered authentic-Tanglish pairs plus 2,394 general-QA pairs — full-parameter fine-tune, prompt tokens masked to -100. Final validation loss 8.98.

The SFT set is small by design: authentic Tanglish is inherently scarce (on the order of tens of MB worldwide), so scaling this track is bounded by data that does not exist, not by compute.

Tokenizer

56,000-piece SentencePiece model trained on the same corpus. Fertility (tokens per word, lower is better):

	fertility	target
Tamil	1.962	2.00	✅ met
Tanglish	2.296	1.60	❌ missed
English	1.526	1.40	❌ missed

The Tanglish and English misses are expected rather than defects: romanised Tamil has no orthographic standard, so a subword vocabulary cannot compress it the way it compresses native script.

Limitations

1. Repetition loops. The base model degenerates into repetition under plain top-k sampling on out-of-distribution prompts. This is a property of the base model, not of the fine-tune — it was verified directly on the un-tuned checkpoint. repetition_penalty=1.3 is set in generation_config.json and is not optional in practice.

2. Answers drift and do not stop. Every SFT response was a complete extracted source passage, so the model never saw an example of answering and then stopping. Left to run, it keeps going and drifts off topic — often starting a fresh Question: of its own. stop_strings in the generation config cuts this. A proper fix requires retraining on concise answers; it is a data problem, not a sampling one.

3. Factual accuracy. At 1.24B, factual reliability is limited. It answers well-known questions (state capitals, major literary figures) reasonably and is unreliable beyond that. It has no retrieval and no tool use; everything comes from parametric memory with a fixed training cutoff.

4. Tanglish register skew. Roughly 93% of held-out Tanglish tokens are web-scraped tam_Latn, which skews adult and long-form. This shapes what the model produces in Tanglish — see Safety.

5. Context length 2048. Short by current standards.

Safety

This model has a known, measured content issue in Tanglish, and no safety tuning has been applied.

A spot check during pretraining found that a mild Tanglish prompt produced explicit sexual content. A parallel prompt in Tamil script did not. Corpus-wide scanning then quantified it: of 19,194,543 Tanglish-bucket documents, 55,695 were flagged (0.29% raw) — but natural-origin Tanglish flags at 3.19% versus 0.27% for synthetically transliterated text, roughly 11.7× more likely.

No corpus documents were removed on the basis of that scan; it was a measurement, not a filter. The flagged content is therefore in the training data of this model.

Part of the authentic Tanglish also derives from offensive-language shared tasks and is abusive by construction — a small fraction by word count, present in both training and validation.

There is no Tamil or Tanglish safety benchmark to evaluate against. To our knowledge one does not meaningfully exist, which is itself a gap worth naming. This model has consequently not been evaluated for safety in either language.

Dual use. A Tamil/Tanglish generative model could be used for disinformation or harassment in a language with comparatively little automated moderation. We regard this as a real risk.

Do not deploy this model in a user-facing product without your own content filtering.

Privacy

Emails, Indian phone-number formats and Aadhaar-like identifiers were masked in the corpus. This is mitigation, not a guarantee — web text at this scale certainly retains personal information in forms those patterns do not match.

Data provenance and licence

The corpus mixes sources under differing licences and is released as a derived mixture; downstream users must review upstream terms. An in-copyright film-lyrics dataset was excluded; a site declaring Content-Signal: ai-train=no was not crawled; YouTube transcripts were not extracted.

⚠️ The corpus licence has not yet been confirmed by counsel. The licence field on this repository is a placeholder and will be updated. Treat the licensing status of these weights as unresolved until it is.

Intended use

Research into Tamil and code-mixed language modelling, tokenizer design for Indic scripts, and as a base for further fine-tuning. It is not a general-purpose assistant and is not suitable for production deployment as-is.

Citation

A resource paper describing the corpus, tokenizer and a negative result on synthetic transliteration is in preparation. Citation details to follow.

---

Built by Balaji (mvbalaji) · 1111AI