| --- |
| language: [hi, en] |
| license: mit |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: [hindi, hinglish, gpt, from-scratch, semantic-parsing, safetensors] |
| datasets: [wikimedia/wikipedia, findnitai/english-to-hinglish] |
| --- |
| |
| # TinyGPT — a Hindi & Hinglish GPT, built from scratch |
|
|
| Two small models from [TinyGPT](https://github.com/TanushLukha/TinyGPT), where every part — the BPE |
| tokenizer, the Transformer, training, decoding, LoRA and the serving code — is written by hand in PyTorch. |
|
|
| | folder | model | what it does | |
| |---|---|---| |
| | `lm/` | `gpt-llama`, 13.8M parameters | continues Hindi or Hinglish text | |
| | `parser/` | `gpt-llama-clean` fully fine-tuned on Hinglish-TOP | turns a Hinglish request to a voice assistant into intent + slots | |
|
|
| **Architecture:** decoder-only Transformer, Llama recipe (RoPE, RMSNorm, SwiGLU), 6 layers, |
| d_model 384, 6 heads, context 256 tokens, tied embeddings; |
| 8,192-token byte-level BPE with a Devanagari-aware pre-tokeniser. |
| |
| **Training data:** 18.2M tokens — Hindi Wikipedia (`wikimedia/wikipedia`, 20231101.hi, CC BY-SA) and |
| Hinglish sentences (`findnitai/english-to-hinglish`, Apache-2.0). The parser: the human-annotated part of |
| [Hinglish-TOP](https://github.com/google-research-datasets/Hinglish-TOP-Dataset) (Apache-2.0), 2,993 |
| requests; its base model was pretrained without Hinglish-TOP's validation and test requests. |
| |
| ## Scores |
| |
| | | | |
| |---|---| |
| | `lm`: bits per byte, held-out text | 0.733 (Hindi 0.689, Hinglish 1.147) | |
| | `parser`: exact match (whole tree right), test, 6,513 requests | 41.0% | |
| | `parser`: intent accuracy, test | 83.9% | |
| | `parser`: slot F1, test | 56.8% | |
| | `parser`: exact match, validation | 51.0% | |
| |
| ## Use |
| |
| ```python |
| # pip install torch safetensors "git+https://github.com/TanushLukha/TinyGPT" (the model's code) |
| from huggingface_hub import snapshot_download |
| from tinygpt.serve.engine import TextGenerator, Parser, GenerateParams |
| |
| path = snapshot_download("tanushlukha/tinygpt") |
| lm = TextGenerator.from_bundle(f"{path}/lm") |
| print("".join(p.text for p in lm.stream("भारत की राजधानी", GenerateParams(max_new_tokens=60)) |
| if hasattr(p, "text"))) |
| print(Parser.from_bundle(f"{path}/parser").parse(["kal subah 7 baje ka alarm laga do"])[0].parse) |
| ``` |
| |
| Each folder holds `model.safetensors`, `config.json` (architecture, task, provenance, scores, SHA-256 of |
| each file — checked on load) and `tokenizer.json`. |
| |
| ## Limitations |
| |
| A 13.8M-parameter model trained for 53 minutes on a laptop: it writes fluent Hindi |
| but has learned the language, not the facts — names, dates and places in its text are invented |
| ([what it knows](https://github.com/TanushLukha/TinyGPT/blob/main/docs/results/m6_facts.md)) — and greedy |
| decoding repeats itself. The parser gets fewer than half of |
| unseen requests exactly right; its most common mistake is a slot missing, extra or mislabelled, and about |
| one mistake in five puts words in a slot that the request doesn't contain. For learning and demos, not for |
| products. |
| |
