Upload README.md with huggingface_hub

README.md

@@ -14,70 +14,66 @@ tags:
 license: mit
 ---
 
-# MANAS (Model for Awadhi Natural Autoregressive Sequences)
+# MANAS
+### Model for Awadhi Natural Autoregressive Sequences
 
-## Model Description
-**MANAS (Model for Awadhi Natural Autoregressive Sequences)** is a custom **Character-level Causal Transformer (Language Model)** trained exclusively on the classical poetry of Goswami Tulsidas (including the *Ramcharitmanas* and *Vinayapatrika*). 
+MANAS is a small experimental **character-level causal Transformer** trained on a corpus centered on the literary works of Goswami Tulsidas.
 
-Unlike modern token-based LLMs (which use subwords/BPE), this model processes and predicts text **one character at a time**. It learned the grammar, vocabulary, and poetic meter of 16th-century Awadhi entirely from scratch, mapping complex Devanagari characters into coherent poetry.
+The model operates directly on individual Unicode characters rather than using a subword tokenizer such as BPE. The project is intended primarily as an educational and research experiment for understanding how Transformer language models learn character-level patterns in classical Devanagari literary text.
 
-- **Parameters:** ~10 Million
-- **Vocab Size:** 101 (Pure Devanagari, punctuation, numbers. 0 Latin characters)
-- **Architecture:** GPT-style Decoder-only Transformer
-- **Language:** Awadhi, Braj Bhasha
-- **Format:** PyTorch (`.pth` state_dict)
+## Model Details
 
-## Model Architecture
-The model was built using PyTorch with the following hyperparameter configuration:
-- `n_embd` (Embedding Dimension): 384
-- `n_layer` (Transformer Blocks): 6
-- `n_head` (Attention Heads): 6
-- `block_size` (Context Window): 256 characters
-- `dropout`: 0.2
+- **Architecture:** GPT-style decoder-only causal Transformer
+- **Parameters:** ~10.9M
+- **Embedding dimension:** 384
+- **Transformer layers:** 6
+- **Attention heads:** 6
+- **Context window:** 256 characters
+- **Dropout:** 0.2
+- **Vocabulary size:** 101 characters
+- **Framework:** PyTorch
+- **Checkpoint:** `tulsidas_model.pth`
+
+## Corpus
+
+The current corpus contains approximately **2.83 million characters** and is focused on Tulsidas-related literary material.
+
+The project currently includes material associated with works such as:
+- Ramcharitmanas
+- Vinay Patrika
+- Kavitavali
+- Parvatimangal
+- additional works contained in the Tulsi-Granthavali compilation
+
+The corpus is character-level and contains no Latin alphabet characters. However, the current corpus should be considered an **experimental OCR-derived corpus**, and some source-extraction/OCR errors or non-literary material may remain.
+
+Future dataset revisions will improve:
+- OCR accuracy
+- literary-text extraction
+- removal of commentary and publication material
+- Unicode cleanup
+- work-level segmentation
+- manual page verification
 
 ## Intended Use
-This model is built as an experimental, educational exploration into character-level causal modeling for low-resource classical Indian dialects.
-- **Text Generation:** Given a prompt in Devanagari (e.g., `श्री राम`), the model will autoregressively generate poetry in the style of Tulsidas.
-- **Linguistic AI Research:** Useful for observing how transformers learn Devanagari conjuncts (मात्राएँ and संयुक्ताक्षर) without the crutch of subword tokenizers.
-
-## Training Data
-The model was trained on the `Awadhi-Tulsidas-Literary-Corpus`, a heavily purified 2.8 million character dataset. The training set is 100% devoid of English and Latin characters to prevent domain confusion.
-- **Train tokens:** 2,550,420
-- **Validation tokens:** 283,381
-
-## Training Procedure
-- **Optimizer:** AdamW
-- **Learning Rate:** 3e-4
-- **Batch Size:** 64
-- **Training Steps:** 15,000 iterations
-- **Hardware:** Trained on single consumer GPU.
-
-## How to Use (Inference)
-Because the model vocabulary does not contain English, **all inputs must be in Devanagari script.** 
-*(An ITRANS-to-Devanagari transliteration layer is recommended for users typing on English keyboards).*
-
-```python
-import torch
-from chat import LanguageModel, encode, decode
-
-device = 'cuda' if torch.cuda.is_available() else 'cpu'
-
-# 1. Initialize model
-model = LanguageModel()
-model.load_state_dict(torch.load('tulsidas_model.pth', map_location=device))
-model.to(device)
-model.eval()
-
-# 2. Prompt in Devanagari
-prompt = "श्री राम"
-context = torch.tensor([encode(prompt)], dtype=torch.long, device=device)
-
-# 3. Generate
-generated_tokens = model.generate(context, max_new_tokens=200)[0].tolist()
-print(decode(generated_tokens))
-```
-
-## Limitations & Bias
-- **Hallucinations:** As a 10M parameter character-level model, it occasionally invents novel (but morphologically plausible) Awadhi words.
-- **Grammar drift:** Over long generations (>500 chars), the model may lose semantic track, though it heavily preserves phonetic and rhythmic flow.
-- **Zero Modern Knowledge:** This model cannot answer questions about the modern world, nor can it speak conversational modern Hindi. It exists solely to predict 16th-century poetic strings.
+
+MANAS is intended for:
+- educational exploration of Transformer language models
+- character-level language-model research
+- experimentation with Devanagari text generation
+- studying learned character and sequence patterns in Tulsidas-related literature
+
+## Inference Note
+Since `chat.py` and the vocabulary arrays are not included in this minimal repository, inference requires manually reconstructing the `LanguageModel` architecture in PyTorch (6 layers, 6 heads, 384 embed dim, 101 vocab size) and loading `tulsidas_model.pth`. It also requires the 101-character mapping array derived from the dataset to properly encode/decode.
+
+## Limitations
+- **Small model:** At approximately 10.9M parameters, MANAS is a small experimental language model and should not be compared directly with modern large language models.
+- **Character-level generation:** Character-level modeling allows the experiment to operate directly on Devanagari characters, but it is less efficient than modern subword tokenization.
+- **Corpus quality:** The current corpus was assembled from scanned literary sources and is still undergoing OCR and extraction-quality improvements. Some OCR errors, editorial material, and commentary may remain.
+- **Long-range coherence:** Generation quality can degrade during long samples because the model has limited capacity and a 256-character context window.
+- **No general knowledge:** The model is not designed to answer modern factual questions or function as a general-purpose conversational model.
+
+## Project Status
+**MANAS is an experimental educational model (Checkpoint 1).**
+
+The next major improvement is a rebuilt, page-verified corpus using OCR for scanned sources with unreliable PDF text layers, followed by cleaner literary-text segmentation and retraining.