| --- |
| license: apache-2.0 |
| --- |
| # NanoRush Chat |
|
|
| NanoRush Chat is a 283M parameter GPT-style causal language model fine-tuned for conversational AI. |
|
|
| Github- https://github.com/Amogh1221/NanoRush |
|
|
| Live- https://nano-chat-web.vercel.app |
|
|
| ## Model Details & Configuration |
| | Detail | Value | |
| | --- | --- | |
| | **Parameters** | 283M | |
| | **Architecture** | GPT-2 style | |
| | **Precision** | FP16 / BFloat16 | |
| | **Context Window** | Up to 4096 tokens | |
| | **Vocabulary Size** | 32,768 | |
| | **Embedding Dimension (n_embd)** | 768 | |
| | **Number of Heads (n_head)** | 12 | |
| | **Number of Layers (n_layer)** | 36 | |
| | **Base Model** | Custom pre-trained | |
| | **Pre-Training Dataset** | `HuggingFaceTB/cosmopedia` | |
| | **Fine-tuning Dataset** | `HuggingFaceTB/smoltalk` | |
| |
| ## Evaluation Results |
| |
| The model was evaluated using standard zero-shot accuracy metrics. |
| |
| | Groups / Tasks | Version | n-shot | Metric | Value | Stderr | |
| |---|---|---|---|---|---| |
| | **mmlu** | 2 | 0 | acc | 0.2297 | ± 0.0035 | |
| | - humanities | 2 | 0 | acc | 0.2438 | ± 0.0063 | |
| | - other | 2 | 0 | acc | 0.2375 | ± 0.0076 | |
| | - social sciences | 2 | 0 | acc | 0.2184 | ± 0.0074 | |
| | - stem | 2 | 0 | acc | 0.2119 | ± 0.0073 | |
| | **arc_challenge** | 1 | 0 | acc | 0.2295 | ± 0.0123 | |
| | **hellaswag** | 1 | 0 | acc | 0.3116 | ± 0.0046 | |
| | **truthfulqa_mc2** | 3 | 0 | acc | 0.4320 | ± 0.0153 | |
| | **winogrande** | 1 | 0 | acc | 0.5107 | ± 0.0140 | |
| |
| |
| ## Usage |
| |
| This model has been exported to be fully compatible with the Hugging Face `transformers` library. |
| You can load it using the standard `AutoModelForCausalLM` pipeline. |
| |
| ### Installation |
| Make sure you have the latest version of the `transformers` and `torch` libraries installed: |
| ```bash |
| pip install torch transformers |
| ``` |
| |
| ### Example Code |
| |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from transformers import StoppingCriteria, StoppingCriteriaList |
| from transformers.generation.streamers import TextIteratorStreamer |
| import threading |
| |
| # Load the model and tokenizer from Hugging Face |
| model_id = "Amogh1221/nano-chat" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| torch_dtype=torch.float32, |
| device_map="cpu" |
| ) |
| system_prompt = |
| """ |
| You are NanoRush, an AI assistant, you are a helpful, respectful, and intelligent conversational |
| partner. You must never pretend to be a human, and you must carefully pay attention to the conversation history. |
| """ |
| |
| class StopOnUser(StoppingCriteria): |
| def __init__(self, prompt_length): |
| self.prompt_length = prompt_length |
| def __call__(self, input_ids, scores, **kwargs): |
| generated_tokens = input_ids[0][self.prompt_length:] |
| tail = tokenizer.decode(generated_tokens[-10:]) |
| return "\nUser:" in tail or "User:" in tail |
| |
| # Format your prompt |
| prompt = f"System: {system_prompt}\\n\\nUser: What is Quantum Computing?\\nAssistant:" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
| stop_criteria = StoppingCriteriaList([StopOnUser(prompt_length=inputs["input_ids"].shape[1])]) |
| streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) |
|
|
| generation_kwargs = dict( |
| **inputs, |
| max_new_tokens=512, |
| temperature=0.7, |
| top_k=50, |
| top_p=0.9, |
| do_sample=True, |
| repetition_penalty=1.15, |
| pad_token_id=tokenizer.eos_token_id, |
| prompt_lookup_num_tokens=3, |
| stopping_criteria=stop_criteria, |
| streamer=streamer, |
| ) |
| |
| # Run generation in a background thread |
| thread = threading.Thread(target=model.generate, kwargs=generation_kwargs) |
| thread.start() |
| |
| print("Assistant: ", end="") |
| for text in streamer: |
| print(text, end="", flush=True) |
| print() |
| ``` |
| |
