Fix config hyperparameters for 80k architecture and tie embeddings

README.md

@@ -1,3 +1,63 @@
----
-license: mit
----
+---
+language:
+- en
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- tiny-models
+- custom-architecture
+- story-generation
+- experimental
+---
+
+# Spin-80k
+
+**Spin-80k** is a lightweight, 80k-parameter decoder-only language model built from scratch by **Quantech** to demonstrate custom Transformer architecture 
+
+---
+
+## Model Specifications
+
+* **Organization:** Quantech
+* **Architecture:** Custom Decoder-only Transformer
+* **Total Parameters:** ~80,112
+* **Layers:** 2
+* **Hidden Dimension ($d_{\text{model}}$):** 48
+* **Attention Heads:** 4
+* **Feed-Forward Dimension ($d_{\text{ff}}$):** 128
+* **Positional Encoding:** Rotary Position Embeddings (RoPE)
+* **Normalization:** RMSNorm ($\epsilon = 10^{-5}$)
+* **Activation:** SwiGLU
+* **Vocabulary:** 512 Byte-Pair Encoding (BPE) tokens
+* **Context Length:** 256 tokens
+
+---
+
+## Quickstart
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "Quantech/spin-80k"
+
+# Load Tokenizer & Model
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
+model.eval()
+
+# ChatML Format
+prompt = "<|im_start|>user\nWrite a short story about a dog.<|im_end|>\n<|im_start|>assistant\n"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=50,
+        temperature=0.7,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+print(tokenizer.decode(outputs[0]))