Update README.md

README.md

@@ -1,118 +1,149 @@
-<model-card>
-  <model-name>Llama‑3 8B GRPO‑GSM8K‑Merged</model-name>
-  <overview>
-    This model is a **Meta Llama‑3 8B‑Instruct** base checkpoint that was
-    post‑trained with **Group‑wise Reinforcement Policy Optimization (GRPO)**
-    on the **GSM8K grade‑school math dataset**.  We merged the LoRA adapter
-    into the full‑precision base weights, so no external adapter file is
-    required at inference time.
-  </overview>
-
-  <model-details>
-    <architecture>Meta Llama‑3 8B‑Instruct</architecture>
-    <parameter-count>8.03 B (after LoRA merge; same as base)</parameter-count>
-    <training-frameworks>Transformers 4.47, PEFT 0.10, TRL 0.8</training-frameworks>
-    <license>LLAMA 3 Community License v1 + MIT (for GSM8K)</license>
-    <language>English</language>
-  </model-details>
-
-  <dataset>
-    <name>openai/gsm8k&nbsp;("main" split)</name>
-    <size>7 473 train, 1 319 test</size>
-    <data-processing>
-      Each example was converted to a chat‑style prompt with a fixed system
-      instruction that forces the model to wrap its chain‑of‑thought in
-      <thinking>…</thinking> and the final numeric answer in
-      <answer>…</answer>.  The gold integer (extracted from the “####&nbsp;”
-      line) was stored separately for reward computation.
-    </data-processing>
-  </dataset>
-
-  <training-procedure>
-    <algorithm>GRPO (Group‑wise Reinforcement Policy Optimization)</algorithm>
-    <num-generations>2 completions per prompt</num-generations>
-    <num-epochs>3</num-epochs>
-    <learning-rate>2 × 10⁻⁴, cosine schedule, 500 warm‑up steps</learning-rate>
-    <kl-beta>0.02 (KL penalty vs. frozen reference)</kl-beta>
-    <batching>
-      Per‑device batch = 8, grad‑accum = 4 → effective 32
-    </batching>
-    <hardware>
-      Single NVIDIA A100 80 GB; flash‑attention disabled (SDPA used).
-    </hardware>
-    <reward-functions>
-      (1) strict&nbsp;XML‑format (+1/‑1)  
-      (2) no‑tail‑after‑&lt;/answer&gt; (+0/‑0.5)  
-      (3) integer‑parsable (+1/‑1)  
-      (4) answer‑correctness (+1/0)  
-      summed to produce composite reward in [‑3.5, 4].
-    </reward-functions>
-    <stopping-criteria>
-      3 epochs or early stop if validation reward plateaued for 1 000 steps.
-    </stopping-criteria>
-  </training-procedure>
-
-  <evaluation>
-    <metric>Exact integer match on GSM8K test split</metric>
-    <prompt-style>Greedy decoding, max_new_tokens = 128, temperature = 0</prompt-style>
-    <accuracy>TODO %  <!-- replace with your final number after eval rerun --></accuracy>
-    <note>Accuracy uses the integer after the “####” line as ground truth.</note>
-  </evaluation>
-
-  <intended-uses>
-    <primary>
-      • Tutoring or explanation of basic arithmetic word problems.  
-      • As a reward‑model or verifier for chain‑of‑thought tasks.
-    </primary>
-    <out-of-scope>
-      • High‑stakes numeric reasoning (finance, medicine).  
-      • Non‑English math word problems (not in training set).  
-      • Any use that violates the LLAMA 3 or GSM8K licenses.
-    </out-of-scope>
-  </intended-uses>
-
-  <limitations>
-    • Still hallucinates reasoning steps; only the final integer is enforced.  
-    • Trained solely on short grade‑school problems (< 10 sentences).  
-    • No exposure to multi‑digit multiplication/division beyond GSM8K scope.  
-    • English‑only; performance on other languages unverified.
-  </limitations>
-
-  <ethical-considerations>
-    • The chain‑of‑thought text may leak private reasoning if used on sensitive
-      user queries; consider stripping <thinking> tags in production.  
-    • GSM8K is MIT‑licensed public data, but it contains synthetic examples that
-      may reflect unintended cultural bias.
-  </ethical-considerations>
-
-  <how-to-use>
-    <![CDATA[
-import torch
+# Llama 3.1-8B Mathematical Reasoning (GRPO)
+
+## Model Description
+
+This model is a fine-tuned version of [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) trained using **Group Relative Policy Optimization (GRPO)** to enable autonomous mathematical reasoning capabilities. Unlike traditional supervised fine-tuning approaches, this model learned to reason through reinforcement learning without requiring pre-labeled reasoning examples or teacher model distillation.
+
+## Key Features
+
+- **Structured Reasoning**: Generates step-by-step mathematical reasoning enclosed in `<thinking>` tags
+- **Formatted Answers**: Provides final numerical answers in `<answer>` tags  
+- **Autonomous Learning**: Trained using reward functions rather than human-annotated reasoning examples
+- **Mathematical Focus**: Optimized for grade school math word problems requiring multi-step reasoning
+
+## Training Details
+
+### Training Method
+- **Algorithm**: Group Relative Policy Optimization (GRPO)
+- **Base Model**: meta-llama/Meta-Llama-3.1-8B-Instruct
+- **Fine-tuning**: LoRA with NF4 quantization
+- **Dataset**: GSM8K (Grade School Math 8K)
+
+### Training Configuration
+- **LoRA rank**: 8
+- **LoRA alpha**: 32
+- **Learning rate**: 2e-4
+- **Batch size**: 32 (effective)
+- **Generations per prompt**: 2-4
+- **Max completion length**: 128-256 tokens
+- **Temperature**: 0.7
+
+### Reward Functions
+The model was trained using multiple custom reward functions:
+1. **Format compliance**: Ensures proper `<thinking>` and `<answer>` tag structure
+2. **Answer correctness**: Validates numerical accuracy against ground truth
+3. **Integer validation**: Confirms answers are properly formatted integers
+4. **Output cleanliness**: Penalizes extraneous text after answer tags
+
+## Usage
+
+### Required Format
+The model expects a specific system prompt to maintain the structured output format:
+
+```python
+SYSTEM_PROMPT = """
+Respond in the following format and make sure the entire response is wrapped in <reasoning> and <answer> tags with no other text outside of these tags:
+<thinking>
+your reasoning goes here and do not use newlines so that the entire reasoning becomes a single paragraph
+</thinking>
+<answer>
+your answer goes here
+</answer>
+"""
+```
+
+### Example Usage
+
+```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-repo = "your-handle/llama3-gsm8k-grpo-merged"
-tok  = AutoTokenizer.from_pretrained(repo)
-tok.pad_token = tok.eos_token              # safety
-model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto").cuda()
-
-prompt = [
-    {"role":"system", "content":"Respond in the required format."},
-    {"role":"user",   "content":"If a room has 5 chairs and each chair ..."}]
-
-text = tok.apply_chat_template(prompt, add_generation_prompt=True, tokenize=False)
-out  = model.generate(**tok(text, return_tensors="pt").to(model.device),
-                      max_new_tokens=128, temperature=0)
-print(tok.decode(out[0], skip_special_tokens=True))
-    ]]>
-  </how-to-use>
-
-  <citation>
-    @misc{your2025llama3grpo,
-      title        = {Llama‑3 8B GRPO‑finetuned on GSM8K},
-      author       = {Your Name},
-      year         = {2025},
-      howpublished = {HuggingFace repository},
-      url          = {https://huggingface.co/your-handle/llama3-gsm8k-grpo-merged}
-    }
-  </citation>
-</model-card>
+model = AutoModelForCausalLM.from_pretrained("your-username/llama3-grpo-math-reasoning")
+tokenizer = AutoTokenizer.from_pretrained("your-username/llama3-grpo-math-reasoning")
+
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {"role": "user", "content": "Mark is looking to buy a total of 12 pieces of fruit at the store. He has already chosen 3 apples. He has also selected a bunch of bananas containing 4 bananas. How many oranges does he need to pick out to have 12 total pieces of fruit?"}
+]
+
+input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+inputs = tokenizer(input_text, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=128,
+        temperature=0.7,
+        top_p=0.9,
+        do_sample=True
+    )
+
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+**Expected Output:**
+```
+<thinking>
+Mark wants 12 total pieces of fruit. He already has 3 apples and 4 bananas, which gives him 3 + 4 = 7 pieces of fruit. To reach 12 total pieces, he needs 12 - 7 = 5 more pieces of fruit, which would be oranges.
+</thinking>
+<answer>
+5
+</answer>
+```
+
+## Performance
+
+- **Dataset**: GSM8K test split (1,319 examples)
+- **Evaluation Metric**: Exact match accuracy on final numerical answers
+- **Performance**: [Insert actual accuracy from your evaluation]
+
+## Technical Details
+
+### Model Architecture
+- **Base**: Llama 3.1-8B-Instruct
+- **Parameters**: ~8B total, ~21M trainable (LoRA)
+- **Quantization**: NF4 with double quantization
+- **Precision**: bfloat16
+
+### GRPO Specifics
+- **Group size**: 2-4 completions per prompt
+- **KL coefficient (β)**: 0.02
+- **Advantage normalization**: Group-based mean/std normalization
+- **Policy updates**: Advantage-weighted importance sampling
+
+## Limitations
+
+- **Domain**: Primarily trained on grade school mathematics; may not generalize to advanced mathematical concepts
+- **Format dependency**: Requires specific system prompt for optimal performance
+- **Language**: English only
+- **Reasoning scope**: Optimized for multi-step arithmetic rather than complex mathematical proofs
+
+## Ethical Considerations
+
+- **Educational use**: Designed to assist with mathematical learning, not replace mathematical education
+- **Accuracy**: While trained for correctness, users should verify mathematical solutions for critical applications
+- **Bias**: Inherits potential biases from the base Llama model and GSM8K dataset
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{llama3-grpo-math-reasoning,
+  title={Llama 3.1-8B Mathematical Reasoning via Group Relative Policy Optimization},
+  author={[Your Name]},
+  year={2025},
+  howpublished={\\url{https://huggingface.co/your-username/llama3-grpo-math-reasoning}}
+}
+```
+
+## Acknowledgments
+
+- Meta AI for the Llama 3.1 base model
+- OpenAI for the GSM8K dataset
+- Hugging Face TRL library for GRPO implementation
+- The reinforcement learning and mathematical reasoning research communities
+
+## License
+
+This model inherits the license from meta-llama/Meta-Llama-3.1-8B-Instruct. Please refer to the [original model's license](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) for usage terms.