Create README.md

README.md

@@ -0,0 +1,246 @@
+---
+language:
+- tr
+- en
+license: apache-2.0
+base_model: Qwen/Qwen2.5-1.5B
+tags:
+- axiom
+- qwen
+- qwen2
+- fine-tuned
+- lora
+- sft
+- trl
+- code
+- python
+- text-generation
+pipeline_tag: text-generation
+model_type: qwen2
+library_name: transformers
+---
+
+# Axiom Python 1.5B
+
+**Axiom Python 1.5B** is a text generation (causal language model) fine-tuned on [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) with a focus on Python programming and code generation.
+
+The model was trained using **LoRA + SFT** with the [TRL](https://github.com/huggingface/trl) library on the [CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K) and [PythonCodeInstruct_18K](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca) datasets.
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Base Model | [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) |
+| Architecture | Qwen2ForCausalLM |
+| Parameters | ~1.5B |
+| Hidden Layers | 28 |
+| Hidden Size | 1536 |
+| Attention Heads | 12 |
+| KV Heads | 2 |
+| Vocabulary Size | 151936 |
+| Max Context Length | 131072 |
+| Weight Dtype | float16 (FP16) |
+| Training Method | LoRA (r=16, alpha=32) + SFT |
+| Datasets | CodeAlpaca_20K + PythonCodeInstruct_18K |
+| Languages | Turkish and English (code-focused) |
+
+## Installation
+
+Install the following packages to get started:
+
+```bash
+pip install transformers torch
+```
+
+> If you are using a GPU, make sure you have installed a CUDA-compatible PyTorch version.
+
+## Usage
+
+### 1. Using `pipeline` (Simplest Way)
+
+```python
+from transformers import pipeline
+
+generator = pipeline(
+    "text-generation",
+    model="coderian/axiom-python-1.5B",
+    device_map="auto",
+    torch_dtype="auto",
+)
+
+prompt = """### Instruction:
+Write a Python function that reverses the elements of a list.
+
+### Answer:
+"""
+
+output = generator(
+    prompt,
+    max_new_tokens=256,
+    temperature=0.7,
+    top_p=0.9,
+    do_sample=True,
+)
+
+print(output[0]["generated_text"])
+```
+
+### 2. Using `AutoModelForCausalLM`
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "coderian/axiom-python-1.5B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+model.eval()
+
+prompt = """### Instruction:
+Write a Python function that adds two numbers.
+
+### Answer:
+"""
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=256,
+        temperature=0.7,
+        top_p=0.9,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id,
+    )
+
+response = tokenizer.decode(
+    outputs[0][inputs["input_ids"].shape[1]:],
+    skip_special_tokens=True,
+)
+
+print(response)
+```
+
+### 3. Using the Chat Template
+
+Since the Qwen2.5 tokenizer supports the ChatML format, you can also use the model for chat-style conversations:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "coderian/axiom-python-1.5B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+messages = [
+    {"role": "system", "content": "You are Axiom, a helpful Python coding assistant."},
+    {"role": "user", "content": "Write a Python function to check if a number is prime."},
+]
+
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=256,
+        temperature=0.7,
+        top_p=0.9,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id,
+    )
+
+response = tokenizer.decode(
+    outputs[0][inputs["input_ids"].shape[1]:],
+    skip_special_tokens=True,
+)
+
+print(response)
+```
+
+### Recommended Generation Parameters
+
+| Parameter | Suggested Value | Description |
+|---|---|---|
+| `max_new_tokens` | `512` | Maximum number of new tokens to generate |
+| `temperature` | `0.7` | Lower values produce more deterministic output |
+| `top_p` | `0.9` | Nucleus sampling ratio |
+| `do_sample` | `True` | Enable/disable sampling |
+| `repetition_penalty` | `1.05` | Reduces repetitive output |
+
+## Training Details
+
+| Setting | Value |
+|---|---|
+| Base Model | Qwen/Qwen2.5-1.5B |
+| LoRA Rank (r) | 16 |
+| LoRA Alpha | 32 |
+| LoRA Dropout | 0.05 |
+| Target Modules | q_proj, v_proj |
+| Batch Size | 32 (2 x 4 grad. accumulation) |
+| Training Epochs | 1 |
+| Learning Rate | 2e-4 |
+| Optimizer | AdamW (fused) |
+| Precision | FP16 |
+| Steps | 4000 |
+| Max Sequence Length | 256 |
+| Adapter Location | `axiom-python-1.5B/checkpoint-4000` |
+
+After training, the LoRA adapter was merged into the base model and released as a single file. You can also load the adapter directly using the peft library:
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen2.5-1.5B",
+    torch_dtype="auto",
+    device_map="auto",
+)
+
+model = PeftModel.from_pretrained(base, "path/to/adapter")
+```
+
+## Limitations
+
+- It is a small 1.5B parameter model and may make mistakes on very complex and long code generation tasks.
+- It was trained only on Python-focused datasets; performance in other languages is limited.
+- The training data has a maximum length of 256 tokens; consistency may degrade in very long contexts.
+- Generated code may not always be correct or safe. Review it before running.
+- It may contain known limitations inherited from the training data regarding bias and harmful content.
+
+## Intended Usage Tips
+
+- It performs best on single-line and medium-complexity Python functions.
+- Lower the `temperature` value if you want stable output for code generation.
+- Since the model was trained in a completion format, the `### Instruction:` / `### Answer:` template yields the highest quality output.
+- For batched inference, remember to set `tokenizer.pad_token = tokenizer.eos_token`.
+
+## License
+
+The base model Qwen2.5 is released under the Apache-2.0 license, and this model is also shared under the **Apache-2.0** license.
+
+## Resources
+
+- Base Model: [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
+- Training Library: [TRL](https://github.com/huggingface/trl)
+- Dataset 1: [HuggingFaceH4/CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K)
+- Dataset 2: [iamtarun/python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)