Update README.md

README.md

@@ -23,60 +23,18 @@ tags:
   - parameter-efficient-fine-tuning
 ---
 
-\---
+# Qwen2.5-Coder-7B-Instruct — DPO Fine-Tuned for Python**
 
-language:
-
-  \- en
-
-license: apache-2.0
-
-library_name: transformers
-
-pipeline_tag: text-generation
-
-base_model:
-
-  \- Qwen/Qwen2.5-Coder-7B-Instruct
-
-datasets:
-
-  \- bigcode/bigcodebench
-
-  \- Muennighoff/mbpp
-
-  \- openai/openai_humaneval
-
-tags:
-
-  \- qwen2.5
-
-  \- qwen
-
-  \- code-generation
-
-  \- python
-
-  \- python-code-generation
+**## Model at a Glance**
 
-  \- dpo
-
-  \- direct-preference-optimization
-
-  \- preference-learning
-
-  \- lora
-
-  \- parameter-efficient-fine-tuning
-
-\---
-# Qwen2.5-Coder-7B-Instruct — DPO Fine-Tuned for Python## Model at a Glance
 This model is a ****Python-focused, DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct****, designed to improve the model's ability to generate ****functionally correct Python code****.
 
 The model was trained using ****Direct Preference Optimization (DPO)**** with automatically constructed preference pairs. Candidate Python solutions were generated for programming problems and evaluated by executing them against unit tests. Solutions that passed the required tests were used as preferred examples, while solutions that failed tests were used as rejected examples.
 
 The resulting model was evaluated against the original Qwen2.5-Coder-7B-Instruct model on held-out programming problems.
-### What changed?
+
+**### What changed?**
+
 The base model:
 
 ****Qwen2.5-Coder-7B-Instruct****
@@ -87,28 +45,32 @@ was fine-tuned using:
 
 The resulting model demonstrates improved coding performance on the evaluated held-out benchmark:
 
-| Metric | Qwen2.5-Coder-7B-Instruct | DPO Model | Improvement |
+\| Metric                  | Qwen2.5-Coder-7B-Instruct |  DPO Model |  Improvement |
 
-| ----------------------- | ------------------------: | ---------: | -----------: |
+\| ----------------------- | ------------------------: | ---------: | -----------: |
 
-| Pass@1 | 41.87% | ****45.22%**** | ****+3.35 pp**** |
+\| Pass@1                  |                    41.87% | ****45.22%**** | ****+3.35 pp**** |
 
-| Pass@5 | 48.87% | ****51.10%**** | ****+2.23 pp**** |
+\| Pass@5                  |                    48.87% | ****51.10%**** | ****+2.23 pp**** |
 
-| Pass@10 | 50.99% | ****53.16%**** | ****+2.17 pp**** |
+\| Pass@10                 |                    50.99% | ****53.16%**** | ****+2.17 pp**** |
 
-| Tests passed per answer | 66.20% | ****69.41%**** | ****+3.21 pp**** |
+\| Tests passed per answer |                    66.20% | ****69.41%**** | ****+3.21 pp**** |
 
 The primary result is the improvement in ****Pass@1****, meaning that a single generated solution was more likely to produce a correct solution on the evaluated programming problems.
 
 The model also showed improved Pass@5, Pass@10, and partial test coverage.
 
----
+**---**
 
 ****Base model:**** \`Qwen/Qwen2.5-Coder-7B-Instruct\` at snapshot
 
-\`c03e6d358207e414f1eca0bb1891e29f1db0e242\`. The merge was performed against exactly these weights.
-## Downloading and using### The merged model (what most people want)
+\`c03e6d358207e414f1eca0bb1891e29f1db0e242\`. The adapter was trained against exactly these weights.
+
+**## Downloading and using**
+
+**### The LoRA adapter (recommended release)**
+
 \`\`\`python
 
 from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -136,7 +98,9 @@ Or fetch the files without loading them (~14 GB, four shards):
 hf download ranjanrajib/DPO-trained-Qwen2.5-Coder-7B --exclude "adapter/*" --local-dir ./d4
 
 \`\`\`
-### The LoRA adapter only (308 MB)
+
+**### The LoRA adapter only (308 MB)**
+
 The adapter lives in \`adapter/\`. Applying it to the base model gives the same result as the
 
 merged weights, at a fraction of the download.
@@ -169,24 +133,11 @@ model = PeftModel.from_pretrained(
 
 \`\`\`
 
----
-
-**# ## VRAM Requirements
-
-The base model and the merged trained model use the same VRAM. The merge changes weight values, not tensor shapes: both models have the same 7.6B parameters and architecture, so the model weights occupy the same amount of bf16 memory.
+**---**
 
-| Component | bf16 VRAM |
-| --- | ---: |
-| Weights (base or merged D4) | 14.19 GiB |
-| KV cache, per token | 56 KiB |
-| Activations / workspace | ~0.5–1 GiB |
-| **Practical single-stream total** | **~15–16 GiB** |
+**# Why Use This Model?**
 
----
-
-Why Use This Model?**
-
-If your primary use case is ****Python code generation****, this model is intended to be a stronger alternative to the original Qwen2.5-Coder-7B-Instruct model based on the evaluation performed in this project.
+If your primary use case is ****Python code generation****, this model is intended as an experimental alternative to the original Qwen2.5-Coder-7B-Instruct model, with higher measured performance on the held-out benchmark used in this project.
 
 It is particularly relevant for users who want to:
 
@@ -210,8 +161,10 @@ It was to encourage the model toward ****solutions that work****.
 
 The preference data therefore used actual program execution and unit-test results rather than subjective judgments about whether generated code **looked** correct.
 
----
-# How Is It Different From Qwen2.5-Coder-7B-Instruct?
+**---**
+
+**# How Is It Different From Qwen2.5-Coder-7B-Instruct?**
+
 The underlying model architecture and capabilities come from ****Qwen2.5-Coder-7B-Instruct****.
 
 This model adds a preference-optimization stage on top of that model.
@@ -275,85 +228,121 @@ Qwen2.5-Coder-7B-Instruct
 \`\`\`
 
 The goal is to shift the model toward coding behaviors associated with solutions that successfully execute against the available tests.
-### Important distinction
+
+**### Important distinction**
+
 This is ****not a new foundation model****.
 
 It is a parameter-efficient fine-tuned adapter for Qwen2.5-Coder-7B-Instruct.
 
 The base model remains unchanged.
 
----
-# What Kind of Model Is This?### Primary task
+**---**
+
+**# What Kind of Model Is This?**
+
+**### Primary task**
+
 ****Python code generation****
-### Model family
+
+**### Model family**
+
 Qwen2.5-Coder
-### Base model
+
+**### Base model**
+
 \`Qwen/Qwen2.5-Coder-7B-Instruct\`
-### Fine-tuning
+
+**### Fine-tuning**
+
 Direct Preference Optimization (DPO)
-### Parameter-efficient method
+
+**### Parameter-efficient method**
+
 LoRA
-### Precision
+
+**### Precision**
+
 bfloat16
-### Quantization
+
+**### Quantization**
+
 None
-### Model size
+
+**### Model size**
+
 7.7B-parameter base model
-### Adapter
+
+**### Adapter**
+
 Approximately 80.7M trainable parameters
 
----
-# Performance
+**---**
+
+**# Performance**
+
 The model was evaluated against the original Qwen2.5-Coder-7B-Instruct using a controlled benchmark consisting of 508 held-out BigCodeBench programming problems.
 
 The same prompts, generation settings, random seeds, number of samples, and grading procedure were used for both models.
-## Pass@k
-| Metric | Base Model | DPO Model |
 
-| ------- | ---------: | ---------: |
+**## Pass@k**
 
-| Pass@1 | 41.87% | ****45.22%**** |
+\| Metric  | Base Model |  DPO Model |
 
-| Pass@5 | 48.87% | ****51.10%**** |
+\| ------- | ---------: | ---------: |
 
-| Pass@10 | 50.99% | ****53.16%**** |
+\| Pass@1  |     41.87% | ****45.22%**** |
+
+\| Pass@5  |     48.87% | ****51.10%**** |
+
+\| Pass@10 |     50.99% | ****53.16%**** |
 
 The improvement in Pass@1 is approximately ****3.35 percentage points****.
-## Partial Correctness
+
+**## Partial Correctness**
+
 The average proportion of tests passed by each generated answer increased from:
 
 ****66.20% → 69.41%****
 
 This is a secondary metric intended to provide additional information about functional correctness beyond binary problem-level pass/fail.
 
----
-# Paired Problem-Level Analysis
+**---**
+
+**# Paired Problem-Level Analysis**
+
 The evaluation used the same 508 problems for both models, allowing direct comparison of model behavior on individual problems.
 
 For the first generated answer:
 
-| Base → DPO | Problems |
+\| Base → DPO            | Problems |
 
-| --------------------- | -------: |
+\| --------------------- | -------: |
 
-| Correct → Correct | 186 |
+\| Correct → Correct     |      186 |
 
-| Correct → Incorrect | 17 |
+\| Correct → Incorrect   |       17 |
 
-| Incorrect → Correct | 41 |
+\| Incorrect → Correct   |       41 |
 
-| Incorrect → Incorrect | 264 |
+\| Incorrect → Incorrect |      264 |
 
 The DPO model converted 41 problems from incorrect to correct while 17 previously correct problems became incorrect.
 
 This provides additional evidence that the aggregate improvement is associated with meaningful changes in problem-level behavior rather than simply differences in the evaluated problem sets.
 
----
-# Who Should Use This Model?
+**---**
+
+**# Who Should Use This Model?**
+
 This model is most appropriate for:
-### Python Developers
+
+**### Python Developers**
+
 Developers interested in experimenting with a locally hosted code-generation model that has been specifically fine-tuned toward execution-validated solutions.
-### AI/ML Researchers
+
+**### AI/ML Researchers**
+
 Researchers studying:
 
 * DPO
@@ -369,7 +358,9 @@ Researchers studying:
 * execution-based evaluation
 
 * parameter-efficient fine-tuning
-### LLM Evaluation Researchers
+
+**### LLM Evaluation Researchers**
+
 The model can also be useful as an experimental model for comparing:
 
 * base versus preference-tuned models
@@ -382,8 +373,10 @@ The model can also be useful as an experimental model for comparing:
 
 * preference-learning behavior
 
----
-# When Should You Prefer the Original Qwen Model?
+**---**
+
+**# When Should You Prefer the Original Qwen Model?**
+
 This model should not automatically be considered better for every possible task.
 
 The experiment specifically focused on ****Python code generation****.
@@ -394,7 +387,7 @@ The reported improvement should therefore be interpreted as:
 
 \> ****Improved performance on the evaluated Python coding benchmark, not a guarantee of improvement across every coding or general-purpose task.****
 
----
+**---**
 
 The base model and the trained model use the same VRAM.
 
@@ -424,8 +417,10 @@ architecture — so they are byte-for-byte the same size.
 
   └───────────────────────────────┴────────────┘
 
----
-# Model Architecture and Training
+**---**
+
+**# Model Architecture and Training**
+
 The model starts from:
 
 \`Qwen/Qwen2.5-Coder-7B-Instruct\`
@@ -438,8 +433,12 @@ The resulting adapter contains approximately 80.7 million trainable parameters.
 
 The adapter is approximately 308 MB.
 
----
-# DPO Training## Why DPO?
+**---**
+
+**# DPO Training**
+
+**## Why DPO?**
+
 Direct Preference Optimization provides a method for optimizing a language model directly from preference pairs without requiring a separate reward-model training stage.
 
 For this experiment, a preference pair has the form:
@@ -460,8 +459,10 @@ Problem
 
 DPO then learns to increase the relative likelihood of the chosen response compared with the rejected response.
 
----
-# Preference Data Construction
+**---**
+
+**# Preference Data Construction**
+
 The preference dataset was generated specifically for this experiment.
 
 It was constructed from programming problems drawn from:
@@ -473,16 +474,18 @@ It was constructed from programming problems drawn from:
 * HumanEval
 
 * handwritten programming problems
-### Training Data
-| Dataset | License |
 
-| ------------ | ---------- |
+**### Training Data**
+
+\| Dataset      | License    |
 
-| BigCodeBench | Apache 2.0 |
+\| ------------ | ---------- |
 
-| MBPP | CC BY 4.0 |
+\| BigCodeBench | Apache 2.0 |
 
-| HumanEval | MIT |
+\| MBPP         | CC BY 4.0  |
+
+\| HumanEval    | MIT        |
 
 Candidate solutions were generated automatically.
 
@@ -504,8 +507,10 @@ Every candidate was then executed in a Docker sandbox against the associated uni
 
 The resulting execution outcome was used to construct chosen/rejected preference pairs.
 
----
-# Preference Dataset
+**---**
+
+**# Preference Dataset**
+
 After filtering and deduplication:
 
 * 1,326 preference pairs
@@ -526,34 +531,58 @@ No external LLM was used as a preference judge.
 
 The preference signal was generated from objective unit-test execution.
 
----
-# DPO Configuration
+**---**
+
+**# DPO Configuration**
+
 The primary DPO configuration was:
 
-| Parameter | Value |
-| --- | --- |
-| Rank | 32 |
-| Alpha | 64 |
-| Dropout | 0.05 |
-| Target modules | Attention and MLP projections |
-| Epochs | 2 |
-| Maximum sequence length | 1,024 |
-| Batch size | 4 |
-| Gradient accumulation | 4 |
-| Effective batch size | 16 |
-| Optimizer | AdamW |
-| Gradient clipping | 1.0 |
-| Gradient checkpointing | Enabled |
-## LoRA Configuration
-| Parameter | Value |
-| --- | --- |
-| Rank | 32 |
-| Alpha | 64 |
-| Dropout | 0.05 |
-| Target modules | Attention and MLP projections |
+\| Parameter               | Value   |
+
+\| ----------------------- | ------- |
+
+\| DPO beta                | 0.1     |
+
+\| Learning rate           | 1e-5    |
+
+\| Scheduler               | Cosine  |
+
+\| Warmup                  | 5%      |
+
+\| Epochs                  | 2       |
+
+\| Maximum sequence length | 1,024   |
+
+\| Batch size              | 4       |
+
+\| Gradient accumulation   | 4       |
+
+\| Effective batch size    | 16      |
+
+\| Optimizer               | AdamW   |
+
+\| Gradient clipping       | 1.0     |
+
+\| Gradient checkpointing  | Enabled |
+
+**## LoRA Configuration**
+
+\| Parameter      | Value                         |
+
+\| -------------- | ----------------------------- |
+
+\| Rank           | 32                            |
+
+\| Alpha          | 64                            |
+
+\| Dropout        | 0.05                          |
+
+\| Target modules | Attention and MLP projections |
+
+**---**
+
+**# Training Hardware**
 
----
-# Training Hardware
 DPO training was performed on:
 
 ****NVIDIA RTX A6000 — 48 GB VRAM****
@@ -570,8 +599,10 @@ Training environment:
 
 Peak GPU memory consumption was approximately 33.4 GiB.
 
----
-# Evaluation Methodology
+**---**
+
+**# Evaluation Methodology**
+
 The primary evaluation benchmark consisted of:
 
 ****508 BigCodeBench programming problems****
@@ -579,7 +610,9 @@ The primary evaluation benchmark consisted of:
 The evaluation problems were reserved separately from the preference-training pool.
 
 The same generation and grading procedure was applied to both models.
-### Generation
+
+**### Generation**
+
 * 10 answers per problem
 
 * temperature: 0.2
@@ -593,22 +626,26 @@ The same generation and grading procedure was applied to both models.
 * sampling enabled
 
 * identical random seeds for base and DPO models
-### Pass@k
+
+**### Pass@k**
+
 Pass@k was calculated using the standard unbiased estimator based on the number of correct samples among the generated candidates.
 
----
-# Preference-Learning Diagnostics
+**---**
+
+**# Preference-Learning Diagnostics**
+
 The DPO training objective produced a strong preference signal on the training examples.
 
 However, the validation results were substantially weaker.
 
-| Metric | Training | Validation |
+\| Metric              | Training |            Validation |
 
-| ------------------- | -------: | --------------------: |
+\| ------------------- | -------: | --------------------: |
 
-| Preference accuracy | ~84.4% | ~45.8% |
+\| Preference accuracy |   ~84.4% |                ~45.8% |
 
-| Reward margin | Positive | Approximately neutral |
+\| Reward margin       | Positive | Approximately neutral |
 
 This indicates that the explicit preference signal learned during training did not generalize strongly to the validation preference examples.
 
@@ -616,8 +653,10 @@ At the same time, downstream coding performance improved on the held-out benchma
 
 This is an important observation from the experiment.
 
----
-# What Does This Mean?
+**---**
+
+**# What Does This Mean?**
+
 The results should not be interpreted as proof that the model learned a broadly generalized notion of code quality.
 
 Instead, they suggest that ****optimizing an automatically constructed preference objective can improve downstream coding performance even when explicit preference metrics show weak validation generalization.****
@@ -634,34 +673,48 @@ Several explanations are possible, including:
 
 These are hypotheses for further investigation rather than established conclusions.
 
----
-# Important Limitations## No Human Preference Evaluation
+**---**
+
+**# Important Limitations**
+
+**## No Human Preference Evaluation**
+
 No human preference study was performed.
 
 The preference signal was derived from automated unit-test execution.
 
 Therefore, this model should not be described as being aligned to human preferences based on this experiment.
-## Unit Tests Are Imperfect
+
+**## Unit Tests Are Imperfect**
+
 Passing the available tests does not guarantee that a solution is universally correct.
 
 A test suite may fail to cover some edge cases or undesirable implementation behavior.
-## Benchmark Source Overlap
+
+**## Benchmark Source Overlap**
+
 The evaluation problems were held out at the problem level.
 
 However, the preference-training data includes other BigCodeBench problems.
 
 Therefore, the evaluation is ****problem-held-out****, but not completely source-family-disjoint.
-## Python Focus
+
+**## Python Focus**
+
 The primary objective and evaluation were focused on Python programming.
 
 Performance should not be assumed to generalize to other programming languages without additional evaluation.
-## Experimental Model
+
+**## Experimental Model**
+
 This is an experimental research model.
 
 Users should evaluate it against their own workloads before using it in production.
 
----
-# Reproducibility
+**---**
+
+**# Reproducibility**
+
 The experiment documents:
 
 * base model
@@ -690,8 +743,10 @@ The experiment documents:
 
 The objective is to make the experiment sufficiently transparent for others to reproduce, extend, or critique.
 
----
-# Independent Research Project
+**---**
+
+**# Independent Research Project**
+
 This model was developed as an ****independent, out-of-office AI/ML research project****.
 
 The project was conceived, designed, implemented, trained, evaluated, analyzed, and released independently.
@@ -732,18 +787,22 @@ Model Release
 
 \`\`\`
 
----
-# Model Release
+**---**
+
+**# Model Release**
+
 The trained LoRA adapter and technical documentation are available here:
 
 ****Hugging Face:****
 
 https://huggingface.co/ranjanrajib/DPO-trained-Qwen2.5-Coder-7B
 
-The repository contains the model adapter and documentation required to understand the training methodology, evaluation, results, intended use, and limitations.
+The repository contains the LoRA adapter and documentation required to understand the training methodology, evaluation, results, intended use, and limitations. The base Qwen weights are not redistributed in this repository.
+
+**---**
+
+**# Licensing and Attribution**
 
----
-# Licensing and Attribution
 The model is derived from:
 
 \`Qwen/Qwen2.5-Coder-7B-Instruct\`
@@ -754,12 +813,16 @@ The preference-generation pipeline used programming problems from BigCodeBench,
 
 The respective dataset licenses and attribution requirements should be observed when using or redistributing material derived from those datasets.
 
----
-# Citation
+**---**
+
+**# Citation**
+
 If you use this model or the experimental methodology in your work, please cite the original Qwen2.5-Coder model and the relevant datasets used in the experiment.
 
----
-# Summary
+**---**
+
+**# Summary**
+
 ****What is this?****
 
 A Python-focused DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct.
@@ -787,7 +850,11 @@ A LoRA adapter that can be loaded on top of \`Qwen/Qwen2.5-Coder-7B-Instruct\`.
 ****Why is the experiment interesting?****
 
 The model improved downstream coding performance even though explicit preference metrics showed weak validation generalization, highlighting the distinction between optimizing a preference objective and improving downstream model capability.
-## Licence and attribution
+
+
+
+**## Licence and attribution**
+
 Apache-2.0, inherited from \`Qwen/Qwen2.5-Coder-7B-Instruct\`. This is a modified derivative of that
 
 model; the original is by the Qwen team at Alibaba Cloud. The preference data was generated from the