Instructions to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B")
model = AutoModelForCausalLM.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B

SGLang

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with Docker Model Runner:
```
docker model run hf.co/ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B
```

ranjanrajib commited on Sep 16

Commit

23ba137

verified ·

1 Parent(s): 4d52f6a

Update README.md

Browse files

Files changed (1) hide show

README.md +19 -743

README.md CHANGED Viewed

@@ -23,773 +23,49 @@ tags:
   - parameter-efficient-fine-tuning
 ---
-\---
-language:
-  \- en
-license: apache-2.0
-library_name: transformers
-pipeline_tag: text-generation
-base_model:
-  \- Qwen/Qwen2.5-Coder-7B-Instruct
-datasets:
-  \- bigcode/bigcodebench
-  \- Muennighoff/mbpp
-  \- openai/openai_humaneval
-tags:
-  \- qwen2.5
-  \- qwen
-  \- code-generation
-  \- python
-  \- python-code-generation
-  \- dpo
-  \- direct-preference-optimization
-  \- preference-learning
-  \- lora
-  \- parameter-efficient-fine-tuning
-\---
-# Qwen2.5-Coder-7B-Instruct — DPO Fine-Tuned for Python## Model at a Glance
-This model is a ****Python-focused, DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct****, designed to improve the model's ability to generate ****functionally correct Python code****.
-The model was trained using ****Direct Preference Optimization (DPO)**** with automatically constructed preference pairs. Candidate Python solutions were generated for programming problems and evaluated by executing them against unit tests. Solutions that passed the required tests were used as preferred examples, while solutions that failed tests were used as rejected examples.
 The resulting model was evaluated against the original Qwen2.5-Coder-7B-Instruct model on held-out programming problems.
 ### What changed?
 The base model:
-****Qwen2.5-Coder-7B-Instruct****
 was fine-tuned using:
-****DPO + LoRA + execution-based preference data****
 The resulting model demonstrates improved coding performance on the evaluated held-out benchmark:
 | Metric | Qwen2.5-Coder-7B-Instruct | DPO Model | Improvement |
-| ----------------------- | ------------------------: | ---------: | -----------: |
-| Pass@1 | 41.87% | ****45.22%**** | ****+3.35 pp**** |
-| Pass@5 | 48.87% | ****51.10%**** | ****+2.23 pp**** |
-| Pass@10 | 50.99% | ****53.16%**** | ****+2.17 pp**** |
-| Tests passed per answer | 66.20% | ****69.41%**** | ****+3.21 pp**** |
-The primary result is the improvement in ****Pass@1****, meaning that a single generated solution was more likely to produce a correct solution on the evaluated programming problems.
-The model also showed improved Pass@5, Pass@10, and partial test coverage.
 ---
-****Base model:**** \`Qwen/Qwen2.5-Coder-7B-Instruct\` at snapshot
-\`c03e6d358207e414f1eca0bb1891e29f1db0e242\`. The merge was performed against exactly these weights.
-## Downloading and using### The merged model (what most people want)
-\`\`\`python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 model = AutoModelForCausalLM.from_pretrained(
-    "ranjanrajib/DPO-trained-Qwen2.5-Coder-7B", dtype="bfloat16", device_map="auto")
 tok = AutoTokenizer.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Coder-7B")
 msgs = [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}]
 text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
 out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
-print(tok.decode(out[0], skip_special_tokens=True))
-\`\`\`
-Or fetch the files without loading them (~14 GB, four shards):
-\`\`\`bash
-hf download ranjanrajib/DPO-trained-Qwen2.5-Coder-7B --exclude "adapter/*" --local-dir ./d4
-\`\`\`
-### The LoRA adapter only (308 MB)
-The adapter lives in \`adapter/\`. Applying it to the base model gives the same result as the
-merged weights, at a fraction of the download.
-\`\`\`bash
-hf download ranjanrajib/DPO-trained-Qwen2.5-Coder-7B --include "adapter/*" --local-dir ./d4-lora
-\`\`\`
-\`\`\`python
-import torch
-from peft import PeftModel
-from transformers import AutoModelForCausalLM
-BASE = "Qwen/Qwen2.5-Coder-7B-Instruct"
-REVISION = "c03e6d358207e414f1eca0bb1891e29f1db0e242"   # the snapshot this was trained against
-base = AutoModelForCausalLM.from_pretrained(
-    BASE, revision=REVISION, dtype=torch.bfloat16, device_map="auto")
-model = PeftModel.from_pretrained(
-    base, "ranjanrajib/DPO-trained-Qwen2.5-Coder-7B", subfolder="adapter")
-\`\`\`
----
-**# ## VRAM Requirements
-The base model and the merged trained model use the same VRAM. The merge changes weight values, not tensor shapes: both models have the same 7.6B parameters and architecture, so the model weights occupy the same amount of bf16 memory.
-| Component | bf16 VRAM |
-| --- | ---: |
-| Weights (base or merged D4) | 14.19 GiB |
-| KV cache, per token | 56 KiB |
-| Activations / workspace | ~0.5–1 GiB |
-| **Practical single-stream total** | **~15–16 GiB** |
----
-Why Use This Model?**
-If your primary use case is ****Python code generation****, this model is intended to be a stronger alternative to the original Qwen2.5-Coder-7B-Instruct model based on the evaluation performed in this project.
-It is particularly relevant for users who want to:
-* generate Python functions and implementations
-* solve programming problems
-* generate code that needs to satisfy functional requirements
-* improve first-attempt coding correctness
-* experiment with DPO-trained code-generation models
-* study execution-based preference optimization
-* build research or evaluation pipelines for Python code generation
-The main motivation for the fine-tuning was not simply to make the model generate more code.
-It was to encourage the model toward ****solutions that work****.
-The preference data therefore used actual program execution and unit-test results rather than subjective judgments about whether generated code **looked** correct.
----
-# How Is It Different From Qwen2.5-Coder-7B-Instruct?
-The underlying model architecture and capabilities come from ****Qwen2.5-Coder-7B-Instruct****.
-This model adds a preference-optimization stage on top of that model.
-Conceptually:
-\`\`\`text
-Qwen2.5-Coder-7B-Instruct
-             │
-             ▼
-   Generate multiple Python
-        candidate solutions
-             │
-             ▼
-     Execute candidates
-       against tests
-             │
-       ┌─────┴─────┐
-       ▼           ▼
-    Pass          Fail
-       │           │
-       ▼           ▼
-    Chosen       Rejected
-       │           │
-       └─────┬─────┘
-             ▼
-            DPO
-             │
-             ▼
-       LoRA Adapter
-             │
-             ▼
-   Python-focused model
-\`\`\`
-The goal is to shift the model toward coding behaviors associated with solutions that successfully execute against the available tests.
-### Important distinction
-This is ****not a new foundation model****.
-It is a parameter-efficient fine-tuned adapter for Qwen2.5-Coder-7B-Instruct.
-The base model remains unchanged.
----
-# What Kind of Model Is This?### Primary task
-****Python code generation****
-### Model family
-Qwen2.5-Coder
-### Base model
-\`Qwen/Qwen2.5-Coder-7B-Instruct\`
-### Fine-tuning
-Direct Preference Optimization (DPO)
-### Parameter-efficient method
-LoRA
-### Precision
-bfloat16
-### Quantization
-None
-### Model size
-7.7B-parameter base model
-### Adapter
-Approximately 80.7M trainable parameters
----
-# Performance
-The model was evaluated against the original Qwen2.5-Coder-7B-Instruct using a controlled benchmark consisting of 508 held-out BigCodeBench programming problems.
-The same prompts, generation settings, random seeds, number of samples, and grading procedure were used for both models.
-## Pass@k
-| Metric | Base Model | DPO Model |
-| ------- | ---------: | ---------: |
-| Pass@1 | 41.87% | ****45.22%**** |
-| Pass@5 | 48.87% | ****51.10%**** |
-| Pass@10 | 50.99% | ****53.16%**** |
-The improvement in Pass@1 is approximately ****3.35 percentage points****.
-## Partial Correctness
-The average proportion of tests passed by each generated answer increased from:
-****66.20% → 69.41%****
-This is a secondary metric intended to provide additional information about functional correctness beyond binary problem-level pass/fail.
----
-# Paired Problem-Level Analysis
-The evaluation used the same 508 problems for both models, allowing direct comparison of model behavior on individual problems.
-For the first generated answer:
-| Base → DPO | Problems |
-| --------------------- | -------: |
-| Correct → Correct | 186 |
-| Correct → Incorrect | 17 |
-| Incorrect → Correct | 41 |
-| Incorrect → Incorrect | 264 |
-The DPO model converted 41 problems from incorrect to correct while 17 previously correct problems became incorrect.
-This provides additional evidence that the aggregate improvement is associated with meaningful changes in problem-level behavior rather than simply differences in the evaluated problem sets.
----
-# Who Should Use This Model?
-This model is most appropriate for:
-### Python Developers
-Developers interested in experimenting with a locally hosted code-generation model that has been specifically fine-tuned toward execution-validated solutions.
-### AI/ML Researchers
-Researchers studying:
-* DPO
-* preference learning
-* LLM alignment
-* code-generation models
-* automated preference construction
-* execution-based evaluation
-* parameter-efficient fine-tuning
-### LLM Evaluation Researchers
-The model can also be useful as an experimental model for comparing:
-* base versus preference-tuned models
-* coding correctness
-* Pass@k
-* partial test coverage
-* preference-learning behavior
----
-# When Should You Prefer the Original Qwen Model?
-This model should not automatically be considered better for every possible task.
-The experiment specifically focused on ****Python code generation****.
-If your application requires broad general-purpose instruction following, other programming languages, or capabilities that were not evaluated here, you should benchmark both models for your specific workload.
-The reported improvement should therefore be interpreted as:
-\> ****Improved performance on the evaluated Python coding benchmark, not a guarantee of improvement across every coding or general-purpose task.****
----
-The base model and the trained model use the same VRAM.
-The merge  changed weight values, not shapes — same 7.6B parameters, same
-architecture — so they are byte-for-byte the same size.
-  ┌───────────────────────────────┬────────────┐
-  │                               │ bf16 VRAM  │
-  ├───────────────────────────────┼────────────┤
-  │ Weights (base or merged D4)   │ 14.19 GiB  │
-  ├───────────────────────────────┼────────────┤
-  │ KV cache, per token           │ 56 KiB     │
-  ├───────────────────────────────┼────────────┤
-  │ Activations / workspace       │ ~0.5–1 GiB │
-  ├───────────────────────────────┼────────────┤
-  │ Practical single-stream total │ ~15–16 GiB │
-  └───────────────────────────────┴────────────┘
----
-# Model Architecture and Training
-The model starts from:
-\`Qwen/Qwen2.5-Coder-7B-Instruct\`
-The base model contains approximately 7.7 billion parameters.
-Rather than updating all parameters, the experiment uses ****LoRA-based parameter-efficient fine-tuning****.
-The resulting adapter contains approximately 80.7 million trainable parameters.
-The adapter is approximately 308 MB.
----
-# DPO Training## Why DPO?
-Direct Preference Optimization provides a method for optimizing a language model directly from preference pairs without requiring a separate reward-model training stage.
-For this experiment, a preference pair has the form:
-\`\`\`text
-Problem
-   │
-   ├── Candidate A → passes tests → chosen
-   │
-   └── Candidate B → fails tests  → rejected
-\`\`\`
-DPO then learns to increase the relative likelihood of the chosen response compared with the rejected response.
----
-# Preference Data Construction
-The preference dataset was generated specifically for this experiment.
-It was constructed from programming problems drawn from:
-* BigCodeBench
-* MBPP
-* HumanEval
-* handwritten programming problems
-### Training Data
-| Dataset | License |
-| ------------ | ---------- |
-| BigCodeBench | Apache 2.0 |
-| MBPP | CC BY 4.0 |
-| HumanEval | MIT |
-Candidate solutions were generated automatically.
-Five generation strategies were used:
-* normal
-* straightforward
-* edge-case-focused
-* alternative
-* optimized
-Multiple candidates were generated for each problem.
-Every candidate was then executed in a Docker sandbox against the associated unit tests.
-The resulting execution outcome was used to construct chosen/rejected preference pairs.
----
-# Preference Dataset
-After filtering and deduplication:
-* 1,326 preference pairs
-* 221 programming problems
-* 933 training pairs
-* 114 validation pairs
-The dataset was split by ****problem****, rather than by individual preference pair.
-This prevents different candidate solutions from the same underlying programming problem from being distributed across training and validation.
-No human preference labels were used.
-No external LLM was used as a preference judge.
-The preference signal was generated from objective unit-test execution.
----
-# DPO Configuration
-The primary DPO configuration was:
-| Parameter | Value |
-| --- | --- |
-| Rank | 32 |
-| Alpha | 64 |
-| Dropout | 0.05 |
-| Target modules | Attention and MLP projections |
-| Epochs | 2 |
-| Maximum sequence length | 1,024 |
-| Batch size | 4 |
-| Gradient accumulation | 4 |
-| Effective batch size | 16 |
-| Optimizer | AdamW |
-| Gradient clipping | 1.0 |
-| Gradient checkpointing | Enabled |
-## LoRA Configuration
-| Parameter | Value |
-| --- | --- |
-| Rank | 32 |
-| Alpha | 64 |
-| Dropout | 0.05 |
-| Target modules | Attention and MLP projections |
----
-# Training Hardware
-DPO training was performed on:
-****NVIDIA RTX A6000 — 48 GB VRAM****
-Training environment:
-* CUDA 12.8
-* PyTorch 2.11
-* bfloat16
-* gradient checkpointing
-Peak GPU memory consumption was approximately 33.4 GiB.
----
-# Evaluation Methodology
-The primary evaluation benchmark consisted of:
-****508 BigCodeBench programming problems****
-The evaluation problems were reserved separately from the preference-training pool.
-The same generation and grading procedure was applied to both models.
-### Generation
-* 10 answers per problem
-* temperature: 0.2
-* top-p: 0.95
-* maximum new tokens: 512
-* repetition penalty: 1.0
-* sampling enabled
-* identical random seeds for base and DPO models
-### Pass@k
-Pass@k was calculated using the standard unbiased estimator based on the number of correct samples among the generated candidates.
----
-# Preference-Learning Diagnostics
-The DPO training objective produced a strong preference signal on the training examples.
-However, the validation results were substantially weaker.
-| Metric | Training | Validation |
-| ------------------- | -------: | --------------------: |
-| Preference accuracy | ~84.4% | ~45.8% |
-| Reward margin | Positive | Approximately neutral |
-This indicates that the explicit preference signal learned during training did not generalize strongly to the validation preference examples.
-At the same time, downstream coding performance improved on the held-out benchmark.
-This is an important observation from the experiment.
----
-# What Does This Mean?
-The results should not be interpreted as proof that the model learned a broadly generalized notion of code quality.
-Instead, they suggest that ****optimizing an automatically constructed preference objective can improve downstream coding performance even when explicit preference metrics show weak validation generalization.****
-Several explanations are possible, including:
-* differences between training and validation preference distributions
-* imperfect test-derived preference labels
-* overfitting to characteristics of the training preference pairs
-* changes in generation behavior that are not fully captured by preference accuracy
-These are hypotheses for further investigation rather than established conclusions.
----
-# Important Limitations## No Human Preference Evaluation
-No human preference study was performed.
-The preference signal was derived from automated unit-test execution.
-Therefore, this model should not be described as being aligned to human preferences based on this experiment.
-## Unit Tests Are Imperfect
-Passing the available tests does not guarantee that a solution is universally correct.
-A test suite may fail to cover some edge cases or undesirable implementation behavior.
-## Benchmark Source Overlap
-The evaluation problems were held out at the problem level.
-However, the preference-training data includes other BigCodeBench problems.
-Therefore, the evaluation is ****problem-held-out****, but not completely source-family-disjoint.
-## Python Focus
-The primary objective and evaluation were focused on Python programming.
-Performance should not be assumed to generalize to other programming languages without additional evaluation.
-## Experimental Model
-This is an experimental research model.
-Users should evaluate it against their own workloads before using it in production.
----
-# Reproducibility
-The experiment documents:
-* base model
-* preference-data construction
-* data sources
-* dataset split methodology
-* DPO configuration
-* LoRA configuration
-* training hardware
-* generation configuration
-* evaluation methodology
-* random seed strategy
-* benchmark methodology
-* limitations
-The objective is to make the experiment sufficiently transparent for others to reproduce, extend, or critique.
----
-# Independent Research Project
-This model was developed as an ****independent, out-of-office AI/ML research project****.
-The project was conceived, designed, implemented, trained, evaluated, analyzed, and released independently.
-The work covers the complete experimental lifecycle:
-\`\`\`text
-Research Question
-       ↓
-Experimental Design
-       ↓
-Preference Data Generation
-       ↓
-Automated Evaluation
-       ↓
-DPO + LoRA Training
-       ↓
-Controlled Benchmarking
-       ↓
-Statistical / Paired Analysis
-       ↓
-Model Release
-\`\`\`
----
-# Model Release
-The trained LoRA adapter and technical documentation are available here:
-****Hugging Face:****
-https://huggingface.co/ranjanrajib/DPO-trained-Qwen2.5-Coder-7B
-The repository contains the model adapter and documentation required to understand the training methodology, evaluation, results, intended use, and limitations.
----
-# Licensing and Attribution
-The model is derived from:
-\`Qwen/Qwen2.5-Coder-7B-Instruct\`
-Users should review and comply with the licensing and usage terms of the base model.
-The preference-generation pipeline used programming problems from BigCodeBench, MBPP, HumanEval, and additional handwritten problems.
-The respective dataset licenses and attribution requirements should be observed when using or redistributing material derived from those datasets.
----
-# Citation
-If you use this model or the experimental methodology in your work, please cite the original Qwen2.5-Coder model and the relevant datasets used in the experiment.
----
-# Summary
-****What is this?****
-A Python-focused DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct.
-****What is different?****
-The model was trained using LoRA-based Direct Preference Optimization with automatically constructed preferences derived from unit-test execution.
-****Does it improve coding performance?****
-On the evaluated held-out benchmark, yes. Pass@1 improved from ****41.87% to 45.22%****, with improvements also observed in Pass@5, Pass@10, and partial test coverage.
-****Is it human-aligned?****
-No human preference evaluation was performed. The preference signal was generated automatically from unit-test outcomes.
-****Who should use it?****
-Users primarily interested in Python code generation, coding experiments, DPO research, preference learning, and execution-based LLM evaluation.
-****What is being released?****
-A LoRA adapter that can be loaded on top of \`Qwen/Qwen2.5-Coder-7B-Instruct\`.
-****Why is the experiment interesting?****
-The model improved downstream coding performance even though explicit preference metrics showed weak validation generalization, highlighting the distinction between optimizing a preference objective and improving downstream model capability.
-## Licence and attribution
-Apache-2.0, inherited from \`Qwen/Qwen2.5-Coder-7B-Instruct\`. This is a modified derivative of that
-model; the original is by the Qwen team at Alibaba Cloud. The preference data was generated from the
-base model's own outputs on BigCodeBench problems.

   - parameter-efficient-fine-tuning
 ---
+# Qwen2.5-Coder-7B-Instruct — DPO Fine-Tuned for Python
+## Model at a Glance
+This model is a **Python-focused, DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct**, designed to improve the model's ability to generate **functionally correct Python code**.
+The model was trained using **Direct Preference Optimization (DPO)** with automatically constructed preference pairs. Candidate Python solutions were generated for programming problems and evaluated by executing them against unit tests. Solutions that passed the required tests were used as preferred examples, while solutions that failed tests were used as rejected examples.
 The resulting model was evaluated against the original Qwen2.5-Coder-7B-Instruct model on held-out programming problems.
 ### What changed?
 The base model:
+**`Qwen/Qwen2.5-Coder-7B-Instruct`**
 was fine-tuned using:
+**DPO + LoRA + execution-based preference data**
 The resulting model demonstrates improved coding performance on the evaluated held-out benchmark:
 | Metric | Qwen2.5-Coder-7B-Instruct | DPO Model | Improvement |
+| :--- | :---: | :---: | :---: |
+| Pass@1 | 41.87% | **45.22%** | **+3.35 pp** |
+| Pass@5 | 48.87% | **51.10%** | **+2.23 pp** |
+| Pass@10 | 50.99% | **53.16%** | **+2.17 pp** |
+| Tests passed per answer | 66.20% | **69.41%** | **+3.21 pp** |
+The primary result is the improvement in **Pass@1**, meaning that a single generated solution was more likely to produce a correct solution on the evaluated programming problems. The model also showed improved Pass@5, Pass@10, and partial test coverage.
 ---
+**Base model:** `Qwen/Qwen2.5-Coder-7B-Instruct` at snapshot `c03e6d358207e414f1eca0bb1891e29f1db0e242`. The merge was performed against exactly these weights.
+## Downloading and Using The merged model
+```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 model = AutoModelForCausalLM.from_pretrained(
+    "ranjanrajib/DPO-trained-Qwen2.5-Coder-7B", dtype="bfloat16", device_map="auto"
+)
 tok = AutoTokenizer.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Coder-7B")
 msgs = [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}]
 text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
 out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
+print(tok.decode(out[0], skip_special_tokens=True))