README.md

# PrefDrive: LoRA DPO LLaMa-7B for Autonomous Driving

This repository contains LoRA (Low-Rank Adaptation) parameters for a fine-tuned version of LLaMa-7B using Direct Preference Optimization (DPO). The model is trained to better align with specific driving behaviors and operational requirements through preference learning, significantly improving autonomous driving performance.

## Model Details

- **Base Model**: [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
- **Training Method**: Direct Preference Optimization (DPO)
- **LoRA Parameters**:
  - Rank (r): 16
  - Alpha (α): 16
  - Target Modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `down_proj`, `up_proj`
- **Training Framework**: Unsloth + TRL
- **Training Precision**: 4-bit Quantization

### Training Configuration

| Parameter | Value |
|-----------|-------|
| Base Model | LLaMA2-7B |
| Training Strategy | LoRA |
| Learning Rate | 1e-5 |
| Batch Size | 4 |
| Gradient Accumulation Steps | 2 |
| Training Epochs | 3 |
| Maximum Sequence Length | 2,048 |
| Warmup Ratio | 0.1 |
| Max Gradient Norm | 0.3 |
| DPO Beta (β) | 0.1 |
| Loss Type | Sigmoid |
| Training Data | Chosen & Rejected action pairs |

## Usage

To use this model, you'll need to load both the base model and the LoRA adapter:

```python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model_id = "meta-llama/Llama-2-7b"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id)

# Load LoRA adapter
peft_model_id = "[YOUR_USERNAME]/lora-dpo-llama-7b"
model = PeftModel.from_pretrained(model, peft_model_id)

# Use model for inference
inputs = tokenizer("Hello, please", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

For faster inference with Unsloth:

```python
from unsloth import FastLanguageModel

# Load the model with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
    "meta-llama/Llama-2-7b",
    load_in_4bit=True,
    max_seq_length=2048
)

# Load LoRA adapter
model = FastLanguageModel.get_peft_model(
    model,
    "[YOUR_USERNAME]/lora-dpo-llama-7b",
)

# Use model for inference
inputs = tokenizer("Hello, please", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Dataset

The model was trained on the PrefDrive dataset, a comprehensive collection of 74,040 driving sequences carefully annotated with driving preferences and driving decisions. Each entry in the dataset consists of:

- A driving scenario description (s)
- A preferred/chosen driving action with its reasoning and resulting waypoint (a_p)
- A rejected driving action with its reasoning and resulting waypoint (a_r)

This dataset captures various autonomous driving scenarios with emphasis on proper distance maintenance, trajectory smoothness, traffic rule compliance, and route adherence.

## Training Procedure

The model was trained using the DPO method which directly optimizes a language model to align with driving preferences without requiring a reward model. The training process uses pairwise comparisons between preferred and rejected driving actions to update model parameters.

### Methodology

The PrefDrive methodology for autonomous driving is formulated as:

$\mathcal{L}_{DPO} = -\mathbb{E}_{(s,a_p,a_r)\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(a_p|s)}{\pi_{ref}(a_p|s)} - \beta\log\frac{\pi_\theta(a_r|s)}{\pi_{ref}(a_r|s)}\Big)\Big]$

where:
- $\mathcal{D}$ represents our driving preference dataset
- $s$ denotes the current driving scenario description
- $a_p$ represents the preferred (chosen) driving action with its reasoning and resulting waypoint
- $a_r$ represents the rejected driving action with its reasoning and resulting waypoint
- $\pi_\theta$ is the policy model being trained
- $\pi_{ref}$ is the initial reference model
- $\beta$ controls the preference learning sensitivity (set to 0.1)
- $\sigma$ represents the sigmoid function

This formulation explicitly shows how our model learns to favor chosen driving actions over rejected ones while maintaining reasonable deviation from the reference model's behavior.

### Key Training Parameters
- Learning rate: 1e-5
- Number of epochs: 3
- DPO beta: 0.1
- Loss type: Sigmoid
- Max sequence length: 2048

## Evaluation Results

The model was evaluated in the CARLA simulator across different town environments. Here are the performance metrics:

### Town 01 Performance

| Metric | LMDrive (baseline) | PrefDrive (Ours) | Improvement |
|--------|-------------------|-----------------|-------------|
| Composite Score | 53.00 | 56.12 | +5.9% |
| Penalty Score | 0.86 | 0.88 | +1.5% |
| Route Completion | 59.10 | 64.15 | +8.5% |
| Layout Collisions | 0.73 | 0.27 | -63.5% |
| Traffic Light Violations | 0.22 | 0.16 | -28.1% |
| Route Deviation | 1.32 | 1.36 | +3.0% |
| Vehicle Blocked | 0.11 | 0.00 | -100.0% |

### Town 04 Performance

| Metric | LMDrive (baseline) | PrefDrive (Ours) | Improvement |
|--------|-------------------|-----------------|-------------|
| Composite Score | 60.11 | 65.93 | +9.7% |
| Penalty Score | 0.93 | 0.96 | +3.2% |
| Route Completion | 65.25 | 69.93 | +7.2% |
| Layout Collisions | 0.00 | 0.00 | 0.0% |
| Traffic Light Violations | 0.24 | 0.00 | -100.0% |
| Route Deviation | 1.86 | 1.77 | -4.8% |
| Vehicle Blocked | 0.00 | 0.00 | 0.0% |

The results demonstrate significant improvements in crucial metrics, particularly in reducing traffic light violations and layout collisions while improving route completion.

## Limitations and Biases

This model inherits the limitations and biases from the base LLaMa model. Additionally:

- It's optimized specifically for autonomous driving tasks and may not perform well in unrelated domains
- Performance may vary in driving environments that differ significantly from the training data
- The LoRA adaptation affects specific parameter matrices and may not fully transform the base model's capabilities
- While the model shows improved performance in simulated environments (CARLA), its behavior in real-world driving scenarios would require further validation and safety testing
- The model is designed to work with a specific autonomous driving stack and may require adaptation for different setups

## Ethical Considerations

When using this model, consider:

- The potential for generating harmful, misleading, or biased content
- The limitations in factual accuracy and reasoning abilities
- The need for appropriate content filtering in production applications

## Citations

```bibtex
@INPROCEEDINGS{Li2025,
  title={PrefDrive: A Preference Learning Framework for Autonomous Driving with Large Language Models}, 
  author={Li, Yun and Javanmardi, Ehsan and Thompson, Simon and Katsumata, Kai and Orsholits, Alex and Tsukada, Manabu},
  booktitle = "{2025 IEEE Intelligent Vehicles Symposium (IV)}",
}
```