Update README.md

README.md

@@ -2,6 +2,7 @@
 license: mit
 ---
 ---
+---
 language:
   - en
 tags:
@@ -11,20 +12,24 @@ tags:
   - peft
   - dpo
   - reinforcement-learning
+  - autonomous-driving
 license: apache-2.0
 datasets:
-  - [DATASET_NAME]
+  - prefdrive-preference-dataset
 metrics:
   - dpo-loss
+  - route-completion
+  - traffic-light-violations
+  - layout-collisions
 library_name: unsloth
 library_version: 0.5.0
 base_model: meta-llama/Llama-2-7b
 pipeline_tag: text-generation
 ---
 
-# LoRA DPO LLaMa-7B
+# PrefDrive: LoRA DPO LLaMa-7B for Autonomous Driving
 
-This repository contains LoRA (Low-Rank Adaptation) parameters for a fine-tuned version of LLaMa-7B using Direct Preference Optimization (DPO). The model is trained to better align with human preferences based on pairwise comparison data.
+This repository contains LoRA (Low-Rank Adaptation) parameters for a fine-tuned version of LLaMa-7B using Direct Preference Optimization (DPO). The model is trained to better align with specific driving behaviors and operational requirements through preference learning, significantly improving autonomous driving performance.
 
 ## Model Details
 
@@ -32,11 +37,28 @@ This repository contains LoRA (Low-Rank Adaptation) parameters for a fine-tuned
 - **Training Method**: Direct Preference Optimization (DPO)
 - **LoRA Parameters**:
   - Rank (r): 16
-  - Alpha: 16
+  - Alpha (α): 16
   - Target Modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `down_proj`, `up_proj`
 - **Training Framework**: Unsloth + TRL
 - **Training Precision**: 4-bit Quantization
 
+### Training Configuration
+
+| Parameter | Value |
+|-----------|-------|
+| Base Model | LLaMA2-7B |
+| Training Strategy | LoRA |
+| Learning Rate | 1e-5 |
+| Batch Size | 4 |
+| Gradient Accumulation Steps | 2 |
+| Training Epochs | 3 |
+| Maximum Sequence Length | 2,048 |
+| Warmup Ratio | 0.1 |
+| Max Gradient Norm | 0.3 |
+| DPO Beta (β) | 0.1 |
+| Loss Type | Sigmoid |
+| Training Data | Chosen & Rejected action pairs |
+
 ## Usage
 
 To use this model, you'll need to load both the base model and the LoRA adapter:
@@ -86,33 +108,82 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 
 ## Training Dataset
 
-The model was trained on [DATASET_DESCRIPTION]. The dataset consists of preference pairs for DPO training with prompts, chosen responses, and rejected responses.
+The model was trained on the PrefDrive dataset, a comprehensive collection of 74,040 driving sequences carefully annotated with driving preferences and driving decisions. Each entry in the dataset consists of:
+
+- A driving scenario description (s)
+- A preferred/chosen driving action with its reasoning and resulting waypoint (a_p)
+- A rejected driving action with its reasoning and resulting waypoint (a_r)
+
+This dataset captures various autonomous driving scenarios with emphasis on proper distance maintenance, trajectory smoothness, traffic rule compliance, and route adherence.
 
 ## Training Procedure
 
-The model was trained using the DPO method which directly optimizes a language model to align with human preferences without requiring a reward model. The training process uses pairwise comparisons between responses to update model parameters.
+The model was trained using the DPO method which directly optimizes a language model to align with driving preferences without requiring a reward model. The training process uses pairwise comparisons between preferred and rejected driving actions to update model parameters.
+
+### Methodology
+
+The PrefDrive methodology for autonomous driving is formulated as:
+
+$\mathcal{L}_{DPO} = -\mathbb{E}_{(s,a_p,a_r)\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(a_p|s)}{\pi_{ref}(a_p|s)} - \beta\log\frac{\pi_\theta(a_r|s)}{\pi_{ref}(a_r|s)}\Big)\Big]$
+
+where:
+- $\mathcal{D}$ represents our driving preference dataset
+- $s$ denotes the current driving scenario description
+- $a_p$ represents the preferred (chosen) driving action with its reasoning and resulting waypoint
+- $a_r$ represents the rejected driving action with its reasoning and resulting waypoint
+- $\pi_\theta$ is the policy model being trained
+- $\pi_{ref}$ is the initial reference model
+- $\beta$ controls the preference learning sensitivity (set to 0.1)
+- $\sigma$ represents the sigmoid function
 
-Key training parameters:
+This formulation explicitly shows how our model learns to favor chosen driving actions over rejected ones while maintaining reasonable deviation from the reference model's behavior.
+
+### Key Training Parameters
 - Learning rate: 1e-5
 - Number of epochs: 3
 - DPO beta: 0.1
-- Loss type: exo_pair
+- Loss type: Sigmoid
 - Max sequence length: 2048
 
 ## Evaluation Results
 
-| Metric | Value |
-|--------|-------|
-| Validation Loss | [VALIDATION_LOSS] |
-| [ADDITIONAL_METRIC] | [VALUE] |
+The model was evaluated in the CARLA simulator across different town environments. Here are the performance metrics:
+
+### Town 01 Performance
+
+| Metric | LMDrive (baseline) | PrefDrive (Ours) | Improvement |
+|--------|-------------------|-----------------|-------------|
+| Composite Score | 53.00 | 56.12 | +5.9% |
+| Penalty Score | 0.86 | 0.88 | +1.5% |
+| Route Completion | 59.10 | 64.15 | +8.5% |
+| Layout Collisions | 0.73 | 0.27 | -63.5% |
+| Traffic Light Violations | 0.22 | 0.16 | -28.1% |
+| Route Deviation | 1.32 | 1.36 | +3.0% |
+| Vehicle Blocked | 0.11 | 0.00 | -100.0% |
+
+### Town 04 Performance
+
+| Metric | LMDrive (baseline) | PrefDrive (Ours) | Improvement |
+|--------|-------------------|-----------------|-------------|
+| Composite Score | 60.11 | 65.93 | +9.7% |
+| Penalty Score | 0.93 | 0.96 | +3.2% |
+| Route Completion | 65.25 | 69.93 | +7.2% |
+| Layout Collisions | 0.00 | 0.00 | 0.0% |
+| Traffic Light Violations | 0.24 | 0.00 | -100.0% |
+| Route Deviation | 1.86 | 1.77 | -4.8% |
+| Vehicle Blocked | 0.00 | 0.00 | 0.0% |
+
+The results demonstrate significant improvements in crucial metrics, particularly in reducing traffic light violations and layout collisions while improving route completion.
 
 ## Limitations and Biases
 
 This model inherits the limitations and biases from the base LLaMa model. Additionally:
 
-- It's optimized for the specific domain and tasks covered in the training dataset
-- Performance on out-of-domain tasks may vary
+- It's optimized specifically for autonomous driving tasks and may not perform well in unrelated domains
+- Performance may vary in driving environments that differ significantly from the training data
 - The LoRA adaptation affects specific parameter matrices and may not fully transform the base model's capabilities
+- While the model shows improved performance in simulated environments (CARLA), its behavior in real-world driving scenarios would require further validation and safety testing
+- The model is designed to work with a specific autonomous driving stack and may require adaptation for different setups
 
 ## Ethical Considerations
 
@@ -142,4 +213,4 @@ When using this model, consider:
       archivePrefix={arXiv},
       primaryClass={cs.CL}
 }
-```
+```