Create README.md

README.md

@@ -0,0 +1,161 @@
+---
+license: mit
+base_model: microsoft/deberta-v3-large
+library_name: transformers
+pipeline_tag: text-classification
+tags:
+- deberta
+- deberta-v3
+- multiple-choice
+- question-answering
+- awp
+metrics:
+- map@3
+---
+
+# QSolver_Encoder_V2
+
+QSolver_Encoder_V2 is an encoder-based language model fine-tuned for 5-choice scientific multiple-choice question answering. Built on top of `microsoft/deberta-v3-large`, the model utilizes partial layer freezing, dynamic 3D sequence collating, 8-bit AdamW optimization, and custom **Adversarial Weight Perturbation (AWP)** to prevent overfitting and improve generalization across distribution shifts.
+
+Evaluation is performed using 5-Fold Stratified Cross-Validation monitored with Mean Average Precision at 3 (MAP@3).
+
+---
+
+## Model Details
+
+- **Model Name:** QSolver_Encoder_V2
+- **Repository ID:** `dahaludba/QSolver_Encoder_V2`
+- **Base Model:** `microsoft/deberta-v3-large`
+- **Architecture:** `AutoModelForMultipleChoice`
+- **Maximum Sequence Length:** 512 tokens
+- **Optimization Precision:** 32-bit Floating Point (FP32)
+- **Optimizer:** 8-bit AdamW (`optim="adamw_8bit"` via `bitsandbytes`)
+- **Total Training Duration:** 13 hours 5 minutes
+- **License:** MIT License
+
+---
+
+## Training Strategy & Features
+
+### 1. Partial Encoder Layer Freezing
+To preserve low-level semantic representation and prevent catastrophic forgetting, the bottom 12 encoder layers (`deberta.encoder.layer.0` through `11`) out of 24 layers were completely frozen. Only the top 12 layers (`12` through `23`) and the multiple-choice classification head were fine-tuned.
+
+### 2. Adversarial Weight Perturbation (AWP)
+To increase regularization and combat overfitting on small datasets, training uses a custom `AWPTrainer`. 
+- **Activation:** Starts after completing epoch 1.0.
+- **Target Layers:** Active `word_embeddings` parameters.
+- **Step Size ($\alpha$):** `1e-3`
+- **Epsilon Bound ($\epsilon$):** `1e-2` (limits weight perturbation magnitude to within 1% of original parameter weights).
+
+During each backward pass, weight embeddings are perturbed in the direction of maximum loss increase, gradients are computed, and original clean weights are restored prior to executing the optimizer step.
+
+### 3. Dynamic 3D Sequence Collating
+Question-option pairs are tokenized into 5 choice paths per sample. A `CustomDataCollator` dynamically pads sequences within each batch and reshapes tensors into 3D shapes `(batch_size, num_choices, max_seq_len)` for compute-efficient batch passes through `AutoModelForMultipleChoice`.
+
+---
+
+## Hyperparameters
+
+| Hyperparameter | Value |
+| :--- | :--- |
+| Peak Learning Rate | 8e-6 |
+| LR Scheduler | Cosine Decay |
+| Warmup Steps | 30 |
+| Optimizer | `adamw_8bit` (bitsandbytes) |
+| Weight Decay | 0.01 |
+| Per Device Train Batch Size | 1 |
+| Per Device Eval Batch Size | 1 |
+| Gradient Accumulation Steps | 16 (Effective Batch Size = 16) |
+| Training Epochs | 4 per fold |
+| Floating Point Precision | Full FP32 (`fp16=False`) |
+| Gradient Checkpointing | Enabled (`use_reentrant=False`, `use_cache=False`) |
+| Frozen Layers | Encoder layers 0 to 11 |
+| Metric Monitored | MAP@3 (`eval_map@3`) |
+| Seed | 42 |
+
+---
+
+## Experiment Tracking & Cross-Validation Results
+
+Total training across all 5 folds completed in **13 hours and 5 minutes**. Full run logs, learning rate curves, and loss trajectories are available via Weights & Biases:
+
+- **Fold 1:** [W&B Run chh1vxqe](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/chh1vxqe?nw=nwuser24f2002963)
+- **Fold 2:** [W&B Run eu537lgm](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/eu537lgm?nw=nwuser24f2002963)
+- **Fold 3:** [W&B Run t3gjczof](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/t3gjczof?nw=nwuser24f2002963)
+- **Fold 4:** [W&B Run qkukzynk](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/qkukzynk?nw=nwuser24f2002963)
+- **Fold 5:** [W&B Run 92ksg28n](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/92ksg28n?nw=nwuser24f2002963)
+
+---
+
+## Evaluation Metric
+
+Performance is measured using Mean Average Precision at 3 (MAP@3):
+
+$$\text{MAP@3} = \frac{1}{U} \sum_{i=1}^{U} \sum_{k=1}^{\min(P, 3)} P(k) \times \text{rel}(k)$$
+
+Where $P(k)$ is the precision at rank $k$, and $\text{rel}(k)$ indicates whether the option at rank $k$ is the ground-truth label.
+
+---
+
+## How to Load and Run Inference
+
+```python
+import torch
+import numpy as np
+import itertools
+from transformers import AutoTokenizer, AutoModelForMultipleChoice
+
+REPO_ID = "dahaludba/QSolver_Encoder_V2"
+SUBFOLDER = "fold_1"
+
+# Load Tokenizer & Model
+tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
+model = AutoModelForMultipleChoice.from_pretrained(REPO_ID, subfolder=SUBFOLDER)
+model.eval()
+
+# Sample Question and Options
+question = "Which organelle is responsible for cellular respiration in eukaryotic cells?"
+options = [
+    "Lysosome",
+    "Mitochondria",
+    "Chloroplast",
+    "Golgi Apparatus",
+    "Peroxisome"
+]
+
+# Pair Question with Options
+first_sentences = [question] * 5
+second_sentences = options
+
+# Tokenize paired sequences
+inputs = tokenizer(
+    first_sentences,
+    second_sentences,
+    truncation=True,
+    max_length=512,
+    padding=True,
+    return_tensors="pt"
+)
+
+# Reshape inputs to 3D tensors (batch_size=1, num_choices=5, seq_len)
+batch_inputs = {
+    k: v.unsqueeze(0) for k, v in inputs.items()
+}
+
+with torch.no_grad():
+    outputs = model(**batch_inputs)
+    logits = outputs.logits.cpu().numpy()[0]
+
+# Rank top choices
+option_letters = ["A", "B", "C", "D", "E"]
+top3_indices = np.argsort(-logits)[:3]
+top3_predictions = [f"{option_letters[idx]}: {options[idx]}" for idx in top3_indices]
+
+print("Top 3 Predictions:", top3_predictions)
+```
+
+---
+
+## License
+
+This model and all repository contents are distributed under the **MIT License**.