DeBERTa-v3-base + LoRA — IMDB Sentiment Classification

LoRA-adapted microsoft/deberta-v3-base for binary sentiment classification on the IMDB Large Movie Review Dataset. One of four base models in a parameter-efficient stacking ensemble described in "Parameter-Efficient Fine-Tuning Ensembles for Sentiment Analysis" (IDAP 2026).

Results (25,000-example IMDB test set)

Metric Value
Accuracy 95.92%
F1 95.95%
Precision 95.23%
Recall 96.69%
ROC-AUC 99.14%

LoRA Configuration

Hyperparameter Value
Rank (r) 32
Alpha 64
Dropout 0.1
Target modules query_proj, value_proj
Trainable parameters 1,181,186 (0.64% of 185.6M total)
Precision fp32 (fp16 causes gradient instability due to ELECTRA-style pretraining)

Files

  • adapter_model.safetensors — LoRA A/B matrices
  • extra_weights.pt — classifier and pooler weights, saved separately (see note below)

⚠️ Loading note

DeBERTa-v3 with task_type=SEQ_CLS does not save the pooler/classifier weights inside the standard PEFT adapter checkpoint. Loading only adapter_model.safetensors via PeftModel.from_pretrained() results in randomly initialized classifier/pooler layers and near-random predictions. Load as follows:

import torch
import safetensors.torch as st
from transformers import DebertaV2Tokenizer, DebertaV2ForSequenceClassification
from peft import LoraConfig, get_peft_model, TaskType
from huggingface_hub import hf_hub_download

REPO = "Denizhaan/imdb-deberta-lora-v2-fixed"

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")
base = DebertaV2ForSequenceClassification.from_pretrained(
    "microsoft/deberta-v3-base", num_labels=2, torch_dtype=torch.float32,
    ignore_mismatched_sizes=True
)

lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS, r=32, lora_alpha=64, lora_dropout=0.1,
    target_modules=["query_proj", "value_proj"], bias="none"
)
model = get_peft_model(base, lora_config)

adapter_path = hf_hub_download(REPO, "adapter_model.safetensors")
extra_path = hf_hub_download(REPO, "extra_weights.pt")
weights = st.load_file(adapter_path)
extra = torch.load(extra_path, map_location="cpu")

new_weights = {}
for k, v in weights.items():
    if "lora_A.weight" in k:
        new_weights[k.replace("lora_A.weight", "lora_A.default.weight")] = v
    elif "lora_B.weight" in k:
        new_weights[k.replace("lora_B.weight", "lora_B.default.weight")] = v

new_weights["base_model.model.classifier.modules_to_save.default.weight"] = extra["classifier.weight"]
new_weights["base_model.model.classifier.modules_to_save.default.bias"] = extra["classifier.bias"]
new_weights["base_model.model.classifier.original_module.weight"] = extra["classifier.weight"]
new_weights["base_model.model.classifier.original_module.bias"] = extra["classifier.bias"]
new_weights["base_model.model.pooler.dense.weight"] = extra["pooler.weight"]
new_weights["base_model.model.pooler.dense.bias"] = extra["pooler.bias"]

model.load_state_dict(new_weights, strict=False)
model = model.to(torch.float32)
model.eval()

Citation

If you use this model, please cite our paper (details in the GitHub repository).

Framework versions

  • PEFT 0.19.1
Downloads last month
12
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Denizhaan/imdb-deberta-lora-v2-fixed

Adapter
(34)
this model