Instructions to use Denizhaan/imdb-deberta-lora-v2-fixed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Denizhaan/imdb-deberta-lora-v2-fixed with PEFT:
from peft import PeftModel from transformers import AutoModelForSequenceClassification base_model = AutoModelForSequenceClassification.from_pretrained("microsoft/deberta-v3-base") model = PeftModel.from_pretrained(base_model, "Denizhaan/imdb-deberta-lora-v2-fixed") - Notebooks
- Google Colab
- Kaggle
DeBERTa-v3-base + LoRA — IMDB Sentiment Classification
LoRA-adapted microsoft/deberta-v3-base for binary sentiment classification on the IMDB Large Movie Review Dataset. One of four base models in a parameter-efficient stacking ensemble described in "Parameter-Efficient Fine-Tuning Ensembles for Sentiment Analysis" (IDAP 2026).
Results (25,000-example IMDB test set)
| Metric | Value |
|---|---|
| Accuracy | 95.92% |
| F1 | 95.95% |
| Precision | 95.23% |
| Recall | 96.69% |
| ROC-AUC | 99.14% |
LoRA Configuration
| Hyperparameter | Value |
|---|---|
| Rank (r) | 32 |
| Alpha | 64 |
| Dropout | 0.1 |
| Target modules | query_proj, value_proj |
| Trainable parameters | 1,181,186 (0.64% of 185.6M total) |
| Precision | fp32 (fp16 causes gradient instability due to ELECTRA-style pretraining) |
Files
adapter_model.safetensors— LoRA A/B matricesextra_weights.pt— classifier and pooler weights, saved separately (see note below)
⚠️ Loading note
DeBERTa-v3 with task_type=SEQ_CLS does not save the pooler/classifier weights inside the standard PEFT adapter checkpoint. Loading only adapter_model.safetensors via PeftModel.from_pretrained() results in randomly initialized classifier/pooler layers and near-random predictions. Load as follows:
import torch
import safetensors.torch as st
from transformers import DebertaV2Tokenizer, DebertaV2ForSequenceClassification
from peft import LoraConfig, get_peft_model, TaskType
from huggingface_hub import hf_hub_download
REPO = "Denizhaan/imdb-deberta-lora-v2-fixed"
tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")
base = DebertaV2ForSequenceClassification.from_pretrained(
"microsoft/deberta-v3-base", num_labels=2, torch_dtype=torch.float32,
ignore_mismatched_sizes=True
)
lora_config = LoraConfig(
task_type=TaskType.SEQ_CLS, r=32, lora_alpha=64, lora_dropout=0.1,
target_modules=["query_proj", "value_proj"], bias="none"
)
model = get_peft_model(base, lora_config)
adapter_path = hf_hub_download(REPO, "adapter_model.safetensors")
extra_path = hf_hub_download(REPO, "extra_weights.pt")
weights = st.load_file(adapter_path)
extra = torch.load(extra_path, map_location="cpu")
new_weights = {}
for k, v in weights.items():
if "lora_A.weight" in k:
new_weights[k.replace("lora_A.weight", "lora_A.default.weight")] = v
elif "lora_B.weight" in k:
new_weights[k.replace("lora_B.weight", "lora_B.default.weight")] = v
new_weights["base_model.model.classifier.modules_to_save.default.weight"] = extra["classifier.weight"]
new_weights["base_model.model.classifier.modules_to_save.default.bias"] = extra["classifier.bias"]
new_weights["base_model.model.classifier.original_module.weight"] = extra["classifier.weight"]
new_weights["base_model.model.classifier.original_module.bias"] = extra["classifier.bias"]
new_weights["base_model.model.pooler.dense.weight"] = extra["pooler.weight"]
new_weights["base_model.model.pooler.dense.bias"] = extra["pooler.bias"]
model.load_state_dict(new_weights, strict=False)
model = model.to(torch.float32)
model.eval()
Citation
If you use this model, please cite our paper (details in the GitHub repository).
Framework versions
- PEFT 0.19.1
- Downloads last month
- 12
Model tree for Denizhaan/imdb-deberta-lora-v2-fixed
Base model
microsoft/deberta-v3-base