fnc-distilbert-LoRA

A LoRA adapter for DistilBERT (uncased) that sorts consumer complaints about Bank of America into two product categories: Mortgage or Credit card or prepaid card. It was trained on 1,600 complaint narratives from the U.S. Consumer Financial Protection Bureau (CFPB) complaint database and reaches 0.94 accuracy on 200 held-out complaints.

This is one of three adapters from the project Lightweight Fine-Tuning of Foundation Models, which compares parameter-efficient fine-tuning methods on the same task. It scored highest of the three and trained fastest (about 3 minutes on one consumer GPU). The other two are fnc-gpt2-LoRA (GPT-2 with LoRA) and fnc-gpt2q4-QLoRA (GPT-2 with QLoRA).

Model details

Developed by	Vassilis Tsoubris (@vtsou359)
Model type	LoRA adapter (PEFT) for binary sequence classification
Base model	distilbert/distilbert-base-uncased
Language	English
License	Apache 2.0, same as the base model
Trainable parameters	887,042 of 67.8M (1.31%)
Adapter size	3.56 MB (adapter_model.safetensors)
Code	GitHub: Lightweight_FineTuning_FoundationModel
Collection	Business Case: Consumer Complaints in Banking

Labels

ID	Label
0	Mortgage
1	Credit card or prepaid card

The adapter does not store label names, so pass them when you load the base model, as in the example below.

How to use

pip install torch transformers peft

import torch
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "distilbert/distilbert-base-uncased"
adapter_id = "vtsou359/fnc-distilbert-LoRA"
id2label = {0: "Mortgage", 1: "Credit card or prepaid card"}
label2id = {label: i for i, label in id2label.items()}

tokenizer = AutoTokenizer.from_pretrained(base_id)
# The warning about newly initialized classifier weights is expected:
# the adapter supplies the trained pre_classifier and classifier layers.
base_model = AutoModelForSequenceClassification.from_pretrained(
    base_id, num_labels=2, id2label=id2label, label2id=label2id
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

complaints = [
    "I paid my credit card balance in full before the due date, but Bank of America still charged me a late fee and interest.",
    "Bank of America has not applied my last two mortgage payments and is now threatening to start foreclosure on my home.",
]
inputs = tokenizer(complaints, truncation=True, padding=True, return_tensors="pt").to(device)

with torch.no_grad():
    logits = model(**inputs).logits
probs = logits.softmax(dim=-1)

for text, p in zip(complaints, probs):
    label_id = int(p.argmax())
    print(f"{id2label[label_id]} ({float(p[label_id]):.2f}): {text[:70]}...")

To fold the adapter into the base weights for slightly faster inference, run model = model.merge_and_unload() after loading.

Evaluation

Data

Both evaluation sets are balanced between the two classes:

• Test set (200 complaints, 100 per class). Sampled from the same filtered complaints after removing every narrative that appears in the 2,000-sample training pool. It was not used for training or model selection.
• Validation set (400 complaints). A stratified 20% split of the 2,000-sample pool (called test in the training notebook). It was used to evaluate each epoch and to keep the checkpoint with the lowest validation loss.

Metrics

Accuracy, plus precision, recall and F1 for the Credit card or prepaid card class (label 1), computed with scikit-learn's binary defaults.

Results

Data	Loss	Accuracy	Precision	Recall	F1
Test (n=200)	0.2681	0.9400	0.8929	1.0000	0.9434
Validation (n=400)	0.2784	0.9400	0.9000	0.9900	0.9429
Base model, untrained head: test	0.6967	0.5000	0.5000	0.0100	0.0196
Base model, untrained head: validation	0.6899	0.4975	0.0000	0.0000	0.0000

The base-model rows use a randomly initialized classification head, so they show chance level rather than a meaningful baseline. With 200 test complaints, the 95% Wilson interval for test accuracy is 0.90 to 0.97.

Confusion matrix on the test set:

True \ Predicted	Mortgage	Credit card or prepaid card
Mortgage	88	12
Credit card or prepaid card	0	100

Every test error is a mortgage complaint labeled as a credit card complaint, and 22 of the 24 validation errors go the same way.

Comparison with the sibling adapters (test set)

Adapter	Base model	Method	Trainable parameters	Accuracy	F1
fnc-distilbert-LoRA (this model)	DistilBERT	LoRA	887,042 (1.31%)	0.9400	0.9434
fnc-gpt2q4-QLoRA	GPT-2, 4-bit during training	QLoRA	886,272 (0.71%)	0.9150	0.9163
fnc-gpt2-LoRA	GPT-2	LoRA	886,272 (0.71%)	0.7550	0.7562

Intended uses and limitations

Intended uses

• Classifying English complaint narratives about Bank of America, written in the style of CFPB submissions, as mortgage or credit card/prepaid card complaints.
• A compact, reproducible example of LoRA fine-tuning for text classification, and a baseline for complaint-routing experiments.

Out of scope

• Other products. The model has no "other" class: complaints about checking accounts, auto loans, debt collection or any other product still receive one of the two labels.
• Other institutions and text types. Performance on other banks, emails, chat or call transcripts, and non-English text has not been tested.
• Unreviewed decisions about people. Do not use it to make or prioritize decisions that affect consumers without human review.

Limitations

• Small, narrow training set: 1,600 complaints, one company, two products.
• Small evaluation sets: a handful of errors moves the scores noticeably (see the interval above).
• Skewed errors: misclassifications run almost entirely from mortgage to credit card. If missing a mortgage complaint is costly, review low-confidence predictions or tune the decision threshold.
• Truncation: inputs are cut at 512 tokens. The median narrative is about 195 words, but the end of long complaints is ignored.
• Redacted text: the CFPB replaces personal details with XXXX, and about 89% of the training narratives contain such markers. Unredacted text is a shift from training conditions.
• One training run: trained once with seed 42; variance across seeds was not measured.

Bias and risks

The CFPB database contains only complaints that consumers chose to file, and narratives are published only when the consumer consents, so the data does not represent all customers or problems. Performance across demographic groups, regions or writing styles was not evaluated. Narratives can still contain personal details despite redaction.

Training data

The source is BEE-spoke-data/consumer-finance-complaints, a Hub copy of the CFPB Consumer Complaint Database (CC0-1.0; 4.71M complaints received between December 2011 and February 2024). The project kept complaints that:

1. name BANK OF AMERICA, NATIONAL ASSOCIATION as the company,
2. concern the product Credit card or prepaid card or Mortgage, and
3. include a consumer complaint narrative.

It then sampled 1,000 complaints per product (shuffle seed 42) and kept only the narrative (input) and product (label) columns. The 2,000 complaints were split 80/20, stratified by label (seed 23): 1,600 for training and 400 for validation. Narratives have a median length of 195 words (mean 263, maximum 2,475).

Training procedure

Preprocessing

Narratives were tokenized with the DistilBERT uncased tokenizer, truncated to 512 tokens and padded.

LoRA configuration

Setting	Value
Rank (r)	8
lora_alpha	16
lora_dropout	0.1
Target modules	q_lin, k_lin, v_lin, out_lin (all attention projections in the 6 transformer blocks)
Bias	none
Task type	SEQ_CLS
Fully trained layers	pre_classifier, classifier

The 887,042 trainable parameters are 294,912 LoRA weights plus the full pre_classifier (590,592) and classifier (1,538) layers. For sequence classification, PEFT trains and saves the classification head (modules_to_save); because it matches module names by suffix, pre_classifier is included as well.

Hyperparameters

Setting	Value
Epochs	11
Learning rate	2e-5, linear decay, no warmup
Batch size	16 per device × 4 gradient accumulation steps = 64
Optimizer	AdamW (Trainer default)
Weight decay	0.01
Precision	fp32
Seed	42
Optimizer steps	275
Checkpoint selection	Lowest validation loss (load_best_model_at_end), which was the last epoch

Training log (validation set)

Epoch	Training loss	Validation loss	Accuracy	Precision	Recall	F1
1	0.6821	0.6687	0.8375	0.8000	0.9000	0.8471
2	0.6645	0.6474	0.8625	0.8607	0.8650	0.8628
3	0.6390	0.6194	0.8825	0.8626	0.9100	0.8856
4	0.6081	0.5788	0.9000	0.8810	0.9250	0.9024
5	0.5629	0.5234	0.9150	0.8952	0.9400	0.9171
6	0.5022	0.4559	0.9300	0.8981	0.9700	0.9327
7	0.4376	0.3895	0.9350	0.8955	0.9850	0.9381
8	0.3785	0.3372	0.9400	0.9037	0.9850	0.9426
9	0.3323	0.3026	0.9400	0.9000	0.9900	0.9429
10	0.3063	0.2843	0.9400	0.9000	0.9900	0.9429
11	0.2921	0.2784	0.9400	0.9000	0.9900	0.9429

Validation accuracy plateaued at 0.94 from epoch 8, while validation loss kept falling slowly.

Compute

One NVIDIA GeForce RTX 4070 Ti on Windows. Training took 183 seconds (about 3 minutes). Peak allocated GPU memory over the notebook session was 2,675 MiB.

Framework versions

• PEFT 0.14.0
• Transformers 4.48
• PyTorch 2.5.1 (CUDA 12.4)

Citation

@misc{tsoubris2025fncdistilbertlora,
  author       = {Tsoubris, Vassilis},
  title        = {fnc-distilbert-LoRA: A DistilBERT LoRA Adapter for Classifying Banking Complaints},
  year         = {2025},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vtsou359/fnc-distilbert-LoRA}}
}

References

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021), arXiv:2106.09685
• Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019), arXiv:1910.01108
• Consumer Financial Protection Bureau, Consumer Complaint Database

Contact

Questions and feedback are welcome in the Community tab.