fnc-gpt2-LoRA

A LoRA adapter for GPT-2 (124M parameters) that sorts consumer complaints about Bank of America into two product categories: Mortgage or Credit card or prepaid card. It was trained on 1,600 complaint narratives from the U.S. Consumer Financial Protection Bureau (CFPB) complaint database and reaches 0.755 accuracy on 200 held-out complaints.

This is one of three adapters from the project Lightweight Fine-Tuning of Foundation Models, which compares parameter-efficient fine-tuning methods on the same task. It is the weakest of the three and serves mainly as a comparison point: on the same data, fnc-distilbert-LoRA reached 0.94 and fnc-gpt2q4-QLoRA, which uses the same LoRA settings on a 4-bit GPT-2, reached 0.915.

Model details

Developed by	Vassilis Tsoubris (@vtsou359)
Model type	LoRA adapter (PEFT) for binary sequence classification
Base model	openai-community/gpt2
Language	English
License	MIT, same as the base model
Trainable parameters	886,272 of 125.3M (0.71%)
Adapter size	3.56 MB (adapter_model.safetensors)
Code	GitHub: Lightweight_FineTuning_FoundationModel
Collection	Business Case: Consumer Complaints in Banking

Labels

ID	Label
0	Mortgage
1	Credit card or prepaid card

The adapter does not store label names, so pass them when you load the base model, as in the example below.

How to use

pip install torch transformers peft

import torch
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "openai-community/gpt2"
adapter_id = "vtsou359/fnc-gpt2-LoRA"
id2label = {0: "Mortgage", 1: "Credit card or prepaid card"}
label2id = {label: i for i, label in id2label.items()}

tokenizer = AutoTokenizer.from_pretrained(base_id)
tokenizer.pad_token = tokenizer.eos_token  # GPT-2 has no pad token; training padded with EOS

# The warning about a newly initialized score layer is expected:
# the adapter supplies the trained classification head.
base_model = AutoModelForSequenceClassification.from_pretrained(
    base_id, num_labels=2, id2label=id2label, label2id=label2id
)
base_model.config.pad_token_id = tokenizer.pad_token_id  # required for batched inputs
# PEFT may warn that it is setting fan_in_fan_out=True for GPT-2's Conv1D layers; this is expected.
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

complaints = [
    "I paid my credit card balance in full before the due date, but Bank of America still charged me a late fee and interest.",
    "Bank of America has not applied my last two mortgage payments and is now threatening to start foreclosure on my home.",
]
inputs = tokenizer(complaints, truncation=True, padding=True, return_tensors="pt").to(device)

with torch.no_grad():
    logits = model(**inputs).logits
probs = logits.softmax(dim=-1)

for text, p in zip(complaints, probs):
    label_id = int(p.argmax())
    print(f"{id2label[label_id]} ({float(p[label_id]):.2f}): {text[:70]}...")

GPT-2 classifies from the last non-padding token, which is why pad_token_id must be set for batched inputs.

Evaluation

Data

Both evaluation sets are balanced between the two classes:

• Test set (200 complaints, 100 per class). Sampled from the same filtered complaints after removing every narrative that appears in the 2,000-sample training pool. It was not used for training or model selection.
• Validation set (400 complaints). A stratified 20% split of the 2,000-sample pool (called test in the training notebook). It was used to evaluate each epoch and to keep the checkpoint with the lowest validation loss.

Metrics

Accuracy, plus precision, recall and F1 for the Credit card or prepaid card class (label 1), computed with scikit-learn's binary defaults.

Results

Data	Loss	Accuracy	Precision	Recall	F1
Test (n=200)	0.5197	0.7550	0.7525	0.7600	0.7562
Validation (n=400)	0.5573	0.7275	0.7136	0.7600	0.7361
GPT-2, untrained head: test	4.1444	0.5000	0.5000	1.0000	0.6667
GPT-2, untrained head: validation	2.8749	0.5075	1.0000	0.0150	0.0296

The GPT-2 rows use a randomly initialized classification head, which labels almost everything as one class, so they show chance level rather than a meaningful baseline. With 200 test complaints, the 95% Wilson interval for test accuracy is 0.69 to 0.81.

Confusion matrix on the test set:

True \ Predicted	Mortgage	Credit card or prepaid card
Mortgage	75	25
Credit card or prepaid card	24	76

Errors are split almost evenly between the two directions.

Comparison with the sibling adapters (test set)

Adapter	Base model	Method	Trainable parameters	Accuracy	F1
fnc-distilbert-LoRA	DistilBERT	LoRA	887,042 (1.31%)	0.9400	0.9434
fnc-gpt2q4-QLoRA	GPT-2, 4-bit during training	QLoRA	886,272 (0.71%)	0.9150	0.9163
fnc-gpt2-LoRA (this model)	GPT-2	LoRA	886,272 (0.71%)	0.7550	0.7562

Intended uses and limitations

Intended uses

• A reference point for comparing LoRA and QLoRA on GPT-2, and LoRA on GPT-2 versus DistilBERT, for the same classification task.
• Classifying English complaint narratives about Bank of America as mortgage or credit card/prepaid card complaints. For practical use, fnc-distilbert-LoRA is both more accurate and cheaper to run.

Out of scope

• Other products. The model has no "other" class: complaints about checking accounts, auto loans, debt collection or any other product still receive one of the two labels.
• Other institutions and text types. Performance on other banks, emails, chat or call transcripts, and non-English text has not been tested.
• Unreviewed decisions about people. Do not use it to make or prioritize decisions that affect consumers without human review.

Limitations

• Likely under-trained: validation loss was still falling at the last epoch (0.5621 to 0.5573), and accuracy leveled off around 0.73 on validation.
• Small, narrow training set: 1,600 complaints, one company, two products.
• Small evaluation sets: a handful of errors moves the scores noticeably (see the interval above).
• Truncation: inputs are cut at 1,024 tokens, so the end of very long complaints is ignored.
• Redacted text: the CFPB replaces personal details with XXXX, and about 89% of the training narratives contain such markers. Unredacted text is a shift from training conditions.
• One training run: trained once with seed 42; variance across seeds was not measured.

Bias and risks

The CFPB database contains only complaints that consumers chose to file, and narratives are published only when the consumer consents, so the data does not represent all customers or problems. Performance across demographic groups, regions or writing styles was not evaluated. Narratives can still contain personal details despite redaction. GPT-2 was pretrained on unfiltered web text and carries the biases documented in its model card.

Training data

The source is BEE-spoke-data/consumer-finance-complaints, a Hub copy of the CFPB Consumer Complaint Database (CC0-1.0; 4.71M complaints received between December 2011 and February 2024). The project kept complaints that:

1. name BANK OF AMERICA, NATIONAL ASSOCIATION as the company,
2. concern the product Credit card or prepaid card or Mortgage, and
3. include a consumer complaint narrative.

It then sampled 1,000 complaints per product (shuffle seed 42) and kept only the narrative (input) and product (label) columns. The 2,000 complaints were split 80/20, stratified by label (seed 23): 1,600 for training and 400 for validation. Narratives have a median length of 195 words (mean 263, maximum 2,475).

Training procedure

Preprocessing

Narratives were tokenized with the GPT-2 byte-level BPE tokenizer and truncated to 1,024 tokens. GPT-2 has no padding token, so the end-of-text token was used for padding and set as pad_token_id in the model config.

LoRA configuration

Setting	Value
Rank (r)	6
lora_alpha	32
lora_dropout	0.1
Target modules	c_attn, c_proj, c_fc in all 12 transformer blocks
Bias	none
Task type	SEQ_CLS
Fully trained layers	score (classification head)

c_attn is the fused query/key/value projection, c_fc the MLP input projection, and c_proj matches both the attention output and the MLP output projections. The 886,272 trainable parameters are 884,736 LoRA weights plus the 1,536-parameter score head, which PEFT trains and saves with the adapter (modules_to_save).

Hyperparameters

Setting	Value
Epochs	10
Learning rate	2e-5, linear decay, no warmup
Batch size	8 per device × 4 gradient accumulation steps = 32
Optimizer	AdamW (Trainer default)
Weight decay	0.01
Precision	fp32
Seed	42
Optimizer steps	500
Checkpoint selection	Lowest validation loss (load_best_model_at_end), which was the last epoch

Training log (validation set)

Epoch	Training loss	Validation loss	Accuracy	Precision	Recall	F1
1	2.1949	1.4603	0.5025	0.5333	0.0400	0.0744
2	0.9612	0.7796	0.5450	0.5474	0.5200	0.5333
3	0.7497	0.6950	0.5900	0.6023	0.5300	0.5638
4	0.6856	0.6601	0.6250	0.6190	0.6500	0.6341
5	0.6506	0.6345	0.6700	0.6545	0.7200	0.6857
6	0.6231	0.6117	0.6950	0.6840	0.7250	0.7039
7	0.6119	0.5929	0.7025	0.6919	0.7300	0.7105
8	0.5818	0.5736	0.7250	0.7064	0.7700	0.7368
9	0.5683	0.5621	0.7300	0.7190	0.7550	0.7366
10	0.5461	0.5573	0.7275	0.7136	0.7600	0.7361

Compute

One NVIDIA GeForce RTX 4070 Ti on Windows. Training took 1,157 seconds (about 19 minutes). Peak allocated GPU memory over the notebook session was 10,636 MiB, versus 3,108 MiB for the QLoRA run.

Framework versions

• PEFT 0.14.0
• Transformers 4.48
• PyTorch 2.5.1 (CUDA 12.4)

Citation

@misc{tsoubris2025fncgpt2lora,
  author       = {Tsoubris, Vassilis},
  title        = {fnc-gpt2-LoRA: A GPT-2 LoRA Adapter for Classifying Banking Complaints},
  year         = {2025},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vtsou359/fnc-gpt2-LoRA}}
}

References

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021), arXiv:2106.09685
• Radford et al., Language Models are Unsupervised Multitask Learners (2019), OpenAI
• Consumer Financial Protection Bureau, Consumer Complaint Database

Contact

Questions and feedback are welcome in the Community tab.