Consumer Complaint Classifier (DistilBERT)

distilbert-base-uncased fine-tuned to route a customer's written complaint to one of 5 product categories:

credit_card · credit_reporting · debt_collection · mortgages_and_loans · retail_banking

Results (held-out test set, 12,430 complaints)

Metric Score
Accuracy 0.8612
Macro precision 0.8499
Macro recall 0.8486
Macro F1 0.8491

Per-class results:

Class Precision Recall F1 Support
credit_card 0.8302 0.7902 0.8097 1,497
credit_reporting 0.8917 0.8967 0.8942 5,617
debt_collection 0.7936 0.7871 0.7903 2,100
mortgages_and_loans 0.8555 0.8615 0.8585 1,870
retail_banking 0.8784 0.9071 0.8925 1,346

Comparison with models trained from scratch (same split)

Model Accuracy Macro F1
SimpleRNN 0.7973 0.7747
LSTM 0.8191 0.8063
GRU 0.8250 0.8184
DistilBERT (this model) 0.8612 0.8491

The SimpleRNN, LSTM and GRU baselines are Keras models (128-d embedding, 128 units, ~3.9M parameters) trained on the same train/validation/test split. Scores for the from-scratch models vary by about ±0.5 points between training runs; the ranking (DistilBERT > GRU ≈ LSTM > SimpleRNN) was the same in every run.

Training

  • Data: Kaggle Consumer Complaints Dataset for NLP. 162,421 rows reduced to 124,300 complaints after removing empty narratives, exact duplicates and texts with conflicting labels.
  • Split: stratified 80/10/10 (99,440 train / 12,430 validation / 12,430 test), seed 42.
  • Class imbalance: credit_reporting is ~45% of the data, so training used class-weighted cross-entropy (weights computed from the training set only).
  • Hyperparameters: learning rate 2e-5, weight decay 0.01, warmup 900 steps, batch size 32, 3 epochs, fp16, max length 256. The best checkpoint was selected by validation macro-F1.

Usage

The model expects text cleaned the same way as the training data: lowercase, no punctuation or numbers, no stop words, lemmatized.

import re, nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from transformers import pipeline

nltk.download("stopwords"); nltk.download("wordnet")
STOP = set(stopwords.words("english"))
lemm = WordNetLemmatizer()

def clean_text(text):
    text = re.sub(r"[^a-z\s]", " ", str(text).lower())
    return " ".join(lemm.lemmatize(w) for w in text.split() if w not in STOP)

clf = pipeline("text-classification",
               model="mai-data-ai/complaint-distilbert", top_k=None)

complaint = "A debt collector keeps calling me about a loan I already paid off."
print(clf(clean_text(complaint), truncation=True, max_length=256))

Limitations

  • Categories overlap. Most of the 1,725 test errors involve credit_reporting and debt_collection (credit_reporting → debt_collection: 325, debt_collection → credit_reporting: 295).
  • Trained on English complaint text from a single source (US consumer complaints), so performance on other domains or languages is not guaranteed.
  • Only 5 coarse product categories; it does not predict issue type or sub-product.
  • Intended for assisting complaint routing, not for fully automated decisions without human review.
Downloads last month
97
Safetensors
Model size
67M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mai-data-ai/complaint-distilbert

Finetuned
(12,888)
this model

Space using mai-data-ai/complaint-distilbert 1