🎬 IMDb Sentiment Analysis with DistilBERT

A Comparative Study of Embedding-Based and Transformer-Based Architectures for IMDb Sentiment Analysis

This repository contains a fine-tuned DistilBERT model for binary sentiment classification (Positive/Negative) of movie reviews. The model was developed as part of a comparative study evaluating simple neural networks, recurrent networks (BiLSTM), and transformer-based architectures on the IMDB dataset.


πŸ‘€ Author

Samad Sohrab


πŸ“Œ Model Summary

  • Base Model: distilbert-base-uncased
  • Task: Binary Text Classification (Sentiment Analysis)
  • Dataset: IMDB Movie Reviews
  • Test Accuracy: 89.95%
  • Framework: PyTorch + Hugging Face Transformers

πŸ—οΈ Model Architecture

The model is built on top of distilbert-base-uncased with a classification head:

Component Details
Base Transformer DistilBERT (6 layers, 768 hidden size, 12 attention heads)
Classifier Head Linear layer (768 β†’ 2) with Dropout
Max Token Length 256
Output Binary labels (LABEL_0 = Negative, LABEL_1 = Positive)

πŸ“Š Performance Comparison

The following table compares the proposed DistilBERT model with baseline architectures evaluated in the same study:

Model Architecture Test Accuracy Final Validation Loss
Baseline Embedding + GlobalAveragePooling + Dense 86.04% 0.323
BiLSTM Embedding + Bidirectional LSTM + Dense 85.44% 0.328
DistilBERT (This Model) Fine-tuned Transformer 89.95% 0.336

Key Finding: While recurrent and simple feedforward networks struggle to generalize on IMDB, the transformer-based DistilBERT achieves the highest accuracy, demonstrating the effectiveness of pre-trained language representations.


πŸš€ How to Use

1. Install Dependencies

pip install transformers torch

2. Load the Model and Predict

from transformers import pipeline classifier = pipeline("text-classification", model="smd20/imdb-distilbert-sentiment") result = classifier("This movie was absolutely fantastic!") print(result)

3. Using PyTorch Directly

import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification model_name = "smd20/imdb-distilbert-sentiment" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) text = "An unforgettable cinematic experience." inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=256) with torch.no_grad(): logits = model(**inputs).logits predicted_class = torch.argmax(logits, dim=-1).item() print(f"Predicted Label: {'Positive' if predicted_class == 1 else 'Negative'}")


πŸ‹οΈ Training Procedure

Hyperparameters

Parameter Value
Epochs 2
Batch Size 16
Learning Rate 5e-5 (AdamW)
Max Sequence Length 256
Training Samples 10,000
Validation Split 10,000 (from test set used as validation)

Training Results

Epoch Training Loss Validation Loss Validation Accuracy
1 0.3598 0.2688 89.58%
2 0.1919 0.3362 89.95%

Note: The model shows slight overfitting after epoch 1, suggesting that training for 1 epoch may be sufficient for optimal generalization.


⚠️ Limitations & Bias

  • Language: The model is trained on English text only. It may not perform well on non-English reviews.
  • Domain: The model is specialized for movie reviews. Performance may degrade on other domains (e.g., product reviews, social media posts).
  • Length: Reviews longer than 256 tokens are truncated, which may lose important context.
  • Sarcasm & Irony: The model may struggle with sarcastic or ironic statements, as is common in sentiment analysis.

πŸ“š Citation

If you use this model in your work, please cite:

@misc{sohrab2026imdb, author = {Samad Sohrab}, title = {IMDb Sentiment Analysis with DistilBERT}, year = {2026}, publisher = {Hugging Face}, howpublished = {https://huggingface.co/smd20/imdb-distilbert-sentiment} }


πŸ“„ License

This model is released under the Apache 2.0 license.

Downloads last month
38
Safetensors
Model size
67M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for smd20/imdb-distilbert-sentiment

Finetuned
(12,906)
this model

Dataset used to train smd20/imdb-distilbert-sentiment