README.md

---
language:
- en
license: mit
base_model: prajjwal1/bert-tiny
tags:
- pytorch
- movie-review-sentiment
- BertForSequenceClassification
- generated_from_trainer
metrics:
- accuracy
- matthews_correlation
model-index:
- name: tiny-imdb
  results:
    - task:
        type: text-classification
      metrics:
        - type: accuracy
          value: 0.8944
          name: accuracy
        - type: accuracy
          value: 0.7888
          name: matthews_correlation 
datasets:
- imdb
library_name: transformers
pipeline_tag: text-classification
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-tiny-imdb

This model is a fine-tuned version of [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) on the imdb dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2775
- Accuracy: 0.8944
- Matthews Correlation: 0.7888

## Model description

This is the smallest version of BERT model suggested by Google in this [GitHub Repo](https://github.com/google-research/bert), this model contains 2 transformer layers and an a hidden layer output length of 128, ie __(L=2, H=128)__. There are a total 4.39 million paramteres in the model.

## Intended uses & limitations

This model should be used for text classification tasks specifically on movie reviews or other such text data. Also you can use this model for other downstream tasks like:

- Sentiment Analysis
- Named Entity Recognition or Token Classification

This model should not be used for any tasks other than the above mentioned or any language other than English.

### How to use the Model

__Pytorch Model__

```python
from transformers import pipeline

# load pipeline
tiny_bert = pipeline("text-classification", "arnabdhar/tinybert-imdb")

# perform inference
results = pipeline(input_text, truncation=True, max_length=128)
```

__ONNX Model__

```python
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForSequenceClassification

# load tokenizer & model
model_name = "arnabdhar/tinybert-imdb"
tokenizer = AutoTokenizer.from_pretrained(model_name)
onnx_model = ORTModelForSequenceClassification.from_pretrained(model_name)

# build pipeline
tiny_bert_onnx = pipeline(
  task = "text-classification",
  tokenizer = tokenizer,
  model = onnx_model
)

# perform inference
results = tiny_bert_onnx(input_text, truncation=True, max_length=128)
```

## Training

The model was finetuned on Google Colab using the NVIDIA V100 GPU and was trained for 9 epochs, it took around 12 minutes to finish finetuning.

This model has been trained on the [imdb](https://huggingface.co/datasets/imdb) dataset which has 25,000 data text data for each training set and testing set, but I have combined both the partitions and then split the dataset in 80:20 ratio and used it for finetuning. This approach gave me a larger dataset to finetune the model.


### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 320
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 9

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy | Matthews Correlation |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------------------:|
| 0.4927        | 1.0   | 1250  | 0.3557          | 0.8484   | 0.7016               |
| 0.298         | 2.0   | 2500  | 0.2874          | 0.8866   | 0.7732               |
| 0.2555        | 3.0   | 3750  | 0.2799          | 0.8912   | 0.7828               |
| 0.2132        | 4.0   | 5000  | 0.2775          | 0.8944   | 0.7888               |
| 0.1779        | 5.0   | 6250  | 0.3065          | 0.891    | 0.7835               |
| 0.1508        | 6.0   | 7500  | 0.3331          | 0.889    | 0.7811               |
| 0.1304        | 7.0   | 8750  | 0.3451          | 0.8926   | 0.7870               |
| 0.119         | 8.0   | 10000 | 0.3670          | 0.8915   | 0.7852               |
| 0.1118        | 9.0   | 11250 | 0.3655          | 0.891    | 0.7840               |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0