*Real-time facial emotion detection using deep learning*
---
This project is a **realโtime facial emotion recognition system** built around a **fineโtuned MobileNetV2 model**.
We start from an **ImageNetโpretrained MobileNetV2**, fineโtune it on a custom facial emotion dataset, save the weights as `emotion_recognition_model.pth`, and then use that model behind a **FastAPI backend** and **browser-based webcam UI** to detect emotions such as ๐ *Angry*, ๐จ *Fear*, ๐ *Happy*, ๐ข *Sad,* and ๐ฒ *Surprise* from a live camera feed.
---
## โจ Features
- ๐ค **Fineโtuned deep learning model** โ MobileNetV2 is fineโtuned on a labeled facial emotion dataset to specialize it for emotion recognition.
- ๐น **Realโtime detection** from your webcam (browser or local OpenCV).
- ๐ **FastAPI REST API** endpoint for image-based emotion prediction.
- ๐จ **Simple HTML/JS frontend** that streams frames to the backend and displays the predicted emotion.
- ๐ป **Standalone webcam script** (`webcam.py`) if you prefer running everything locally without the browser.
---
## ๐ Project Structure
```
๐ฆ Emotion Recognition Project
โโโ ๐ frontend/
โ โโโ ๐ index.html # Minimal UI with a video element and live emotion text
โ โโโ ๐ script.js # Grabs webcam frames, sends them to the backend
โ โโโ ๐ style.css # Basic styling for the page and video element
โโโ ๐ model/
โ โโโ ๐ api.py # FastAPI app exposing POST /api/predict
โ โโโ ๐ inference.py # Loads model and defines predict(image) function
โ โโโ ๐ model.py # MobileNetV2 architecture definition
โ โโโ ๐ webcam.py # OpenCV-based realโtime emotion recognition
โ โโโ ๐ working.ipynb # Jupyter notebook for fineโtuning MobileNetV2
โ โโโ ๐พ emotion_recognition_model.pth # Fineโtuned PyTorch checkpoint
โ โโโ ๐ Data/ # Dataset folders (Angry, Fear, Happy, Sad, Surprise)
โโโ ๐ณ DockerFile # Docker image definition for deployment
โโโ ๐ requirements.txt # Python dependencies
```
---
## ๐ ๏ธ Requirements
### System Requirements
- ๐ **Python** 3.8+ (recommended)
- ๐ฆ **pip** for installing dependencies
- ๐ท A **webcam**
- โก (Optional) **GPU with CUDA** for faster inference, otherwise CPU will be used.
### Python Packages
Python packages (also listed in `requirements.txt`):
- `torch`, `torchvision` ๐ฅ
- `numpy` ๐ข
- `opencv-python` ๐น
- `matplotlib` ๐
- `pillow` ๐ผ๏ธ
- `tqdm` ๐
- `requests` ๐
- `fastapi`, `uvicorn` โก (install explicitly if missing)
### Installation
You can install everything with:
```bash
pip install -r requirements.txt fastapi uvicorn
```
---
## ๐ How to Run
### ๐ Web API + Frontend
1. **Navigate to the model folder**:
```bash
cd model
```
2. **Start the FastAPI server** (with Uvicorn):
```bash
uvicorn api:app --host 0.0.0.0 --port 8000 --reload
```
3. **Open the frontend**:
- Option A: The API mounts the `frontend` folder as static files.
Open your browser and go to:
```text
http://localhost:8000/
```
- If needed, ensure the working directory is such that `../frontend` (from `model/api.py`) points to the `frontend` folder in this repo.
4. **Allow camera access** ๐ท in the browser when prompted.
5. You should see the **live video** and the **predicted emotion** updating underneath! ๐
---
### ๐ป Local Webcam Script (No Browser)
If you prefer a pure Python / OpenCV pipeline:
1. Make sure dependencies are installed:
```bash
pip install -r requirements.txt
```
2. From the `model` directory, run:
```bash
python webcam.py
```
3. A window called **"Emotion Recognition"** will appear:
- ๐ค Detected faces will be highlighted with a bounding box.
- ๐ท๏ธ The predicted emotion label will be shown next to each face.
- Press **`q`** to quit.
---
### ๐ณ Run with Docker (Optional)
If you prefer not to manage Python and system dependencies manually, you can use the provided `DockerFile`:
1. **Build the image** from the project root:
```bash
docker build -t emotion-recognition -f DockerFile .
```
2. **Run the container** and expose the API on port `8000`:
```bash
docker run --rm -p 8000:8000 emotion-recognition
```
3. Open your browser at:
```text
http://localhost:8000/
```
and allow camera access. The behavior is the same as in the "Web API + Frontend" section above.
---
## ๐ง Model & Inference Details
- The model is a **MobileNetV2** classifier whose final layer is adapted to the number of emotion classes (e.g. ๐ Angry, ๐จ Fear, ๐ Happy, ๐ข Sad, ๐ฒ Surprise).
- The **fineโtuned weights and class names** are stored in `emotion_recognition_model.pth`, which is produced by the training notebook (`working.ipynb`).
- Images are preprocessed with:
- ๐ Resize to \(224 \times 224\)
- ๐ Conversion to tensor
- ๐ Normalization with ImageNet mean and std
- Inference is done by `inference.py` via:
- `predict(pil_image)` โ returns a string label, e.g. `"Happy"`.
### Testing the Model
You can test the model directly with a static image (from the `model` directory):
```bash
python inference.py
```
This will load `Image.jpg` and print the predicted emotion.
---
## ๐ก API Reference
### Endpoints
- **`GET /`**
- Returns a simple JSON status: `{"status": "API running"}`.
- **`POST /api/predict`**
- **Body**: `multipart/form-data` with a single field:
- `file`: image file (e.g. JPEG/PNG).
- **Response**:
```json
{ "emotion": "Happy" }
```
The frontend uses this endpoint to send frames from your webcam (as blobs) approximately once per second.
---
## ๐ Dataset & Training (FineโTuning Overview)
- The (optional) `Data` directory is expected to contain labeled images organized by emotion:
- ๐ `Angry/`, ๐จ `Fear/`, ๐ `Happy/`, ๐ข `Sad/`, ๐ฒ `Suprise/`.
- The **fineโtuning process is captured in `working.ipynb`**:
- Load an **ImageNetโpretrained MobileNetV2** from `torchvision`.
- Replace the final classification layer so that its output dimension matches the number of emotion classes.
- Create PyTorch `Dataset`/`DataLoader` objects from the `Data` folders with standard augmentations and preprocessing.
- Train the network (typically using crossโentropy loss and an optimizer like Adam/SGD) while monitoring validation accuracy.
- Save the best model checkpoint and class mapping to `emotion_recognition_model.pth`.
- You can adapt the model for new datasets by:
- Updating the `Data` folders and class list.
- Reโrunning or modifying `working.ipynb` and, if needed, adjusting `model.py` for a different backbone or number of classes.
**Dataset Source**: [Kaggle - Human Face Emotions](https://www.kaggle.com/datasets/samithsachidanandan/human-face-emotions)
---
## ๐ง Troubleshooting
- **๐ท Camera access denied (browser)**
- Check browser permissions and ensure you're using `http://localhost` (not `file://`).
- **๐ "Backend not reachable" in the frontend**
- Confirm the FastAPI server is running on the same host/port that `script.js` expects (`/api/predict` โ default `http://localhost:8000/api/predict`).
- Check for CORS issues or port conflicts.
- **๐ฆ Model file not found**
- Ensure `emotion_recognition_model.pth` is present in the `model` directory when running any Python scripts there.
---
## ๐ License & Credits
- The project uses **PyTorch**, **FastAPI**, **OpenCV**, and **PIL** under their respective licenses.
- Dataset images in `Data/` should respect their original source licenses (not provided here).
- Feel free to modify or extend this project for research, learning, or personal use.
---
Made with โค๏ธ using PyTorch, FastAPI, and OpenCV