Spaces:

ZenithsArch
/

MOCKMATE

Runtime error

MOCKMATE / README.md

sriyanshcodes-dev

Initial clean commit for Hugging Face Spaces

fd13a7c 3 months ago

10.3 kB

---
title: MockMate AI - AI Interview Coach
emoji: 🎤
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.40.0
app_file: app.py
pinned: false
license: mit
short_description: End-to-end AI-powered interview assessment platform.
---
# 🎤 AI Interview Analyzer


> An end-to-end AI-powered interview assessment platform that evaluates a candidate's communication, confidence, facial expressions, eye contact, posture, gestures, and overall body language using Computer Vision and Machine Learning.

---

# 📖 Overview

Traditional mock interviews provide little objective feedback beyond the interviewer's opinion. This project aims to automate interview analysis using Computer Vision, Human Pose Estimation, Facial Landmark Detection, and AI-based scoring.

The system records (or uploads) an interview video and analyzes every aspect of the candidate's visual behavior.

Instead of simply saying

> "Maintain eye contact."

the system provides measurable metrics like

```
Eye Contact        : 81%
Face Visibility    : 98%
Head Stability     : 90%
Posture Score      : 86%
Gesture Score      : 79%
Overall Score      : 84/100
```

and personalized coaching recommendations.

---

# 🎯 Project Goals

The objective is to create an AI assistant capable of evaluating a candidate's non-verbal communication during interviews.

The project focuses on

- Face Visibility
- Eye Contact
- Head Pose
- Facial Expressions
- Body Posture
- Hand Gestures
- Body Stability
- Video Quality
- Personalized Feedback

The final goal is to generate an AI Interview Report.

---

# 🏗 High Level Architecture

```
```
                    User
                      │
                      ▼
              Video Recording
                      │
          ┌───────────┴───────────┐
          │                       │
          ▼                       ▼
     Live Webcam            Uploaded Video
          │                       │
          └───────────┬───────────┘
                      │
               Frame Source
                      │
                      ▼
             Video Processing
                      │
                      ▼
           Computer Vision Pipeline
                      │
    ┌──────────────────────────────────┐
    │ Face Detection                   │
    │ Face Mesh                        │
    │ Pose Estimation                  │
    │ Hand Tracking                    │
    │ Video Quality                    │
    └──────────────────────────────────┘
                      │
                      ▼
            Behavioral Analysis
                      │
    ┌──────────────────────────────────┐
    │ Eye Contact                      │
    │ Facial Emotion                   │
    │ Head Pose                        │
    │ Body Stability                   │
    │ Hand Gestures                    │
    │ Posture                          │
    └──────────────────────────────────┘
                      │
                      ▼
              Scoring Engine
                      │
                      ▼
             Report Generation
                      │
                      ▼
               Interactive Dashboard
```
---

# 📂 Project Structure

```
```
AI-Interview/

├── backend/
│
├── frontend/
│
├── uploads/
│
├── reports/
│
├── temp/
│
├── README.md
│
└── .gitignore
```
---
# Backend Structure
```

```
backend/
│
├── api/
│   ├── upload.py
│   ├── analyze.py
│   ├── report.py
│
├── services/
│
│   ├── video/
│   │      extractor.py
│   │
│   ├── vision/
│   │
│   │     detector.py
│   │     landmarks.py
│   │
│   │     face/
│   │          visibility.py
│   │          eye_contact.py
│   │          expression.py
│   │          head_pose.py
│   │
│   │     body/
│   │          posture.py
│   │          stability.py
│   │
│   │     hands/
│   │          gestures.py
│   │
│   │     quality/
│   │          blur.py
│   │          lighting.py
│   │          camera_angle.py
│
│   ├── scoring/
│   │      scorer.py
│   │
│   └── reporting/
│          report_generator.py
│
├── models/
│
├── utils/
│
├── core/
│
└── main.py
```

---

# 📹 Input Sources

The system supports multiple input sources.

## Live Webcam

```
```
Laptop Webcam

↓

Real-time Analysis
```
---
## Uploaded Video
```

```
Interview.mp4
↓
Offline Analysis
```

---

## IP Camera (Future)

```
```
RTSP Stream

↓

Analysis
```
---
# 🎥 Video Processing Pipeline
Instead of extracting every frame and storing thousands of images,
the system processes the video as a stream.
```

```
Video
↓
VideoCapture
↓
Frame
↓
Analysis
↓
Discard
↓
Next Frame
```

Advantages

- Low memory usage
- Faster processing
- Real-time compatible

---

# 🖼 Frame Source

Every frame enters the pipeline through a common interface.

```
```
Video

↓

FrameSource

↓

Generator

↓

Frame 1

↓

Frame 2

↓

Frame 3
```
This abstraction allows the same analysis code to work for
- webcam
- recorded video
- RTSP streams
without modification.
---
# 👤 Face Detection
Purpose
Determine whether a face is visible throughout the interview.
Output
```

```
Face Found
Bounding Box
Confidence
Face Visibility %
```

Technologies

- MediaPipe Face Detection

---

# 😀 Face Mesh

The most important module.

Instead of returning only a face,

MediaPipe Face Mesh returns

```
```
468 Facial Landmarks
```
Each landmark contains
```

```
x
y
z
```

These landmarks are shared across multiple downstream modules.

```
```
Face Mesh

↓

Landmarks

├── Eye Contact

├── Head Pose

├── Expression

└── Face Visibility
```
The Face Mesh model is executed **once per frame**, and all modules reuse its output to avoid redundant computation.
---
# 👀 Eye Contact Analysis
Using facial landmarks, the system estimates
- Looking Left
- Looking Right
- Looking Up
- Looking Down
- Looking At Camera
Output
```

```
Eye Contact
82%
Looking Away
18%
```

---

# 🙂 Facial Expression Analysis

Emotion recognition estimates

- Neutral
- Happy
- Smile
- Nervous
- Sad
- Angry
- Surprise

Future improvements

- FER
- Hume AI
- Azure Face API

---

# 🧠 Head Pose Estimation

The head orientation is represented using

- Pitch
- Yaw
- Roll

```
```
          Up
           ▲
Left ◄─────┼────► Right

           ▼
         Down
```
Metrics

- Head Stability
- Sudden Turns
- Frequent Looking Down

---

# 🧍 Pose Estimation

MediaPipe Pose estimates

- Neck
- Shoulders
- Elbows
- Hips

The system determines

- Upright
- Leaning
- Slouching

---

# ✋ Hand Tracking

Tracks

- Hand Visibility
- Gesture Count
- Natural Gestures
- Excessive Gestures

---

# 🚶 Body Stability

The body is compared across consecutive frames.

```
```
Frame i

↓

Pose

↓

Frame i+1

↓

Pose

↓

Distance
```
Small distance
↓
Stable
Large distance
↓
Excessive movement
---
# 📸 Video Quality
Checks
- Brightness
- Blur
- Camera Angle
- Resolution
- FPS
- Background
---
# 📊 Scoring Engine
Every module outputs metrics.
Example
```

```
Eye Contact
78%
```

```
```
Head Stability

90%
```
```

```
Posture
85%
```

These are combined into

```
```
Overall Interview Score
```
Possible weighting
```

```
Eye Contact        30%
Posture            20%
Expression         20%
Hand Movement      15%
Body Stability     10%
Video Quality       5%
```

---

# 📄 Report Generation

The report contains

- Overall Score
- Face Analysis
- Eye Contact
- Head Pose
- Body Language
- Video Quality
- Personalized Coaching
- Recommendations

Output formats

- JSON
- PDF
- Dashboard

---

# 🌐 Frontend

Responsibilities

- Record webcam
- Upload interview
- Live preview
- Dashboard
- Display metrics
- Download report

Technologies

- React
- React Webcam
- Tailwind CSS

---

# ⚙ Backend

Responsibilities

- Receive uploads
- Process video
- Run MediaPipe
- Calculate metrics
- Generate reports

Technologies

- FastAPI
- OpenCV
- MediaPipe
- NumPy

---

# 📚 Technologies

| Category | Technology |
|-----------|------------|
| Backend | FastAPI |
| Frontend | React |
| Video | OpenCV |
| Face Detection | MediaPipe |
| Face Landmarks | MediaPipe Face Mesh |
| Pose Estimation | MediaPipe Pose |
| Hand Tracking | MediaPipe Hands |
| Numerical Computing | NumPy |
| Reporting | Python |

---

# 🚀 Future Improvements

- Speech Analysis
- Voice Emotion Recognition
- Lip Synchronization
- Object Detection
- Eye Blink Rate
- Attention Detection
- Stress Detection
- Real-time Dashboard
- Multi-person Detection
- Interview Difficulty Estimation
- LLM-powered Coaching
- Historical Performance Tracking
- Resume-aware Feedback
- Personalized Practice Sessions

---

# 💡 Design Philosophy

The system follows a **streaming pipeline architecture**.

Instead of repeatedly running expensive AI models for each analysis task, each frame is processed once, producing reusable intermediate representations (such as facial landmarks and body keypoints). These intermediate results are then shared across specialized analysis modules.

This design provides:

- High performance
- Low memory consumption
- Modular components
- Easy extensibility
- Support for both live and recorded interviews
- Separation of concerns
- Production-ready architecture

The goal is to build an extensible Computer Vision platform rather than a collection of independent scripts, making it straightforward to integrate new models, replace existing algorithms, or support additional input sources in the future.

	---
	title: MockMate AI - AI Interview Coach
	emoji: 🎤
	colorFrom: blue
	colorTo: indigo
	sdk: gradio
	sdk_version: 4.40.0
	app_file: app.py
	pinned: false
	license: mit
	short_description: End-to-end AI-powered interview assessment platform.
	---

	# 🎤 AI Interview Analyzer


	> An end-to-end AI-powered interview assessment platform that evaluates a candidate's communication, confidence, facial expressions, eye contact, posture, gestures, and overall body language using Computer Vision and Machine Learning.

	---

	# 📖 Overview

	Traditional mock interviews provide little objective feedback beyond the interviewer's opinion. This project aims to automate interview analysis using Computer Vision, Human Pose Estimation, Facial Landmark Detection, and AI-based scoring.

	The system records (or uploads) an interview video and analyzes every aspect of the candidate's visual behavior.

	Instead of simply saying

	> "Maintain eye contact."

	the system provides measurable metrics like

	```
	Eye Contact : 81%
	Face Visibility : 98%
	Head Stability : 90%
	Posture Score : 86%
	Gesture Score : 79%
	Overall Score : 84/100
	```

	and personalized coaching recommendations.

	---

	# 🎯 Project Goals

	The objective is to create an AI assistant capable of evaluating a candidate's non-verbal communication during interviews.

	The project focuses on

	- Face Visibility
	- Eye Contact
	- Head Pose
	- Facial Expressions
	- Body Posture
	- Hand Gestures
	- Body Stability
	- Video Quality
	- Personalized Feedback

	The final goal is to generate an AI Interview Report.

	---

	# 🏗 High Level Architecture

	```

	```
	User
	│
	▼
	Video Recording
	│
	┌───────────┴───────────┐
	│ │
	▼ ▼
	Live Webcam Uploaded Video
	│ │
	└───────────┬───────────┘
	│
	Frame Source
	│
	▼
	Video Processing
	│
	▼
	Computer Vision Pipeline
	│
	┌──────────────────────────────────┐
	│ Face Detection │
	│ Face Mesh │
	│ Pose Estimation │
	│ Hand Tracking │
	│ Video Quality │
	└──────────────────────────────────┘
	│
	▼
	Behavioral Analysis
	│
	┌──────────────────────────────────┐
	│ Eye Contact │
	│ Facial Emotion │
	│ Head Pose │
	│ Body Stability │
	│ Hand Gestures │
	│ Posture │
	└──────────────────────────────────┘
	│
	▼
	Scoring Engine
	│
	▼
	Report Generation
	│
	▼
	Interactive Dashboard
	```

	---

	# 📂 Project Structure

	```

	```
	AI-Interview/

	├── backend/
	│
	├── frontend/
	│
	├── uploads/
	│
	├── reports/
	│
	├── temp/
	│
	├── README.md
	│
	└── .gitignore
	```

	---

	# Backend Structure

	```

	```
	backend/

	│
	├── api/
	│ ├── upload.py
	│ ├── analyze.py
	│ ├── report.py
	│
	├── services/
	│
	│ ├── video/
	│ │ extractor.py
	│ │
	│ ├── vision/
	│ │
	│ │ detector.py
	│ │ landmarks.py
	│ │
	│ │ face/
	│ │ visibility.py
	│ │ eye_contact.py
	│ │ expression.py
	│ │ head_pose.py
	│ │
	│ │ body/
	│ │ posture.py
	│ │ stability.py
	│ │
	│ │ hands/
	│ │ gestures.py
	│ │
	│ │ quality/
	│ │ blur.py
	│ │ lighting.py
	│ │ camera_angle.py
	│
	│ ├── scoring/
	│ │ scorer.py
	│ │
	│ └── reporting/
	│ report_generator.py
	│
	├── models/
	│
	├── utils/
	│
	├── core/
	│
	└── main.py
	```

	---

	# 📹 Input Sources

	The system supports multiple input sources.

	## Live Webcam

	```

	```
	Laptop Webcam

	↓

	Real-time Analysis
	```

	---

	## Uploaded Video

	```

	```
	Interview.mp4

	↓

	Offline Analysis
	```

	---

	## IP Camera (Future)

	```

	```
	RTSP Stream

	↓

	Analysis
	```

	---

	# 🎥 Video Processing Pipeline

	Instead of extracting every frame and storing thousands of images,

	the system processes the video as a stream.

	```

	```
	Video

	↓

	VideoCapture

	↓

	Frame

	↓

	Analysis

	↓

	Discard

	↓

	Next Frame
	```

	Advantages

	- Low memory usage
	- Faster processing
	- Real-time compatible

	---

	# 🖼 Frame Source

	Every frame enters the pipeline through a common interface.

	```

	```
	Video

	↓

	FrameSource

	↓

	Generator

	↓

	Frame 1

	↓

	Frame 2

	↓

	Frame 3
	```

	This abstraction allows the same analysis code to work for

	- webcam
	- recorded video
	- RTSP streams

	without modification.

	---

	# 👤 Face Detection

	Purpose

	Determine whether a face is visible throughout the interview.

	Output

	```

	```
	Face Found

	Bounding Box

	Confidence

	Face Visibility %
	```

	Technologies

	- MediaPipe Face Detection

	---

	# 😀 Face Mesh

	The most important module.

	Instead of returning only a face,

	MediaPipe Face Mesh returns

	```

	```
	468 Facial Landmarks
	```

	Each landmark contains

	```

	```
	x
	y
	z
	```

	These landmarks are shared across multiple downstream modules.

	```

	```
	Face Mesh

	↓

	Landmarks

	├── Eye Contact

	├── Head Pose

	├── Expression

	└── Face Visibility
	```

	The Face Mesh model is executed once per frame, and all modules reuse its output to avoid redundant computation.

	---

	# 👀 Eye Contact Analysis

	Using facial landmarks, the system estimates

	- Looking Left
	- Looking Right
	- Looking Up
	- Looking Down
	- Looking At Camera

	Output

	```

	```
	Eye Contact

	82%

	Looking Away

	18%
	```

	---

	# 🙂 Facial Expression Analysis

	Emotion recognition estimates

	- Neutral
	- Happy
	- Smile
	- Nervous
	- Sad
	- Angry
	- Surprise

	Future improvements

	- FER
	- Hume AI
	- Azure Face API

	---

	# 🧠 Head Pose Estimation

	The head orientation is represented using

	- Pitch
	- Yaw
	- Roll

	```

	```
	Up

	▲

	Left ◄─────┼────► Right

	▼

	Down
	```

	Metrics

	- Head Stability
	- Sudden Turns
	- Frequent Looking Down

	---

	# 🧍 Pose Estimation

	MediaPipe Pose estimates

	- Neck
	- Shoulders
	- Elbows
	- Hips

	The system determines

	- Upright
	- Leaning
	- Slouching

	---

	# ✋ Hand Tracking

	Tracks

	- Hand Visibility
	- Gesture Count
	- Natural Gestures
	- Excessive Gestures

	---

	# 🚶 Body Stability

	The body is compared across consecutive frames.

	```

	```
	Frame i

	↓

	Pose

	↓

	Frame i+1

	↓

	Pose

	↓

	Distance
	```

	Small distance

	↓

	Stable

	Large distance

	↓

	Excessive movement

	---

	# 📸 Video Quality

	Checks

	- Brightness
	- Blur
	- Camera Angle
	- Resolution
	- FPS
	- Background

	---

	# 📊 Scoring Engine

	Every module outputs metrics.

	Example

	```

	```
	Eye Contact

	78%
	```

	```

	```
	Head Stability

	90%
	```

	```

	```
	Posture

	85%
	```

	These are combined into

	```

	```
	Overall Interview Score
	```

	Possible weighting

	```

	```
	Eye Contact 30%

	Posture 20%

	Expression 20%

	Hand Movement 15%

	Body Stability 10%

	Video Quality 5%
	```

	---

	# 📄 Report Generation

	The report contains

	- Overall Score
	- Face Analysis
	- Eye Contact
	- Head Pose
	- Body Language
	- Video Quality
	- Personalized Coaching
	- Recommendations

	Output formats

	- JSON
	- PDF
	- Dashboard

	---

	# 🌐 Frontend

	Responsibilities

	- Record webcam
	- Upload interview
	- Live preview
	- Dashboard
	- Display metrics
	- Download report

	Technologies

	- React
	- React Webcam
	- Tailwind CSS

	---

	# ⚙ Backend

	Responsibilities

	- Receive uploads
	- Process video
	- Run MediaPipe
	- Calculate metrics
	- Generate reports

	Technologies

	- FastAPI
	- OpenCV
	- MediaPipe
	- NumPy

	---

	# 📚 Technologies

	\| Category \| Technology \|
	\|-----------\|------------\|
	\| Backend \| FastAPI \|
	\| Frontend \| React \|
	\| Video \| OpenCV \|
	\| Face Detection \| MediaPipe \|
	\| Face Landmarks \| MediaPipe Face Mesh \|
	\| Pose Estimation \| MediaPipe Pose \|
	\| Hand Tracking \| MediaPipe Hands \|
	\| Numerical Computing \| NumPy \|
	\| Reporting \| Python \|

	---

	# 🚀 Future Improvements

	- Speech Analysis
	- Voice Emotion Recognition
	- Lip Synchronization
	- Object Detection
	- Eye Blink Rate
	- Attention Detection
	- Stress Detection
	- Real-time Dashboard
	- Multi-person Detection
	- Interview Difficulty Estimation
	- LLM-powered Coaching
	- Historical Performance Tracking
	- Resume-aware Feedback
	- Personalized Practice Sessions

	---

	# 💡 Design Philosophy

	The system follows a streaming pipeline architecture.

	Instead of repeatedly running expensive AI models for each analysis task, each frame is processed once, producing reusable intermediate representations (such as facial landmarks and body keypoints). These intermediate results are then shared across specialized analysis modules.

	This design provides:

	- High performance
	- Low memory consumption
	- Modular components
	- Easy extensibility
	- Support for both live and recorded interviews
	- Separation of concerns
	- Production-ready architecture

	The goal is to build an extensible Computer Vision platform rather than a collection of independent scripts, making it straightforward to integrate new models, replace existing algorithms, or support additional input sources in the future.