Spaces:
Runtime error
A newer version of the Gradio SDK is available: 6.30.0
title: MockMate AI - AI Interview Coach
emoji: 🎤
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.40.0
app_file: app.py
pinned: false
license: mit
short_description: End-to-end AI-powered interview assessment platform.
🎤 AI Interview Analyzer
An end-to-end AI-powered interview assessment platform that evaluates a candidate's communication, confidence, facial expressions, eye contact, posture, gestures, and overall body language using Computer Vision and Machine Learning.
📖 Overview
Traditional mock interviews provide little objective feedback beyond the interviewer's opinion. This project aims to automate interview analysis using Computer Vision, Human Pose Estimation, Facial Landmark Detection, and AI-based scoring.
The system records (or uploads) an interview video and analyzes every aspect of the candidate's visual behavior.
Instead of simply saying
"Maintain eye contact."
the system provides measurable metrics like
Eye Contact : 81%
Face Visibility : 98%
Head Stability : 90%
Posture Score : 86%
Gesture Score : 79%
Overall Score : 84/100
and personalized coaching recommendations.
🎯 Project Goals
The objective is to create an AI assistant capable of evaluating a candidate's non-verbal communication during interviews.
The project focuses on
- Face Visibility
- Eye Contact
- Head Pose
- Facial Expressions
- Body Posture
- Hand Gestures
- Body Stability
- Video Quality
- Personalized Feedback
The final goal is to generate an AI Interview Report.
🏗 High Level Architecture
User
│
▼
Video Recording
│
┌───────────┴───────────┐
│ │
▼ ▼
Live Webcam Uploaded Video
│ │
└───────────┬───────────┘
│
Frame Source
│
▼
Video Processing
│
▼
Computer Vision Pipeline
│
┌──────────────────────────────────┐
│ Face Detection │
│ Face Mesh │
│ Pose Estimation │
│ Hand Tracking │
│ Video Quality │
└──────────────────────────────────┘
│
▼
Behavioral Analysis
│
┌──────────────────────────────────┐
│ Eye Contact │
│ Facial Emotion │
│ Head Pose │
│ Body Stability │
│ Hand Gestures │
│ Posture │
└──────────────────────────────────┘
│
▼
Scoring Engine
│
▼
Report Generation
│
▼
Interactive Dashboard
---
# 📂 Project Structure
AI-Interview/
├── backend/
│
├── frontend/
│
├── uploads/
│
├── reports/
│
├── temp/
│
├── README.md
│
└── .gitignore
Backend Structure
backend/
│ ├── api/ │ ├── upload.py │ ├── analyze.py │ ├── report.py │ ├── services/ │ │ ├── video/ │ │ extractor.py │ │ │ ├── vision/ │ │ │ │ detector.py │ │ landmarks.py │ │ │ │ face/ │ │ visibility.py │ │ eye_contact.py │ │ expression.py │ │ head_pose.py │ │ │ │ body/ │ │ posture.py │ │ stability.py │ │ │ │ hands/ │ │ gestures.py │ │ │ │ quality/ │ │ blur.py │ │ lighting.py │ │ camera_angle.py │ │ ├── scoring/ │ │ scorer.py │ │ │ └── reporting/ │ report_generator.py │ ├── models/ │ ├── utils/ │ ├── core/ │ └── main.py
---
# 📹 Input Sources
The system supports multiple input sources.
## Live Webcam
Laptop Webcam
↓
Real-time Analysis
Uploaded Video
Interview.mp4
↓
Offline Analysis
---
## IP Camera (Future)
RTSP Stream
↓
Analysis
🎥 Video Processing Pipeline
Instead of extracting every frame and storing thousands of images,
the system processes the video as a stream.
Video
↓
VideoCapture
↓
Frame
↓
Analysis
↓
Discard
↓
Next Frame
Advantages
- Low memory usage
- Faster processing
- Real-time compatible
---
# 🖼 Frame Source
Every frame enters the pipeline through a common interface.
Video
↓
FrameSource
↓
Generator
↓
Frame 1
↓
Frame 2
↓
Frame 3
This abstraction allows the same analysis code to work for
- webcam
- recorded video
- RTSP streams
without modification.
👤 Face Detection
Purpose
Determine whether a face is visible throughout the interview.
Output
Face Found
Bounding Box
Confidence
Face Visibility %
Technologies
- MediaPipe Face Detection
---
# 😀 Face Mesh
The most important module.
Instead of returning only a face,
MediaPipe Face Mesh returns
468 Facial Landmarks
Each landmark contains
x y z
These landmarks are shared across multiple downstream modules.
Face Mesh
↓
Landmarks
├── Eye Contact
├── Head Pose
├── Expression
└── Face Visibility
The Face Mesh model is executed once per frame, and all modules reuse its output to avoid redundant computation.
👀 Eye Contact Analysis
Using facial landmarks, the system estimates
- Looking Left
- Looking Right
- Looking Up
- Looking Down
- Looking At Camera
Output
Eye Contact
82%
Looking Away
18%
---
# 🙂 Facial Expression Analysis
Emotion recognition estimates
- Neutral
- Happy
- Smile
- Nervous
- Sad
- Angry
- Surprise
Future improvements
- FER
- Hume AI
- Azure Face API
---
# 🧠 Head Pose Estimation
The head orientation is represented using
- Pitch
- Yaw
- Roll
Up
▲
Left ◄─────┼────► Right
▼
Down
Metrics
- Head Stability
- Sudden Turns
- Frequent Looking Down
🧍 Pose Estimation
MediaPipe Pose estimates
- Neck
- Shoulders
- Elbows
- Hips
The system determines
- Upright
- Leaning
- Slouching
✋ Hand Tracking
Tracks
- Hand Visibility
- Gesture Count
- Natural Gestures
- Excessive Gestures
🚶 Body Stability
The body is compared across consecutive frames.
Frame i
↓
Pose
↓
Frame i+1
↓
Pose
↓
Distance
Small distance
↓
Stable
Large distance
↓
Excessive movement
---
# 📸 Video Quality
Checks
- Brightness
- Blur
- Camera Angle
- Resolution
- FPS
- Background
---
# 📊 Scoring Engine
Every module outputs metrics.
Example
Eye Contact
78%
Head Stability
90%
Posture
85%
These are combined into
Overall Interview Score
Possible weighting
Eye Contact 30%
Posture 20%
Expression 20%
Hand Movement 15%
Body Stability 10%
Video Quality 5%
📄 Report Generation
The report contains
- Overall Score
- Face Analysis
- Eye Contact
- Head Pose
- Body Language
- Video Quality
- Personalized Coaching
- Recommendations
Output formats
- JSON
- Dashboard
🌐 Frontend
Responsibilities
- Record webcam
- Upload interview
- Live preview
- Dashboard
- Display metrics
- Download report
Technologies
- React
- React Webcam
- Tailwind CSS
⚙ Backend
Responsibilities
- Receive uploads
- Process video
- Run MediaPipe
- Calculate metrics
- Generate reports
Technologies
- FastAPI
- OpenCV
- MediaPipe
- NumPy
📚 Technologies
| Category | Technology |
|---|---|
| Backend | FastAPI |
| Frontend | React |
| Video | OpenCV |
| Face Detection | MediaPipe |
| Face Landmarks | MediaPipe Face Mesh |
| Pose Estimation | MediaPipe Pose |
| Hand Tracking | MediaPipe Hands |
| Numerical Computing | NumPy |
| Reporting | Python |
🚀 Future Improvements
- Speech Analysis
- Voice Emotion Recognition
- Lip Synchronization
- Object Detection
- Eye Blink Rate
- Attention Detection
- Stress Detection
- Real-time Dashboard
- Multi-person Detection
- Interview Difficulty Estimation
- LLM-powered Coaching
- Historical Performance Tracking
- Resume-aware Feedback
- Personalized Practice Sessions
💡 Design Philosophy
The system follows a streaming pipeline architecture.
Instead of repeatedly running expensive AI models for each analysis task, each frame is processed once, producing reusable intermediate representations (such as facial landmarks and body keypoints). These intermediate results are then shared across specialized analysis modules.
This design provides:
- High performance
- Low memory consumption
- Modular components
- Easy extensibility
- Support for both live and recorded interviews
- Separation of concerns
- Production-ready architecture
The goal is to build an extensible Computer Vision platform rather than a collection of independent scripts, making it straightforward to integrate new models, replace existing algorithms, or support additional input sources in the future.