--- title: MockMate AI - AI Interview Coach emoji: 🎤 colorFrom: blue colorTo: indigo sdk: gradio sdk_version: 4.40.0 app_file: app.py pinned: false license: mit short_description: End-to-end AI-powered interview assessment platform. --- # 🎤 AI Interview Analyzer > An end-to-end AI-powered interview assessment platform that evaluates a candidate's communication, confidence, facial expressions, eye contact, posture, gestures, and overall body language using Computer Vision and Machine Learning. --- # 📖 Overview Traditional mock interviews provide little objective feedback beyond the interviewer's opinion. This project aims to automate interview analysis using Computer Vision, Human Pose Estimation, Facial Landmark Detection, and AI-based scoring. The system records (or uploads) an interview video and analyzes every aspect of the candidate's visual behavior. Instead of simply saying > "Maintain eye contact." the system provides measurable metrics like ``` Eye Contact : 81% Face Visibility : 98% Head Stability : 90% Posture Score : 86% Gesture Score : 79% Overall Score : 84/100 ``` and personalized coaching recommendations. --- # 🎯 Project Goals The objective is to create an AI assistant capable of evaluating a candidate's non-verbal communication during interviews. The project focuses on - Face Visibility - Eye Contact - Head Pose - Facial Expressions - Body Posture - Hand Gestures - Body Stability - Video Quality - Personalized Feedback The final goal is to generate an AI Interview Report. --- # 🏗 High Level Architecture ``` ``` User │ ▼ Video Recording │ ┌───────────┴───────────┐ │ │ ▼ ▼ Live Webcam Uploaded Video │ │ └───────────┬───────────┘ │ Frame Source │ ▼ Video Processing │ ▼ Computer Vision Pipeline │ ┌──────────────────────────────────┐ │ Face Detection │ │ Face Mesh │ │ Pose Estimation │ │ Hand Tracking │ │ Video Quality │ └──────────────────────────────────┘ │ ▼ Behavioral Analysis │ ┌──────────────────────────────────┐ │ Eye Contact │ │ Facial Emotion │ │ Head Pose │ │ Body Stability │ │ Hand Gestures │ │ Posture │ └──────────────────────────────────┘ │ ▼ Scoring Engine │ ▼ Report Generation │ ▼ Interactive Dashboard ``` --- # 📂 Project Structure ``` ``` AI-Interview/ ├── backend/ │ ├── frontend/ │ ├── uploads/ │ ├── reports/ │ ├── temp/ │ ├── README.md │ └── .gitignore ``` --- # Backend Structure ``` ``` backend/ │ ├── api/ │ ├── upload.py │ ├── analyze.py │ ├── report.py │ ├── services/ │ │ ├── video/ │ │ extractor.py │ │ │ ├── vision/ │ │ │ │ detector.py │ │ landmarks.py │ │ │ │ face/ │ │ visibility.py │ │ eye_contact.py │ │ expression.py │ │ head_pose.py │ │ │ │ body/ │ │ posture.py │ │ stability.py │ │ │ │ hands/ │ │ gestures.py │ │ │ │ quality/ │ │ blur.py │ │ lighting.py │ │ camera_angle.py │ │ ├── scoring/ │ │ scorer.py │ │ │ └── reporting/ │ report_generator.py │ ├── models/ │ ├── utils/ │ ├── core/ │ └── main.py ``` --- # 📹 Input Sources The system supports multiple input sources. ## Live Webcam ``` ``` Laptop Webcam ↓ Real-time Analysis ``` --- ## Uploaded Video ``` ``` Interview.mp4 ↓ Offline Analysis ``` --- ## IP Camera (Future) ``` ``` RTSP Stream ↓ Analysis ``` --- # 🎥 Video Processing Pipeline Instead of extracting every frame and storing thousands of images, the system processes the video as a stream. ``` ``` Video ↓ VideoCapture ↓ Frame ↓ Analysis ↓ Discard ↓ Next Frame ``` Advantages - Low memory usage - Faster processing - Real-time compatible --- # 🖼 Frame Source Every frame enters the pipeline through a common interface. ``` ``` Video ↓ FrameSource ↓ Generator ↓ Frame 1 ↓ Frame 2 ↓ Frame 3 ``` This abstraction allows the same analysis code to work for - webcam - recorded video - RTSP streams without modification. --- # 👤 Face Detection Purpose Determine whether a face is visible throughout the interview. Output ``` ``` Face Found Bounding Box Confidence Face Visibility % ``` Technologies - MediaPipe Face Detection --- # 😀 Face Mesh The most important module. Instead of returning only a face, MediaPipe Face Mesh returns ``` ``` 468 Facial Landmarks ``` Each landmark contains ``` ``` x y z ``` These landmarks are shared across multiple downstream modules. ``` ``` Face Mesh ↓ Landmarks ├── Eye Contact ├── Head Pose ├── Expression └── Face Visibility ``` The Face Mesh model is executed **once per frame**, and all modules reuse its output to avoid redundant computation. --- # 👀 Eye Contact Analysis Using facial landmarks, the system estimates - Looking Left - Looking Right - Looking Up - Looking Down - Looking At Camera Output ``` ``` Eye Contact 82% Looking Away 18% ``` --- # 🙂 Facial Expression Analysis Emotion recognition estimates - Neutral - Happy - Smile - Nervous - Sad - Angry - Surprise Future improvements - FER - Hume AI - Azure Face API --- # 🧠 Head Pose Estimation The head orientation is represented using - Pitch - Yaw - Roll ``` ``` Up ▲ Left ◄─────┼────► Right ▼ Down ``` Metrics - Head Stability - Sudden Turns - Frequent Looking Down --- # 🧍 Pose Estimation MediaPipe Pose estimates - Neck - Shoulders - Elbows - Hips The system determines - Upright - Leaning - Slouching --- # ✋ Hand Tracking Tracks - Hand Visibility - Gesture Count - Natural Gestures - Excessive Gestures --- # 🚶 Body Stability The body is compared across consecutive frames. ``` ``` Frame i ↓ Pose ↓ Frame i+1 ↓ Pose ↓ Distance ``` Small distance ↓ Stable Large distance ↓ Excessive movement --- # 📸 Video Quality Checks - Brightness - Blur - Camera Angle - Resolution - FPS - Background --- # 📊 Scoring Engine Every module outputs metrics. Example ``` ``` Eye Contact 78% ``` ``` ``` Head Stability 90% ``` ``` ``` Posture 85% ``` These are combined into ``` ``` Overall Interview Score ``` Possible weighting ``` ``` Eye Contact 30% Posture 20% Expression 20% Hand Movement 15% Body Stability 10% Video Quality 5% ``` --- # 📄 Report Generation The report contains - Overall Score - Face Analysis - Eye Contact - Head Pose - Body Language - Video Quality - Personalized Coaching - Recommendations Output formats - JSON - PDF - Dashboard --- # 🌐 Frontend Responsibilities - Record webcam - Upload interview - Live preview - Dashboard - Display metrics - Download report Technologies - React - React Webcam - Tailwind CSS --- # ⚙ Backend Responsibilities - Receive uploads - Process video - Run MediaPipe - Calculate metrics - Generate reports Technologies - FastAPI - OpenCV - MediaPipe - NumPy --- # 📚 Technologies | Category | Technology | |-----------|------------| | Backend | FastAPI | | Frontend | React | | Video | OpenCV | | Face Detection | MediaPipe | | Face Landmarks | MediaPipe Face Mesh | | Pose Estimation | MediaPipe Pose | | Hand Tracking | MediaPipe Hands | | Numerical Computing | NumPy | | Reporting | Python | --- # 🚀 Future Improvements - Speech Analysis - Voice Emotion Recognition - Lip Synchronization - Object Detection - Eye Blink Rate - Attention Detection - Stress Detection - Real-time Dashboard - Multi-person Detection - Interview Difficulty Estimation - LLM-powered Coaching - Historical Performance Tracking - Resume-aware Feedback - Personalized Practice Sessions --- # 💡 Design Philosophy The system follows a **streaming pipeline architecture**. Instead of repeatedly running expensive AI models for each analysis task, each frame is processed once, producing reusable intermediate representations (such as facial landmarks and body keypoints). These intermediate results are then shared across specialized analysis modules. This design provides: - High performance - Low memory consumption - Modular components - Easy extensibility - Support for both live and recorded interviews - Separation of concerns - Production-ready architecture The goal is to build an extensible Computer Vision platform rather than a collection of independent scripts, making it straightforward to integrate new models, replace existing algorithms, or support additional input sources in the future.