Spaces:
Running on Zero
Running on Zero
File size: 2,166 Bytes
dc2e00e 2ee32cf dc2e00e 2ee32cf 8ed9a1c d776254 dc2e00e 2ee32cf 8ed9a1c 2ee32cf 8ed9a1c 2ee32cf 3448150 2ee32cf 3448150 2ee32cf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
title: SignLingo ASL Coach
emoji: 🤟
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: mit
hardware: zero-gpu
tags:
- asl
- sign-language
- mediapipe
- minicpm
- education
- backyard-ai
- build-small
- off-brand
- tiny-titan
- best-demo
- codex
- track:backyard
- sponsor:openbmb
- sponsor:openai
- achievement:offgrid
- achievement:offbrand
---
# SignLingo ASL Coach
Learn American Sign Language with real-time AI coaching. Built for the **Hugging Face × Gradio Build Small** hackathon.
## What it does
- Watch the reference ASL sign on the left
- Sign in front of your webcam on the right
- **MediaPipe** detects your hands automatically — recording starts when you raise your hands, stops when you drop them
- **DTW (Dynamic Time Warping)** scores your attempt against the reference in real time
- On a failed attempt, frames from your recording are sent to **MiniCPM-V** which returns one specific coaching tip
## How it works
- Hand + pose landmarks extracted in-browser via MediaPipe WASM (no server round-trip for detection)
- Normalized landmark sequences compared with DTW for a 0–100 similarity score
- Failed attempts: 1-2fps frames → MiniCPM-V 4.6 → natural language coaching tip
- No cloud APIs — scoring runs entirely in the browser; coaching runs on HF Zero GPU
## Tech Stack
| Component | Choice |
|-----------|--------|
| Frontend | React + Vite served via `gr.Server` |
| Hand detection | MediaPipe Tasks Vision (WASM, in-browser) |
| Scoring | DTW on normalized landmark sequences |
| Coaching VLM | MiniCPM-V 4.6 on HF Zero GPU |
| Reference Signs | Lifeprint ASL Dictionary |
## Vocabulary (23 signs)
Hello, Yes, No, Thank You, Help, Pain, Head, Back, Sick, Fever, Medicine, Water, Blood, Doctor, Nurse, Emergency, Bank, Money, Pay, Credit Card, Account, Deposit, Withdraw
## Demo Video
https://youtu.be/qynGluC2ZLc
## Social Post
https://x.com/JaspalS44414885/status/2066665273739092445
## Built for
**Hugging Face × Gradio "Build Small" Hackathon** — Track: Backyard AI (Practical)
Made with care for the deaf and hard-of-hearing community.
|