| --- |
| title: LumiSign |
| emoji: 🚀 |
| colorFrom: blue |
| colorTo: red |
| sdk: docker |
| sdk_version: 4.19.2 |
| python_version: 3.9 |
| app_file: app.py |
| pinned: false |
| short_description: Indian Sign Language Recognition Web UI |
| --- |
| |
| # LumiSign |
|
|
| Indian Sign Language recognition with FastAPI + React UI, using a Transformer model trained on `isl-split-dataset`. |
|
|
| This README is written for developers who are new to the project. |
|
|
| ## What You Need |
|
|
| 1. Python `3.9` |
| 2. Node.js `18+` and npm |
| 3. Git |
| 4. A Linux/macOS shell (or WSL on Windows) |
|
|
| ## Project Layout |
|
|
| ```text |
| LumiSign/ |
| app.py |
| inference.py |
| runner.py |
| prepare_custom_dataset.py |
| check_split_leakage.py |
| transformer_large.pth |
| label_maps/ |
| label_map_isl_split_dataset.json |
| isl-split-dataset/ |
| train/ |
| eval/ |
| test/ |
| ui/ |
| ``` |
|
|
| ## Dataset Layout Expected |
|
|
| `prepare_custom_dataset.py` expects this format: |
|
|
| ```text |
| isl-split-dataset/ |
| train/ |
| bank/ |
| court/ |
| store or shop/ |
| eval/ |
| bank/ |
| court/ |
| store or shop/ |
| test/ |
| bank/ |
| court/ |
| store or shop/ |
| ``` |
|
|
| Labels are normalized internally: |
| - `store or shop` becomes `storeorshop` |
|
|
| ## Quick Start (Use Existing `transformer_large.pth`) |
| |
| Run these from the project root. |
| |
| ### 1) Create and install environment |
| |
| ```bash |
| python3.9 -m venv venv |
| source venv/bin/activate |
| venv/bin/pip install --upgrade pip |
| venv/bin/pip install -r requirements.txt |
| ``` |
| |
| ### 2) Build frontend once |
| |
| ```bash |
| cd ui |
| npm install |
| npm run build |
| cd .. |
| ``` |
| |
| ### 3) Start backend with ISL model |
| |
| ```bash |
| MODEL_CHECKPOINT=./transformer_large.pth \ |
| MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ |
| MODEL_DATASET=isl_split_dataset \ |
| MODEL_TYPE=transformer \ |
| MODEL_TRANSFORMER_SIZE=large \ |
| MODEL_MAX_FRAME_LEN=169 \ |
| venv/bin/python -m uvicorn app:app --host 0.0.0.0 --port 8070 --reload |
| ``` |
| |
| Open: |
| - UI: `http://localhost:8070` |
| - API docs: `http://localhost:8070/docs` |
| |
| ## Full Pipeline (Prepare Data -> Train -> Evaluate) |
| |
| Use this if you want to regenerate keypoints and retrain. |
| |
| ### 1) Set paths |
| |
| ```bash |
| PROJECT_ROOT="$(pwd)" |
| ISL_SPLIT_ROOT="$PROJECT_ROOT/isl-split-dataset" |
| ISL_DATASET_NAME="isl_split_dataset" |
| ISL_PROCESSED_DIR="$PROJECT_ROOT/processed_data_islsplit" |
| ISL_CKPT_DIR="$PROJECT_ROOT/checkpoints_islsplit" |
| |
| mkdir -p "$ISL_PROCESSED_DIR" "$ISL_CKPT_DIR" |
| ``` |
| |
| ### 2) Convert split folders to keypoint JSON files |
| |
| ```bash |
| venv/bin/python prepare_custom_dataset.py \ |
| --data_dir "$ISL_SPLIT_ROOT" \ |
| --save_dir "$ISL_PROCESSED_DIR" \ |
| --dataset_name "$ISL_DATASET_NAME" \ |
| --jobs 4 \ |
| --use_holistic \ |
| --face_mode full \ |
| --write_placeholders |
| ``` |
| |
| This creates: |
| - `processed_data_islsplit/isl_split_dataset_train_keypoints` |
| - `processed_data_islsplit/isl_split_dataset_val_keypoints` |
| - `processed_data_islsplit/isl_split_dataset_test_keypoints` |
| - `label_maps/label_map_isl_split_dataset.json` |
|
|
| ### 3) Check split leakage |
|
|
| ```bash |
| venv/bin/python check_split_leakage.py \ |
| --data_dir "$ISL_PROCESSED_DIR" \ |
| --dataset "$ISL_DATASET_NAME" \ |
| --dark_suffix __dark |
| ``` |
|
|
| ### 4) Train Transformer (large) |
|
|
| ```bash |
| venv/bin/python runner.py \ |
| --dataset "$ISL_DATASET_NAME" \ |
| --model transformer \ |
| --transformer_size large \ |
| --max_frame_len 169 \ |
| --data_dir "$ISL_PROCESSED_DIR" \ |
| --save_path "$ISL_CKPT_DIR" \ |
| --batch_size 8 \ |
| --early_stop_metric val_loss \ |
| --early_stop_patience 5 |
| ``` |
|
|
| ### 5) Evaluate on test split |
|
|
| ```bash |
| venv/bin/python runner.py \ |
| --dataset "$ISL_DATASET_NAME" \ |
| --model transformer \ |
| --transformer_size large \ |
| --max_frame_len 169 \ |
| --data_dir "$ISL_PROCESSED_DIR" \ |
| --save_path "$ISL_CKPT_DIR" \ |
| --batch_size 1 \ |
| --epochs 0 \ |
| --eval_split test |
| ``` |
|
|
| ## Single Video CLI Inference |
|
|
| ```bash |
| venv/bin/python inference.py \ |
| --video /absolute/path/to/video.mp4 \ |
| --dataset isl_split_dataset \ |
| --model transformer \ |
| --transformer_size large \ |
| --checkpoint ./transformer_large.pth \ |
| --label_map_path ./label_maps/label_map_isl_split_dataset.json \ |
| --max_frame_len 169 |
| ``` |
|
|
| ## Run UI + API in Development |
|
|
| Terminal 1: |
|
|
| ```bash |
| MODEL_CHECKPOINT=./transformer_large.pth \ |
| MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ |
| MODEL_DATASET=isl_split_dataset \ |
| MODEL_TYPE=transformer \ |
| MODEL_TRANSFORMER_SIZE=large \ |
| MODEL_MAX_FRAME_LEN=169 \ |
| venv/bin/python -m uvicorn app:app --host 0.0.0.0 --port 8070 --reload |
| ``` |
|
|
| Terminal 2: |
|
|
| ```bash |
| cd ui |
| npm install |
| npm run dev |
| ``` |
|
|
| If Vite starts on `5173`, open `http://localhost:5173`. |
|
|
| ## Common Errors and Fixes |
|
|
| ### 1) `size mismatch for l2.weight` when loading checkpoint |
|
|
| Cause: |
| - Label map class count does not match checkpoint output classes. |
|
|
| Fix: |
|
|
| ```bash |
| MODEL_CHECKPOINT=./transformer_large.pth \ |
| MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ |
| MODEL_DATASET=isl_split_dataset |
| ``` |
|
|
| Your ISL checkpoint expects 3 classes: `bank`, `court`, `storeorshop`. |
|
|
| ### 2) CUDA warning like `Error 804: forward compatibility was attempted` |
|
|
| Cause: |
| - GPU driver/CUDA compatibility mismatch. |
|
|
| Fix: |
| - Run on CPU, or align NVIDIA driver + CUDA + PyTorch versions. |
| - This warning is not the same as the class mismatch error. |
|
|
| ### 3) `No label map found for dataset ...` |
|
|
| Fix: |
| - Ensure `label_maps/label_map_isl_split_dataset.json` exists. |
| - Pass `MODEL_LABEL_MAP_PATH` explicitly. |
|
|
| ## Useful Entrypoints |
|
|
| - `app.py`: FastAPI server startup and model loading |
| - `inference.py`: model load and single-video prediction |
| - `prepare_custom_dataset.py`: convert split video folders to keypoint JSON |
| - `check_split_leakage.py`: verify train/val/test split leakage |
| - `runner.py`: train and evaluate models |
|
|
| ## Reference Commands |
|
|
| - Full command history used in this project is in `commands.txt`. |
|
|
| ## Citation |
|
|
| If you use this work, cite INCLUDE : |
|
|
| ```bibtex |
| @inproceedings{10.1145/3394171.3413528, |
| author = {Sridhar, Advaith and Ganesan, Rohith Gandhi and Kumar, Pratyush and Khapra, Mitesh}, |
| title = {INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition}, |
| year = {2020}, |
| isbn = {9781450379885}, |
| publisher = {Association for Computing Machinery}, |
| doi = {10.1145/3394171.3413528}, |
| series = {MM '20} |
| } |
| ``` |
|
|
