Release SignBridge-TSL-STGCN with compliance Model Card
Browse files- README.md +4 -0
- README_zh.md +81 -0
README.md
CHANGED
|
@@ -19,6 +19,10 @@ pipeline_tag: video-classification
|
|
| 19 |
|
| 20 |
# SignBridge-TSL-STGCN: Taiwanese Sign Language Recognition via ST-GCN
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
## 1. Model Summary
|
| 23 |
|
| 24 |
SignBridge-TSL-STGCN is a lightweight Spatial-Temporal Graph Convolutional Network (ST-GCN) engineered for isolated Taiwanese Sign Language (TSL) word recognition. The model models the spatial configuration of skeletal joints and their temporal evolution across frames, enabling real-time, low-latency sign language recognition suitable for mobile and edge deployments.
|
|
|
|
| 19 |
|
| 20 |
# SignBridge-TSL-STGCN: Taiwanese Sign Language Recognition via ST-GCN
|
| 21 |
|
| 22 |
+
<p align="center">
|
| 23 |
+
<b>English</b> | <a href="README_zh.md">繁體中文</a>
|
| 24 |
+
</p>
|
| 25 |
+
|
| 26 |
## 1. Model Summary
|
| 27 |
|
| 28 |
SignBridge-TSL-STGCN is a lightweight Spatial-Temporal Graph Convolutional Network (ST-GCN) engineered for isolated Taiwanese Sign Language (TSL) word recognition. The model models the spatial configuration of skeletal joints and their temporal evolution across frames, enabling real-time, low-latency sign language recognition suitable for mobile and edge deployments.
|
README_zh.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SignBridge-TSL-STGCN: 台灣手語時空圖卷積識別模型
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<a href="README.md">English</a> | <b>繁體中文</b>
|
| 5 |
+
</p>
|
| 6 |
+
|
| 7 |
+
## 1. 模型摘要 (Model Summary)
|
| 8 |
+
|
| 9 |
+
SignBridge-TSL-STGCN 是專為「台灣手語(Taiwanese Sign Language, TSL)」獨立詞辨識研發之輕量化時空圖卷積神經網路(ST-GCN)模型。本模型透過人體骨架關節點之拓撲圖卷積與時間卷積,捕捉手語動作在時空序列中的動態特徵,具備邊緣端與即時環境下低延遲、高穩定度的辨識能力。
|
| 10 |
+
|
| 11 |
+
- **研發單位**:SignBridge 專案團隊 (Taiwan)
|
| 12 |
+
- **模型架構**:時空圖卷積神經網路(Spatial-Temporal Graph Convolutional Network, ST-GCN)
|
| 13 |
+
- **輸入維度**:`(Batch_Size, Channels=2, Frames=64, Nodes=55)`
|
| 14 |
+
- **關節拓撲**:55 節點(12 上半身姿態 + 21 左手掌指 + 21 右手掌指 + 1 雙肩中點虛擬根節點)
|
| 15 |
+
- **主要權重檔**:`best_model.pt`
|
| 16 |
+
- **開源授權**:Apache License 2.0
|
| 17 |
+
- **官方儲存庫**:[https://huggingface.co/dYang1/SignBridge-TSL-STGCN](https://huggingface.co/dYang1/SignBridge-TSL-STGCN)
|
| 18 |
+
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
## 2. 開源與非中資合規聲明 (Compliance and Provenance)
|
| 22 |
+
|
| 23 |
+
為符合數位發展部數位產業署「開源 AI 模型應用組」之競賽規範與資通安全查核,本模型聲明如下:
|
| 24 |
+
|
| 25 |
+
1. **自主研發與非中資來源**:
|
| 26 |
+
- 本模型之網路架構設計、特徵管線邏輯、訓練流程及推論權重,均由台灣本地團隊自主研發完成。
|
| 27 |
+
- 核心框架均採用開源且具國際公信力之主流專案(PyTorch、OpenCV、MediaPipe),完全無使用任何源自中國大陸或受國際制裁實體之專有模型、私有權重或黑箱相依套件。
|
| 28 |
+
2. **訓練資料在地合規**:
|
| 29 |
+
- 訓練語料皆採集自台灣手語(TSL)詞彙與日常會話情境,遵循台灣在地手語文法,無跨境不合規數據。
|
| 30 |
+
3. **商業友善與完整開源**:
|
| 31 |
+
- 依 Apache-2.0 協議完整開源模型權重、圖架構代碼(`model.py`、`graph.py`)、特徵前處理邏輯(`extract.py`)與標籤對應表(`sign_language_words.json`)。
|
| 32 |
+
|
| 33 |
+
---
|
| 34 |
+
|
| 35 |
+
## 3. 骨架拓撲與特徵前處理 (Skeletal Layout & Preprocessing)
|
| 36 |
+
|
| 37 |
+
特徵提取採用 55 個空間座標點(透過 MediaPipe Holistic 擷取):
|
| 38 |
+
- **上半身姿態 (節點 0–11)**:肩部、手肘、手腕、髖部關鍵節點。
|
| 39 |
+
- **左手掌與指關節 (節點 12–32)**:完整 21 個手部關節鏈結。
|
| 40 |
+
- **右手掌與指關節 (節點 33–53)**:完整 21 個手部關節鏈結。
|
| 41 |
+
- **虛擬頸點 (節點 54)**:取雙肩中點作為全域空間幾何對齊與正規化原點。
|
| 42 |
+
|
| 43 |
+
所有輸入序列皆在時間軸上等距重採樣至 64 幀,並依據雙肩距離進行空間尺度正規化。
|
| 44 |
+
|
| 45 |
+
---
|
| 46 |
+
|
| 47 |
+
## 4. 快速推論範例代碼 (Inference Snippet)
|
| 48 |
+
|
| 49 |
+
```python
|
| 50 |
+
import json
|
| 51 |
+
import torch
|
| 52 |
+
from model import build_from_checkpoint
|
| 53 |
+
|
| 54 |
+
# 1. 載入模型檢查點
|
| 55 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 56 |
+
checkpoint = torch.load("best_model.pt", map_location=device)
|
| 57 |
+
model, config = build_from_checkpoint(checkpoint)
|
| 58 |
+
model.eval()
|
| 59 |
+
|
| 60 |
+
# 2. 讀取台灣手語詞彙標籤
|
| 61 |
+
with open("sign_language_words.json", "r", encoding="utf-8") as f:
|
| 62 |
+
vocab = json.load(f)
|
| 63 |
+
|
| 64 |
+
# 3. 模擬輸入骨架特徵: (Batch=1, Channels=2, Frames=64, Nodes=55)
|
| 65 |
+
dummy_skeleton = torch.randn(1, 2, 64, 55).to(device)
|
| 66 |
+
|
| 67 |
+
# 4. 執行前向傳播
|
| 68 |
+
with torch.no_grad():
|
| 69 |
+
logits = model(dummy_skeleton)
|
| 70 |
+
probabilities = torch.softmax(logits, dim=-1)
|
| 71 |
+
predicted_index = torch.argmax(probabilities, dim=-1).item()
|
| 72 |
+
|
| 73 |
+
print(f"辨識結果 Gloss: {vocab[predicted_index]}")
|
| 74 |
+
print(f"信心度 Confidence: {probabilities[0, predicted_index]:.4f}")
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
---
|
| 78 |
+
|
| 79 |
+
## 5. 引用與授權 (Citation & License)
|
| 80 |
+
|
| 81 |
+
本專案依 Apache License 2.0 條款授權釋出。若您在研究或商業專案中使用本權重或架構,請註明引用本儲存庫。
|