Claw commited on
Commit ·
b4917e3
1
Parent(s): 1084b8c
Replace Comic-Gen with Chinese→Malay Video Dubber (CPU Basic)
Browse files- Dockerfile +4 -3
- README.md +16 -58
- app.py +220 -212
- requirements.txt +7 -14
Dockerfile
CHANGED
|
@@ -1,9 +1,10 @@
|
|
| 1 |
# HF-recommended base for Gradio Spaces
|
| 2 |
FROM python:3.10-slim
|
| 3 |
|
| 4 |
-
# Install system dependencies
|
| 5 |
RUN apt-get update && apt-get install -y --no-install-recommends \
|
| 6 |
git \
|
|
|
|
| 7 |
&& rm -rf /var/lib/apt/lists/*
|
| 8 |
|
| 9 |
# Create user with UID 1000 (HF requirement)
|
|
@@ -14,7 +15,7 @@ ENV HOME=/home/user \
|
|
| 14 |
|
| 15 |
WORKDIR $HOME/app
|
| 16 |
|
| 17 |
-
# Copy requirements and install
|
| 18 |
COPY --chown=user requirements.txt .
|
| 19 |
RUN pip install --no-cache-dir -r requirements.txt \
|
| 20 |
"uvicorn>=0.14.0" \
|
|
@@ -27,4 +28,4 @@ ENV GRADIO_SERVER_NAME=0.0.0.0 \
|
|
| 27 |
GRADIO_SERVER_PORT=7860 \
|
| 28 |
PYTHONUNBUFFERED=1
|
| 29 |
|
| 30 |
-
CMD ["python", "app.py"]
|
|
|
|
| 1 |
# HF-recommended base for Gradio Spaces
|
| 2 |
FROM python:3.10-slim
|
| 3 |
|
| 4 |
+
# Install system dependencies (ffmpeg for audio/video processing)
|
| 5 |
RUN apt-get update && apt-get install -y --no-install-recommends \
|
| 6 |
git \
|
| 7 |
+
ffmpeg \
|
| 8 |
&& rm -rf /var/lib/apt/lists/*
|
| 9 |
|
| 10 |
# Create user with UID 1000 (HF requirement)
|
|
|
|
| 15 |
|
| 16 |
WORKDIR $HOME/app
|
| 17 |
|
| 18 |
+
# Copy requirements and install
|
| 19 |
COPY --chown=user requirements.txt .
|
| 20 |
RUN pip install --no-cache-dir -r requirements.txt \
|
| 21 |
"uvicorn>=0.14.0" \
|
|
|
|
| 28 |
GRADIO_SERVER_PORT=7860 \
|
| 29 |
PYTHONUNBUFFERED=1
|
| 30 |
|
| 31 |
+
CMD ["python", "app.py"]
|
README.md
CHANGED
|
@@ -1,70 +1,28 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
-
sdk:
|
| 7 |
-
sdk_version: "5.0.0"
|
| 8 |
app_file: app.py
|
| 9 |
pinned: false
|
| 10 |
-
license: apache-2.0
|
| 11 |
-
tags:
|
| 12 |
-
- stable-diffusion
|
| 13 |
-
- sdxl
|
| 14 |
-
- noobai
|
| 15 |
-
- manga
|
| 16 |
-
- anime
|
| 17 |
-
- cpu-basic
|
| 18 |
---
|
| 19 |
|
| 20 |
-
#
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
-
|
| 27 |
-
- **NoobAI-XL 1.1 SDNQ UINT4**:量化模型 ~2.56 GB,Diffusers 仓库 ~3.63 GB,适配 16GB RAM
|
| 28 |
-
- **CPU Basic 优化**:
|
| 29 |
-
- `low_cpu_mem_usage=True` 全链路
|
| 30 |
-
- VAE slicing + tiling
|
| 31 |
-
- Attention slicing
|
| 32 |
-
- `enable_model_cpu_offload()` 逐层卸载
|
| 33 |
-
- 单张生成,关闭并行
|
| 34 |
-
- **默认配置**:640×960、20 steps、Euler Ancestral、guidance 5.0
|
| 35 |
-
- **输出**:PNG 格式
|
| 36 |
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
| 40 |
-
2. 上传三个文件:`app.py`、`requirements.txt`、`README.md`
|
| 41 |
-
3. 等待 Build 完成(首次下载模型约 3.6 GB)
|
| 42 |
-
4. 打开网页,输入中文场景,点击 Generate
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
|
| 47 |
-
|------|------|--------|-------|
|
| 48 |
-
| 一只愤怒的霸王龙站在暴雨中的史前森林里仰头怒吼,闪电照亮天空,低角度镜头,强烈的漫画构图,动态线条,丰富背景细节 | Manga / Anime | 640×960 | 20 |
|
| 49 |
-
| 赛博朋克城市夜晚,霓虹灯招牌倒映在湿漉漉的街道上,孤独的少女撑着透明伞,电影感构图 | Manga / Anime | 640×960 | 20 |
|
| 50 |
-
| 古风仙侠,白衣剑仙御剑飞行穿越云海,日出金光洒在云层上,宏大壮观,工笔画风 | Manga / Anime | 640×960 | 25 |
|
| 51 |
-
|
| 52 |
-
## 性能预期 (CPU Basic: 2 vCPU / 16GB RAM)
|
| 53 |
-
|
| 54 |
-
| 分辨率 | Steps | 预估耗时 |
|
| 55 |
-
|--------|-------|----------|
|
| 56 |
-
| 640×960 | 20 | ~3-5 分钟 |
|
| 57 |
-
| 640×960 | 30 | ~5-7 分钟 |
|
| 58 |
-
| 512×768 | 20 | ~2-3 分钟 |
|
| 59 |
-
|
| 60 |
-
> 首次冷启动需额外下载模型权重(~3.6 GB),之后缓存复用。
|
| 61 |
-
|
| 62 |
-
## 模型来源
|
| 63 |
-
|
| 64 |
-
- **NoobAI-XL/NoobAI-XL-1.1-SDNQ-UINT4** (Apache-2.0)
|
| 65 |
-
- SDNQ 量化:INT4 权重 + FP16 激活,体积约 1/3 原版
|
| 66 |
-
|
| 67 |
-
## 许可证
|
| 68 |
-
|
| 69 |
-
Apache-2.0(遵循 NoobAI-XL 原模型许可)
|
| 70 |
-
# Rebuild trigger Tue Sep 1 15:58:37 UTC 2026
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Chinese To Malay Video Dubber CPU
|
| 3 |
+
emoji: 🇨🇳🇲🇾
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: green
|
| 6 |
+
sdk: gradio
|
|
|
|
| 7 |
app_file: app.py
|
| 8 |
pinned: false
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
+
# Chinese → Bahasa Melayu Video Dubber — CPU Basic
|
| 12 |
|
| 13 |
+
Designed for Hugging Face Spaces CPU Basic (2 vCPU / 16 GB RAM).
|
| 14 |
|
| 15 |
+
Pipeline:
|
| 16 |
|
| 17 |
+
Chinese MP4 → Whisper Small → Chinese → Malay translation → Meta MMS Malay TTS → FFmpeg → Malay MP4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
+
Models:
|
| 20 |
+
- `openai/whisper-small`
|
| 21 |
+
- `Helsinki-NLP/opus-mt-zh-ms`
|
| 22 |
+
- `facebook/mms-tts-zlm`
|
| 23 |
|
| 24 |
+
This version does not clone the original speaker's voice. It is intentionally lightweight for CPU Basic.
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
Recommended test: 1–3 minute videos. Longer videos can take a long time on CPU.
|
| 27 |
|
| 28 |
+
FFmpeg/ffprobe are required by the Space runtime.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
app.py
CHANGED
|
@@ -1,221 +1,229 @@
|
|
| 1 |
import os
|
| 2 |
-
import
|
| 3 |
-
import
|
| 4 |
-
|
| 5 |
-
from
|
| 6 |
-
from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
|
| 7 |
-
from safetensors.torch import load_file
|
| 8 |
-
import gc
|
| 9 |
-
|
| 10 |
-
# ─── Config ──────────────────────────────────────────────────────────
|
| 11 |
-
MODEL_ID = "Laxhar/noobai-XL-1.1" # HF repo (public, diffusers-compatible)
|
| 12 |
-
DEVICE = "cpu"
|
| 13 |
-
DTYPE = torch.bfloat16 # CPU 不支持 fp16 运算,改用 bfloat16(同样 2 字节/参数,省内存且 CPU 原生支持)
|
| 14 |
-
DEFAULT_W, DEFAULT_H = 640, 960
|
| 15 |
-
DEFAULT_STEPS = 20
|
| 16 |
-
DEFAULT_GUIDANCE = 5.0
|
| 17 |
-
|
| 18 |
-
# ─── Prompt enhancement (Chinese → English + quality tags) ───────────
|
| 19 |
-
QUALITY_TAGS = (
|
| 20 |
-
"masterpiece, best quality, ultra high res, "
|
| 21 |
-
"dynamic composition, dramatic lighting, "
|
| 22 |
-
"detailed background, manga style, anime style, "
|
| 23 |
-
"sharp focus, vivid colors, professional illustration"
|
| 24 |
-
)
|
| 25 |
-
|
| 26 |
-
NEGATIVE_PROMPT = (
|
| 27 |
-
"low quality, worst quality, bad anatomy, bad composition, "
|
| 28 |
-
"blurry, pixelated, watermark, text, signature, logo, "
|
| 29 |
-
"ugly, deformed, noise, artifacts, lowres, error, "
|
| 30 |
-
"missing fingers, extra digits, fewer digits, cropped, "
|
| 31 |
-
"jpeg artifacts, out of frame, mutation, mutated, "
|
| 32 |
-
"extra limbs, malformed limbs, long neck, disfigured"
|
| 33 |
-
)
|
| 34 |
-
|
| 35 |
-
STYLE_PROMPTS = {
|
| 36 |
-
"Manga / Anime": "manga style, anime style, screentone, ink lines, dynamic panel composition",
|
| 37 |
-
"Western Comics": "western comic style, bold outlines, halftone dots, vibrant colors, dynamic action",
|
| 38 |
-
"Manhwa": "manhwa style, webtoon style, clean lines, soft shading, vertical composition",
|
| 39 |
-
"Sketch / Lineart": "sketch style, lineart, black and white, pencil drawing, detailed strokes",
|
| 40 |
-
}
|
| 41 |
-
|
| 42 |
-
def enhance_prompt(user_prompt: str, style: str) -> str:
|
| 43 |
-
"""中文场景 → 英文提示词 + 质量标签 + 风格标签"""
|
| 44 |
-
# 简单的中文关键词映射(可扩展为更完整的翻译/扩写)
|
| 45 |
-
cn_en_map = {
|
| 46 |
-
"恐龙": "dinosaur",
|
| 47 |
-
"霸王龙": "T-Rex",
|
| 48 |
-
"暴雨": "heavy rain",
|
| 49 |
-
"史前森林": "prehistoric forest",
|
| 50 |
-
"仰头": "looking up",
|
| 51 |
-
"怒吼": "roaring",
|
| 52 |
-
"闪电": "lightning",
|
| 53 |
-
"照亮": "illuminating",
|
| 54 |
-
"天空": "sky",
|
| 55 |
-
"低角度": "low angle",
|
| 56 |
-
"镜头": "camera angle",
|
| 57 |
-
"强烈": "intense",
|
| 58 |
-
"漫画": "manga",
|
| 59 |
-
"构图": "composition",
|
| 60 |
-
"动态": "dynamic",
|
| 61 |
-
"线条": "lines",
|
| 62 |
-
"丰富": "rich",
|
| 63 |
-
"背景": "background",
|
| 64 |
-
"细节": "details",
|
| 65 |
-
"愤怒": "angry",
|
| 66 |
-
"站在": "standing in",
|
| 67 |
-
"里": "in",
|
| 68 |
-
"的": "",
|
| 69 |
-
"一只": "a",
|
| 70 |
-
"在": "in",
|
| 71 |
-
"中": "in",
|
| 72 |
-
"和": "and",
|
| 73 |
-
"与": "and",
|
| 74 |
-
}
|
| 75 |
-
# 粗略翻译:把常见中文词替换成英文
|
| 76 |
-
translated = user_prompt
|
| 77 |
-
for cn, en in cn_en_map.items():
|
| 78 |
-
translated = translated.replace(cn, en)
|
| 79 |
-
# 如果全是中文,做简单拼接;如果已经有英文,保留
|
| 80 |
-
if any('\u4e00' <= c <= '\u9fff' for c in translated):
|
| 81 |
-
# 还含有中文,简单处理
|
| 82 |
-
prompt_en = translated
|
| 83 |
-
else:
|
| 84 |
-
prompt_en = translated
|
| 85 |
-
style_tag = STYLE_PROMPTS.get(style, STYLE_PROMPTS["Manga / Anime"])
|
| 86 |
-
return f"{prompt_en}, {style_tag}, {QUALITY_TAGS}"
|
| 87 |
-
|
| 88 |
-
# ─── Load pipeline (lazy, cached) ────────────────────────────────────
|
| 89 |
-
_pipe = None
|
| 90 |
-
|
| 91 |
-
def get_pipeline():
|
| 92 |
-
global _pipe
|
| 93 |
-
if _pipe is not None:
|
| 94 |
-
return _pipe
|
| 95 |
-
|
| 96 |
-
print("Loading NoobAI-XL SDNQ pipeline on CPU (bf16)...")
|
| 97 |
-
|
| 98 |
-
# 1) VAE (单独加载,启用 slicing)
|
| 99 |
-
vae = AutoencoderKL.from_pretrained(
|
| 100 |
-
MODEL_ID,
|
| 101 |
-
subfolder="vae",
|
| 102 |
-
torch_dtype=DTYPE,
|
| 103 |
-
low_cpu_mem_usage=True,
|
| 104 |
-
)
|
| 105 |
-
vae.enable_slicing()
|
| 106 |
-
vae.enable_tiling()
|
| 107 |
-
|
| 108 |
-
# 2) Text encoders & tokenizers (SDXL 双编码器)
|
| 109 |
-
tokenizer_1 = CLIPTokenizer.from_pretrained(MODEL_ID, subfolder="tokenizer")
|
| 110 |
-
tokenizer_2 = CLIPTokenizer.from_pretrained(MODEL_ID, subfolder="tokenizer_2")
|
| 111 |
-
text_encoder_1 = CLIPTextModel.from_pretrained(
|
| 112 |
-
MODEL_ID, subfolder="text_encoder", torch_dtype=DTYPE, low_cpu_mem_usage=True
|
| 113 |
-
)
|
| 114 |
-
text_encoder_2 = CLIPTextModelWithProjection.from_pretrained(
|
| 115 |
-
MODEL_ID, subfolder="text_encoder_2", torch_dtype=DTYPE, low_cpu_mem_usage=True
|
| 116 |
-
)
|
| 117 |
-
|
| 118 |
-
# 3) Pipeline
|
| 119 |
-
pipe = StableDiffusionXLPipeline.from_pretrained(
|
| 120 |
-
MODEL_ID,
|
| 121 |
-
vae=vae,
|
| 122 |
-
text_encoder=text_encoder_1,
|
| 123 |
-
text_encoder_2=text_encoder_2,
|
| 124 |
-
tokenizer=tokenizer_1,
|
| 125 |
-
tokenizer_2=tokenizer_2,
|
| 126 |
-
scheduler=EulerAncestralDiscreteScheduler.from_pretrained(MODEL_ID, subfolder="scheduler"),
|
| 127 |
-
torch_dtype=DTYPE,
|
| 128 |
-
low_cpu_mem_usage=True,
|
| 129 |
-
)
|
| 130 |
-
|
| 131 |
-
# CPU 优化
|
| 132 |
-
# 注意:enable_model_cpu_offload() 依赖 CUDA(会把模块按需搬到 GPU),
|
| 133 |
-
# 纯 CPU 环境不能用,否则会报 "Found no NVIDIA driver"
|
| 134 |
-
pipe.enable_vae_slicing()
|
| 135 |
-
pipe.enable_vae_tiling()
|
| 136 |
-
pipe.enable_attention_slicing("auto") # 注意力切片
|
| 137 |
-
# 关闭 xformers / flash attention(CPU 不支持)
|
| 138 |
-
try:
|
| 139 |
-
pipe.disable_xformers_memory_efficient_attention()
|
| 140 |
-
except Exception:
|
| 141 |
-
pass
|
| 142 |
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
def generate(prompt, style, width, height, steps, guidance, progress=gr.Progress()):
|
| 150 |
-
if not prompt.strip():
|
| 151 |
-
return None, "请输入场景描述"
|
| 152 |
|
| 153 |
-
|
| 154 |
-
|
|
|
|
| 155 |
|
| 156 |
-
|
| 157 |
-
|
| 158 |
|
| 159 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
with torch.inference_mode():
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
""
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
with gr.Row():
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
value="Manga / Anime",
|
| 202 |
-
)
|
| 203 |
-
with gr.Row():
|
| 204 |
-
width = gr.Slider(label="Width", minimum=320, maximum=1024, value=DEFAULT_W, step=64)
|
| 205 |
-
height = gr.Slider(label="Height", minimum=320, maximum=1536, value=DEFAULT_H, step=64)
|
| 206 |
-
steps = gr.Slider(label="Steps", minimum=10, maximum=50, value=DEFAULT_STEPS, step=1)
|
| 207 |
-
guidance = gr.Slider(label="Guidance Scale", minimum=1.0, maximum=15.0, value=DEFAULT_GUIDANCE, step=0.5)
|
| 208 |
-
btn = gr.Button("Generate", elem_classes="generate-btn", variant="primary")
|
| 209 |
-
|
| 210 |
-
with gr.Column(scale=1):
|
| 211 |
-
output_img = gr.Image(label="生成的漫画场景", type="filepath", interactive=False)
|
| 212 |
-
status = gr.Textbox(label="Status", interactive=False)
|
| 213 |
-
|
| 214 |
-
btn.click(
|
| 215 |
-
generate,
|
| 216 |
-
inputs=[prompt, style, width, height, steps, guidance],
|
| 217 |
-
outputs=[output_img, status],
|
| 218 |
-
)
|
| 219 |
|
| 220 |
if __name__ == "__main__":
|
| 221 |
-
demo.
|
|
|
|
| 1 |
import os
|
| 2 |
+
import re
|
| 3 |
+
import subprocess
|
| 4 |
+
import tempfile
|
| 5 |
+
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
+
import gradio as gr
|
| 8 |
+
import torch
|
| 9 |
+
import torchaudio
|
| 10 |
+
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
|
| 11 |
+
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
|
| 12 |
+
from transformers import VitsModel
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
+
ASR_MODEL = os.getenv("ASR_MODEL", "openai/whisper-small")
|
| 15 |
+
TRANSLATION_MODEL = os.getenv("TRANSLATION_MODEL", "Helsinki-NLP/opus-mt-zh-ms")
|
| 16 |
+
TTS_MODEL = os.getenv("TTS_MODEL", "facebook/mms-tts-zlm")
|
| 17 |
|
| 18 |
+
torch.set_num_threads(2)
|
| 19 |
+
DEVICE = "cpu"
|
| 20 |
|
| 21 |
+
_asr_processor = _asr_model = None
|
| 22 |
+
_tr_tokenizer = _tr_model = None
|
| 23 |
+
_tts_tokenizer = _tts_model = None
|
| 24 |
+
|
| 25 |
+
def run(cmd):
|
| 26 |
+
p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
| 27 |
+
if p.returncode:
|
| 28 |
+
raise RuntimeError(p.stderr[-5000:])
|
| 29 |
+
return p.stdout
|
| 30 |
+
|
| 31 |
+
def video_duration(path):
|
| 32 |
+
return float(run(["ffprobe","-v","error","-show_entries","format=duration",
|
| 33 |
+
"-of","default=noprint_wrappers=1:nokey=1",str(path)]).strip())
|
| 34 |
+
|
| 35 |
+
def extract_audio(video, wav):
|
| 36 |
+
run(["ffmpeg","-y","-i",str(video),"-vn","-ac","1","-ar","16000",str(wav)])
|
| 37 |
+
|
| 38 |
+
def load_asr():
|
| 39 |
+
global _asr_processor, _asr_model
|
| 40 |
+
if _asr_model is None:
|
| 41 |
+
_asr_processor = AutoProcessor.from_pretrained(ASR_MODEL)
|
| 42 |
+
_asr_model = AutoModelForSpeechSeq2Seq.from_pretrained(
|
| 43 |
+
ASR_MODEL, low_cpu_mem_usage=True
|
| 44 |
+
).to(DEVICE).eval()
|
| 45 |
+
return _asr_processor, _asr_model
|
| 46 |
+
|
| 47 |
+
def transcribe(wav):
|
| 48 |
+
processor, model = load_asr()
|
| 49 |
+
audio, sr = torchaudio.load(str(wav))
|
| 50 |
+
audio = audio.mean(0)
|
| 51 |
+
if sr != 16000:
|
| 52 |
+
audio = torchaudio.functional.resample(audio, sr, 16000)
|
| 53 |
+
|
| 54 |
+
total = audio.numel() / 16000
|
| 55 |
+
pieces = []
|
| 56 |
+
chunk = 30
|
| 57 |
+
|
| 58 |
+
for start in range(0, int(total + 0.999), chunk):
|
| 59 |
+
x = audio[start*16000:min(int(total*16000),(start+chunk)*16000)].numpy()
|
| 60 |
+
if len(x) < 400:
|
| 61 |
+
continue
|
| 62 |
+
inputs = processor(x, sampling_rate=16000, return_tensors="pt")
|
| 63 |
+
with torch.inference_mode():
|
| 64 |
+
ids = model.generate(
|
| 65 |
+
inputs.input_features,
|
| 66 |
+
language="zh",
|
| 67 |
+
task="transcribe",
|
| 68 |
+
max_new_tokens=256
|
| 69 |
+
)
|
| 70 |
+
text = processor.batch_decode(ids, skip_special_tokens=True)[0].strip()
|
| 71 |
+
if text:
|
| 72 |
+
pieces.append(text)
|
| 73 |
+
return " ".join(pieces)
|
| 74 |
+
|
| 75 |
+
def load_translation():
|
| 76 |
+
global _tr_tokenizer, _tr_model
|
| 77 |
+
if _tr_model is None:
|
| 78 |
+
_tr_tokenizer = AutoTokenizer.from_pretrained(TRANSLATION_MODEL)
|
| 79 |
+
_tr_model = AutoModelForSeq2SeqLM.from_pretrained(
|
| 80 |
+
TRANSLATION_MODEL, low_cpu_mem_usage=True
|
| 81 |
+
).to(DEVICE).eval()
|
| 82 |
+
return _tr_tokenizer, _tr_model
|
| 83 |
+
|
| 84 |
+
def translate(text):
|
| 85 |
+
tokenizer, model = load_translation()
|
| 86 |
+
parts = [x.strip() for x in re.split(r'(?<=[。!?!?])\s*', text) if x.strip()]
|
| 87 |
+
if not parts:
|
| 88 |
+
parts = [text]
|
| 89 |
+
|
| 90 |
+
result = []
|
| 91 |
+
for part in parts:
|
| 92 |
+
inputs = tokenizer(part, return_tensors="pt", truncation=True, max_length=256)
|
| 93 |
+
with torch.inference_mode():
|
| 94 |
+
ids = model.generate(**inputs, max_new_tokens=256, num_beams=4,
|
| 95 |
+
renormalize_logits=True)
|
| 96 |
+
result.append(tokenizer.decode(ids[0], skip_special_tokens=True).strip())
|
| 97 |
+
return " ".join(result)
|
| 98 |
+
|
| 99 |
+
def load_tts():
|
| 100 |
+
global _tts_tokenizer, _tts_model
|
| 101 |
+
if _tts_model is None:
|
| 102 |
+
_tts_tokenizer = AutoTokenizer.from_pretrained(TTS_MODEL)
|
| 103 |
+
_tts_model = VitsModel.from_pretrained(
|
| 104 |
+
TTS_MODEL, low_cpu_mem_usage=True
|
| 105 |
+
).to(DEVICE).eval()
|
| 106 |
+
return _tts_tokenizer, _tts_model
|
| 107 |
+
|
| 108 |
+
def tts(text, wav):
|
| 109 |
+
tokenizer, model = load_tts()
|
| 110 |
+
inputs = tokenizer(text, return_tensors="pt")
|
| 111 |
with torch.inference_mode():
|
| 112 |
+
audio = model(**inputs).waveform.cpu()
|
| 113 |
+
torchaudio.save(str(wav), audio, model.config.sampling_rate)
|
| 114 |
+
|
| 115 |
+
def fit_audio(src, dst, target):
|
| 116 |
+
target = max(float(target), 0.1)
|
| 117 |
+
ratio = video_duration(src) / target
|
| 118 |
+
filters = []
|
| 119 |
+
while ratio > 2:
|
| 120 |
+
filters.append("atempo=2.0")
|
| 121 |
+
ratio /= 2
|
| 122 |
+
while ratio < 0.5:
|
| 123 |
+
filters.append("atempo=0.5")
|
| 124 |
+
ratio /= 0.5
|
| 125 |
+
filters.append(f"atempo={ratio:.6f}")
|
| 126 |
+
run(["ffmpeg","-y","-i",str(src),"-filter:a",",".join(filters),
|
| 127 |
+
"-t",f"{target:.3f}","-ar","24000","-ac","1",str(dst)])
|
| 128 |
+
|
| 129 |
+
def make_dub(ms, total, work):
|
| 130 |
+
parts = [x.strip() for x in re.split(r'(?<=[。!?!?])\s*|(?<=[,,])\s+', ms) if x.strip()]
|
| 131 |
+
if not parts:
|
| 132 |
+
parts = [ms]
|
| 133 |
+
weights = [max(1,len(re.sub(r'\s+','',x))) for x in parts]
|
| 134 |
+
total_w = sum(weights)
|
| 135 |
+
|
| 136 |
+
silent = work/"silent.wav"
|
| 137 |
+
run(["ffmpeg","-y","-f","lavfi","-i","anullsrc=r=24000:cl=mono",
|
| 138 |
+
"-t",f"{total:.3f}","-ar","24000","-ac","1",str(silent)])
|
| 139 |
+
|
| 140 |
+
inputs = ["-i",str(silent)]
|
| 141 |
+
labels = ["[0:a]"]
|
| 142 |
+
start = 0.0
|
| 143 |
+
|
| 144 |
+
for i, part in enumerate(parts, 1):
|
| 145 |
+
slot = total * weights[i-1] / total_w
|
| 146 |
+
raw, fitted = work/f"tts_{i}.wav", work/f"fit_{i}.wav"
|
| 147 |
+
tts(part, raw)
|
| 148 |
+
fit_audio(raw, fitted, slot)
|
| 149 |
+
inputs += ["-i",str(fitted)]
|
| 150 |
+
delay = int(start*1000)
|
| 151 |
+
labels.append(f"[{i}:a]adelay={delay}|{delay}[d{i}]")
|
| 152 |
+
start += slot
|
| 153 |
+
|
| 154 |
+
mix = "[0:a]" + "".join(f"[d{i}]" for i in range(1,len(parts)+1))
|
| 155 |
+
filt = ";".join(labels) + f";{mix}amix=inputs={len(parts)+1}:duration=first[a]"
|
| 156 |
+
out = work/"dub.wav"
|
| 157 |
+
run(["ffmpeg","-y",*inputs,"-filter_complex",filt,"-map","[a]",
|
| 158 |
+
"-ar","24000","-ac","1",str(out)])
|
| 159 |
+
return out
|
| 160 |
+
|
| 161 |
+
def process(video, keep_original, original_volume, progress=gr.Progress()):
|
| 162 |
+
if not video:
|
| 163 |
+
raise gr.Error("请上传中文视频。")
|
| 164 |
+
work = Path(tempfile.mkdtemp(prefix="zhms_cpu_"))
|
| 165 |
+
try:
|
| 166 |
+
src = Path(video)
|
| 167 |
+
audio = work/"source.wav"
|
| 168 |
+
progress(0.05, desc="提取声音…")
|
| 169 |
+
extract_audio(src, audio)
|
| 170 |
+
total = video_duration(src)
|
| 171 |
+
|
| 172 |
+
progress(0.15, desc="中文语音识别(CPU)…")
|
| 173 |
+
zh = transcribe(audio)
|
| 174 |
+
if not zh:
|
| 175 |
+
raise RuntimeError("没有识别到中文语音。")
|
| 176 |
+
|
| 177 |
+
progress(0.42, desc="翻译成 Bahasa Melayu…")
|
| 178 |
+
ms = translate(zh)
|
| 179 |
+
|
| 180 |
+
progress(0.58, desc="生成马来文配音…")
|
| 181 |
+
dub = make_dub(ms, total, work)
|
| 182 |
+
|
| 183 |
+
out = work/"malay_dub.mp4"
|
| 184 |
+
progress(0.9, desc="合成 MP4…")
|
| 185 |
+
if keep_original:
|
| 186 |
+
filt = f"[0:a]volume={float(original_volume):.3f}[o];[o][1:a]amix=inputs=2:duration=first[a]"
|
| 187 |
+
run(["ffmpeg","-y","-i",str(src),"-i",str(dub),"-filter_complex",filt,
|
| 188 |
+
"-map","0:v:0","-map","[a]","-c:v","copy","-c:a","aac","-b:a","160k",
|
| 189 |
+
"-shortest",str(out)])
|
| 190 |
+
else:
|
| 191 |
+
run(["ffmpeg","-y","-i",str(src),"-i",str(dub),
|
| 192 |
+
"-map","0:v:0","-map","1:a:0","-c:v","copy","-c:a","aac",
|
| 193 |
+
"-b:a","160k","-shortest",str(out)])
|
| 194 |
+
|
| 195 |
+
txt = work/"translation.txt"
|
| 196 |
+
txt.write_text("=== 中文识别 ===\n"+zh+"\n\n=== Bahasa Melayu ===\n"+ms,encoding="utf-8")
|
| 197 |
+
progress(1, desc="完成")
|
| 198 |
+
return str(out), str(txt), zh, ms
|
| 199 |
+
except Exception as e:
|
| 200 |
+
raise gr.Error(str(e))
|
| 201 |
+
|
| 202 |
+
with gr.Blocks(title="中文 → Bahasa Melayu(CPU Basic)") as demo:
|
| 203 |
+
gr.Markdown("""# 🇨🇳 → 🇲🇾 中文视频 → Bahasa Melayu
|
| 204 |
+
**Hugging Face CPU Basic 专用版本**
|
| 205 |
+
|
| 206 |
+
上传中文视频 → 中文语音识别 → 马来文翻译 → 马来文 TTS → 输出 MP4。
|
| 207 |
+
|
| 208 |
+
- 不需要 GPU
|
| 209 |
+
- 不需要 API Key
|
| 210 |
+
- 不使用声音克隆
|
| 211 |
+
- 使用专门的 Chinese → Malay 翻译模型
|
| 212 |
+
- 使用 Meta MMS Malay TTS
|
| 213 |
+
|
| 214 |
+
⚠️ CPU Basic 只有 2 vCPU / 16 GB RAM,建议先测试 1–3 分钟视频。
|
| 215 |
+
""")
|
| 216 |
+
video = gr.Video(label="上传中文视频", sources=["upload"], type="filepath")
|
| 217 |
with gr.Row():
|
| 218 |
+
keep = gr.Checkbox(label="保留原声音作为背景", value=False)
|
| 219 |
+
volume = gr.Slider(0,0.3,value=0.06,step=0.01,label="原声音量")
|
| 220 |
+
button = gr.Button("🚀 开始翻译成马来文", variant="primary")
|
| 221 |
+
output = gr.Video(label="🇲🇾 马来文视频")
|
| 222 |
+
text_file = gr.File(label="识别 + 翻译文本")
|
| 223 |
+
with gr.Row():
|
| 224 |
+
zh_box = gr.Textbox(label="中文识别结果", lines=8)
|
| 225 |
+
ms_box = gr.Textbox(label="Bahasa Melayu", lines=8)
|
| 226 |
+
button.click(process,[video,keep,volume],[output,text_file,zh_box,ms_box])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 227 |
|
| 228 |
if __name__ == "__main__":
|
| 229 |
+
demo.queue(max_size=2).launch()
|
requirements.txt
CHANGED
|
@@ -1,14 +1,7 @@
|
|
| 1 |
-
|
| 2 |
-
torch=
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
accelerate==0.33.0
|
| 9 |
-
safetensors==0.4.4
|
| 10 |
-
pillow==10.4.0
|
| 11 |
-
|
| 12 |
-
# Gradio (use latest 5.x with latest gradio_client)
|
| 13 |
-
gradio>=5.0.0
|
| 14 |
-
gradio_client>=1.0.0
|
|
|
|
| 1 |
+
gradio>=5,<7
|
| 2 |
+
torch>=2.2
|
| 3 |
+
torchaudio>=2.2
|
| 4 |
+
transformers>=4.48,<5
|
| 5 |
+
sentencepiece>=0.2
|
| 6 |
+
safetensors>=0.4
|
| 7 |
+
numpy<2.3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|