BonusLockSMith commited on
Commit
95dcd0c
·
verified ·
1 Parent(s): e12db0b

switch faster-whisper to CPU int8 (base) — no CUDA libs; free CPU Space

Browse files
Files changed (3) hide show
  1. README.md +6 -9
  2. app.py +15 -31
  3. requirements.txt +0 -2
README.md CHANGED
@@ -23,12 +23,11 @@ so every quote is citable by the minute), a short **summary**, and a list of **a
23
  ![demo](demo.png)
24
 
25
  ## How it works
26
- - **Transcribe** with **faster-whisper** (`small`, float16) on **Hugging Face ZeroGPU** — the GPU is held
27
- only for the transcription itself.
28
  - **Summarize** with one **Claude** call using **forced tool-use**, so the summary + action items come
29
  back as schema-valid JSON every time (no brittle parsing).
30
- - **Two-stage pipeline:** transcription (GPU) → summarization (CPU/API), so the GPU is released before
31
- the network call.
32
 
33
  ## Two tiers
34
  - **Free (this demo)** — short clips → transcript + summary + action items.
@@ -39,14 +38,12 @@ so every quote is citable by the minute), a short **summary**, and a list of **a
39
  ```bash
40
  pip install -r requirements.txt
41
  export ANTHROPIC_API_KEY=sk-ant-... # for the summary + action items
42
- python app.py # http://127.0.0.1:7860 (transcription needs a CUDA GPU)
43
  ```
44
- The `spaces` import is shimmed to a no-op locally, so the UI builds without a GPU — but transcription
45
- requires CUDA + faster-whisper.
46
 
47
  ## Deploy (Hugging Face Space)
48
- 1. Create a **Gradio** Space, hardware **ZeroGPU**.
49
- 2. Add secrets **`ANTHROPIC_API_KEY`** (summary) and, if models are gated, **`HF_TOKEN`**.
50
  3. Push these files; point **notes.gritai.solutions** at the Space (CNAME → `hf.space`, DNS-only).
51
 
52
  ---
 
23
  ![demo](demo.png)
24
 
25
  ## How it works
26
+ - **Transcribe** with **faster-whisper** (`base`, int8) on **CPU** — reliable and quick for short clips,
27
+ no GPU/CUDA libraries required.
28
  - **Summarize** with one **Claude** call using **forced tool-use**, so the summary + action items come
29
  back as schema-valid JSON every time (no brittle parsing).
30
+ - **Two-stage pipeline:** transcription → summarization, each a clean step.
 
31
 
32
  ## Two tiers
33
  - **Free (this demo)** — short clips → transcript + summary + action items.
 
38
  ```bash
39
  pip install -r requirements.txt
40
  export ANTHROPIC_API_KEY=sk-ant-... # for the summary + action items
41
+ python app.py # http://127.0.0.1:7860
42
  ```
 
 
43
 
44
  ## Deploy (Hugging Face Space)
45
+ 1. Create a **Gradio** Space — **CPU basic** hardware is enough (no GPU needed).
46
+ 2. Add secret **`ANTHROPIC_API_KEY`** for the summary + action items.
47
  3. Push these files; point **notes.gritai.solutions** at the Space (CNAME → `hf.space`, DNS-only).
48
 
49
  ---
app.py CHANGED
@@ -17,22 +17,9 @@ import os
17
 
18
  import gradio as gr
19
 
20
- try:
21
- import spaces # ZeroGPU runtime
22
- _HAS_SPACES = True
23
- except Exception:
24
- _HAS_SPACES = False
25
-
26
- class spaces: # local no-op shim so the UI builds without GPU
27
- @staticmethod
28
- def GPU(*a, **k):
29
- def wrap(fn):
30
- return fn
31
- return wrap
32
-
33
-
34
- WHISPER_SIZE = "small" # good accuracy/speed balance; float16 on ZeroGPU
35
- WHISPER_REPO = "Systran/faster-whisper-small"
36
  CLAUDE_MODEL = os.environ.get("ANTHROPIC_MODEL", "claude-haiku-4-5")
37
  MAX_TRANSCRIPT_CHARS = 24000 # keep the summarize call comfortably in-context
38
 
@@ -40,23 +27,21 @@ _MODEL = None
40
 
41
 
42
  def _get_model():
43
- """Construct the faster-whisper model on CUDA (cached). Weights are pre-fetched at startup so this
44
- only loads from the warm disk cache, not the network, inside the GPU budget."""
45
  global _MODEL
46
  if _MODEL is None:
47
  from faster_whisper import WhisperModel
48
- _MODEL = WhisperModel(WHISPER_SIZE, device="cuda", compute_type="float16")
49
  return _MODEL
50
 
51
 
52
- # Pre-fetch weights at startup, ONLY on the real Space (never during local UI preview).
53
- if _HAS_SPACES:
54
  try:
55
- from huggingface_hub import snapshot_download
56
- snapshot_download(WHISPER_REPO)
57
- print("[startup] faster-whisper weights cached; GPU attaches per request", flush=True)
58
  except Exception as _e:
59
- print(f"[startup] whisper prefetch failed (will retry on first call): {_e}", flush=True)
60
 
61
 
62
  def _mmss(seconds):
@@ -64,15 +49,14 @@ def _mmss(seconds):
64
  return f"{seconds // 60:02d}:{seconds % 60:02d}"
65
 
66
 
67
- @spaces.GPU(duration=120)
68
  def _transcribe(audio_path):
69
- import torch
70
- if not torch.cuda.is_available():
 
71
  raise gr.Error(
72
- "🎙️ Local UI preview — transcription runs on the ZeroGPU Space (needs a GPU). "
73
- "The controls and Studio waitlist here are live."
74
  )
75
- model = _get_model()
76
  segments, info = model.transcribe(audio_path, vad_filter=True, beam_size=5)
77
  rows = [(seg.start, seg.end, seg.text.strip()) for seg in segments]
78
  return rows, getattr(info, "language", "?")
 
17
 
18
  import gradio as gr
19
 
20
+ ON_SPACE = bool(os.environ.get("SPACE_ID")) # True when running on a Hugging Face Space
21
+
22
+ WHISPER_SIZE = "base" # int8 on CPU — fast + reliable for short clips, no CUDA libs needed
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  CLAUDE_MODEL = os.environ.get("ANTHROPIC_MODEL", "claude-haiku-4-5")
24
  MAX_TRANSCRIPT_CHARS = 24000 # keep the summarize call comfortably in-context
25
 
 
27
 
28
 
29
  def _get_model():
30
+ """Construct the faster-whisper model on CPU (int8), cached. Runs anywhere — no GPU / CUDA libs."""
 
31
  global _MODEL
32
  if _MODEL is None:
33
  from faster_whisper import WhisperModel
34
+ _MODEL = WhisperModel(WHISPER_SIZE, device="cpu", compute_type="int8")
35
  return _MODEL
36
 
37
 
38
+ # Warm the model at startup on the Space so the first request isn't slow.
39
+ if ON_SPACE:
40
  try:
41
+ _get_model()
42
+ print("[startup] faster-whisper (base, cpu/int8) loaded", flush=True)
 
43
  except Exception as _e:
44
+ print(f"[startup] whisper load failed (will retry on first call): {_e}", flush=True)
45
 
46
 
47
  def _mmss(seconds):
 
49
  return f"{seconds // 60:02d}:{seconds % 60:02d}"
50
 
51
 
 
52
  def _transcribe(audio_path):
53
+ try:
54
+ model = _get_model()
55
+ except ImportError:
56
  raise gr.Error(
57
+ "🎙️ Local UI preview — transcription runs on the Space (faster-whisper isn't installed here). "
58
+ "The controls and Studio waitlist are live."
59
  )
 
60
  segments, info = model.transcribe(audio_path, vad_filter=True, beam_size=5)
61
  rows = [(seg.start, seg.end, seg.text.strip()) for seg in segments]
62
  return rows, getattr(info, "language", "?")
requirements.txt CHANGED
@@ -1,5 +1,3 @@
1
  # gradio comes from the Space (sdk_version in README) — don't pin it here.
2
- spaces>=0.30
3
- torch
4
  faster-whisper>=1.0
5
  anthropic>=0.40
 
1
  # gradio comes from the Space (sdk_version in README) — don't pin it here.
 
 
2
  faster-whisper>=1.0
3
  anthropic>=0.40