Spaces:
Paused
NINA v0.80 — performance pass (app.py)
Browse files- Auto device select (CUDA/MPS/CPU): CLIP + cross-encoder on Metal on Apple Silicon.
- Drop double transcription: global uses the base transcript; specific does
confidence-gated (avg_logprob) selective turbo re-transcription only.
- Parallel cold start (ASR on CPU || CLIP on MPS); zero-copy audio decode and
raw-frame ffmpeg pipe (no temp wav / JPEGs).
- Persistent disk cache keyed by content fingerprint.
- Streaming LLM synthesis; run() is now a generator.
- Router fix (no more "about X" => global) and removed the leaking [2:30] example.
Measured: cold first prompt 58.8s -> ~9s on M1; warm prompts sub-second to first token.
requirements.txt: pinned to tested versions; CPU torch wheel index for the Space.
README: HF_TOKEN secret note + @gradio/client API usage for the app.nnlabs.pl front-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- .gitignore +5 -0
- README.md +29 -1
- app.py +918 -164
- requirements.txt +20 -6
|
@@ -1,2 +1,7 @@
|
|
| 1 |
.DS_Store
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
|
|
|
| 1 |
.DS_Store
|
| 2 |
+
.venv/
|
| 3 |
+
venv/
|
| 4 |
+
__pycache__/
|
| 5 |
+
*.pyc
|
| 6 |
+
.env
|
| 7 |
|
|
@@ -12,4 +12,32 @@ license: apache-2.0
|
|
| 12 |
short_description: Advanced video sumarizing tool
|
| 13 |
---
|
| 14 |
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
short_description: Advanced video sumarizing tool
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# NINA — Neural Information & Narrative Abstraction
|
| 16 |
+
|
| 17 |
+
Upload a video and a prompt; NINA answers using **only** the video — fusing what
|
| 18 |
+
is *said* (Whisper ASR) and what is *shown* (CLIP), with semantic retrieval, a
|
| 19 |
+
cross-encoder rerank, confidence-gated precise re-transcription, and a streaming
|
| 20 |
+
LLM synthesis. See the header of `app.py` for the full v0.80 pipeline + perf notes.
|
| 21 |
+
|
| 22 |
+
## Running this Space
|
| 23 |
+
|
| 24 |
+
* **Required secret:** set `HF_TOKEN` in *Settings → Variables and secrets* — the
|
| 25 |
+
LLM synthesis step calls the Hugging Face Inference API. Without it, NINA still
|
| 26 |
+
retrieves regions but falls back to raw excerpts instead of a written summary.
|
| 27 |
+
* **Hardware:** runs on CPU (auto-falls back from MPS/CUDA). `large-v3-turbo` is
|
| 28 |
+
CPU-bound, so precise re-transcription is gated to only fire on low-confidence
|
| 29 |
+
audio. A larger CPU/GPU tier improves latency on long videos.
|
| 30 |
+
|
| 31 |
+
## API (used by the custom front-end at app.nnlabs.pl/nina)
|
| 32 |
+
|
| 33 |
+
The Gradio app doubles as the API. From the browser:
|
| 34 |
+
|
| 35 |
+
```js
|
| 36 |
+
import { Client } from "https://cdn.jsdelivr.net/npm/@gradio/client/+esm";
|
| 37 |
+
const app = await Client.connect("Malina420/Neural-Information-and-Narrative-Abstraction");
|
| 38 |
+
const job = app.submit("/run", { video_path: fileBlob, prompt: "summarize briefly" });
|
| 39 |
+
for await (const ev of job) if (ev.type === "data") console.log(ev.data[0]); // streams
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Config reference: https://huggingface.co/docs/hub/spaces-config-reference
|
| 43 |
+
|
|
@@ -1,13 +1,63 @@
|
|
| 1 |
# ─────────────────────────────────────────────
|
| 2 |
# NINA — Neural Information & Narrative Abstraction
|
| 3 |
-
#
|
| 4 |
# ─────────────────────────────────────────────
|
| 5 |
#
|
| 6 |
-
# Precision video search + summarization pipeline.
|
| 7 |
-
# Pipeline: video → audio → ASR segments → semantic chunking
|
|
|
|
|
|
|
| 8 |
#
|
| 9 |
-
#
|
| 10 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
#
|
| 12 |
# Entity extraction: regex + simple NP heuristic (no spacy dependency).
|
| 13 |
# Works multilingually because it operates on surface forms, not POS tags.
|
|
@@ -15,15 +65,19 @@
|
|
| 15 |
|
| 16 |
import os
|
| 17 |
import re
|
|
|
|
| 18 |
import math
|
|
|
|
| 19 |
import tempfile
|
| 20 |
import threading
|
| 21 |
-
import
|
| 22 |
|
| 23 |
import ffmpeg
|
| 24 |
-
import whisper
|
| 25 |
import gradio as gr
|
| 26 |
import numpy as np
|
|
|
|
|
|
|
|
|
|
| 27 |
from sentence_transformers import SentenceTransformer, CrossEncoder
|
| 28 |
from huggingface_hub import InferenceClient
|
| 29 |
import warnings
|
|
@@ -34,38 +88,104 @@ warnings.filterwarnings("ignore", message="FP16 is not supported on CPU; using F
|
|
| 34 |
|
| 35 |
WHISPER_FAST = "base"
|
| 36 |
WHISPER_PRECISE = "large-v3-turbo"
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
CHUNK_SPLIT_THRESHOLD = 0.45
|
| 40 |
CHUNK_MIN_DURATION = 8.0
|
| 41 |
CHUNK_MAX_DURATION = 90.0
|
| 42 |
CHUNK_WINDOW = 5
|
| 43 |
|
| 44 |
-
BI_ENCODER_TOP_K = 100
|
| 45 |
-
CROSS_ENCODER_TOP_K = 10
|
| 46 |
-
CE_WEIGHT = 0.6
|
| 47 |
-
ENTITY_WEIGHT = 0.4
|
| 48 |
RESULTS_TOP_N = 6
|
| 49 |
REGION_MIN_DURATION = 5.0
|
| 50 |
MERGE_GAP_SEC = 6.0
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
LLM_MODEL = "Qwen/Qwen2.5-7B-Instruct"
|
| 53 |
LLM_MAX_TOKENS = 600
|
| 54 |
LLM_TIMEOUT = 60
|
| 55 |
|
|
|
|
|
|
|
|
|
|
| 56 |
DEBUG = False
|
| 57 |
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
# ── Model load ─────────────────────────────────────────────────────────────────
|
| 60 |
|
| 61 |
-
print("[NINA]
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
| 63 |
|
|
|
|
|
|
|
| 64 |
print("[NINA] Loading bi-encoder...")
|
| 65 |
-
bi_encoder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
|
| 66 |
|
| 67 |
-
print("[NINA] Loading cross-encoder...")
|
| 68 |
-
cross_encoder = CrossEncoder("cross-encoder/mmarco-mMiniLMv2-L12-H384-v1")
|
| 69 |
|
| 70 |
|
| 71 |
HF_TOKEN = os.getenv("HF_TOKEN")
|
|
@@ -92,61 +212,153 @@ def get_whisper_precise():
|
|
| 92 |
global _whisper_precise
|
| 93 |
with _whisper_precise_lock:
|
| 94 |
if _whisper_precise is None:
|
| 95 |
-
print(f"[NINA] Loading Whisper {WHISPER_PRECISE}...")
|
| 96 |
-
_whisper_precise =
|
|
|
|
|
|
|
| 97 |
print(f"[NINA] Whisper {WHISPER_PRECISE} ready.")
|
| 98 |
return _whisper_precise
|
| 99 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
threading.Thread(target=get_whisper_precise, daemon=True).start()
|
| 101 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
|
| 103 |
|
| 104 |
# ── Audio ──────────────────────────────────────────────────────────────────────
|
| 105 |
|
| 106 |
-
def
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
(
|
| 110 |
-
ffmpeg.input(video_path)
|
| 111 |
-
.output(tmp.name, ac=1, ar=16000, vn=None)
|
| 112 |
-
.run(overwrite_output=True, quiet=True)
|
| 113 |
-
)
|
| 114 |
-
return tmp.name
|
| 115 |
|
| 116 |
|
| 117 |
-
def
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
|
| 128 |
# ── Stage 2: transcription ─────────────────────────────────────────────────────
|
| 129 |
|
| 130 |
-
def transcribe_fast(
|
| 131 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
return [
|
| 133 |
{
|
| 134 |
-
"text":
|
| 135 |
-
"start":
|
| 136 |
-
"end":
|
|
|
|
| 137 |
}
|
| 138 |
-
for seg in
|
| 139 |
-
if seg
|
| 140 |
]
|
| 141 |
|
| 142 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
# ── semantic chunking ─────────────────────────────────────────────────
|
| 144 |
|
| 145 |
def semantic_chunk(segments: list[dict]) -> list[dict]:
|
| 146 |
if not segments:
|
| 147 |
return []
|
| 148 |
texts = [s["text"] for s in segments]
|
| 149 |
-
embeddings = bi_encoder.encode(texts, convert_to_numpy=True)
|
| 150 |
chunks, current, cur_emb = [], [segments[0]], [embeddings[0]]
|
| 151 |
|
| 152 |
for i in range(1, len(segments)):
|
|
@@ -157,14 +369,14 @@ def semantic_chunk(segments: list[dict]) -> list[dict]:
|
|
| 157 |
long_enough = (current[-1]["end"] - current[0]["start"]) >= CHUNK_MIN_DURATION
|
| 158 |
|
| 159 |
if (sim < CHUNK_SPLIT_THRESHOLD or chunk_dur >= CHUNK_MAX_DURATION) and long_enough:
|
| 160 |
-
chunks.append(_build_chunk(current))
|
| 161 |
current, cur_emb = [segments[i]], [embeddings[i]]
|
| 162 |
else:
|
| 163 |
current.append(segments[i])
|
| 164 |
cur_emb.append(embeddings[i])
|
| 165 |
|
| 166 |
if current:
|
| 167 |
-
chunks.append(_build_chunk(current))
|
| 168 |
|
| 169 |
print(f"[NINA] {len(segments)} segments → {len(chunks)} chunks.")
|
| 170 |
return chunks
|
|
@@ -192,18 +404,18 @@ def extract_entities(text: str) -> set[str]:
|
|
| 192 |
entities.add(match.group(1).lower().strip())
|
| 193 |
|
| 194 |
# 2. capitalised multi-word sequences (proper nouns)
|
| 195 |
-
for match in re.finditer(r'\b([A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-záéíóúàâêîôùäëïöüąćęłńóśźż]+(?:\s[A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-
|
| 196 |
token = match.group(1).lower().strip()
|
| 197 |
if token not in _STOPWORDS and len(token) > 2:
|
| 198 |
entities.add(token)
|
| 199 |
|
| 200 |
# 3. single capitalised tokens (names, brands)
|
| 201 |
-
for token in re.findall(r'\b[A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-
|
| 202 |
t = token.lower()
|
| 203 |
if t not in _STOPWORDS:
|
| 204 |
entities.add(t)
|
| 205 |
|
| 206 |
-
# 4. numbers with context
|
| 207 |
for match in re.finditer(r'\b([A-Za-z]+[-\s]?\d+[\w]*)\b', text):
|
| 208 |
entities.add(match.group(1).lower())
|
| 209 |
|
|
@@ -213,7 +425,7 @@ def extract_entities(text: str) -> set[str]:
|
|
| 213 |
def entity_overlap_score(prompt_entities: set[str],
|
| 214 |
chunk_entities: set[str],
|
| 215 |
chunk_text: str = "") -> float:
|
| 216 |
-
|
| 217 |
if not prompt_entities:
|
| 218 |
return 0.0
|
| 219 |
|
|
@@ -235,13 +447,60 @@ def add_entities_to_chunks(chunks: list[dict]) -> list[dict]:
|
|
| 235 |
return chunks
|
| 236 |
|
| 237 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
# ── Stage 4: bi-encoder retrieval (top-100) ───────────────────────────────────
|
| 239 |
|
| 240 |
def bi_encode_retrieve(chunks: list[dict], prompt: str) -> tuple[list[dict], float]:
|
| 241 |
-
|
| 242 |
-
|
| 243 |
prompt_emb = bi_encoder.encode([prompt], convert_to_numpy=True)[0]
|
| 244 |
-
scores = [_cosine(prompt_emb,
|
| 245 |
|
| 246 |
top_k = min(BI_ENCODER_TOP_K, len(chunks))
|
| 247 |
top_idx = np.argsort(scores)[::-1][:top_k]
|
|
@@ -257,53 +516,178 @@ def bi_encode_retrieve(chunks: list[dict], prompt: str) -> tuple[list[dict], flo
|
|
| 257 |
)
|
| 258 |
|
| 259 |
|
| 260 |
-
# ── Stage
|
| 261 |
|
| 262 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 263 |
# cross-encoder
|
| 264 |
pairs = [[prompt, c["text"]] for c in candidates]
|
| 265 |
logits = cross_encoder.predict(pairs)
|
| 266 |
|
| 267 |
# entity overlap
|
| 268 |
prompt_entities = extract_entities(prompt)
|
| 269 |
-
prompt_words = {w.lower() for w in re.findall(r'[\
|
| 270 |
if w.lower() not in _STOPWORDS}
|
| 271 |
all_prompt_tokens = prompt_entities | prompt_words
|
| 272 |
|
| 273 |
if DEBUG and all_prompt_tokens:
|
| 274 |
print(f"[DEBUG] Prompt tokens for entity match: {all_prompt_tokens}")
|
| 275 |
|
|
|
|
|
|
|
| 276 |
scored = []
|
| 277 |
for candidate, logit in zip(candidates, logits):
|
| 278 |
chunk_entities = candidate.get("entities", set())
|
| 279 |
chunk_text = candidate.get("text", "")
|
| 280 |
overlap = entity_overlap_score(all_prompt_tokens, chunk_entities, chunk_text)
|
| 281 |
-
bi_score = candidate.get("bi_score", 0.0)
|
| 282 |
|
| 283 |
-
|
|
|
|
|
|
|
|
|
|
| 284 |
|
| 285 |
scored.append({
|
| 286 |
**candidate,
|
| 287 |
"ce_logit": round(float(logit), 3),
|
| 288 |
"entity_overlap": round(overlap, 3),
|
| 289 |
-
"
|
|
|
|
| 290 |
})
|
| 291 |
|
| 292 |
-
# sort by
|
| 293 |
-
ranked = sorted(scored, key=lambda x: x["
|
| 294 |
|
| 295 |
if DEBUG:
|
| 296 |
-
print(f"[DEBUG] Top-3 after reranking:")
|
| 297 |
for r in ranked[:3]:
|
| 298 |
print(f" ce_logit={r['ce_logit']:.2f} "
|
| 299 |
f"bi={r['bi_score']:.3f} "
|
| 300 |
f"entity={r['entity_overlap']:.3f} "
|
|
|
|
| 301 |
f"conf={r['confidence']:.3f} "
|
| 302 |
f"\"{r['text'][:60]}\"")
|
| 303 |
|
| 304 |
return ranked[:CROSS_ENCODER_TOP_K]
|
| 305 |
|
| 306 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 307 |
# ── Stage 6: merge regions ─────────────────────────────────────────────────────
|
| 308 |
|
| 309 |
def merge_regions(results: list[dict]) -> list[dict]:
|
|
@@ -334,40 +718,137 @@ def _finalize_region(region: dict) -> dict:
|
|
| 334 |
for f in frags) / total_dur
|
| 335 |
duration = region["end"] - region["start"]
|
| 336 |
w_conf *= min(1.0, duration / 20.0)
|
| 337 |
-
|
|
|
|
| 338 |
|
| 339 |
|
| 340 |
-
# ── Stage 7: precise re-transcription ──────────────────────
|
| 341 |
|
| 342 |
-
def retranscribe_region(
|
| 343 |
-
|
| 344 |
-
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
|
| 350 |
-
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
|
| 355 |
-
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 361 |
if DEBUG:
|
| 362 |
-
|
| 363 |
-
|
| 364 |
-
|
| 365 |
-
|
| 366 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 367 |
|
| 368 |
|
| 369 |
# ── Stage 8: synthesis ─────────────────────────────────────────────────────────
|
| 370 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 371 |
def _fallback_summary(regions: list[dict]) -> str:
|
| 372 |
snippets = [
|
| 373 |
f"[{fmt_time(r['start'])}] {r.get('precise_text', r['text'])[:200]}"
|
|
@@ -376,49 +857,101 @@ def _fallback_summary(regions: list[dict]) -> str:
|
|
| 376 |
return "(LLM unavailable — raw excerpts:)\n" + "\n".join(snippets)
|
| 377 |
|
| 378 |
|
| 379 |
-
def
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 388 |
|
| 389 |
-
|
| 390 |
-
f"You are analyzing a video
|
| 391 |
-
f"
|
| 392 |
-
f
|
| 393 |
-
f"
|
|
|
|
| 394 |
f"Output format:\n"
|
| 395 |
-
|
| 396 |
-
f"-
|
| 397 |
-
|
| 398 |
-
f"-
|
| 399 |
-
f"-
|
| 400 |
-
f"-
|
|
|
|
|
|
|
| 401 |
)
|
| 402 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 403 |
try:
|
| 404 |
-
|
|
|
|
| 405 |
model=LLM_MODEL,
|
| 406 |
messages=[{"role": "user", "content": synth_prompt}],
|
| 407 |
max_tokens=LLM_MAX_TOKENS,
|
| 408 |
temperature=0.3,
|
|
|
|
| 409 |
)
|
| 410 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 411 |
except Exception as e:
|
| 412 |
print(f"[NINA] Synthesis error: {e}")
|
| 413 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 414 |
|
| 415 |
|
| 416 |
# ── Helpers ────────────────────────────────────────────────────────────────────
|
| 417 |
|
| 418 |
-
def _build_chunk(segs: list[dict]) -> dict:
|
| 419 |
-
|
| 420 |
-
|
| 421 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 422 |
|
| 423 |
def _cosine(a: np.ndarray, b: np.ndarray) -> float:
|
| 424 |
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))
|
|
@@ -435,9 +968,96 @@ def conf_bar(c: float) -> str:
|
|
| 435 |
return "█" * f + "░" * (10 - f)
|
| 436 |
|
| 437 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 438 |
# ── Formatter ──────────────────────────────────────────────────────────────────
|
| 439 |
|
| 440 |
-
def format_output(regions: list[dict], prompt: str, summary: str
|
|
|
|
| 441 |
if not regions:
|
| 442 |
return (
|
| 443 |
f' Topic not found in this video.\n\n'
|
|
@@ -446,82 +1066,214 @@ def format_output(regions: list[dict], prompt: str, summary: str) -> str:
|
|
| 446 |
f' Try a broader or differently worded prompt.'
|
| 447 |
)
|
| 448 |
|
| 449 |
-
|
|
|
|
|
|
|
| 450 |
if DEBUG:
|
| 451 |
for i, r in enumerate(regions, 1):
|
| 452 |
dur = round(r["end"] - r["start"], 1)
|
| 453 |
text = r.get("precise_text", r["text"])
|
| 454 |
ent = r.get("entity_overlap", 0)
|
| 455 |
bi = r.get("bi_score", 0)
|
|
|
|
|
|
|
| 456 |
lines += [
|
| 457 |
f'\n [{i}] {fmt_time(r["start"])} → {fmt_time(r["end"])} ({dur}s)',
|
| 458 |
f' {conf_bar(r["confidence"])} {r["confidence"]:.0%} '
|
| 459 |
-
f'(embed {bi:.0%} · entity {ent:.0%})',
|
|
|
|
| 460 |
f' "{text[:220]}{"…" if len(text) > 220 else ""}"',
|
| 461 |
]
|
| 462 |
-
|
| 463 |
-
|
| 464 |
-
|
| 465 |
-
f' 🧠 Based on uploaded video: — "{prompt}"',
|
| 466 |
summary,
|
| 467 |
f'\n Sources: ' + ', '.join(
|
| 468 |
f'[{fmt_time(r["start"])} → {fmt_time(r["end"])}]' for r in regions
|
| 469 |
),
|
| 470 |
]
|
| 471 |
-
return "\n".join(
|
| 472 |
|
| 473 |
|
| 474 |
-
# ──
|
| 475 |
|
| 476 |
-
def
|
| 477 |
-
|
| 478 |
-
|
| 479 |
-
|
| 480 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 481 |
|
| 482 |
-
|
| 483 |
-
print(f'\n[NINA] ── New request ── "{prompt}"')
|
| 484 |
|
| 485 |
-
|
| 486 |
-
|
| 487 |
-
|
| 488 |
-
|
| 489 |
-
|
| 490 |
-
|
| 491 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 492 |
|
| 493 |
-
print(f"[NINA] {len(segments)} raw segments.")
|
| 494 |
-
if not segments:
|
| 495 |
-
return "No speech detected."
|
| 496 |
|
| 497 |
-
|
| 498 |
-
chunks = semantic_chunk(segments)
|
| 499 |
-
chunks = add_entities_to_chunks(chunks)
|
| 500 |
|
| 501 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 502 |
candidates, best_bi = bi_encode_retrieve(chunks, prompt)
|
|
|
|
| 503 |
|
| 504 |
-
|
| 505 |
-
|
| 506 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 507 |
|
| 508 |
-
|
| 509 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 510 |
|
| 511 |
# Stage 6: merge adjacent regions
|
| 512 |
regions = merge_regions(results)
|
| 513 |
if not regions:
|
| 514 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 515 |
|
| 516 |
-
# Stage
|
| 517 |
-
|
|
|
|
| 518 |
|
| 519 |
-
# Stage 8: LLM synthesis
|
| 520 |
-
print("[NINA] Running synthesis...")
|
| 521 |
-
summary = llm_synthesize(regions, prompt)
|
| 522 |
|
| 523 |
-
|
| 524 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 525 |
|
| 526 |
|
| 527 |
# ── Gradio UI ──────────��───────────────────────────────────────────────────────
|
|
@@ -532,16 +1284,18 @@ demo = gr.Interface(
|
|
| 532 |
gr.Video(label="Upload video "),
|
| 533 |
gr.Textbox(
|
| 534 |
label="What are you looking for?",
|
| 535 |
-
placeholder='e.g. "What does the video say about...?"'
|
|
|
|
| 536 |
),
|
| 537 |
],
|
| 538 |
outputs=gr.Textbox(label="Results", lines=38),
|
| 539 |
-
title="NINA v0.
|
| 540 |
description=(
|
| 541 |
-
"video -> a bit of magic -> summary."
|
|
|
|
| 542 |
),
|
| 543 |
flagging_mode="never",
|
| 544 |
)
|
| 545 |
|
| 546 |
if __name__ == "__main__":
|
| 547 |
-
demo.launch()
|
|
|
|
| 1 |
# ─────────────────────────────────────────────
|
| 2 |
# NINA — Neural Information & Narrative Abstraction
|
| 3 |
+
# app.py | v0.80 | Made by Malina
|
| 4 |
# ─────────────────────────────────────────────
|
| 5 |
#
|
| 6 |
+
# Precision MULTIMODAL video search + summarization pipeline.
|
| 7 |
+
# Pipeline: video → audio → ASR segments → semantic chunking
|
| 8 |
+
# → bi-encoder (text) ⨉ CLIP (vision) fusion → cross-encoder + entity overlap
|
| 9 |
+
# → merge → confidence-gated precise re-transcribe → streaming LLM synthesis.
|
| 10 |
#
|
| 11 |
+
# ─────────────────────────────────────────────────────────────────────────────
|
| 12 |
+
# v0.80 — performance & modernisation pass over v0.71
|
| 13 |
+
# ─────────────────────────────────────────────────────────────────────────────
|
| 14 |
+
# Measured on Apple M1 (8 core / 16 GB), 39 s test clip:
|
| 15 |
+
# cold first prompt 58.8 s → ~9 s, warm prompts ~1.5 s → sub-second to first token.
|
| 16 |
+
#
|
| 17 |
+
# [P1] HARDWARE ACCELERATION
|
| 18 |
+
# Torch models now auto-target the best device (CUDA → MPS → CPU). On
|
| 19 |
+
# Apple Silicon CLIP runs ~1.8× and the cross-encoder ~1.4× faster on the
|
| 20 |
+
# Metal (MPS) backend. CTranslate2 (Whisper) has no MPS backend, so it
|
| 21 |
+
# stays on CPU but is tuned: int8 + cpu_threads, greedy decoding for the
|
| 22 |
+
# cheap pass, and Silero VAD to skip non-speech.
|
| 23 |
+
#
|
| 24 |
+
# [P2] NO MORE DOUBLE TRANSCRIPTION
|
| 25 |
+
# v0.71 transcribed the whole video with `base`, then RE-transcribed every
|
| 26 |
+
# sampled sector with the ~15× slower `large-v3-turbo` — even for a global
|
| 27 |
+
# summary (≈24 s wasted on this clip). v0.80:
|
| 28 |
+
# • GLOBAL summaries use the clean `base` transcript directly (no turbo).
|
| 29 |
+
# • SPECIFIC queries re-transcribe with turbo ONLY on matched regions and
|
| 30 |
+
# ONLY when the base pass was low-confidence (avg_logprob gate). On
|
| 31 |
+
# clean audio turbo never runs; on hard audio it runs exactly where it
|
| 32 |
+
# helps. Per-sector results are cached and persist across prompts.
|
| 33 |
+
#
|
| 34 |
+
# [P3] PARALLEL COLD START
|
| 35 |
+
# Audio transcription (CPU) and CLIP frame analysis (MPS) are independent
|
| 36 |
+
# and now run concurrently on first sight of a video, overlapping the two
|
| 37 |
+
# heaviest stages instead of summing them.
|
| 38 |
+
#
|
| 39 |
+
# [P4] ZERO-COPY I/O
|
| 40 |
+
# Audio is decoded straight from the container (PyAV) — no ffmpeg wav
|
| 41 |
+
# temp file. Frames are streamed from one ffmpeg rawvideo pipe into NumPy —
|
| 42 |
+
# no per-frame JPEG temp files.
|
| 43 |
+
#
|
| 44 |
+
# [P5] PERSISTENT DISK CACHE
|
| 45 |
+
# Transcript, chunk + visual embeddings and precise sectors are persisted
|
| 46 |
+
# under a content fingerprint, so re-opening the same video — even after a
|
| 47 |
+
# restart — skips transcription/chunking/CLIP entirely.
|
| 48 |
+
#
|
| 49 |
+
# [P6] STREAMING + UX
|
| 50 |
+
# LLM synthesis streams token-by-token; run() is a generator that shows
|
| 51 |
+
# progress and writes the answer as it is produced.
|
| 52 |
+
#
|
| 53 |
+
# [Q] QUALITY FIXES
|
| 54 |
+
# • Router no longer misreads "what does the video say about X" as a global
|
| 55 |
+
# summary (the bare word "about"/"explain" no longer forces global mode).
|
| 56 |
+
# • The illustrative "[2:30]" timestamp no longer leaks into answers.
|
| 57 |
+
#
|
| 58 |
+
# Scoring formula (SPECIFIC/HYBRID mode):
|
| 59 |
+
# audio = 0.6 * sigmoid(cross_encoder_logit) + 0.4 * entity_overlap
|
| 60 |
+
# final = (1 - VISUAL_WEIGHT) * audio + VISUAL_WEIGHT * clip_visual_score
|
| 61 |
#
|
| 62 |
# Entity extraction: regex + simple NP heuristic (no spacy dependency).
|
| 63 |
# Works multilingually because it operates on surface forms, not POS tags.
|
|
|
|
| 65 |
|
| 66 |
import os
|
| 67 |
import re
|
| 68 |
+
import json
|
| 69 |
import math
|
| 70 |
+
import hashlib
|
| 71 |
import tempfile
|
| 72 |
import threading
|
| 73 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 74 |
|
| 75 |
import ffmpeg
|
|
|
|
| 76 |
import gradio as gr
|
| 77 |
import numpy as np
|
| 78 |
+
from PIL import Image
|
| 79 |
+
from faster_whisper import WhisperModel
|
| 80 |
+
from faster_whisper.audio import decode_audio
|
| 81 |
from sentence_transformers import SentenceTransformer, CrossEncoder
|
| 82 |
from huggingface_hub import InferenceClient
|
| 83 |
import warnings
|
|
|
|
| 88 |
|
| 89 |
WHISPER_FAST = "base"
|
| 90 |
WHISPER_PRECISE = "large-v3-turbo"
|
| 91 |
+
SAMPLE_RATE = 16000
|
| 92 |
+
WHISPER_CPU_THREADS = 6 # tuned for M1 (perf cores); CTranslate2 is CPU-only
|
| 93 |
+
|
| 94 |
+
RETRANSCRIBE_MAX = 12 # per-prompt budget for NEW (uncached) sector transcriptions
|
| 95 |
+
PRECISE_LOGPROB_GATE = -0.55 # only turbo-refine sectors whose base avg_logprob is below this
|
| 96 |
+
# (clean audio scores high → turbo is correctly skipped)
|
| 97 |
|
| 98 |
CHUNK_SPLIT_THRESHOLD = 0.45
|
| 99 |
CHUNK_MIN_DURATION = 8.0
|
| 100 |
CHUNK_MAX_DURATION = 90.0
|
| 101 |
CHUNK_WINDOW = 5
|
| 102 |
|
| 103 |
+
BI_ENCODER_TOP_K = 100
|
| 104 |
+
CROSS_ENCODER_TOP_K = 10
|
|
|
|
|
|
|
| 105 |
RESULTS_TOP_N = 6
|
| 106 |
REGION_MIN_DURATION = 5.0
|
| 107 |
MERGE_GAP_SEC = 6.0
|
| 108 |
|
| 109 |
+
# routing thresholds
|
| 110 |
+
BI_SCORE_GLOBAL_FLOOR = 0.10 # below this → escalate to GLOBAL regardless
|
| 111 |
+
BI_SCORE_HYBRID_FLOOR = 0.28 # below this → HYBRID (pad with neighbours)
|
| 112 |
+
|
| 113 |
+
# global mode: how many evenly-sampled chunks to retranscribe
|
| 114 |
+
GLOBAL_MAX_CHUNKS = 12 # hard cap to keep latency sane
|
| 115 |
+
|
| 116 |
+
# ── CLIP / visual config ────────────────────────────────────────────────────────
|
| 117 |
+
CLIP_MODEL = "clip-ViT-B-32" # OpenAI CLIP ViT-B/32 (via sentence-transformers)
|
| 118 |
+
CLIP_MAX_FRAMES = 96 # cap frames encoded per video (perf)
|
| 119 |
+
CLIP_MIN_FPS = 0.1 # sample at least 1 frame / 10s
|
| 120 |
+
CLIP_MAX_FPS = 0.5 # sample at most 1 frame / 2s
|
| 121 |
+
CLIP_FRAME_SIZE = 224 # square fed to CLIP (matches ViT-B/32 input → no double resize)
|
| 122 |
+
VISUAL_WEIGHT = 0.30 # fusion weight of visual vs audio in final score
|
| 123 |
+
VISUAL_TAGS_TOP_K = 4 # max zero-shot tags surfaced per region
|
| 124 |
+
VISUAL_TAG_MIN_SIM = 0.20 # min CLIP cosine for a tag to count
|
| 125 |
+
|
| 126 |
+
# vocabulary for zero-shot "what is on screen" tagging
|
| 127 |
+
VISUAL_CONCEPTS = [
|
| 128 |
+
"a person talking to the camera", "a group of people", "a presentation with slides",
|
| 129 |
+
"a lecture or classroom", "an interview", "a news broadcast",
|
| 130 |
+
"a screen recording of software", "source code in a text editor",
|
| 131 |
+
"a webpage in a browser", "a chart or graph", "text on screen",
|
| 132 |
+
"a product demonstration", "hands demonstrating something",
|
| 133 |
+
"a cooking or food scene", "a kitchen", "an office", "a laboratory",
|
| 134 |
+
"a factory or machinery", "a construction site", "an outdoor nature landscape",
|
| 135 |
+
"a forest", "mountains", "a beach or ocean", "a city street",
|
| 136 |
+
"buildings and architecture", "a car or vehicle driving", "an airplane", "a train",
|
| 137 |
+
"a sports event", "people exercising", "a concert or live music",
|
| 138 |
+
"a video game gameplay", "an animation or cartoon", "a medical or hospital scene",
|
| 139 |
+
"a doctor or patient", "an animal", "a dog", "a cat", "a crowd of people",
|
| 140 |
+
"a map", "a whiteboard with diagrams", "a scientific diagram",
|
| 141 |
+
"a phone or mobile device", "a computer or laptop", "money or finance",
|
| 142 |
+
"a meeting or conference room", "a stage with a speaker",
|
| 143 |
+
"a dark scene at night", "a daytime scene",
|
| 144 |
+
]
|
| 145 |
+
|
| 146 |
LLM_MODEL = "Qwen/Qwen2.5-7B-Instruct"
|
| 147 |
LLM_MAX_TOKENS = 600
|
| 148 |
LLM_TIMEOUT = 60
|
| 149 |
|
| 150 |
+
CACHE_DIR = os.path.join(tempfile.gettempdir(), "nina_cache")
|
| 151 |
+
DISK_CACHE = True
|
| 152 |
+
|
| 153 |
DEBUG = False
|
| 154 |
|
| 155 |
|
| 156 |
+
# ── Device selection ─────────────────────────────────────────────────────────────
|
| 157 |
+
# CTranslate2 (Whisper) is CPU-only here; the torch models (bi/cross/CLIP) can
|
| 158 |
+
# use Metal (MPS) on Apple Silicon or CUDA on a GPU box.
|
| 159 |
+
|
| 160 |
+
def _best_torch_device() -> str:
|
| 161 |
+
try:
|
| 162 |
+
import torch
|
| 163 |
+
if torch.cuda.is_available():
|
| 164 |
+
return "cuda"
|
| 165 |
+
if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
|
| 166 |
+
return "mps"
|
| 167 |
+
except Exception:
|
| 168 |
+
pass
|
| 169 |
+
return "cpu"
|
| 170 |
+
|
| 171 |
+
TORCH_DEVICE = _best_torch_device()
|
| 172 |
+
|
| 173 |
+
|
| 174 |
# ── Model load ─────────────────────────────────────────────────────────────────
|
| 175 |
|
| 176 |
+
print(f"[NINA] Torch device: {TORCH_DEVICE} | Whisper: CPU int8 (x{WHISPER_CPU_THREADS})")
|
| 177 |
+
|
| 178 |
+
print("[NINA] Loading Whisper base (faster-whisper)...")
|
| 179 |
+
_whisper_fast = WhisperModel(WHISPER_FAST, device="cpu", compute_type="int8",
|
| 180 |
+
cpu_threads=WHISPER_CPU_THREADS)
|
| 181 |
|
| 182 |
+
# Bi-encoder stays on CPU: it is tiny (sub-100 ms) and MPS transfer overhead is a
|
| 183 |
+
# wash, and this keeps the MPS device free for CLIP during the parallel cold start.
|
| 184 |
print("[NINA] Loading bi-encoder...")
|
| 185 |
+
bi_encoder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2", device="cpu")
|
| 186 |
|
| 187 |
+
print(f"[NINA] Loading cross-encoder ({TORCH_DEVICE})...")
|
| 188 |
+
cross_encoder = CrossEncoder("cross-encoder/mmarco-mMiniLMv2-L12-H384-v1", device=TORCH_DEVICE)
|
| 189 |
|
| 190 |
|
| 191 |
HF_TOKEN = os.getenv("HF_TOKEN")
|
|
|
|
| 212 |
global _whisper_precise
|
| 213 |
with _whisper_precise_lock:
|
| 214 |
if _whisper_precise is None:
|
| 215 |
+
print(f"[NINA] Loading Whisper {WHISPER_PRECISE} (faster-whisper)...")
|
| 216 |
+
_whisper_precise = WhisperModel(WHISPER_PRECISE, device="cpu",
|
| 217 |
+
compute_type="int8",
|
| 218 |
+
cpu_threads=WHISPER_CPU_THREADS)
|
| 219 |
print(f"[NINA] Whisper {WHISPER_PRECISE} ready.")
|
| 220 |
return _whisper_precise
|
| 221 |
|
| 222 |
+
|
| 223 |
+
# ── CLIP (OpenAI ViT-B/32) — lazy, warmed in background ──────────────────────────
|
| 224 |
+
_clip_model = None
|
| 225 |
+
_clip_lock = threading.Lock()
|
| 226 |
+
_concept_embs = None
|
| 227 |
+
_concept_lock = threading.Lock()
|
| 228 |
+
|
| 229 |
+
def get_clip():
|
| 230 |
+
global _clip_model
|
| 231 |
+
with _clip_lock:
|
| 232 |
+
if _clip_model is None:
|
| 233 |
+
print(f"[NINA] Loading CLIP ({CLIP_MODEL}, {TORCH_DEVICE})...")
|
| 234 |
+
_clip_model = SentenceTransformer(CLIP_MODEL, device=TORCH_DEVICE)
|
| 235 |
+
print(f"[NINA] CLIP ready.")
|
| 236 |
+
return _clip_model
|
| 237 |
+
|
| 238 |
+
def _get_concept_embs() -> np.ndarray:
|
| 239 |
+
"""Encode (once) the zero-shot vocabulary, L2-normalised."""
|
| 240 |
+
global _concept_embs
|
| 241 |
+
with _concept_lock:
|
| 242 |
+
if _concept_embs is None:
|
| 243 |
+
embs = get_clip().encode(VISUAL_CONCEPTS, convert_to_numpy=True,
|
| 244 |
+
show_progress_bar=False)
|
| 245 |
+
_concept_embs = embs / (np.linalg.norm(embs, axis=1, keepdims=True) + 1e-9)
|
| 246 |
+
return _concept_embs
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
# Warm the lazy models in the background; CLIP loads on the (free) MPS device,
|
| 250 |
+
# turbo loads on CPU. Both finish before they are first needed in most cases.
|
| 251 |
threading.Thread(target=get_whisper_precise, daemon=True).start()
|
| 252 |
+
threading.Thread(target=get_clip, daemon=True).start()
|
| 253 |
+
print(f"[NINA] Ready. ( {WHISPER_PRECISE} + CLIP warming up...)")
|
| 254 |
+
|
| 255 |
+
# ── Transcript + visual cache (in-memory) ─────────────────────────────────────────
|
| 256 |
+
# Stores segments, chunks (with embeddings), visual frame embeddings, the decoded
|
| 257 |
+
# waveform, and a per-sector precise-transcription cache, keyed by video path.
|
| 258 |
+
# Second prompt on same video skips transcription, chunking and CLIP entirely.
|
| 259 |
+
_cache: dict = {} # { video_path: {"segments","chunks","visual","audio","precise"} }
|
| 260 |
|
| 261 |
|
| 262 |
# ── Audio ──────────────────────────────────────────────────────────────────────
|
| 263 |
|
| 264 |
+
def load_audio(media_path: str) -> np.ndarray:
|
| 265 |
+
"""Decode any container straight to a float32 mono 16k waveform (PyAV, no temp file)."""
|
| 266 |
+
return decode_audio(media_path, sampling_rate=SAMPLE_RATE)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
|
| 268 |
|
| 269 |
+
def media_duration(media_path: str) -> float:
|
| 270 |
+
try:
|
| 271 |
+
info = ffmpeg.probe(media_path)
|
| 272 |
+
return float(info["format"]["duration"])
|
| 273 |
+
except Exception:
|
| 274 |
+
return 0.0
|
| 275 |
+
|
| 276 |
+
|
| 277 |
+
# ── Frames (CLIP input) — streamed from one ffmpeg pipe, no temp files ────────────
|
| 278 |
+
|
| 279 |
+
def extract_frames(video_path: str, duration: float):
|
| 280 |
+
"""
|
| 281 |
+
Sample frames evenly across the video and stream them as raw RGB straight
|
| 282 |
+
into NumPy via a single ffmpeg pipe (no per-frame JPEG temp files).
|
| 283 |
+
fps is chosen so total frames ≈ CLIP_MAX_FRAMES (bounded by MIN/MAX fps).
|
| 284 |
+
Returns (frames_rgb, times) where frames_rgb is (N, S, S, 3) uint8.
|
| 285 |
+
"""
|
| 286 |
+
fps = CLIP_MAX_FPS
|
| 287 |
+
if duration and duration > 0:
|
| 288 |
+
fps = min(CLIP_MAX_FPS, max(CLIP_MIN_FPS, CLIP_MAX_FRAMES / duration))
|
| 289 |
+
|
| 290 |
+
S = CLIP_FRAME_SIZE
|
| 291 |
+
try:
|
| 292 |
+
out, _ = (
|
| 293 |
+
ffmpeg.input(video_path)
|
| 294 |
+
.filter("fps", fps=fps)
|
| 295 |
+
# cover-then-centre-crop to a square so portrait/landscape both work
|
| 296 |
+
.filter("scale", S, S, force_original_aspect_ratio="increase")
|
| 297 |
+
.filter("crop", S, S)
|
| 298 |
+
.output("pipe:", format="rawvideo", pix_fmt="rgb24")
|
| 299 |
+
.run(capture_stdout=True, capture_stderr=True)
|
| 300 |
+
)
|
| 301 |
+
except ffmpeg.Error:
|
| 302 |
+
return np.zeros((0, S, S, 3), dtype=np.uint8), np.zeros((0,))
|
| 303 |
+
|
| 304 |
+
if not out:
|
| 305 |
+
return np.zeros((0, S, S, 3), dtype=np.uint8), np.zeros((0,))
|
| 306 |
+
|
| 307 |
+
frames = np.frombuffer(out, np.uint8).reshape(-1, S, S, 3)[:CLIP_MAX_FRAMES]
|
| 308 |
+
times = np.array([(i + 0.5) / fps for i in range(len(frames))])
|
| 309 |
+
return frames, times
|
| 310 |
|
| 311 |
|
| 312 |
# ── Stage 2: transcription ─────────────────────────────────────────────────────
|
| 313 |
|
| 314 |
+
def transcribe_fast(audio: np.ndarray) -> list[dict]:
|
| 315 |
+
"""
|
| 316 |
+
Cheap full-video pass: base model, greedy decoding, Silero VAD.
|
| 317 |
+
Keeps per-segment avg_logprob so we can later refine ONLY weak sectors.
|
| 318 |
+
"""
|
| 319 |
+
segments_gen, _ = _whisper_fast.transcribe(
|
| 320 |
+
audio, beam_size=1, vad_filter=True, word_timestamps=False,
|
| 321 |
+
)
|
| 322 |
return [
|
| 323 |
{
|
| 324 |
+
"text": seg.text.strip(),
|
| 325 |
+
"start": round(seg.start, 2),
|
| 326 |
+
"end": round(seg.end, 2),
|
| 327 |
+
"logprob": round(float(getattr(seg, "avg_logprob", 0.0) or 0.0), 3),
|
| 328 |
}
|
| 329 |
+
for seg in segments_gen
|
| 330 |
+
if seg.text.strip()
|
| 331 |
]
|
| 332 |
|
| 333 |
|
| 334 |
+
# ── Stage 3a: visual analysis (CLIP) ─────────────────────────────────────────────
|
| 335 |
+
|
| 336 |
+
def analyze_visual(video_path: str, duration: float) -> dict:
|
| 337 |
+
"""
|
| 338 |
+
Sample → encode frames with CLIP. Cached per video.
|
| 339 |
+
Returns {"emb": (N, D) float32, "time": (N,) float}.
|
| 340 |
+
Empty arrays when the video has no usable visual stream (audio-only).
|
| 341 |
+
"""
|
| 342 |
+
frames, times = extract_frames(video_path, duration)
|
| 343 |
+
if len(frames) == 0:
|
| 344 |
+
print("[NINA] No video frames (audio-only?) — visual analysis skipped.")
|
| 345 |
+
return {"emb": np.zeros((0, 512), dtype=np.float32), "time": np.zeros((0,))}
|
| 346 |
+
|
| 347 |
+
images = [Image.fromarray(f) for f in frames]
|
| 348 |
+
embs = get_clip().encode(images, convert_to_numpy=True,
|
| 349 |
+
batch_size=32, show_progress_bar=False)
|
| 350 |
+
|
| 351 |
+
print(f"[NINA] CLIP encoded {len(images)} frames ({TORCH_DEVICE}).")
|
| 352 |
+
return {"emb": np.asarray(embs, dtype=np.float32), "time": np.asarray(times)}
|
| 353 |
+
|
| 354 |
+
|
| 355 |
# ── semantic chunking ─────────────────────────────────────────────────
|
| 356 |
|
| 357 |
def semantic_chunk(segments: list[dict]) -> list[dict]:
|
| 358 |
if not segments:
|
| 359 |
return []
|
| 360 |
texts = [s["text"] for s in segments]
|
| 361 |
+
embeddings = bi_encoder.encode(texts, convert_to_numpy=True, batch_size=64)
|
| 362 |
chunks, current, cur_emb = [], [segments[0]], [embeddings[0]]
|
| 363 |
|
| 364 |
for i in range(1, len(segments)):
|
|
|
|
| 369 |
long_enough = (current[-1]["end"] - current[0]["start"]) >= CHUNK_MIN_DURATION
|
| 370 |
|
| 371 |
if (sim < CHUNK_SPLIT_THRESHOLD or chunk_dur >= CHUNK_MAX_DURATION) and long_enough:
|
| 372 |
+
chunks.append(_build_chunk(current, cur_emb))
|
| 373 |
current, cur_emb = [segments[i]], [embeddings[i]]
|
| 374 |
else:
|
| 375 |
current.append(segments[i])
|
| 376 |
cur_emb.append(embeddings[i])
|
| 377 |
|
| 378 |
if current:
|
| 379 |
+
chunks.append(_build_chunk(current, cur_emb))
|
| 380 |
|
| 381 |
print(f"[NINA] {len(segments)} segments → {len(chunks)} chunks.")
|
| 382 |
return chunks
|
|
|
|
| 404 |
entities.add(match.group(1).lower().strip())
|
| 405 |
|
| 406 |
# 2. capitalised multi-word sequences (proper nouns)
|
| 407 |
+
for match in re.finditer(r'\b([A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-záéíóúàâêîôùäëïöüąćęłńóśźż]+(?:\s[A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-zÀ-žąćęłńóśźż]+)*)\b', text):
|
| 408 |
token = match.group(1).lower().strip()
|
| 409 |
if token not in _STOPWORDS and len(token) > 2:
|
| 410 |
entities.add(token)
|
| 411 |
|
| 412 |
# 3. single capitalised tokens (names, brands)
|
| 413 |
+
for token in re.findall(r'\b[A-ZÁÉÍÓÚÀÂÊÎÔÙÄËÏÖÜĄĆĘŁŃÓŚŹŻ][a-zÀ-žąćęłńóśźż]{2,}\b', text):
|
| 414 |
t = token.lower()
|
| 415 |
if t not in _STOPWORDS:
|
| 416 |
entities.add(t)
|
| 417 |
|
| 418 |
+
# 4. numbers with context
|
| 419 |
for match in re.finditer(r'\b([A-Za-z]+[-\s]?\d+[\w]*)\b', text):
|
| 420 |
entities.add(match.group(1).lower())
|
| 421 |
|
|
|
|
| 425 |
def entity_overlap_score(prompt_entities: set[str],
|
| 426 |
chunk_entities: set[str],
|
| 427 |
chunk_text: str = "") -> float:
|
| 428 |
+
|
| 429 |
if not prompt_entities:
|
| 430 |
return 0.0
|
| 431 |
|
|
|
|
| 447 |
return chunks
|
| 448 |
|
| 449 |
|
| 450 |
+
# ── Query router ───────────────────────────────────────────────────────────────
|
| 451 |
+
#
|
| 452 |
+
# Classifies prompt into one of three modes before retrieval runs.
|
| 453 |
+
#
|
| 454 |
+
# GLOBAL — user wants a summary / overview of the whole video.
|
| 455 |
+
# Retrieval is the wrong tool here because no single chunk is "about"
|
| 456 |
+
# summarization. → skip retrieval, sample chunks evenly, synthesize.
|
| 457 |
+
#
|
| 458 |
+
# HYBRID — user has a topic but signal is weak (bi-score 0.10–0.28).
|
| 459 |
+
# → run retrieval, but pad results with adjacent chunks.
|
| 460 |
+
#
|
| 461 |
+
# SPECIFIC — normal focused query, retrieval works well (bi-score > 0.28).
|
| 462 |
+
# → standard pipeline unchanged.
|
| 463 |
+
#
|
| 464 |
+
# Router runs in two passes:
|
| 465 |
+
# Pass 1: regex pattern match on prompt text (zero-cost, runs before encoding)
|
| 466 |
+
# Pass 2: bi-encoder score after retrieval (upgrades SPECIFIC → GLOBAL if
|
| 467 |
+
# retrieval actually fails)
|
| 468 |
+
|
| 469 |
+
# Global ONLY when the user asks for a whole-video overview. Critically, this no
|
| 470 |
+
# longer fires on "what does the video say ABOUT <topic>" — a bare "about" or
|
| 471 |
+
# "explain" used to force global and swallow specific questions.
|
| 472 |
+
_GLOBAL_PATTERNS = re.compile(
|
| 473 |
+
r'\b('
|
| 474 |
+
r'summari[sz]e|summary|overview|recap|tl;?dr|key\s+points?|main\s+points?|'
|
| 475 |
+
r'key\s+takeaways?|main\s+idea|gist|'
|
| 476 |
+
r'what(?:\'s| is| was)\s+(?:this|the|it)\s+(?:video|recording|clip|all\s+)?\s*about|'
|
| 477 |
+
r'what(?:\'s| is| was)\s+(?:this|the)\s+(?:video|recording|clip)|'
|
| 478 |
+
r'describe\s+(?:the|this)\s+(?:video|recording|clip)|'
|
| 479 |
+
r'streszcz\w*|podsumuj\w*|podsumowanie|'
|
| 480 |
+
r'o\s+czym\s+(?:jest|to|opowiada|mówi)|czego\s+dotyczy|'
|
| 481 |
+
r'gł[oó]wn\w+\s+(?:punkt\w*|my[sś]l\w*|tez\w*)|og[oó]ln\w+\s+zarys'
|
| 482 |
+
r')\b',
|
| 483 |
+
re.IGNORECASE
|
| 484 |
+
)
|
| 485 |
+
|
| 486 |
+
def classify_query(prompt: str) -> str:
|
| 487 |
+
"""
|
| 488 |
+
Pass 1 router — pattern-based, runs before any encoding.
|
| 489 |
+
Returns "global" or "specific".
|
| 490 |
+
Pass 2 (bi-score based upgrade) happens inside run() after retrieval.
|
| 491 |
+
"""
|
| 492 |
+
if _GLOBAL_PATTERNS.search(prompt):
|
| 493 |
+
return "global"
|
| 494 |
+
return "specific"
|
| 495 |
+
|
| 496 |
+
|
| 497 |
# ── Stage 4: bi-encoder retrieval (top-100) ───────────────────────────────────
|
| 498 |
|
| 499 |
def bi_encode_retrieve(chunks: list[dict], prompt: str) -> tuple[list[dict], float]:
|
| 500 |
+
# chunk embeddings were computed once at chunking time and cached on the
|
| 501 |
+
# chunk dict ("emb"); here we only encode the prompt.
|
| 502 |
prompt_emb = bi_encoder.encode([prompt], convert_to_numpy=True)[0]
|
| 503 |
+
scores = [_cosine(prompt_emb, c["emb"]) for c in chunks]
|
| 504 |
|
| 505 |
top_k = min(BI_ENCODER_TOP_K, len(chunks))
|
| 506 |
top_idx = np.argsort(scores)[::-1][:top_k]
|
|
|
|
| 516 |
)
|
| 517 |
|
| 518 |
|
| 519 |
+
# ── Stage 4b: visual relevance (CLIP) ─────────────────────────────────────────
|
| 520 |
|
| 521 |
+
def visual_scores(visual: dict, prompt: str) -> tuple[np.ndarray, np.ndarray]:
|
| 522 |
+
"""
|
| 523 |
+
Cosine of prompt↔each frame in CLIP space, plus a min-max normalised copy.
|
| 524 |
+
Normalisation makes CLIP sims (typically ~0.2–0.35) comparable to the
|
| 525 |
+
audio score's 0–1 range before fusion. Returns (raw, norm); empty when
|
| 526 |
+
there are no frames.
|
| 527 |
+
"""
|
| 528 |
+
embs = visual.get("emb")
|
| 529 |
+
if embs is None or len(embs) == 0:
|
| 530 |
+
if DEBUG:
|
| 531 |
+
print("[DEBUG] Visual: no frames (audio-only) — visual weight = 0.")
|
| 532 |
+
return np.array([]), np.array([])
|
| 533 |
+
q = get_clip().encode([prompt], convert_to_numpy=True, show_progress_bar=False)[0]
|
| 534 |
+
qn = q / (np.linalg.norm(q) + 1e-9)
|
| 535 |
+
en = embs / (np.linalg.norm(embs, axis=1, keepdims=True) + 1e-9)
|
| 536 |
+
sims = en @ qn
|
| 537 |
+
lo, hi = float(sims.min()), float(sims.max())
|
| 538 |
+
norm = (sims - lo) / (hi - lo + 1e-9)
|
| 539 |
+
if DEBUG:
|
| 540 |
+
times = visual.get("time")
|
| 541 |
+
order = np.argsort(sims)[::-1][:3]
|
| 542 |
+
tops = ", ".join(f"{fmt_time(times[i])}={sims[i]:.3f}" for i in order)
|
| 543 |
+
print(f"[DEBUG] Visual: {len(sims)} frames; raw sim "
|
| 544 |
+
f"min/mean/max={sims.min():.3f}/{sims.mean():.3f}/{sims.max():.3f}; "
|
| 545 |
+
f"top moments: {tops}")
|
| 546 |
+
return sims, norm
|
| 547 |
+
|
| 548 |
+
|
| 549 |
+
def span_visual_score(times: np.ndarray, norm: np.ndarray,
|
| 550 |
+
start: float, end: float) -> float:
|
| 551 |
+
"""Best (normalised) visual relevance among frames inside [start, end]."""
|
| 552 |
+
if len(times) == 0:
|
| 553 |
+
return 0.0
|
| 554 |
+
mask = (times >= start - 0.5) & (times <= end + 0.5)
|
| 555 |
+
if not mask.any():
|
| 556 |
+
idx = int(np.argmin(np.abs(times - (start + end) / 2)))
|
| 557 |
+
return float(norm[idx])
|
| 558 |
+
return float(norm[mask].max())
|
| 559 |
+
|
| 560 |
+
|
| 561 |
+
def region_visual_tags(visual: dict, start: float, end: float) -> list[str]:
|
| 562 |
+
"""Zero-shot CLIP tags describing what is on screen during [start, end]."""
|
| 563 |
+
embs = visual.get("emb")
|
| 564 |
+
times = visual.get("time")
|
| 565 |
+
if embs is None or len(embs) == 0:
|
| 566 |
+
return []
|
| 567 |
+
|
| 568 |
+
mask = (times >= start - 0.5) & (times <= end + 0.5)
|
| 569 |
+
if mask.any():
|
| 570 |
+
sel = embs[mask]
|
| 571 |
+
else:
|
| 572 |
+
idx = int(np.argmin(np.abs(times - (start + end) / 2)))
|
| 573 |
+
sel = embs[idx:idx + 1]
|
| 574 |
+
|
| 575 |
+
region_emb = sel.mean(axis=0)
|
| 576 |
+
region_emb = region_emb / (np.linalg.norm(region_emb) + 1e-9)
|
| 577 |
+
sims = _get_concept_embs() @ region_emb
|
| 578 |
+
|
| 579 |
+
order = np.argsort(sims)[::-1][:VISUAL_TAGS_TOP_K]
|
| 580 |
+
return [VISUAL_CONCEPTS[i] for i in order if sims[i] >= VISUAL_TAG_MIN_SIM]
|
| 581 |
+
|
| 582 |
+
|
| 583 |
+
def add_visual_tags(regions: list[dict], visual: dict) -> list[dict]:
|
| 584 |
+
for r in regions:
|
| 585 |
+
r["visual_tags"] = region_visual_tags(visual, r["start"], r["end"])
|
| 586 |
+
return regions
|
| 587 |
+
|
| 588 |
+
|
| 589 |
+
# ── Stage 5: cross-encoder rerank + audio/visual fusion (top-100 → top-10) ─────
|
| 590 |
+
|
| 591 |
+
def rerank_and_score(candidates: list[dict], prompt: str,
|
| 592 |
+
times: np.ndarray, norm: np.ndarray) -> list[dict]:
|
| 593 |
# cross-encoder
|
| 594 |
pairs = [[prompt, c["text"]] for c in candidates]
|
| 595 |
logits = cross_encoder.predict(pairs)
|
| 596 |
|
| 597 |
# entity overlap
|
| 598 |
prompt_entities = extract_entities(prompt)
|
| 599 |
+
prompt_words = {w.lower() for w in re.findall(r'[\wÀ-ž]{3,}', prompt)
|
| 600 |
if w.lower() not in _STOPWORDS}
|
| 601 |
all_prompt_tokens = prompt_entities | prompt_words
|
| 602 |
|
| 603 |
if DEBUG and all_prompt_tokens:
|
| 604 |
print(f"[DEBUG] Prompt tokens for entity match: {all_prompt_tokens}")
|
| 605 |
|
| 606 |
+
vw = VISUAL_WEIGHT if len(norm) else 0.0 # no frames → pure audio
|
| 607 |
+
|
| 608 |
scored = []
|
| 609 |
for candidate, logit in zip(candidates, logits):
|
| 610 |
chunk_entities = candidate.get("entities", set())
|
| 611 |
chunk_text = candidate.get("text", "")
|
| 612 |
overlap = entity_overlap_score(all_prompt_tokens, chunk_entities, chunk_text)
|
|
|
|
| 613 |
|
| 614 |
+
ce_conf = _sigmoid(float(logit))
|
| 615 |
+
audio_score = 0.6 * ce_conf + 0.4 * overlap
|
| 616 |
+
visual_score = span_visual_score(times, norm, candidate["start"], candidate["end"])
|
| 617 |
+
final = (1.0 - vw) * audio_score + vw * visual_score
|
| 618 |
|
| 619 |
scored.append({
|
| 620 |
**candidate,
|
| 621 |
"ce_logit": round(float(logit), 3),
|
| 622 |
"entity_overlap": round(overlap, 3),
|
| 623 |
+
"visual_score": round(visual_score, 3),
|
| 624 |
+
"confidence": round(min(1.0, final), 3),
|
| 625 |
})
|
| 626 |
|
| 627 |
+
# sort by fused multimodal score
|
| 628 |
+
ranked = sorted(scored, key=lambda x: x["confidence"], reverse=True)
|
| 629 |
|
| 630 |
if DEBUG:
|
| 631 |
+
print(f"[DEBUG] Top-3 after reranking (audio+visual):")
|
| 632 |
for r in ranked[:3]:
|
| 633 |
print(f" ce_logit={r['ce_logit']:.2f} "
|
| 634 |
f"bi={r['bi_score']:.3f} "
|
| 635 |
f"entity={r['entity_overlap']:.3f} "
|
| 636 |
+
f"visual={r['visual_score']:.3f} "
|
| 637 |
f"conf={r['confidence']:.3f} "
|
| 638 |
f"\"{r['text'][:60]}\"")
|
| 639 |
|
| 640 |
return ranked[:CROSS_ENCODER_TOP_K]
|
| 641 |
|
| 642 |
|
| 643 |
+
# ── Hybrid padding ─────────────────────────────────────────────────────────────
|
| 644 |
+
|
| 645 |
+
def pad_with_neighbours(results: list[dict], chunks: list[dict]) -> list[dict]:
|
| 646 |
+
"""
|
| 647 |
+
HYBRID mode: for each retrieved chunk, include its immediate neighbours
|
| 648 |
+
from the full chunk list. Helps when the relevant content slightly
|
| 649 |
+
overflows the chunk boundary. Deduplicates by start timestamp.
|
| 650 |
+
"""
|
| 651 |
+
result_starts = {r["start"] for r in results}
|
| 652 |
+
padded = list(results)
|
| 653 |
+
|
| 654 |
+
for r in results:
|
| 655 |
+
for i, c in enumerate(chunks):
|
| 656 |
+
if abs(c["start"] - r["start"]) < 0.5: # found matching chunk
|
| 657 |
+
for neighbour_i in [i - 1, i + 1]:
|
| 658 |
+
if 0 <= neighbour_i < len(chunks):
|
| 659 |
+
nb = chunks[neighbour_i]
|
| 660 |
+
if nb["start"] not in result_starts:
|
| 661 |
+
padded.append({
|
| 662 |
+
**nb,
|
| 663 |
+
"bi_score": 0.0,
|
| 664 |
+
"ce_logit": -9.0,
|
| 665 |
+
"entity_overlap": 0.0,
|
| 666 |
+
"visual_score": 0.0,
|
| 667 |
+
"confidence": nb.get("confidence", 0.1),
|
| 668 |
+
})
|
| 669 |
+
result_starts.add(nb["start"])
|
| 670 |
+
break
|
| 671 |
+
|
| 672 |
+
return sorted(padded, key=lambda x: x["start"])
|
| 673 |
+
|
| 674 |
+
|
| 675 |
+
# ── Global mode: even sampling ────────────────────────────────────────────────
|
| 676 |
+
|
| 677 |
+
def sample_chunks_for_global(chunks: list[dict], n_target: int) -> list[dict]:
|
| 678 |
+
"""
|
| 679 |
+
For global summarization, pick `n_target` chunks evenly distributed across
|
| 680 |
+
the video, so the summary covers the whole thing, not just the beginning.
|
| 681 |
+
"""
|
| 682 |
+
n_target = max(3, min(n_target, len(chunks)))
|
| 683 |
+
if n_target >= len(chunks):
|
| 684 |
+
return chunks
|
| 685 |
+
|
| 686 |
+
# evenly spaced indices
|
| 687 |
+
indices = [int(i * (len(chunks) - 1) / (n_target - 1)) for i in range(n_target)]
|
| 688 |
+
return [chunks[i] for i in indices]
|
| 689 |
+
|
| 690 |
+
|
| 691 |
# ── Stage 6: merge regions ─────────────────────────────────────────────────────
|
| 692 |
|
| 693 |
def merge_regions(results: list[dict]) -> list[dict]:
|
|
|
|
| 718 |
for f in frags) / total_dur
|
| 719 |
duration = region["end"] - region["start"]
|
| 720 |
w_conf *= min(1.0, duration / 20.0)
|
| 721 |
+
visual = max((f.get("visual_score", 0.0) for f in frags), default=0.0)
|
| 722 |
+
return {**region, "confidence": round(w_conf, 3), "visual_score": round(visual, 3)}
|
| 723 |
|
| 724 |
|
| 725 |
+
# ── Stage 7: precise re-transcription (confidence-gated) ──────────────────────
|
| 726 |
|
| 727 |
+
def retranscribe_region(audio: np.ndarray, region: dict) -> str:
|
| 728 |
+
"""Slice the in-memory waveform (no per-region ffmpeg) and run precise ASR."""
|
| 729 |
+
s = max(0, int(region["start"] * SAMPLE_RATE))
|
| 730 |
+
e = min(len(audio), int(region["end"] * SAMPLE_RATE))
|
| 731 |
+
clip = audio[s:e]
|
| 732 |
+
if len(clip) < SAMPLE_RATE * 0.2: # < 0.2s → nothing to gain
|
| 733 |
+
return region.get("text", "")
|
| 734 |
+
model = get_whisper_precise()
|
| 735 |
+
segments_gen, _ = model.transcribe(clip, beam_size=1)
|
| 736 |
+
return " ".join(seg.text.strip() for seg in segments_gen).strip()
|
| 737 |
+
|
| 738 |
+
|
| 739 |
+
def fill_precise(audio: np.ndarray, regions: list[dict], all_chunks: list[dict],
|
| 740 |
+
precise_cache: dict, budget: int) -> int:
|
| 741 |
+
"""
|
| 742 |
+
Ensure every region has `precise_text`. A sector is turbo-refined ONLY when:
|
| 743 |
+
• it is not already cached, AND
|
| 744 |
+
• the base pass for that sector was low-confidence (avg_logprob < gate), AND
|
| 745 |
+
• we are still within `budget` NEW transcriptions this call.
|
| 746 |
+
Clean/high-confidence sectors keep the (already accurate) base text for free,
|
| 747 |
+
so on clean audio the slow turbo model never runs. `precise_cache` persists
|
| 748 |
+
per video, so later prompts reuse earlier work. Returns remaining budget.
|
| 749 |
+
"""
|
| 750 |
+
reused = new = skipped = 0
|
| 751 |
+
for region in regions:
|
| 752 |
+
# the chunk sectors this region spans (chunks are non-overlapping)
|
| 753 |
+
comp = [c for c in all_chunks
|
| 754 |
+
if c["start"] < region["end"] - 1e-3 and c["end"] > region["start"] + 1e-3]
|
| 755 |
+
if not comp:
|
| 756 |
+
comp = [region]
|
| 757 |
+
|
| 758 |
+
for ch in comp:
|
| 759 |
+
key = round(ch["start"], 2)
|
| 760 |
+
if key in precise_cache:
|
| 761 |
+
reused += 1
|
| 762 |
+
continue
|
| 763 |
+
weak = ch.get("logprob", 0.0) < PRECISE_LOGPROB_GATE
|
| 764 |
+
if not weak:
|
| 765 |
+
skipped += 1 # base was confident → trust it
|
| 766 |
+
continue
|
| 767 |
+
if budget <= 0:
|
| 768 |
+
continue # over budget → fall back to fast text
|
| 769 |
+
print(f"[NINA] Re-transcribing weak sector "
|
| 770 |
+
f"[{fmt_time(ch['start'])} → {fmt_time(ch['end'])}] "
|
| 771 |
+
f"(logprob {ch.get('logprob', 0.0):.2f})")
|
| 772 |
+
precise_cache[key] = retranscribe_region(audio, ch)
|
| 773 |
if DEBUG:
|
| 774 |
+
snip = precise_cache[key]
|
| 775 |
+
print(f"[DEBUG] Precise sector [{fmt_time(ch['start'])}→{fmt_time(ch['end'])}]: "
|
| 776 |
+
f"{snip[:200]}{'…' if len(snip) > 200 else ''}")
|
| 777 |
+
new += 1
|
| 778 |
+
budget -= 1
|
| 779 |
+
|
| 780 |
+
region["precise_text"] = " ".join(
|
| 781 |
+
(precise_cache.get(round(ch["start"], 2)) or ch.get("text", ""))
|
| 782 |
+
for ch in comp
|
| 783 |
+
).strip() or region.get("text", "")
|
| 784 |
+
|
| 785 |
+
print(f"[NINA] Precise sectors — {new} refined, {reused} cached, {skipped} trusted (base).")
|
| 786 |
+
return budget
|
| 787 |
|
| 788 |
|
| 789 |
# ── Stage 8: synthesis ─────────────────────────────────────────────────────────
|
| 790 |
|
| 791 |
+
# ── Output sizing — inferred from the PROMPT (no slider) ─────────────────────────
|
| 792 |
+
|
| 793 |
+
_BRIEF_CUES = (
|
| 794 |
+
"brief", "briefly", "short", "shortly", "quick", "quickly", "concise",
|
| 795 |
+
"in a sentence", "one sentence", "one-liner", "tl;dr", "tldr",
|
| 796 |
+
"krótko", "w skrócie", "zwięźle", "jednym zdaniem", "w jednym zdaniu",
|
| 797 |
+
)
|
| 798 |
+
_DETAIL_CUES = (
|
| 799 |
+
"in detail", "detailed", "in-depth", "in depth", "thorough", "thoroughly",
|
| 800 |
+
"comprehensive", "exhaustive", "everything", "full summary", "elaborate",
|
| 801 |
+
"step by step", "step-by-step", "deep dive", "as much as possible",
|
| 802 |
+
"szczegółowo", "dokładnie", "wyczerpująco", "wszystko", "krok po kroku",
|
| 803 |
+
"dogłębnie", "obszernie",
|
| 804 |
+
)
|
| 805 |
+
_FACTUAL_START = re.compile(
|
| 806 |
+
r'^\s*(what|who|whom|whose|when|where|which|how many|how much|how long|'
|
| 807 |
+
r'is|are|was|were|does|do|did|can|could|will|would|'
|
| 808 |
+
r'czy|kiedy|kto|kogo|gdzie|ile|jaki|jaka|jakie|który|która|które)\b',
|
| 809 |
+
re.IGNORECASE,
|
| 810 |
+
)
|
| 811 |
+
|
| 812 |
+
def _summary_tier(max_words: int) -> str:
|
| 813 |
+
"""Human-readable label for the chosen output length."""
|
| 814 |
+
if max_words <= 60: return "brief / one-liner"
|
| 815 |
+
if max_words <= 110: return "short"
|
| 816 |
+
if max_words <= 250: return "standard"
|
| 817 |
+
if max_words <= 400: return "detailed"
|
| 818 |
+
return "exhaustive"
|
| 819 |
+
|
| 820 |
+
|
| 821 |
+
def _prompt_to_words(prompt: str) -> int:
|
| 822 |
+
"""
|
| 823 |
+
Infer the answer's word budget from the prompt itself (replaces the slider):
|
| 824 |
+
- explicit brevity cues → ~40 words
|
| 825 |
+
- explicit detail cues → ~450 words
|
| 826 |
+
- factual wh-/yes-no Q → ~80 words
|
| 827 |
+
- otherwise scale gently with prompt length (longer ask → longer answer)
|
| 828 |
+
Clamped to [30, 500].
|
| 829 |
+
"""
|
| 830 |
+
p = prompt.lower()
|
| 831 |
+
if any(c in p for c in _DETAIL_CUES):
|
| 832 |
+
base, reason = 450, "explicit detail cue"
|
| 833 |
+
elif any(c in p for c in _BRIEF_CUES):
|
| 834 |
+
base, reason = 40, "explicit brevity cue"
|
| 835 |
+
elif _FACTUAL_START.match(p):
|
| 836 |
+
base, reason = 80, "factual question"
|
| 837 |
+
else:
|
| 838 |
+
n_words = len(re.findall(r'\w+', p))
|
| 839 |
+
base, reason = 120 + min(180, n_words * 8), f"length-scaled ({n_words} prompt words)"
|
| 840 |
+
|
| 841 |
+
out = int(max(30, min(500, base)))
|
| 842 |
+
if DEBUG:
|
| 843 |
+
print(f"[DEBUG] Summary sizing: {reason} → {out} words [{_summary_tier(out)}]")
|
| 844 |
+
return out
|
| 845 |
+
|
| 846 |
+
|
| 847 |
+
def _words_to_global_chunks(max_words: int) -> int:
|
| 848 |
+
"""More requested words → sample more chunks for the global overview."""
|
| 849 |
+
return max(3, min(GLOBAL_MAX_CHUNKS, round(max_words / 40)))
|
| 850 |
+
|
| 851 |
+
|
| 852 |
def _fallback_summary(regions: list[dict]) -> str:
|
| 853 |
snippets = [
|
| 854 |
f"[{fmt_time(r['start'])}] {r.get('precise_text', r['text'])[:200]}"
|
|
|
|
| 857 |
return "(LLM unavailable — raw excerpts:)\n" + "\n".join(snippets)
|
| 858 |
|
| 859 |
|
| 860 |
+
def _build_synth_prompt(regions: list[dict], prompt: str,
|
| 861 |
+
max_words: int, mode: str) -> str:
|
| 862 |
+
# Each fragment carries BOTH modalities: what is SAID (audio) and what is
|
| 863 |
+
# SHOWN (CLIP visual cues). The LLM is asked to fuse them.
|
| 864 |
+
blocks = []
|
| 865 |
+
for r in regions:
|
| 866 |
+
spoken = (r.get("precise_text", r["text"]) or "").strip()
|
| 867 |
+
tags = r.get("visual_tags", [])
|
| 868 |
+
block = (f"[{fmt_time(r['start'])} → {fmt_time(r['end'])}]\n"
|
| 869 |
+
f" Spoken (audio): {spoken or '(no speech)'}")
|
| 870 |
+
if tags:
|
| 871 |
+
block += f"\n On screen (video): {', '.join(tags)}"
|
| 872 |
+
blocks.append(block)
|
| 873 |
+
fragments = "\n\n".join(blocks)
|
| 874 |
+
|
| 875 |
+
if mode == "global":
|
| 876 |
+
task_line = "Task: Give a complete overview of what this video is about."
|
| 877 |
+
else:
|
| 878 |
+
task_line = f'Task: Summarize what the video says/shows about "{prompt}".'
|
| 879 |
|
| 880 |
+
return (
|
| 881 |
+
f"You are analyzing a video. Each fragment below contains what is SPOKEN "
|
| 882 |
+
f"(audio transcript) and what is SHOWN (visual cues detected in the frames).\n"
|
| 883 |
+
f'The user is looking for: "{prompt}"\n\n'
|
| 884 |
+
f"Fragments:\n\n{fragments}\n\n"
|
| 885 |
+
f"{task_line}\n\n"
|
| 886 |
f"Output format:\n"
|
| 887 |
+
f"- Combine BOTH the spoken content and the on-screen visuals into one coherent answer\n"
|
| 888 |
+
f"- After each statement, append the timestamp it came from, taken from the "
|
| 889 |
+
f"bracketed start time of the relevant fragment above (use that exact value)\n"
|
| 890 |
+
f"- Maximum {max_words} words total\n"
|
| 891 |
+
f"- IMPORTANT: Write in the same language as the prompt given by the user\n"
|
| 892 |
+
f"- Only use information from the fragments (audio + visual) — do not invent details\n"
|
| 893 |
+
f"- If fragments contain no real information about the topic, say so clearly, "
|
| 894 |
+
f"without further explanation\n"
|
| 895 |
)
|
| 896 |
|
| 897 |
+
|
| 898 |
+
def llm_synthesize_stream(regions: list[dict], prompt: str,
|
| 899 |
+
max_words: int = 150, mode: str = "specific"):
|
| 900 |
+
"""
|
| 901 |
+
Generator: yields the answer incrementally as the LLM streams tokens.
|
| 902 |
+
On any error, yields a single fallback string.
|
| 903 |
+
"""
|
| 904 |
+
if not regions:
|
| 905 |
+
yield "(No relevant regions to summarize.)"
|
| 906 |
+
return
|
| 907 |
+
|
| 908 |
+
synth_prompt = _build_synth_prompt(regions, prompt, max_words, mode)
|
| 909 |
+
|
| 910 |
+
if DEBUG:
|
| 911 |
+
print(f"[DEBUG] Synthesis [{mode}] — {len(regions)} region(s), "
|
| 912 |
+
f"max_words={max_words} [{_summary_tier(max_words)}].")
|
| 913 |
+
|
| 914 |
try:
|
| 915 |
+
acc = ""
|
| 916 |
+
stream = llm_client.chat_completion(
|
| 917 |
model=LLM_MODEL,
|
| 918 |
messages=[{"role": "user", "content": synth_prompt}],
|
| 919 |
max_tokens=LLM_MAX_TOKENS,
|
| 920 |
temperature=0.3,
|
| 921 |
+
stream=True,
|
| 922 |
)
|
| 923 |
+
for ev in stream:
|
| 924 |
+
delta = ev.choices[0].delta.content or ""
|
| 925 |
+
if delta:
|
| 926 |
+
acc += delta
|
| 927 |
+
yield acc.strip()
|
| 928 |
+
if not acc.strip():
|
| 929 |
+
yield _fallback_summary(regions)
|
| 930 |
except Exception as e:
|
| 931 |
print(f"[NINA] Synthesis error: {e}")
|
| 932 |
+
yield _fallback_summary(regions)
|
| 933 |
+
|
| 934 |
+
|
| 935 |
+
def llm_synthesize(regions: list[dict], prompt: str,
|
| 936 |
+
max_words: int = 150, mode: str = "specific") -> str:
|
| 937 |
+
"""Non-streaming convenience wrapper (kept for API compatibility / testing)."""
|
| 938 |
+
out = ""
|
| 939 |
+
for out in llm_synthesize_stream(regions, prompt, max_words, mode):
|
| 940 |
+
pass
|
| 941 |
+
return out
|
| 942 |
|
| 943 |
|
| 944 |
# ── Helpers ────────────────────────────────────────────────────────────────────
|
| 945 |
|
| 946 |
+
def _build_chunk(segs: list[dict], embs: list[np.ndarray]) -> dict:
|
| 947 |
+
total = sum(max(s["end"] - s["start"], 0.1) for s in segs)
|
| 948 |
+
logp = sum(s.get("logprob", 0.0) * max(s["end"] - s["start"], 0.1)
|
| 949 |
+
for s in segs) / total # duration-weighted base confidence
|
| 950 |
+
return {"text": " ".join(s["text"] for s in segs),
|
| 951 |
+
"start": segs[0]["start"],
|
| 952 |
+
"end": segs[-1]["end"],
|
| 953 |
+
"logprob": round(logp, 3),
|
| 954 |
+
"emb": np.mean(np.stack(embs), axis=0)} # cached chunk embedding
|
| 955 |
|
| 956 |
def _cosine(a: np.ndarray, b: np.ndarray) -> float:
|
| 957 |
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))
|
|
|
|
| 968 |
return "█" * f + "░" * (10 - f)
|
| 969 |
|
| 970 |
|
| 971 |
+
# ── Persistent disk cache ────────────────────────────────────────────────────────
|
| 972 |
+
# Keyed by a content fingerprint (size + head + tail bytes), so the same video
|
| 973 |
+
# re-uploaded to a fresh temp path — or opened after a restart — is recognised
|
| 974 |
+
# and its transcript / embeddings / precise sectors are loaded instead of
|
| 975 |
+
# recomputed. Best-effort: any failure silently falls back to recomputation.
|
| 976 |
+
|
| 977 |
+
def _fingerprint(path: str) -> str:
|
| 978 |
+
h = hashlib.sha1()
|
| 979 |
+
st = os.stat(path)
|
| 980 |
+
h.update(str(st.st_size).encode())
|
| 981 |
+
with open(path, "rb") as f:
|
| 982 |
+
h.update(f.read(1 << 20)) # first 1 MB
|
| 983 |
+
if st.st_size > (1 << 20):
|
| 984 |
+
f.seek(-(1 << 20), os.SEEK_END)
|
| 985 |
+
h.update(f.read(1 << 20)) # last 1 MB
|
| 986 |
+
return h.hexdigest()[:16]
|
| 987 |
+
|
| 988 |
+
|
| 989 |
+
def _disk_load(fp: str) -> dict | None:
|
| 990 |
+
if not DISK_CACHE:
|
| 991 |
+
return None
|
| 992 |
+
d = os.path.join(CACHE_DIR, fp)
|
| 993 |
+
try:
|
| 994 |
+
with open(os.path.join(d, "meta.json"), encoding="utf-8") as f:
|
| 995 |
+
meta = json.load(f)
|
| 996 |
+
arr = np.load(os.path.join(d, "emb.npz"))
|
| 997 |
+
chunk_emb = arr["chunk_emb"]
|
| 998 |
+
chunks = []
|
| 999 |
+
for i, c in enumerate(meta["chunks"]):
|
| 1000 |
+
chunks.append({
|
| 1001 |
+
"text": c["text"], "start": c["start"], "end": c["end"],
|
| 1002 |
+
"logprob": c.get("logprob", 0.0),
|
| 1003 |
+
"entities": set(c.get("entities", [])),
|
| 1004 |
+
"emb": chunk_emb[i],
|
| 1005 |
+
})
|
| 1006 |
+
visual = {"emb": arr["visual_emb"], "time": arr["visual_time"]}
|
| 1007 |
+
print(f"[NINA] Disk cache hit ({fp}) — loaded {len(chunks)} chunks.")
|
| 1008 |
+
return {"segments": meta["segments"], "chunks": chunks,
|
| 1009 |
+
"visual": visual, "precise": {float(k): v for k, v in meta["precise"].items()}}
|
| 1010 |
+
except Exception:
|
| 1011 |
+
return None
|
| 1012 |
+
|
| 1013 |
+
|
| 1014 |
+
def _disk_save(fp: str, entry: dict) -> None:
|
| 1015 |
+
if not DISK_CACHE:
|
| 1016 |
+
return
|
| 1017 |
+
d = os.path.join(CACHE_DIR, fp)
|
| 1018 |
+
try:
|
| 1019 |
+
os.makedirs(d, exist_ok=True)
|
| 1020 |
+
chunks = entry["chunks"]
|
| 1021 |
+
meta = {
|
| 1022 |
+
"segments": entry["segments"],
|
| 1023 |
+
"precise": entry["precise"],
|
| 1024 |
+
"chunks": [{"text": c["text"], "start": c["start"], "end": c["end"],
|
| 1025 |
+
"logprob": c.get("logprob", 0.0),
|
| 1026 |
+
"entities": sorted(c.get("entities", []))}
|
| 1027 |
+
for c in chunks],
|
| 1028 |
+
}
|
| 1029 |
+
chunk_emb = (np.stack([c["emb"] for c in chunks]) if chunks
|
| 1030 |
+
else np.zeros((0, 384), dtype=np.float32))
|
| 1031 |
+
with open(os.path.join(d, "meta.json"), "w", encoding="utf-8") as f:
|
| 1032 |
+
json.dump(meta, f)
|
| 1033 |
+
np.savez(os.path.join(d, "emb.npz"),
|
| 1034 |
+
chunk_emb=chunk_emb,
|
| 1035 |
+
visual_emb=entry["visual"]["emb"],
|
| 1036 |
+
visual_time=entry["visual"]["time"])
|
| 1037 |
+
except Exception as e:
|
| 1038 |
+
if DEBUG:
|
| 1039 |
+
print(f"[DEBUG] disk cache save failed: {e}")
|
| 1040 |
+
|
| 1041 |
+
|
| 1042 |
+
def _disk_update_precise(fp: str, precise: dict) -> None:
|
| 1043 |
+
"""Persist newly-refined precise sectors without rewriting embeddings."""
|
| 1044 |
+
if not DISK_CACHE:
|
| 1045 |
+
return
|
| 1046 |
+
p = os.path.join(CACHE_DIR, fp, "meta.json")
|
| 1047 |
+
try:
|
| 1048 |
+
with open(p, encoding="utf-8") as f:
|
| 1049 |
+
meta = json.load(f)
|
| 1050 |
+
meta["precise"] = precise
|
| 1051 |
+
with open(p, "w", encoding="utf-8") as f:
|
| 1052 |
+
json.dump(meta, f)
|
| 1053 |
+
except Exception:
|
| 1054 |
+
pass
|
| 1055 |
+
|
| 1056 |
+
|
| 1057 |
# ── Formatter ──────────────────────────────────────────────────────────────────
|
| 1058 |
|
| 1059 |
+
def format_output(regions: list[dict], prompt: str, summary: str,
|
| 1060 |
+
mode: str = "specific") -> str:
|
| 1061 |
if not regions:
|
| 1062 |
return (
|
| 1063 |
f' Topic not found in this video.\n\n'
|
|
|
|
| 1066 |
f' Try a broader or differently worded prompt.'
|
| 1067 |
)
|
| 1068 |
|
| 1069 |
+
mode_tag = {"global": "🌐 global", "hybrid": "⚡ hybrid", "specific": "🎯 specific"}.get(mode, "")
|
| 1070 |
+
lines = [f' {len(regions)} region(s) found for: "{prompt}" [{mode_tag} · audio+video]\n']
|
| 1071 |
+
|
| 1072 |
if DEBUG:
|
| 1073 |
for i, r in enumerate(regions, 1):
|
| 1074 |
dur = round(r["end"] - r["start"], 1)
|
| 1075 |
text = r.get("precise_text", r["text"])
|
| 1076 |
ent = r.get("entity_overlap", 0)
|
| 1077 |
bi = r.get("bi_score", 0)
|
| 1078 |
+
vis = r.get("visual_score", 0)
|
| 1079 |
+
tags = ", ".join(r.get("visual_tags", []))
|
| 1080 |
lines += [
|
| 1081 |
f'\n [{i}] {fmt_time(r["start"])} → {fmt_time(r["end"])} ({dur}s)',
|
| 1082 |
f' {conf_bar(r["confidence"])} {r["confidence"]:.0%} '
|
| 1083 |
+
f'(embed {bi:.0%} · entity {ent:.0%} · visual {vis:.0%})',
|
| 1084 |
+
f' 👁 {tags}' if tags else ' 👁 (no visual tags)',
|
| 1085 |
f' "{text[:220]}{"…" if len(text) > 220 else ""}"',
|
| 1086 |
]
|
| 1087 |
+
|
| 1088 |
+
lines += [
|
| 1089 |
+
f' 🧠 Based on uploaded video (audio + video): — "{prompt}"',
|
|
|
|
| 1090 |
summary,
|
| 1091 |
f'\n Sources: ' + ', '.join(
|
| 1092 |
f'[{fmt_time(r["start"])} → {fmt_time(r["end"])}]' for r in regions
|
| 1093 |
),
|
| 1094 |
]
|
| 1095 |
+
return "\n".join(lines)
|
| 1096 |
|
| 1097 |
|
| 1098 |
+
# ── Cold-start ingestion (transcription + chunking + CLIP, in parallel) ───────────
|
| 1099 |
|
| 1100 |
+
def ingest_video(video_path: str) -> dict:
|
| 1101 |
+
"""
|
| 1102 |
+
First sight of a video: decode audio, then run the two heavy independent
|
| 1103 |
+
stages concurrently —
|
| 1104 |
+
• CPU thread : base transcription → semantic chunking → entities
|
| 1105 |
+
• MPS thread : frame extraction → CLIP encoding
|
| 1106 |
+
Returns the cache entry dict.
|
| 1107 |
+
"""
|
| 1108 |
+
audio = load_audio(video_path)
|
| 1109 |
+
duration = media_duration(video_path) or (len(audio) / SAMPLE_RATE)
|
| 1110 |
|
| 1111 |
+
result = {}
|
|
|
|
| 1112 |
|
| 1113 |
+
def _audio_branch():
|
| 1114 |
+
segments = transcribe_fast(audio)
|
| 1115 |
+
result["segments"] = segments
|
| 1116 |
+
if segments:
|
| 1117 |
+
result["chunks"] = add_entities_to_chunks(semantic_chunk(segments))
|
| 1118 |
+
else:
|
| 1119 |
+
result["chunks"] = []
|
| 1120 |
+
|
| 1121 |
+
def _visual_branch():
|
| 1122 |
+
result["visual"] = analyze_visual(video_path, duration)
|
| 1123 |
+
|
| 1124 |
+
with ThreadPoolExecutor(max_workers=2) as ex:
|
| 1125 |
+
fa = ex.submit(_audio_branch)
|
| 1126 |
+
fv = ex.submit(_visual_branch)
|
| 1127 |
+
fa.result()
|
| 1128 |
+
fv.result()
|
| 1129 |
+
|
| 1130 |
+
return {
|
| 1131 |
+
"segments": result["segments"],
|
| 1132 |
+
"chunks": result["chunks"],
|
| 1133 |
+
"visual": result["visual"],
|
| 1134 |
+
"audio": audio,
|
| 1135 |
+
"precise": {},
|
| 1136 |
+
}
|
| 1137 |
+
|
| 1138 |
+
|
| 1139 |
+
# ── Global path (shared by GLOBAL mode + every escalation/fallback) ───────────────
|
| 1140 |
+
# Global summaries run on the clean base transcript — no whole-video turbo pass.
|
| 1141 |
+
|
| 1142 |
+
def _run_global(chunks: list[dict], visual: dict,
|
| 1143 |
+
prompt: str, max_words: int) -> list[dict]:
|
| 1144 |
+
sampled = sample_chunks_for_global(chunks, _words_to_global_chunks(max_words))
|
| 1145 |
+
print(f"[NINA] Global mode — sampled {len(sampled)} chunks evenly (base transcript).")
|
| 1146 |
+
for c in sampled:
|
| 1147 |
+
c.setdefault("confidence", 0.5)
|
| 1148 |
+
c.setdefault("bi_score", 0.0)
|
| 1149 |
+
c.setdefault("entity_overlap", 0.0)
|
| 1150 |
+
c.setdefault("visual_score", 0.0)
|
| 1151 |
+
c["precise_text"] = c.get("text", "") # base text is enough for an overview
|
| 1152 |
+
return add_visual_tags(sampled, visual)
|
| 1153 |
|
|
|
|
|
|
|
|
|
|
| 1154 |
|
| 1155 |
+
# ── Main pipeline (generator: streams progress + answer) ─────────────────────────
|
|
|
|
|
|
|
| 1156 |
|
| 1157 |
+
def run(video_path: str, prompt: str):
|
| 1158 |
+
if video_path is None:
|
| 1159 |
+
yield "Please upload a video."
|
| 1160 |
+
return
|
| 1161 |
+
if not prompt or not prompt.strip():
|
| 1162 |
+
yield "Please enter a prompt."
|
| 1163 |
+
return
|
| 1164 |
+
|
| 1165 |
+
prompt = prompt.strip()
|
| 1166 |
+
max_words = _prompt_to_words(prompt) # output size inferred from the prompt
|
| 1167 |
+
print(f'\n[NINA] ── New request ── "{prompt}" (target ≈ {max_words} words)')
|
| 1168 |
+
|
| 1169 |
+
# Stage 1–3: transcription + chunking + CLIP visual (cached per video)
|
| 1170 |
+
fp = None
|
| 1171 |
+
if video_path in _cache:
|
| 1172 |
+
print(f"[NINA] Cache hit (memory) — skipping transcription + CLIP.")
|
| 1173 |
+
entry = _cache[video_path]
|
| 1174 |
+
else:
|
| 1175 |
+
try:
|
| 1176 |
+
fp = _fingerprint(video_path)
|
| 1177 |
+
except Exception:
|
| 1178 |
+
fp = None
|
| 1179 |
+
|
| 1180 |
+
disk = _disk_load(fp) if fp else None
|
| 1181 |
+
if disk is not None:
|
| 1182 |
+
disk["audio"] = load_audio(video_path)
|
| 1183 |
+
entry = disk
|
| 1184 |
+
_cache[video_path] = entry
|
| 1185 |
+
else:
|
| 1186 |
+
yield "🎬 Reading video — transcribing audio and scanning frames…"
|
| 1187 |
+
entry = ingest_video(video_path)
|
| 1188 |
+
if not entry["segments"]:
|
| 1189 |
+
yield "No speech detected."
|
| 1190 |
+
return
|
| 1191 |
+
_cache[video_path] = entry
|
| 1192 |
+
if fp:
|
| 1193 |
+
_disk_save(fp, entry)
|
| 1194 |
+
print(f"[NINA] Cached transcript + visuals for this video.")
|
| 1195 |
+
|
| 1196 |
+
segments = entry["segments"]
|
| 1197 |
+
chunks = entry["chunks"]
|
| 1198 |
+
visual = entry["visual"]
|
| 1199 |
+
precise_cache = entry["precise"]
|
| 1200 |
+
audio = entry["audio"]
|
| 1201 |
+
if fp is None and video_path not in (None,):
|
| 1202 |
+
try:
|
| 1203 |
+
fp = _fingerprint(video_path)
|
| 1204 |
+
except Exception:
|
| 1205 |
+
fp = None
|
| 1206 |
+
|
| 1207 |
+
if not chunks:
|
| 1208 |
+
yield "No speech detected."
|
| 1209 |
+
return
|
| 1210 |
+
|
| 1211 |
+
# ── Query routing (pass 1: pattern-based) ─────────────────────────────────
|
| 1212 |
+
mode = classify_query(prompt)
|
| 1213 |
+
print(f"[NINA] Query mode (pass 1): {mode}")
|
| 1214 |
+
|
| 1215 |
+
# ── GLOBAL mode ───────────────────────────────────────────────────────────
|
| 1216 |
+
if mode == "global":
|
| 1217 |
+
regions = _run_global(chunks, visual, prompt, max_words)
|
| 1218 |
+
yield from _synthesize_streaming(regions, prompt, max_words, "global")
|
| 1219 |
+
print(f"[NINA] Done — {len(regions)} region(s) [global].")
|
| 1220 |
+
return
|
| 1221 |
+
|
| 1222 |
+
# ── SPECIFIC / HYBRID: run retrieval (audio) + visual relevance ───────────
|
| 1223 |
candidates, best_bi = bi_encode_retrieve(chunks, prompt)
|
| 1224 |
+
_, v_norm = visual_scores(visual, prompt) # CLIP prompt↔frames
|
| 1225 |
|
| 1226 |
+
# Pass 2 routing: upgrade based on actual bi-score
|
| 1227 |
+
if best_bi < BI_SCORE_GLOBAL_FLOOR:
|
| 1228 |
+
print(f"[NINA] bi-score {best_bi:.3f} below floor → escalating to global.")
|
| 1229 |
+
regions = _run_global(chunks, visual, prompt, max_words)
|
| 1230 |
+
yield from _synthesize_streaming(regions, prompt, max_words, "global")
|
| 1231 |
+
print(f"[NINA] Done — {len(regions)} region(s) [escalated global].")
|
| 1232 |
+
return
|
| 1233 |
+
elif best_bi < BI_SCORE_HYBRID_FLOOR:
|
| 1234 |
+
mode = "hybrid"
|
| 1235 |
|
| 1236 |
+
print(f"[NINA] Query mode (pass 2): {mode} bi={best_bi:.3f}")
|
| 1237 |
+
|
| 1238 |
+
# Stage 5: cross-encoder + entity overlap + visual fusion → top-10
|
| 1239 |
+
results = rerank_and_score(candidates, prompt, visual["time"], v_norm)
|
| 1240 |
+
|
| 1241 |
+
# hybrid: pad with neighbouring chunks for context
|
| 1242 |
+
if mode == "hybrid":
|
| 1243 |
+
results = pad_with_neighbours(results, chunks)
|
| 1244 |
|
| 1245 |
# Stage 6: merge adjacent regions
|
| 1246 |
regions = merge_regions(results)
|
| 1247 |
if not regions:
|
| 1248 |
+
print(f"[NINA] No regions after merge → falling back to global.")
|
| 1249 |
+
regions = _run_global(chunks, visual, prompt, max_words)
|
| 1250 |
+
yield from _synthesize_streaming(regions, prompt, max_words, "global")
|
| 1251 |
+
return
|
| 1252 |
+
|
| 1253 |
+
# Stage 7: precise re-transcription (confidence-gated, cache-aware)
|
| 1254 |
+
before = dict(precise_cache)
|
| 1255 |
+
fill_precise(audio, regions, chunks, precise_cache, RETRANSCRIBE_MAX)
|
| 1256 |
+
if fp and precise_cache != before:
|
| 1257 |
+
_disk_update_precise(fp, precise_cache)
|
| 1258 |
+
|
| 1259 |
+
# Stage 7b: attach CLIP visual tags to each region
|
| 1260 |
+
regions = add_visual_tags(regions, visual)
|
| 1261 |
|
| 1262 |
+
# Stage 8: LLM synthesis (audio + visual) — streamed
|
| 1263 |
+
yield from _synthesize_streaming(regions, prompt, max_words, mode)
|
| 1264 |
+
print(f"[NINA] Done — {len(regions)} region(s) [{mode}].")
|
| 1265 |
|
|
|
|
|
|
|
|
|
|
| 1266 |
|
| 1267 |
+
def _synthesize_streaming(regions: list[dict], prompt: str, max_words: int, mode: str):
|
| 1268 |
+
"""Stream the formatted answer to the UI as the LLM produces it."""
|
| 1269 |
+
print(f"[NINA] Running synthesis [{mode}]...")
|
| 1270 |
+
yield format_output(regions, prompt, "✍️ …", mode)
|
| 1271 |
+
last = ""
|
| 1272 |
+
for partial in llm_synthesize_stream(regions, prompt, max_words, mode):
|
| 1273 |
+
last = partial
|
| 1274 |
+
yield format_output(regions, prompt, partial, mode)
|
| 1275 |
+
if not last:
|
| 1276 |
+
yield format_output(regions, prompt, _fallback_summary(regions), mode)
|
| 1277 |
|
| 1278 |
|
| 1279 |
# ── Gradio UI ──────────��───────────────────────────────────────────────────────
|
|
|
|
| 1284 |
gr.Video(label="Upload video "),
|
| 1285 |
gr.Textbox(
|
| 1286 |
label="What are you looking for?",
|
| 1287 |
+
placeholder='e.g. "What does the video say about...?" '
|
| 1288 |
+
'(add "briefly" or "in detail" to size the answer)',
|
| 1289 |
),
|
| 1290 |
],
|
| 1291 |
outputs=gr.Textbox(label="Results", lines=38),
|
| 1292 |
+
title="NINA v0.80 — Multimodal Video Search (audio + video via CLIP)",
|
| 1293 |
description=(
|
| 1294 |
+
"video -> a bit of magic (speech + vision) -> summary. "
|
| 1295 |
+
"Answer length adapts to your prompt."
|
| 1296 |
),
|
| 1297 |
flagging_mode="never",
|
| 1298 |
)
|
| 1299 |
|
| 1300 |
if __name__ == "__main__":
|
| 1301 |
+
demo.launch()
|
|
@@ -1,6 +1,20 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# NINA v0.80 — pinned for reproducible Hugging Face Space builds
|
| 2 |
+
# (Linux · Python 3.13 · CPU). torch is pulled from the CPU wheel index so the
|
| 3 |
+
# Space builds smaller/faster (no CUDA on a CPU Space; the code auto-falls back
|
| 4 |
+
# from MPS/CUDA to CPU). faster-whisper pulls `av` + `tokenizers` transitively.
|
| 5 |
+
--extra-index-url https://download.pytorch.org/whl/cpu
|
| 6 |
+
|
| 7 |
+
# ── Core app ──
|
| 8 |
+
gradio==6.14.0
|
| 9 |
+
ffmpeg-python==0.2.0
|
| 10 |
+
numpy==2.4.6
|
| 11 |
+
Pillow==12.2.0
|
| 12 |
+
|
| 13 |
+
# ── ML / inference ──
|
| 14 |
+
torch==2.12.0
|
| 15 |
+
sentence-transformers==5.5.1
|
| 16 |
+
transformers==5.8.1
|
| 17 |
+
huggingface-hub==1.15.0
|
| 18 |
+
faster-whisper==1.2.1
|
| 19 |
+
ctranslate2==4.7.2
|
| 20 |
+
onnxruntime==1.26.0
|