Spaces:
Paused
Paused
Commit History
-
NINA: VLM-centric synthesis β the model now actually sees the videoβ
Malina +1: Claude Opus 4.8Pivot the answer engine from CLIP-tags + text-LLM to a vision-language model (Qwen3-VL-8B via HF Inference) that consumes sampled keyframes + the transcript: - frames.py: pick representative timestamps (region midpoints + even context) and extract downscaled base64 keyframes. - synthesis.vlm_synthesize_stream: sends frames + speech transcript + speaker labels to the VLM; reads on-screen text natively, describes who/what is shown, fuses vision with speech. Falls back to the text LLM if the VLM is unavailable. One call per request β no extra inference cost beyond the image tokens. - pipeline: route synthesis through the VLM with keyframes; disable the OCR rung (the VLM reads text far better); add a per-video ingest lock to stop the duplicate concurrent ingestion seen in the logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>33ec6e1