--- title: LectureLens emoji: 🎞️ colorFrom: green colorTo: blue sdk: gradio sdk_version: 6.30.0 python_version: '3.12.12' app_file: app.py pinned: false short_description: Ask videos with hierarchical retrieval and frame citations --- # LectureLens Upload a short video, import a YouTube clip, or try the bundled gradient descent demo. Ask your own question or choose an example, then inspect the cited evidence, original frames, and replay clips. ## Answer engines and quotas - **Auto (Groq + local fallback):** use Groq when available; fall back to a local Qwen2.5-1.5B-Instruct model if Groq is limited, unavailable, or unconfigured. Audio recognition falls back to local Whisper Tiny. Optional visual descriptions can be skipped while speech and OCR remain available. - **Local models (no Groq calls):** generate answers with local Qwen and transcribe with local Whisper. This text model uses speech and OCR evidence; it does not inspect frame images. It can be less accurate than the larger Groq models. - Repeated successful questions reuse a cache isolated to the current video, answer engine, and browser session. Known Groq cooldowns honor `Retry-After` without repeatedly sending rejected requests. Local inference removes Groq's API quota dependency, but free ZeroGPU still has its own per-user GPU allowance and queue. This Space is not unlimited. Running the app on hardware you control avoids external inference-provider quotas; CPU inference is slower. ## YouTube clips Open **Import a YouTube clip**, paste a public video link, choose the start time in seconds and a length of up to 120 seconds, then load it. Long videos are supported through a selected clip; the app does not analyze the entire long video. Citation links and displayed timestamps point into the original YouTube video; frame replay uses the downloaded clip's local timeline. No YouTube captions are assumed or fabricated: speech recognition runs on the actual clip audio. The importer uses the official [youtube-dl](https://github.com/ytdl-org/youtube-dl) Python package, pinned to source commit `956b8c585591b401a543e409accb163eeaaa1193`. It resolves the video/audio streams, then FFmpeg downloads and accurately cuts the requested interval. It does not rely on `yt-dlp`-specific clip options or fetch an entire long video before trimming it. Only supported public, unrestricted videos are imported. YouTube can reject data-center downloads, require sign-in, or change its delivery system. The app reports blocked/timed-out imports and keeps file upload available. It does not use cookies, credential forwarding to YouTube, proxy rotation, or bypass access restrictions. Import only content you have permission to process. This prototype splits contiguous video frame ranges in half until each leaf represents one original frame. Evidence comes from subtitles or Groq Whisper speech recognition, sampled Tesseract OCR, and sampled Groq visual descriptions. Upper groups get purpose labels. On ZeroGPU, a small open-source MiniLM encoder embeds evidence and questions on GPU; semantic scores combine with lexical scores to route the hierarchy. The CPU/local path uses lexical routing. Broad summaries use coverage across the timeline. ## Limits and grounding - Uploads: up to 2 minutes, 100 MB, and 9,000 decoded frames. - Each session: up to 20 questions; server inference budget defaults to 240 calls/hour across sessions. - Frames use exact decoded ordinals and original presentation timestamps, normalized to the first video frame. Speech timestamps are approximate, and uploaded subtitles must align with that timeline. - Citation IDs and source intervals are validated in code. This cannot prove factual entailment. Sampled visual indexing can miss fast events, and a limited beam can miss relevant branches. - Purpose labels are generated for upper groups; deeper groups use extractive labels. Every original frame is represented, although every frame is not sent to a vision model. ## Models and privacy The Space reads `GROQ_API_KEY` from its server-side secrets. Groq defaults are `openai/gpt-oss-20b` for text, `qwen/qwen3.8-27b` for vision, and `whisper-large-v3-turbo` for transcription. Override `GROQ_TEXT_MODEL` and `GROQ_VISION_MODEL` through Space variables if account availability changes. Local generation uses [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct); local speech recognition uses Faster Whisper Tiny on CPU. These model weights download at first use/startup, with no inference API key needed for local mode. Uploads and imported clips stay in temporary session directories on the Space. Auto mode sends audio, selected frames, and evidence text to Groq when available. Local mode keeps inference on the hosting machine, using ZeroGPU for generation on this Space. Sessions expire after inactivity and Gradio clears expired temporary output files. Do not upload confidential material. No upload or runtime key is committed to either repository. This is a Gradio SDK Space. Groq generates answers, transcribes speech, and describes selected frames. On ZeroGPU, [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) runs locally for semantic retrieval; its weights download at startup. This uses the free ZeroGPU request quota. If semantic inference is temporarily unavailable, lexical routing remains available. Deployment first tries CPU Basic; if Hugging Face requires a paid plan, it tries the documented free personal ZeroGPU allocation. That tier requires a verified account older than 30 days and an available free slot. No paid hardware or subscription is requested. GitHub Actions publishes only the allowlisted app runtime from `aniketqw/lecturelens-research`. ## Run locally Install FFmpeg and Tesseract, install `gradio/requirements.txt`, then run `python gradio/app.py` from the research repository. In the Space, run `python app.py`. Select **Local models (no Groq calls)** to work without a Groq key. On your own machine, local generation defaults to CPU; set `LECTURELENS_LOCAL_DEVICE=cuda` to use your NVIDIA GPU directly. On ZeroGPU, the default local generator uses its managed GPU allowance; setting `LECTURELENS_LOCAL_DEVICE=cpu` selects CPU generation. Initial model downloads require internet access. Model size can be changed with `LECTURELENS_LOCAL_MODEL`; supported instruction models must work with the Qwen-style chat template. Use smaller models for low-memory CPUs, accepting reduced answer quality. For fixture-only development tests without downloading Torch/model weights, install `gradio/requirements-test.txt` and run `python -m unittest discover -s tests -v`.