README.md

metadata

title: Video Intelligence
emoji: 👁
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.31.0
python_version: '3.12'
app_file: app.py
pinned: false
tags:
  - Object Detection
  - Instance Segmentation
  - Zero-Shot Object Detection
  - Object Tracking
  - Image Captioning
  - Video Captioning
short_description: Detect, segment, track and caption images and videos

Video Intelligence

Upload an image or video, pick a task and a model, and get boxes or masks plus captions.

Tasks

• Detection (COCO): RF-DETR Nano to Large. Fixed 80-class COCO head.
• Segmentation (COCO): RF-DETR Seg Nano to XXLarge. Same classes, with masks.
• Zero-shot detection: OWLv2 Base, Grounding DINO Tiny, MM Grounding DINO Tiny, LLMDet Tiny and Base, OmDet-Turbo Tiny. Labels come from a preset vocabulary in labels/ (default, coco, objects365, openimages, nudenet, nudenet-prompts), an uploaded JSON that overrides the preset, or a typed comma-separated list that overrides both. Their scores run lower than RF-DETR's, so the threshold resets to each model's default when you pick it.

Grounding DINO, MM Grounding DINO and LLMDet ground every salient object to some phrase in their text, so past a few dozen classes, or when the vocabulary needs more than one pass through the 256-token text encoder, they return spurious classes. OWLv2 and OmDet-Turbo score each prompt on its own and stay clean on Objects365 and Open Images.

Labels JSON

Each key is a class name and each value lists the text prompts that stand for it. A hit on any prompt counts for the class, and overlapping hits from one class collapse to the best box. Any labels/*.json file becomes a preset.

{
  "person": ["a person", "a cartoon character", "an animated character"],
  "tie": ["a necktie", "a tie"]
}

Upload your own file under Custom labels, or add a file to labels/.

Video runs ByteTrack with temporal smoothing by default; turn it off under Tracking for raw per-frame boxes. Images are captioned by Florence-2. Videos are captioned as a whole by the models you tick under Captions: Qwen3.5-4B, Qwen3.5-9B, or Marlin-2B, which returns a scene description and timestamped events.

Models folder

Every detector and Florence-2 is a Transformers save_pretrained folder under models/, loaded from disk. Spaces cap repo storage at 1 GB, so the stock folders are not committed: the app fills any missing folder from its Hub repo at startup. A committed folder takes precedence.

models/
  rf-detr-nano/ ... rf-detr-large/
  rf-detr-seg-nano/ ... rf-detr-seg-xxlarge/
  owlv2-base-patch16-ensemble/
  grounding-dino-tiny/  mm-grounding-dino-tiny/
  llmdet-tiny/  llmdet-base/
  omdet-turbo-swin-tiny/
  florence-2-large/

To add a model, commit its folder (or point a TASKS entry at a Hub repo) and add one entry to TASKS in detection.py. The model class comes from architectures in the folder's config.json. A fine-tuned RF-DETR (for example on CrowdHuman head boxes) goes in the same way as its own folder and task entry. The Qwen3.5 and Marlin video captioners load from the Hub cache.

Licenses: RF-DETR, OWLv2, Grounding DINO, MM Grounding DINO, LLMDet, OmDet-Turbo, Qwen3.5 and Marlin-2B are Apache 2.0. Florence-2 is MIT.

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference