---
title: Video Intelligence
emoji: 👁
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.31.0
python_version: "3.12"
app_file: app.py
pinned: false
tags:
- Object Detection
- Instance Segmentation
- Zero-Shot Object Detection
- Object Tracking
- Image Captioning
- Video Captioning
short_description: Detect, segment, track and caption images and videos
---

# Video Intelligence

Upload an image or video, pick a task and a model, and get boxes or masks plus
captions.

## Tasks

- **Detection (COCO)**: RF-DETR Nano to Large. Fixed 80-class COCO head.
- **Segmentation (COCO)**: RF-DETR Seg Nano to XXLarge. Same classes, with masks.
- **Zero-shot detection**: OWLv2 Base, Grounding DINO Tiny, MM Grounding DINO
  Tiny, LLMDet Tiny and Base, OmDet-Turbo Tiny. Labels come from a preset
  vocabulary in `labels/` (`default`, `coco`, `objects365`, `openimages`, `nudenet`, `nudenet-prompts`), an
  uploaded JSON that overrides the preset, or a typed comma-separated list that
  overrides both. Their scores run lower than RF-DETR's, so the threshold
  resets to each model's default when you pick it.

Grounding DINO, MM Grounding DINO and LLMDet ground every salient object to
some phrase in their text, so past a few dozen classes, or when the vocabulary
needs more than one pass through the 256-token text encoder, they return
spurious classes. OWLv2 and OmDet-Turbo score each prompt on its own and stay
clean on Objects365 and Open Images.

## Labels JSON

Each key is a class name and each value lists the text prompts that stand for
it. A hit on any prompt counts for the class, and overlapping hits from one
class collapse to the best box. Any `labels/*.json` file becomes a preset.

```json
{
  "person": ["a person", "a cartoon character", "an animated character"],
  "tie": ["a necktie", "a tie"]
}
```

Upload your own file under Custom labels, or add a file to `labels/`.

Video runs ByteTrack with temporal smoothing by default; turn it off under
*Tracking* for raw per-frame boxes. Images are captioned by Florence-2. Videos
are captioned as a whole by the models you tick under *Captions*: Qwen3.5-4B,
Qwen3.5-9B, or Marlin-2B, which returns a scene description and timestamped
events.

## Models folder

Every detector and Florence-2 is a Transformers `save_pretrained` folder under
`models/`, loaded from disk. Spaces cap repo storage at 1 GB, so the stock
folders are not committed: the app fills any missing folder from its Hub repo
at startup. A committed folder takes precedence.

```
models/
  rf-detr-nano/ ... rf-detr-large/
  rf-detr-seg-nano/ ... rf-detr-seg-xxlarge/
  owlv2-base-patch16-ensemble/
  grounding-dino-tiny/  mm-grounding-dino-tiny/
  llmdet-tiny/  llmdet-base/
  omdet-turbo-swin-tiny/
  florence-2-large/
```

To add a model, commit its folder (or point a `TASKS` entry at a Hub repo) and
add one entry to `TASKS` in `detection.py`. The model class comes from
`architectures` in the folder's `config.json`. A fine-tuned RF-DETR (for
example on CrowdHuman head boxes) goes in the same way as its own folder and
task entry. The Qwen3.5 and Marlin video captioners load from the Hub cache.

Licenses: RF-DETR, OWLv2, Grounding DINO, MM Grounding DINO, LLMDet,
OmDet-Turbo, Qwen3.5 and Marlin-2B are Apache 2.0. Florence-2 is MIT.

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference