vuda-vlm / README.md
NeoSapian's picture
Update README.md
f13583c verified
|
Raw History Blame Contribute Delete
1.79 kB
metadata
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - anomaly-detection
  - vuda
  - surveillance
  - lora-merged

VUDA

VUDA is a compact (~500M parameter) VUDA model for anomaly detection. It looks at an image or video frame and identifies and describes unusual or abnormal events in the scene.

Intended use

  • Detecting and describing anomalies in camera or video-frame input
  • Surveillance and monitoring analytics
  • Triage of footage for human review

Quick start

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "NeoSapian/vuda-vlm"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval()

image = Image.open("frame.jpg")
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Is there any anomaly in this scene? Describe it."},
    ],
}]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to(device, torch.bfloat16)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128)

print(processor.batch_decode(out, skip_special_tokens=True)[0])

Limitations

  • Outputs can be wrong or miss events; do not use as the sole basis for safety-critical or enforcement decisions.
  • Performance depends on image quality, camera angle, lighting, and how close the scene is to the training data.
  • Always keep a human in the loop for review.

Training

Fine-tuned with LoRA for anomaly detection, then merged into the base weights.