How to use from the
Use from the
Transformers library
# Gated model: Login with a HF token with gated access permission
hf auth login
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="NeoSapian/vuda-vlm")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("NeoSapian/vuda-vlm")
model = AutoModelForMultimodalLM.from_pretrained("NeoSapian/vuda-vlm", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
Quick Links

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

VUDA

VUDA is a compact (~500M parameter) VUDA model for anomaly detection. It looks at an image or video frame and identifies and describes unusual or abnormal events in the scene.

Intended use

  • Detecting and describing anomalies in camera or video-frame input
  • Surveillance and monitoring analytics
  • Triage of footage for human review

Quick start

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "NeoSapian/vuda-vlm"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval()

image = Image.open("frame.jpg")
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Is there any anomaly in this scene? Describe it."},
    ],
}]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to(device, torch.bfloat16)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128)

print(processor.batch_decode(out, skip_special_tokens=True)[0])

Limitations

  • Outputs can be wrong or miss events; do not use as the sole basis for safety-critical or enforcement decisions.
  • Performance depends on image quality, camera angle, lighting, and how close the scene is to the training data.
  • Always keep a human in the loop for review.

Training

Fine-tuned with LoRA for anomaly detection, then merged into the base weights.

Downloads last month
9
Safetensors
Model size
0.5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support