rskill-omdet_turbo-any-indoor-fp16

OpenRAL rSkill — OmDet-Turbo (Swin-tiny) packaged as an in-process, Apache-2.0 open-vocabulary object detector run over a fixed curated indoor vocabulary (~266 household / kitchen / office / manipulation classes). It is an unprompted background perception producer: it streams ObjectsMetadata to /openral/perception/objects every frame, giving the world model far more object classes than the 80 COCO categories — without any reasoner prompting. No actuators.

This package wraps hf://omlab/omdet-turbo-swin-tiny-hf with a rskill.yaml manifest that adds the fixed-vocabulary detector contract, capability checking, license surfacing, and latency budgets. It does not copy model weights.

What this skill does

Detects objects from a fixed curated indoor vocabulary in every RGB camera frame and publishes 2D detections (ObjectsMetadata) on the perception bus. It emits no action chunks, drives no actuators, and has no proprioception contract. The class list is fixed (not query-driven), so it behaves like a large closed-vocabulary detector: the reasoner does not retarget it.

Field	Value
Actions	detect
Objects	open-vocabulary indoor objects — kitchenware, tableware, appliances, furniture, tools, containers (~266 classes)
Scenes	tabletop, kitchen, indoor, household, office, bathroom
Embodiment	embodiment-agnostic (any RGB camera ≥ 640×480)

How it works

OmDet-Turbo is a real-time, transformer-based open-vocabulary detector (AutoModelForZeroShotObjectDetection). Unlike locateanything-3b-nf4 (a heavy VLM pinned to transformers==4.57.1, run out-of-process in a sidecar), it is a first-class transformers architecture that loads under the OpenRAL runtime's own transformers>=5, so it runs in process: no sidecar venv, no ZMQ.

The OpenRAL backend (OmDetTurboDetector) loads the processor + model on first detect(), moves the model to CUDA when available (CPU fallback otherwise), and runs the manifest's fixed labels vocabulary against each frame via processor.post_process_grounded_object_detection. It is selected as DetectorTier.ZEROSHOT_HF by build_manifest_detector for manifests whose detector.engine is zeroshot_hf, and consumes the same system-memory BGR camera-tee branch as the CPU ONNX and VLM-sidecar tiers (added in the 2026-06-12 detector-runner amendment).

Observation → action contract

Direction	Key	Shape	Notes
in	any RGB camera	(H, W, 3) BGR uint8	system-memory frame from the camera tee; min 640×480. vla_feature_key is intentionally omitted
out	ObjectsMetadata	list of ObjectDetection2D	(label, confidence, bbox_xyxy) per detection on /openral/perception/objects; no action chunk

Upstream model and training

A thin wrapper around the upstream Apache-2.0 OmDet-Turbo checkpoint; weights live upstream and are not copied here.

Field	Value
Source repo	omlab/omdet-turbo-swin-tiny-hf
Base model	OmDet-Turbo, Swin-tiny backbone
Paper	arxiv:2403.06892 — Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head
License	apache-2.0 (commercial use permitted)
Parameters	~115 M
Training data	upstream: Objects365 / GoldG and grounding data per the OmDet-Turbo release

Supported robots

Embodiment-agnostic — the only requirement is an RGB camera stream. rskill.yaml declares the explicit embodiment_tags: ["any"] wildcard.

Robot	Embodiment tag	Status	Notes
any with an RGB camera	any	⚡ experimental	camera-only

Sensors required

Mirrors rskill.yaml::sensors_required.

Key	Modality	Min resolution	Format
any RGB camera	RGB	640 × 480	uint8 BGR frame

Manifest summary

Field	Value
name	OpenRAL/rskill-omdet_turbo-any-indoor-fp16
version	0.1.0
license	apache-2.0
role / kind	s1 / detector
embodiment_tags	["any"] (camera-only)
runtime / quantization.dtype	pytorch / fp16
detector.engine	zeroshot_hf (in-process Transformers zero-shot)
weights_uri	hf://omlab/omdet-turbo-swin-tiny-hf
latency_budget.per_chunk_ms	200 ms
is_commercial_use_allowed (derived from license)	true (Apache-2.0 weights)

Full schema: openral_core.schemas.RSkillManifest.

Quick start

just sync --group omdet   # torch + transformers for the in-process backend

from openral_core.schemas import RSkillManifest, DetectorEngine

manifest = RSkillManifest.from_yaml("rskills/omdet-turbo-indoor/rskill.yaml")
assert manifest.kind == "detector"
assert manifest.detector.engine is DetectorEngine.ZEROSHOT_HF
print(len(manifest.detector.labels), "fixed indoor classes")

Run it in a deploy sim (publishes ObjectsMetadata every frame, no ONNX file and no prompting). It is already openral deploy sim's default continuous detector when the omdet deps are installed; the flag makes the choice explicit:

openral deploy sim --config scenes/deploy/libero_pnp.yaml \
  --object-detector-manifest rskills/omdet-turbo-indoor/rskill.yaml

Reproduction

Packaging-only wrapper — no trained numbers to reproduce. Validate the wiring (manifest + in-process dispatch) without a GPU:

just bootstrap && just sync --group omdet
uv run pytest tests/unit/test_omdet_turbo_detector.py

The GPU-gated end-to-end test (test_e2e_detects_indoor_objects_on_coco_sample) loads the real Apache-2.0 weights and grounds indoor classes on the coco_sample.jpg fixture; it skips on GPU-less hosts (the legitimate CI skip path, CLAUDE.md §1.12).

Evaluation

No benchmarks shipped — packaging-only wrapper (no eval/ results).

License

This rSkill package (rskill.yaml, README.md) is apache-2.0. The wrapped weights at hf://omlab/omdet-turbo-swin-tiny-hf are also released under apache-2.0, so the detector is fully commercial-safe (CLAUDE.md §1.9) — unlike the NVIDIA non-commercial locateanything-3b-nf4 open-vocab detector.

See also

• rskills/locateanything-3b-nf4/ — query-driven open-vocab detector (NVIDIA non-commercial; VLM_SIDECAR tier).
• rskills/rtdetr-coco-r18/ — fixed 80-class COCO RT-DETR detector (ONNX tier).
• CLAUDE.md §3 — rSkill packaging contract.