license: other
license_name: nvidia-license
license_link: LICENSE
base_model: nvidia/LocateAnything-3B
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- rknn
- rkllm
- rk3588
- vlm
- visual-grounding
- object-detection
LocateAnything-3B for RK3588 (RKLLM + RKNN)
Built with Qwen. Converted from nvidia/LocateAnything-3B so it runs on the NPU of Rockchip RK3588 boards (tested on a Radxa Rock 5C, 8 GB).
Non-commercial research and evaluation use only. This is a derivative of NVIDIA's model and is distributed under the NVIDIA License. It also contains Qwen2.5-3B-Instruct weights, which are subject to the Qwen Research License. See NOTICE. This is an unofficial community conversion; it is not endorsed by NVIDIA, Alibaba Cloud or Rockchip.
This is not the same as the other LocateAnything RKNN repository on the Hub (RKNNAI/RK182X-VLM-LocateAnything-3B),
which targets the RK1828/RK182X coprocessor. The files here target RK3588 (.rkllm for runtime 1.3.0 and .rknn).
Files
| File | Size | Description |
|---|---|---|
LocateAnything-3B_LLM_RK3588_w8a8.rkllm |
3.76 GB | Qwen2.5-3B language model (vocabulary extended to 152,681 with box/coordinate tokens), RKLLM 1.3.0 toolkit, w8a8, 3 NPU cores, max context 4096 |
LocateAnything-3B_vision_RK3588.rknn |
0.92 GB | MoonViT-SO-400M + patch merger + mlp1 projector, RKNN-Toolkit2 2.5.0, fp16 (no quantization), fixed 448x448 input, outputs 256 x 2048 |
tokenizer.json |
11.6 MB | Tokenizer (exported from the original) |
rk3588/locate3b.py, rk3588/rkllm_binding.py |
Inference engine and ctypes binding for librkllmrt.so |
|
tools/ |
Conversion and evaluation scripts | |
eval/ |
Evaluation subset (ids and boxes only) and raw predictions |
SHA-256: see SHA256SUMS.
Usage (on the board)
Requirements: RK3588 with the RKNPU driver, librkllmrt.so 1.3.0 in /usr/lib, rknn-toolkit-lite2 2.5.0,
numpy, Pillow, tokenizers. About 4.7 GB of RAM is needed (the 3.8 GB LLM plus the 0.9 GB vision model), so stop
other NPU services first.
python rk3588/locate3b.py image.jpg "Locate all the instances that matches the following description: person." -v
Prompts are the same as in the original model card, for example Locate all the instances that matches the following description: <category>. and Point to: <phrase>.. Boxes are <box><x1><y1><x2><y2></box> with coordinates
normalized to 0-1000. Defaults follow the original batch_infer.py: temperature=0.7, top_p=0.9, repetition_penalty=1.1. Do not use greedy decoding: in my tests it almost never terminates and keeps emitting
boxes (F1 0.38-0.46 on the same subset on a GPU).
How it differs from the original (read this)
- Causal PBD window. LocateAnything predicts a box with Parallel Box Decoding: a 6-token window
(
last token + 5 <text_mask>) that uses bidirectional attention inside the window. RKLLM cannot take a custom attention mask, so here the window is evaluated with a causal mask. The engine sends the accepted tokens plus 5<text_mask>tokens each step and relies on the runtime reusing the KV cache of the common prefix (no explicit rollback). Token positions match the original layout, but the model was trained with bidirectional attention inside the window, so this is an approximation. - Fixed 448x448 input. Images are squashed to 448x448 (256 image tokens). Coordinates are normalized, so boxes map back to the original size, but small objects suffer. The original supports dynamic resolution up to ~25k patches.
- Quantization. The language model is w8a8; the vision encoder runs in fp16 on the NPU.
- The hybrid MTP/AR logic (
handle_pattern,decode_bbox_avg,decode_ref) is a numpy port of NVIDIA's code.
Evaluation
Protocol: 100 images from COCO val2017 (seed 0; one query per image for the most frequent category; 1-8 instances;
all boxes >= 32x32 px; crowd regions excluded). Prompt: Locate all the instances that matches the following description: <category>. Box F1 at IoU 0.5 (greedy matching in output order, micro-averaged), plus mean IoU of matched
boxes. Predictions are truncated at the first box that duplicates an earlier one (IoU > 0.85) and capped at 30 boxes,
applied identically to all rows. Sampling: temperature 0.7, top_p 0.9, repetition_penalty 1.1, one seed.
Because the sampling libraries differ (PyTorch vs numpy), the board and GPU rows do not see the same random draws.
| Setup | Precision | Recall | F1@0.5 | F1@0.75 | mIoU (matched) | s / image |
|---|---|---|---|---|---|---|
| Original model, GPU bf16, bidirectional window, 448x448 | 0.842 | 0.716 | 0.774 | 0.660 | 0.874 | 0.14 |
| Original model, GPU bf16, causal window, 448x448 | 0.760 | 0.716 | 0.738 | 0.624 | 0.871 | 0.14 |
| Original model, GPU bf16, bidirectional, native resolution | 0.747 | 0.712 | 0.729 | 0.645 | 0.885 | 0.19 |
| This conversion, Rock 5C (RK3588) | 0.710 | 0.529 | 0.606 | 0.408 | 0.795 | 5.8 |
On this small subset the conversion reaches about 78% of the original's F1 (0.606 vs 0.774). Roughly 4 F1 points come
from the causal window; the rest comes from quantization, the fp16 NPU vision encoder and other numerical differences,
which I did not separate. n = 100 and a single seed, so differences of a few points are within noise. On the GPU runs all outputs (99-100 of 100) ended on their own with <|im_end|>. On the board, 99 of 100 outputs end with
<|im_end|>, but the engine also appends that token when it stops on a duplicate box (IoU > 0.85), so the board number does
not prove that the model terminated by itself every time; I did not log the stop reason. Rock 5C time per image (mean 5.78 s, median 4.90 s) breaks down into image load + RKNN vision encoding about 2.67 s (nearly constant), RKLLM about 2.92 s (including a first prefill of about 2 s, then roughly 140-160 ms per step, about 7 steps on average) and CPU sampling/decoding about 0.19 s.
Raw predictions are in eval/.
Reproducing the conversion
See tools/: extract_llm.py (split the Qwen2 language model), convert_rkllm.py (RKLLM 1.3.0 toolkit, w8a8),
vit_fixed.py (fixed-size MoonViT + projector with 2D RoPE and position embeddings baked in; verified against the
original, cosine 1.0, max abs diff 5e-4), build_rknn.py (RKNN-Toolkit2 2.5.0). Conversion was done on x86 Linux.
License and attribution
- LocateAnything-3B, Copyright (c) 2026 NVIDIA CORPORATION, NVIDIA License: non-commercial research or evaluation only; redistribution must keep the license and notices. Original: https://huggingface.co/nvidia/LocateAnything-3B
- Qwen2.5-3B-Instruct language model: Qwen Research License. Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved. Built with Qwen. Files in this repository were modified (converted and quantized).
- MoonViT-SO-400M vision encoder, Moonshot AI, MIT License (as declared by the upstream model card).
The weights are provided "as is" without warranty of any kind.