| --- |
| license: other |
| license_name: nvidia-license |
| license_link: LICENSE |
| base_model: nvidia/LocateAnything-3B |
| base_model_relation: quantized |
| pipeline_tag: image-text-to-text |
| tags: |
| - rknn |
| - rkllm |
| - rk3588 |
| - vlm |
| - visual-grounding |
| - object-detection |
| --- |
| |
| # LocateAnything-3B for RK3588 (RKLLM + RKNN) |
|
|
| **Built with Qwen.** Converted from [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) so it |
| runs on the NPU of Rockchip **RK3588** boards (tested on a Radxa Rock 5C, 8 GB). |
|
|
| > **Non-commercial research and evaluation use only.** This is a derivative of NVIDIA's model and is distributed under |
| > the [NVIDIA License](LICENSE). It also contains Qwen2.5-3B-Instruct weights, which are subject to the |
| > [Qwen Research License](LICENSE-Qwen). See [NOTICE](NOTICE). This is an unofficial community conversion; it is not |
| > endorsed by NVIDIA, Alibaba Cloud or Rockchip. |
|
|
| This is **not** the same as the other LocateAnything RKNN repository on the Hub (`RKNNAI/RK182X-VLM-LocateAnything-3B`), |
| which targets the RK1828/RK182X coprocessor. The files here target RK3588 (`.rkllm` for runtime 1.3.0 and `.rknn`). |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| | --- | --- | --- | |
| | `LocateAnything-3B_LLM_RK3588_w8a8.rkllm` | 3.76 GB | Qwen2.5-3B language model (vocabulary extended to 152,681 with box/coordinate tokens), RKLLM 1.3.0 toolkit, w8a8, 3 NPU cores, max context 4096 | |
| | `LocateAnything-3B_vision_RK3588.rknn` | 0.92 GB | MoonViT-SO-400M + patch merger + `mlp1` projector, RKNN-Toolkit2 2.5.0, fp16 (no quantization), fixed **448x448** input, outputs 256 x 2048 | |
| | `tokenizer.json` | 11.6 MB | Tokenizer (exported from the original) | |
| | `rk3588/locate3b.py`, `rk3588/rkllm_binding.py` | | Inference engine and ctypes binding for `librkllmrt.so` | |
| | `tools/` | | Conversion and evaluation scripts | |
| | `eval/` | | Evaluation subset (ids and boxes only) and raw predictions | |
|
|
| SHA-256: see `SHA256SUMS`. |
|
|
| ## Usage (on the board) |
|
|
| Requirements: RK3588 with the RKNPU driver, `librkllmrt.so` **1.3.0** in `/usr/lib`, `rknn-toolkit-lite2` 2.5.0, |
| `numpy`, `Pillow`, `tokenizers`. About 4.7 GB of RAM is needed (the 3.8 GB LLM plus the 0.9 GB vision model), so stop |
| other NPU services first. |
|
|
| ```bash |
| python rk3588/locate3b.py image.jpg "Locate all the instances that matches the following description: person." -v |
| ``` |
|
|
| Prompts are the same as in the original model card, for example `Locate all the instances that matches the following |
| description: <category>.` and `Point to: <phrase>.`. Boxes are `<box><x1><y1><x2><y2></box>` with coordinates |
| normalized to 0-1000. Defaults follow the original `batch_infer.py`: `temperature=0.7, top_p=0.9, |
| repetition_penalty=1.1`. **Do not use greedy decoding**: in my tests it almost never terminates and keeps emitting |
| boxes (F1 0.38-0.46 on the same subset on a GPU). |
|
|
| ## How it differs from the original (read this) |
|
|
| 1. **Causal PBD window.** LocateAnything predicts a box with Parallel Box Decoding: a 6-token window |
| (`last token + 5 <text_mask>`) that uses *bidirectional* attention inside the window. RKLLM cannot take a custom |
| attention mask, so here the window is evaluated with a *causal* mask. The engine sends the accepted tokens plus |
| 5 `<text_mask>` tokens each step and relies on the runtime reusing the KV cache of the common prefix (no explicit |
| rollback). Token positions match the original layout, but the model was trained with bidirectional |
| attention inside the window, so this is an approximation. |
| 2. **Fixed 448x448 input.** Images are squashed to 448x448 (256 image tokens). Coordinates are normalized, so boxes |
| map back to the original size, but small objects suffer. The original supports dynamic resolution up to ~25k patches. |
| 3. **Quantization.** The language model is w8a8; the vision encoder runs in fp16 on the NPU. |
| 4. The hybrid MTP/AR logic (`handle_pattern`, `decode_bbox_avg`, `decode_ref`) is a numpy port of NVIDIA's code. |
|
|
| ## Evaluation |
|
|
| Protocol: 100 images from COCO val2017 (seed 0; one query per image for the most frequent category; 1-8 instances; |
| all boxes >= 32x32 px; crowd regions excluded). Prompt: `Locate all the instances that matches the following |
| description: <category>.` Box F1 at IoU 0.5 (greedy matching in output order, micro-averaged), plus mean IoU of matched |
| boxes. Predictions are truncated at the first box that duplicates an earlier one (IoU > 0.85) and capped at 30 boxes, |
| applied identically to all rows. Sampling: `temperature 0.7, top_p 0.9, repetition_penalty 1.1`, one seed. |
| Because the sampling libraries differ (PyTorch vs numpy), the board and GPU rows do not see the same random draws. |
|
|
| | Setup | Precision | Recall | F1@0.5 | F1@0.75 | mIoU (matched) | s / image | |
| | --- | --- | --- | --- | --- | --- | --- | |
| | Original model, GPU bf16, bidirectional window, 448x448 | 0.842 | 0.716 | 0.774 | 0.660 | 0.874 | 0.14 | |
| | Original model, GPU bf16, **causal** window, 448x448 | 0.760 | 0.716 | 0.738 | 0.624 | 0.871 | 0.14 | |
| | Original model, GPU bf16, bidirectional, native resolution | 0.747 | 0.712 | 0.729 | 0.645 | 0.885 | 0.19 | |
| | **This conversion, Rock 5C (RK3588)** | 0.710 | 0.529 | **0.606** | 0.408 | 0.795 | 5.8 | |
|
|
| On this small subset the conversion reaches about 78% of the original's F1 (0.606 vs 0.774). Roughly 4 F1 points come |
| from the causal window; the rest comes from quantization, the fp16 NPU vision encoder and other numerical differences, |
| which I did not separate. n = 100 and a single seed, so differences of a few points are within noise. On the GPU runs all outputs (99-100 of 100) ended on their own with `<|im_end|>`. On the board, 99 of 100 outputs end with |
| `<|im_end|>`, but the engine also appends that token when it stops on a duplicate box (IoU > 0.85), so the board number does |
| not prove that the model terminated by itself every time; I did not log the stop reason. Rock 5C time per image (mean 5.78 s, median 4.90 s) breaks down into image load + RKNN vision encoding about 2.67 s (nearly constant), RKLLM about 2.92 s (including a first prefill of about 2 s, then roughly 140-160 ms per step, about 7 steps on average) and CPU sampling/decoding about 0.19 s. |
| Raw predictions are in `eval/`. |
|
|
| ## Reproducing the conversion |
|
|
| See `tools/`: `extract_llm.py` (split the Qwen2 language model), `convert_rkllm.py` (RKLLM 1.3.0 toolkit, w8a8), |
| `vit_fixed.py` (fixed-size MoonViT + projector with 2D RoPE and position embeddings baked in; verified against the |
| original, cosine 1.0, max abs diff 5e-4), `build_rknn.py` (RKNN-Toolkit2 2.5.0). Conversion was done on x86 Linux. |
|
|
| ## License and attribution |
|
|
| - **LocateAnything-3B**, Copyright (c) 2026 NVIDIA CORPORATION, [NVIDIA License](LICENSE): non-commercial research or |
| evaluation only; redistribution must keep the license and notices. Original: https://huggingface.co/nvidia/LocateAnything-3B |
| - **Qwen2.5-3B-Instruct** language model: [Qwen Research License](LICENSE-Qwen). *Qwen is licensed under the Qwen |
| RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.* **Built with Qwen.** |
| Files in this repository were modified (converted and quantized). |
| - **MoonViT-SO-400M** vision encoder, Moonshot AI, MIT License (as declared by the upstream model card). |
|
|
| The weights are provided "as is" without warranty of any kind. |
|
|
