README.md

---
license: apple-amlr
pipeline_tag: image-to-text
library_name: llama.cpp
tags:
- fastvlm
- spacemit
- k1
- k3
- gguf
- onnxruntime
---

# FastVLM-0.5B for SpacemiT K1/K3

This repository contains a SpacemiT edge-deployment version of
[apple/FastVLM-0.5B](https://huggingface.co/apple/FastVLM-0.5B). The FastVLM
vision encoder is exported as an ONNX model and runs through the SpacemiT
Execution Provider, while the Qwen2-based 0.5B text decoder is stored as a
Q4_1 GGUF model and runs through the SpacemiT `llama.cpp` backend.

## Model origin and acknowledgement

FastVLM was introduced by Apple in
[FastVLM: Efficient Vision Encoding for Vision Language Models](https://arxiv.org/abs/2412.13303),
published at CVPR 2025. It is a vision-language model for understanding and
describing images. The key FastViTHD encoder is designed to produce fewer
visual tokens and reduce vision encoding latency, while retaining useful image
understanding quality. The FastVLM-0.5B variant is the compact model in the
released family.

This repository is a deployment conversion, not a new base model. We thank the
Apple Machine Learning Research team for releasing the FastVLM model, paper,
and source code:

- [FastVLM paper](https://arxiv.org/abs/2412.13303)
- [Official FastVLM repository](https://github.com/apple/ml-fastvlm)
- [Official FastVLM-0.5B model](https://huggingface.co/apple/FastVLM-0.5B)

The current SpacemiT package exposes offline image understanding through the
OpenAI-compatible `llama-server` chat completions API. The bundled
`humanspeech.jpg` sample has been validated on both K1 and K3.

## Files

```text
FastVLM-0.5B-SpacemiT/
├── fastvlm-text-0.5B-Q4_1.gguf
├── fastvlm_vision.f16.onnx
├── configs/
│   ├── K1/config.json
│   └── K3/config.json
├── humanspeech.jpg
└── README.md
```

The included `humanspeech.jpg` shows a woman speaking at a podium and is used
by the board smoke example below. `fastvlm_vision.f16.onnx` is the vision encoder;
`fastvlm-text-0.5B-Q4_1.gguf` is the text decoder.

## Supported platforms

| Platform | Accelerated cores | SMT config | `llama-server` threads |
|---|---:|---|---:|
| SpacemiT K1 / X60 | `0,1,2,3` | `configs/K1` | `-t 4` |
| SpacemiT K3 / A100 | `8,9,10,11,12,13,14,15` | `configs/K3` | `-t 8` |

The platform-specific `config.json` controls the ONNX vision encoder's
SpaceMIT EP thread count and affinity. The `-t` argument controls the GGUF
text decoder. Do not use the K3 config on K1: K1 has four accelerated cores at
IDs `0-3`, whereas K3 has eight accelerated cores at IDs `8-15`.

The SpacemiT `llama.cpp` runtime automatically detects the accelerated CPU
cores. Normal runs do not require manually setting
`SPACEMIT_PERFER_CORE_ARCH`, `SPACEMIT_PERFER_CORE_ID`, or
`SPACEMIT_MEM_BACKEND`.

## Prerequisites

Two runtime components are required:

1. A SpacemiT ONNX Runtime package containing `libonnxruntime` and the
   SpacemiT Execution Provider:
   [spacemit-com/onnxruntime releases](https://github.com/spacemit-com/onnxruntime/releases)
2. An SMT-enabled SpacemiT `llama.cpp` build containing `llama-server`:
   [spacemit-com/llama.cpp](https://github.com/spacemit-com/llama.cpp)

The validation below used SpacemiT ORT `2.0.6` and a RISC-V
`llama-server` built from the SpacemiT fork.

### Option A: use prebuilt packages

Download and unpack the current RISC-V glibc releases:

```bash
wget https://github.com/spacemit-com/onnxruntime/releases/download/2.0.6/spacemit-ort.riscv64.2.0.6.tar.gz
tar -xf spacemit-ort.riscv64.2.0.6.tar.gz

wget https://github.com/spacemit-com/llama.cpp/releases/download/v0.1.7/spacemit-llama.cpp.riscv64.0.1.7.tar.gz
tar -xf spacemit-llama.cpp.riscv64.0.1.7.tar.gz
```

Check the release pages for newer mutually compatible packages when deploying
this model in another software image.

### Option B: build llama.cpp from source

Cross-compilation also requires a SpacemiT RISC-V toolchain:

```bash
git clone --recursive https://github.com/spacemit-com/llama.cpp.git
cd llama.cpp

export RISCV_ROOT_PATH=/path/to/spacemit-riscv-toolchain
export SPACEMIT_ORT_DIR=/path/to/spacemit-ort.riscv64.2.0.6

bash build_spacemit.sh glibc
```

The installed runtime is generated under `build/installed/`. An equivalent
manual CMake build must enable at least `GGML_CPU_RISCV64_SPACEMIT=ON` and
`LLAMA_SERVER_SMT_MTMD=ON`, and pass `SPACEMIT_ORT_DIR` to CMake.

## Run on a K1 or K3 board

Copy this model folder, the unpacked ORT package, and the prebuilt or locally
built `llama.cpp` installation to the board. Set the following paths to match
their actual locations:

```bash
export MODEL_DIR=/path/to/FastVLM-0.5B-SpacemiT
export ORT_DIR=/path/to/spacemit-ort.riscv64.2.0.6
export LLAMA_DIR=/path/to/spacemit-llama.cpp.riscv64.0.1.7

export LD_LIBRARY_PATH="${LLAMA_DIR}/lib:${ORT_DIR}/lib:${LD_LIBRARY_PATH:-}"
```

### K1

K1 uses four accelerated cores, `0-3`:

```bash
"${LLAMA_DIR}/bin/llama-server" \
  -m "${MODEL_DIR}/fastvlm-text-0.5B-Q4_1.gguf" \
  --media-backend smt \
  --smt-config-dir "${MODEL_DIR}/configs/K1" \
  -t 4 \
  --host 0.0.0.0 \
  --port 8080 \
  --warmup
```

The K1 vision config applies:

```json
"ep_config": {
  "SPACEMIT_EP_INTRA_THREAD_NUM": "4",
  "SPACEMIT_EP_INTER_THREAD_NUM": "1",
  "SPACEMIT_EP_INTRA_THREAD_AFFINITY": "0;1;2;3"
}
```

### K3

K3 uses eight accelerated cores, `8-15`:

```bash
"${LLAMA_DIR}/bin/llama-server" \
  -m "${MODEL_DIR}/fastvlm-text-0.5B-Q4_1.gguf" \
  --media-backend smt \
  --smt-config-dir "${MODEL_DIR}/configs/K3" \
  -t 8 \
  --host 0.0.0.0 \
  --port 8080 \
  --warmup
```

The K3 vision config applies:

```json
"ep_config": {
  "SPACEMIT_EP_INTRA_THREAD_NUM": "8",
  "SPACEMIT_EP_INTER_THREAD_NUM": "1",
  "SPACEMIT_EP_INTRA_THREAD_AFFINITY": "8;9;10;11;12;13;14;15"
}
```

Wait until the server prints a line similar to:

```text
llama_server: listening on http://0.0.0.0:8080
```

If the service is reachable outside a trusted local network, configure an API
key and suitable network access controls instead of exposing an unauthenticated
`0.0.0.0` endpoint.

## Send an image understanding request

The request can be sent from the board itself or another machine that can
reach the board. Set `SERVER_URL=http://127.0.0.1:8080` when testing locally,
or replace `BOARD_IP` when testing remotely.

```bash
export SERVER_URL=http://127.0.0.1:8080

base64 < "${MODEL_DIR}/humanspeech.jpg" | tr -d '\n' | jq -Rs '{
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image_url",
          image_url: {
            url: ("data:image/jpeg;base64," + .)
          }
        },
        {
          type: "text",
          text: "Describe the image content."
        }
      ]
    }
  ],
  max_tokens: 64,
  temperature: 0,
  stream: false,
  chat_template_kwargs: {
    enable_thinking: false
  }
}' | curl "${SERVER_URL}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data-binary @-
```

The generated description is returned in `choices[0].message.content`. Replace
`humanspeech.jpg` with another JPEG/PNG image to describe your own image.

## Verified example

The bundled `humanspeech.jpg` was tested on 2026-08-10 with `--warmup` enabled.
Both boards returned HTTP 200 and generated a description of the speaker,
podium, microphone, clothing, and background banner. With `max_tokens=64`,
the responses were truncated at the requested token limit.

K1 example content:

```text
The image depicts a woman standing at a podium, delivering a speech or presentation. She is positioned at the center of the frame, with her right hand raised, holding a microphone close to her mouth, suggesting she is speaking. She is dressed in a patterned top and dark pants. Behind her, there is a banner
```

K3 example content:

```text
The image depicts a woman standing at a podium, delivering a speech or presentation. She is positioned at the center of the frame, with her right hand raised, holding a microphone close to her mouth, suggesting she is actively speaking. She is dressed in a patterned top and dark pants. Behind her, there is a
```

Observed single-request wall time after server warmup:

| Board | HTTP status | Wall time |
|---|---:|---:|
| K1 (`0-3`, `-t 4`) | 200 | 9.79 s |
| K3 (`8-15`, `-t 8`) | 200 | 1.91 s |

These are functional smoke-test observations rather than a formal benchmark.
Startup and first-request time can be longer while the GGUF model is loaded and
the ONNX graph is compiled.

## Citation

Please cite the original FastVLM work when using this converted model:

```bibtex
@InProceedings{fastvlm2025,
  author = {Pavan Kumar Anasosalu Vasu and Fartash Faghri and Chun-Liang Li and Cem Koc and Nate True and Albert Antony and Gokul Santhanam and James Gabriel and Peter Grasch and Oncel Tuzel and Hadi Pouransari},
  title = {FastVLM: Efficient Vision Encoding for Vision Language Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2025},
}
```

## License

The original FastVLM model is released under Apple's AMLR model license. See
the [official model card](https://huggingface.co/apple/FastVLM-0.5B) and the
official repository's `LICENSE_MODEL` for the terms that apply to the model.
The SpacemiT `llama.cpp` and ONNX Runtime packages are separate dependencies
and remain subject to their respective repository licenses.