Harsha001/LFM3-VL-2B-ROCmFPX

LFM3-VL-2B, ROCmFPX GGUF

LiquidAI/LFM3-VL-2B, Liquid AI's 2B vision-language model for on-device image, document, chart and screen understanding, quantized for the ROCmFPX llama.cpp fork (low-bit weight types for AMD GPUs, Vulkan and HIP/ROCm).

The recommended 6-bit file needs HarshaVardhan3002/ROCmFPX. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds may not load it. The Q8_0 and Q4_K_M files use stock types and load in any llama.cpp build with LFM3 support. These files were checked on the CPU and have not been tested on AMD hardware yet.

Not an official release of Liquid AI. All credit for the model goes to its authors.

Files

File	Size	Weights	KL divergence vs BF16 (lower is better)	Same top token as BF16
LFM3-VL-2B-Q6_K_ROCMFPX.gguf (recommended)	1.43 GB	ROCmFPX 6-bit (Q6_0_ROCMFPX with tensor scales)	0.0162	93.3%
LFM3-VL-2B-Q8_0.gguf	1.85 GB	stock llama.cpp Q8_0	0.0032	96.9%
LFM3-VL-2B-Q4_K_M.gguf	1.10 GB	stock llama.cpp Q4_K_M	0.1248	80.7%
stock Q6_K, for comparison (not uploaded)	1.43 GB	stock llama.cpp Q6_K	0.0167	92.5%

Vision projector (load with --mmproj, needed for images):

File	Size
mmproj-LFM3-VL-2B-Q8_0.gguf (recommended)	0.20 GB
mmproj-LFM3-VL-2B-F16.gguf	0.38 GB

Both projectors were tested on a photo of a newspaper front page (with an 8-bit build of the model) and the model described it correctly. Images with the 6-bit and 4-bit files were not tested separately.

• Measured on: 40 chunks of 512 tokens of WikiText-2 (test split), against the BF16 GGUF, CPU.
• The ROCmFPX 6-bit file beats stock Q6_K at the same size. Its file name says Q6_K_ROCMFPX so that Hugging Face lists it as a 6-bit quant; the weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock Q6_K.
• No ROCmFPX 4-bit file: our 4-bit types came out clearly worse than stock Q4_K_M on this model (KL divergence 1.5 to 1.7 times higher), so we ship Q4_K_M instead.
• Every file passed a short greedy answer test before upload.

How these were made

1. BF16 GGUF converted with the fork's convert_hf_to_gguf.py (plus the vision projector as F16 and Q8_0).
2. Importance matrix from the BF16 file over a general calibration text (calibration_datav3), 512-token chunks.
3. llama-quantize --imatrix from the BF16 file for every file, stock ones included. The 6-bit file was made with LLAMA_ROCMFPX_TENSOR_SCALE=1, which stores a power-of-two scale per weight tensor (the runtime applies it; no setting is needed to run the file), with --token-embedding-type q6_K.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build -j
build/bin/llama-server -m LFM3-VL-2B-Q6_K_ROCMFPX.gguf --mmproj mmproj-LFM3-VL-2B-Q8_0.gguf -ngl 99 --jinja \
  --temp 0.1 --top-k 50 --repeat-penalty 1.1

Liquid AI recommends temperature 0.1, top-k 50, repetition penalty 1.1. llama-mtmd-cli with the same -m and --mmproj works for single images from the command line.

LFM3 support in the fork comes from llama.cpp pull request #31252. LFM3-Omni and LFM3-Audio need further upstream llama.cpp work and will follow later.

License

LFM Open License v1.0, same as the base model. Commercial use is allowed for companies below 10 million US dollars of annual revenue; above that it is not covered by the license. Read LICENSE before you use these files.