Upload optimized GGUF for embeddings.cpp

4cdd43d verified 10 days ago

3.76 kB

base_model: BAAI/bge-m3
library_name: embeddings.cpp
tags:
  - gguf
  - embeddings
  - embeddings.cpp
  - bge
  - bge-m3
  - cpu
pipeline_tag: sentence-similarity

BAAI/bge-m3 GGUF for embeddings.cpp

This repository contains an optimized GGUF artifact for running BAAI/bge-m3 with embeddings.cpp.

The GGUF is intended for embedding inference. It is not a llama.cpp text-generation model.

File

File	Quantization	Size	SHA256
`bge-m3.q4_k_mlp_emb_q8_attn.gguf`	`q4_k` MLP and vocabulary, `q8_0` attention	`365.77 MB`	`74fbdc71de72a6e60305c1f34b2ef810f23b650dbd5b09c417e324169ff8517e`

Mixed quantization: the feed-forward weights and the 250k-token vocabulary table are q4_k, attention stays q8_0. Compared with a plain q8_0 file (584 MB, same units as the table above):

Retrieval quality is unchanged on a Chinese/English agent-history benchmark (2040 documents: top-1 66%, recall@5 78–83%, both files).
Cosine to the fp16 model: mean 0.990, min 0.988 (q8_0: 0.999).
About 40% faster on x86 CPUs with AVX2 (Xeon E5-2673 v3, 12 threads: 2.0 vs 1.4 docs/s) and ~230 MB less RAM.
Apple silicon: use Metal (default on macOS, see below). On an M4 this file runs at 11.5 docs/s with Metal vs ~~5.5 on the CPU, the same speed as q8_0. Only CPU-only Apple builds favour q8_0 (~~18% faster there). Build q8_0 yourself (below) if you need strict alignment with Python.

Python Usage

from embeddings_cpp import load

model = load("BAAI/bge-m3")
vectors = model.batch_encode(["hello world", "你好，世界"])

Install the Hugging Face downloader extra when loading directly from this repo:

pip install "embeddings-cpp[hub]"

Recommended embeddings.cpp Build

Linux and Windows (CPU):

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DEMBEDDINGS_CPP_ENABLE_PYBIND=ON \
  -DEMBEDDINGS_CPP_BUILD_WASM_TOOLS=OFF \
  -DGGML_CPU_REPACK=ON \
  -DGGML_BLAS=OFF \
  -DGGML_OPENMP=OFF \
  -DGGML_NATIVE=OFF \
  -DGGML_CUDA=OFF \
  -DGGML_VULKAN=OFF \
  -DGGML_METAL=OFF

cmake --build build -j "$(nproc)"

On macOS (Apple silicon) use Metal instead: the PyPI wheels and a default CMake configure enable it, and embeddings.cpp picks the Metal backend automatically.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON
cmake --build build -j "$(sysctl -n hw.ncpu)"

Recommended CPU Runtime

The registry applies the recommended runtime environment automatically when using embeddings_cpp.load():

EMBEDDINGS_CPP_CPU_REPACK=1

By default, embeddings.cpp uses the detected CPU concurrency for model inference. Set EMBEDDINGS_CPP_THREADS=N only when pinning a deployment to a measured value for a specific CPU quota or host.

Reproducing The GGUF

From an embeddings.cpp checkout:

uv pip install -r scripts/requirements.txt

uv run scripts/convert.py \
  BAAI/bge-m3 \
  models/bge-m3.f16.gguf \
  f16

EMBEDDINGS_CPP_SKIP_QUANT_PATTERNS='attention.,position_embeddings' \
./build/quantize \
  models/bge-m3.f16.gguf \
  models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
  q4_k

./build/quantize \
  models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
  models/bge-m3.q4_k_mlp_emb_q8_attn.gguf \
  q8_0

For a plain q8_0 file, quantize models/bge-m3.f16.gguf with q8_0 in one step.

The registry-driven artifact builder performs the same conversion and quantization pipeline:

uv run scripts/build_gguf_artifact.py --model-id BAAI/bge-m3

Notes

This model is derived from BAAI/bge-m3. Use the upstream model card and license terms when deciding whether this artifact is appropriate for your use case.