chux0519's picture
Upload optimized GGUF for embeddings.cpp
4cdd43d verified
|
Raw History Blame Contribute Delete
3.76 kB
---
base_model: BAAI/bge-m3
library_name: embeddings.cpp
tags:
- gguf
- embeddings
- embeddings.cpp
- bge
- bge-m3
- cpu
pipeline_tag: sentence-similarity
---
# BAAI/bge-m3 GGUF for embeddings.cpp
This repository contains an optimized GGUF artifact for running
`BAAI/bge-m3` with
[`embeddings.cpp`](https://github.com/daandtu/embeddings.cpp).
The GGUF is intended for embedding inference. It is not a llama.cpp
text-generation model.
## File
| File | Quantization | Size | SHA256 |
|---|---|---:|---|
| `bge-m3.q4_k_mlp_emb_q8_attn.gguf` | `q4_k` MLP and vocabulary, `q8_0` attention | `365.77 MB` | `74fbdc71de72a6e60305c1f34b2ef810f23b650dbd5b09c417e324169ff8517e` |
Mixed quantization: the feed-forward weights and the 250k-token vocabulary
table are `q4_k`, attention stays `q8_0`. Compared with a plain `q8_0` file
(584 MB, same units as the table above):
- Retrieval quality is unchanged on a Chinese/English agent-history
benchmark (2040 documents: top-1 66%, recall@5 78–83%, both files).
- Cosine to the fp16 model: mean 0.990, min 0.988 (`q8_0`: 0.999).
- About 40% faster on x86 CPUs with AVX2 (Xeon E5-2673 v3, 12 threads:
2.0 vs 1.4 docs/s) and ~230 MB less RAM.
- Apple silicon: use Metal (default on macOS, see below). On an M4 this file
runs at 11.5 docs/s with Metal vs ~5.5 on the CPU, the same speed as
`q8_0`. Only CPU-only Apple builds favour `q8_0` (~18% faster there).
Build `q8_0` yourself (below) if you need strict alignment with Python.
## Python Usage
```python
from embeddings_cpp import load
model = load("BAAI/bge-m3")
vectors = model.batch_encode(["hello world", "你好,世界"])
```
Install the Hugging Face downloader extra when loading directly from this repo:
```bash
pip install "embeddings-cpp[hub]"
```
## Recommended embeddings.cpp Build
Linux and Windows (CPU):
```bash
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DEMBEDDINGS_CPP_ENABLE_PYBIND=ON \
-DEMBEDDINGS_CPP_BUILD_WASM_TOOLS=OFF \
-DGGML_CPU_REPACK=ON \
-DGGML_BLAS=OFF \
-DGGML_OPENMP=OFF \
-DGGML_NATIVE=OFF \
-DGGML_CUDA=OFF \
-DGGML_VULKAN=OFF \
-DGGML_METAL=OFF
cmake --build build -j "$(nproc)"
```
On macOS (Apple silicon) use Metal instead: the PyPI wheels and a default
CMake configure enable it, and `embeddings.cpp` picks the Metal backend
automatically.
```bash
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON
cmake --build build -j "$(sysctl -n hw.ncpu)"
```
## Recommended CPU Runtime
The registry applies the recommended runtime environment automatically when
using `embeddings_cpp.load()`:
```bash
EMBEDDINGS_CPP_CPU_REPACK=1
```
By default, `embeddings.cpp` uses the detected CPU concurrency for model
inference. Set `EMBEDDINGS_CPP_THREADS=N` only when pinning a deployment to a
measured value for a specific CPU quota or host.
## Reproducing The GGUF
From an `embeddings.cpp` checkout:
```bash
uv pip install -r scripts/requirements.txt
uv run scripts/convert.py \
BAAI/bge-m3 \
models/bge-m3.f16.gguf \
f16
EMBEDDINGS_CPP_SKIP_QUANT_PATTERNS='attention.,position_embeddings' \
./build/quantize \
models/bge-m3.f16.gguf \
models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
q4_k
./build/quantize \
models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
models/bge-m3.q4_k_mlp_emb_q8_attn.gguf \
q8_0
```
For a plain `q8_0` file, quantize `models/bge-m3.f16.gguf` with `q8_0` in one
step.
The registry-driven artifact builder performs the same conversion and
quantization pipeline:
```bash
uv run scripts/build_gguf_artifact.py --model-id BAAI/bge-m3
```
## Notes
This model is derived from `BAAI/bge-m3`. Use the upstream model card and
license terms when deciding whether this artifact is appropriate for your use
case.