Upload optimized GGUF for embeddings.cpp

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+bge-m3.q4_k_mlp_emb_q8_attn.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,139 @@
+---
+base_model: BAAI/bge-m3
+library_name: embeddings.cpp
+tags:
+- gguf
+- embeddings
+- embeddings.cpp
+- bge
+- bge-m3
+- cpu
+pipeline_tag: sentence-similarity
+---
+
+# BAAI/bge-m3 GGUF for embeddings.cpp
+
+This repository contains an optimized GGUF artifact for running
+`BAAI/bge-m3` with
+[`embeddings.cpp`](https://github.com/daandtu/embeddings.cpp).
+
+The GGUF is intended for embedding inference. It is not a llama.cpp
+text-generation model.
+
+## File
+
+| File | Quantization | Size | SHA256 |
+|---|---|---:|---|
+| `bge-m3.q4_k_mlp_emb_q8_attn.gguf` | `q4_k` MLP and vocabulary, `q8_0` attention | `365.77 MB` | `74fbdc71de72a6e60305c1f34b2ef810f23b650dbd5b09c417e324169ff8517e` |
+
+Mixed quantization: the feed-forward weights and the 250k-token vocabulary
+table are `q4_k`, attention stays `q8_0`. Compared with a plain `q8_0` file
+(584 MB, same units as the table above):
+
+- Retrieval quality is unchanged on a Chinese/English agent-history
+  benchmark (2040 documents: top-1 66%, recall@5 78–83%, both files).
+- Cosine to the fp16 model: mean 0.990, min 0.988 (`q8_0`: 0.999).
+- About 40% faster on x86 CPUs with AVX2 (Xeon E5-2673 v3, 12 threads:
+  2.0 vs 1.4 docs/s) and ~230 MB less RAM.
+- Apple silicon: use Metal (default on macOS, see below). On an M4 this file
+  runs at 11.5 docs/s with Metal vs ~5.5 on the CPU, the same speed as
+  `q8_0`. Only CPU-only Apple builds favour `q8_0` (~18% faster there).
+  Build `q8_0` yourself (below) if you need strict alignment with Python.
+
+## Python Usage
+
+```python
+from embeddings_cpp import load
+
+model = load("BAAI/bge-m3")
+vectors = model.batch_encode(["hello world", "你好，世界"])
+```
+
+Install the Hugging Face downloader extra when loading directly from this repo:
+
+```bash
+pip install "embeddings-cpp[hub]"
+```
+
+## Recommended embeddings.cpp Build
+
+Linux and Windows (CPU):
+
+```bash
+cmake -S . -B build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DEMBEDDINGS_CPP_ENABLE_PYBIND=ON \
+  -DEMBEDDINGS_CPP_BUILD_WASM_TOOLS=OFF \
+  -DGGML_CPU_REPACK=ON \
+  -DGGML_BLAS=OFF \
+  -DGGML_OPENMP=OFF \
+  -DGGML_NATIVE=OFF \
+  -DGGML_CUDA=OFF \
+  -DGGML_VULKAN=OFF \
+  -DGGML_METAL=OFF
+
+cmake --build build -j "$(nproc)"
+```
+
+On macOS (Apple silicon) use Metal instead: the PyPI wheels and a default
+CMake configure enable it, and `embeddings.cpp` picks the Metal backend
+automatically.
+
+```bash
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
+  -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON
+cmake --build build -j "$(sysctl -n hw.ncpu)"
+```
+
+## Recommended CPU Runtime
+
+The registry applies the recommended runtime environment automatically when
+using `embeddings_cpp.load()`:
+
+```bash
+EMBEDDINGS_CPP_CPU_REPACK=1
+```
+
+By default, `embeddings.cpp` uses the detected CPU concurrency for model
+inference. Set `EMBEDDINGS_CPP_THREADS=N` only when pinning a deployment to a
+measured value for a specific CPU quota or host.
+
+## Reproducing The GGUF
+
+From an `embeddings.cpp` checkout:
+
+```bash
+uv pip install -r scripts/requirements.txt
+
+uv run scripts/convert.py \
+  BAAI/bge-m3 \
+  models/bge-m3.f16.gguf \
+  f16
+
+EMBEDDINGS_CPP_SKIP_QUANT_PATTERNS='attention.,position_embeddings' \
+./build/quantize \
+  models/bge-m3.f16.gguf \
+  models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
+  q4_k
+
+./build/quantize \
+  models/bge-m3.q4_k_mlp_emb_attnf16.gguf \
+  models/bge-m3.q4_k_mlp_emb_q8_attn.gguf \
+  q8_0
+```
+
+For a plain `q8_0` file, quantize `models/bge-m3.f16.gguf` with `q8_0` in one
+step.
+
+The registry-driven artifact builder performs the same conversion and
+quantization pipeline:
+
+```bash
+uv run scripts/build_gguf_artifact.py --model-id BAAI/bge-m3
+```
+
+## Notes
+
+This model is derived from `BAAI/bge-m3`. Use the upstream model card and
+license terms when deciding whether this artifact is appropriate for your use
+case.

bge-m3.q4_k_mlp_emb_q8_attn.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74fbdc71de72a6e60305c1f34b2ef810f23b650dbd5b09c417e324169ff8517e
+size 383532480