How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible embeddings server:
llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible embeddings server:
llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible embeddings server:
./build/bin/llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
Quick Links

multilingual-e5-small β€” GGUF (q8_0)

intfloat/multilingual-e5-small converted to GGUF for llama.cpp and quantized to q8_0. It is the model of search by meaning in Brulion, an offline-first Markdown notes app: the app downloads this file and embeds the notes on the device.

File Size SHA-256
multilingual-e5-small-q8_0.gguf 132,440,544 bytes 3a98dd889ee152d923991c60df2fad21e39f7441a5ee302a0a050d5137993b60

118M parameters, 384 dimensions, up to 512 tokens, mean pooling (stored in the file). As with the original model, prefix queries with query: and documents with passage: .

How it was made

From intfloat/multilingual-e5-small at revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, with llama.cpp v0.9.0 and Brulion's scripts/semantic/convert_e5_gguf.py:

python3 scripts/semantic/convert_e5_gguf.py --llama-src <llama.cpp v0.9.0 sources> e5 e5-small-f16.gguf
llama-quantize e5-small-f16.gguf multilingual-e5-small-q8_0.gguf q8_0

llama.cpp's own converter maps the model to a WordPiece vocabulary (BertModel) or to position ids starting at 2 (XLMRobertaModel); both are wrong for this model. The script keeps the SentencePiece vocabulary and positions from 0, so token ids match the Hugging Face tokenizer exactly. On Brulion's benchmark the ranking equals PyTorch's for every query (the same top hit for 341 of 341 queries, scores within 2·10⁻⁴); q8_0 loses nothing measurable. The full procedure: docs/SEMANTIC.md.

Licence

MIT, as the original model (see LICENSE). The weights are those of intfloat/multilingual-e5-small; only their format and the q8_0 quantization differ.

Citation

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}
Downloads last month
-
GGUF
Model size
0.1B params
Architecture
bert
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for janusz-bit/multilingual-e5-small-gguf

Quantized
(318)
this model

Paper for janusz-bit/multilingual-e5-small-gguf