Instructions to use janusz-bit/multilingual-e5-small-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use janusz-bit/multilingual-e5-small-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible embeddings server: llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible embeddings server: llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible embeddings server: ./build/bin/llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings
- Llama new
- LM Studio
- Jan
- Ollama
How to use janusz-bit/multilingual-e5-small-gguf with Ollama:
ollama run hf.co/janusz-bit/multilingual-e5-small-gguf:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use janusz-bit/multilingual-e5-small-gguf with Docker Model Runner:
docker model run hf.co/janusz-bit/multilingual-e5-small-gguf:Q8_0
- Lemonade
How to use janusz-bit/multilingual-e5-small-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull janusz-bit/multilingual-e5-small-gguf:Q8_0
Run and chat with the model
lemonade run user.multilingual-e5-small-gguf-Q8_0
List all available models
lemonade list
- Atomic Chat
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible embeddings server:
llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddingsBuild from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible embeddings server:
./build/bin/llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddingsmultilingual-e5-small β GGUF (q8_0)
intfloat/multilingual-e5-small converted to GGUF for llama.cpp and quantized to q8_0. It is the model of search by meaning in Brulion, an offline-first Markdown notes app: the app downloads this file and embeds the notes on the device.
| File | Size | SHA-256 |
|---|---|---|
multilingual-e5-small-q8_0.gguf |
132,440,544 bytes | 3a98dd889ee152d923991c60df2fad21e39f7441a5ee302a0a050d5137993b60 |
118M parameters, 384 dimensions, up to 512 tokens, mean pooling (stored in the file). As with the
original model, prefix queries with query: and documents with passage: .
How it was made
From intfloat/multilingual-e5-small at revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, with
llama.cpp v0.9.0 and Brulion's
scripts/semantic/convert_e5_gguf.py:
python3 scripts/semantic/convert_e5_gguf.py --llama-src <llama.cpp v0.9.0 sources> e5 e5-small-f16.gguf
llama-quantize e5-small-f16.gguf multilingual-e5-small-q8_0.gguf q8_0
llama.cpp's own converter maps the model to a WordPiece vocabulary (BertModel) or to position
ids starting at 2 (XLMRobertaModel); both are wrong for this model. The script keeps the
SentencePiece vocabulary and positions from 0, so token ids match the Hugging Face tokenizer
exactly. On Brulion's benchmark the ranking equals PyTorch's for every query (the same top hit for
341 of 341 queries, scores within 2Β·10β»β΄); q8_0 loses nothing measurable. The full procedure:
docs/SEMANTIC.md.
Licence
MIT, as the original model (see LICENSE). The weights are those of
intfloat/multilingual-e5-small; only their format and the q8_0 quantization differ.
Citation
@article{wang2024multilingual,
title={Multilingual E5 Text Embeddings: A Technical Report},
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
journal={arXiv preprint arXiv:2402.05672},
year={2024}
}
- Downloads last month
- -
8-bit
Model tree for janusz-bit/multilingual-e5-small-gguf
Base model
intfloat/multilingual-e5-small
Install (macOS, Linux)
# Start a local OpenAI-compatible embeddings server: llama serve -hf janusz-bit/multilingual-e5-small-gguf:Q8_0 --embeddings