multilingual-e5-small in ORT format (memory-mappable)

The weights of intfloat/multilingual-e5-small in the ONNX Runtime ORT format, for Memoa for iOS. Loaded with its initialisers memory-mapped, it embeds with a fraction of the memory the ONNX file needs, and gives bit-identical vectors.

Source intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, onnx/model.onnx (470,268,510 bytes, SHA-256 ca456c06b3a9505ddfd9131408916dd79290368331e7d76bb621f1cba6bc8665)
File model.ort: 470,350,304 bytes, SHA-256 ba3519110cbb648984f04415ebc6e32946de1c55ac85afbe3719864a76b4ef29
Converter ONNX Runtime 1.31.0 (Python wheel, installed by hash), macOS arm64
Session options CPU execution provider only; intra-op and inter-op threads 2; GraphOptimizationLevel.ORT_ENABLE_ALL; optimized_model_filepath with session.save_model_format=ORT
Script scripts/convert-e5-ort.sh in Memoa for iOS (mentisia-memoa-ios)

What it is

It is the same graph ONNX Runtime 1.31.0 builds from the source model.onnx at every load with these options, saved once. Nothing is retrained, quantised or pruned; the weights are unchanged (float32).

To load it with the initialisers mapped from the file, use ONNX Runtime 1.31.0 (CPU) and set session.load_model_format=ORT, session.use_memory_mapped_ort_model=1 and session.use_ort_model_bytes_for_initializers=1.

ONNX Runtime warns that a model optimised at ORT_ENABLE_ALL may hold hardware-specific layout transforms (NCHWc, an x86 convolution layout). e5 has no convolutions, and the vectors below were checked on arm64 (an M1 Mac and an A16 iPad). Two conversions differ only in about 1 KB of flatbuffer metadata, with the same vectors. This file is the one Memoa pins.

Vectors

The 33 reference texts of Memoa's e5 parity fixture (25 passages, 8 queries, made with memoryd's fastembed / ONNX Runtime 1.31.0 stack) were embedded from this file, with mapped initialisers, and from the source model.onnx:

  • every vector is bit-identical to the source model's (M1 Mac: this file; iPad A16: a conversion of the same source by the same ORT version);
  • max |difference| to the reference vectors 9.69e-08, int8 codes (Γ— 640) identical.

On an iPad (A16), the load's peak memory footprint was 264 MB with this file, against 1,351 MB for model.onnx read from its path. The load took 0.64 s instead of 1.03–1.23 s.

Attribution and citation

The model is by intfloat (Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei); see the original model card for training data, intended use, and limitations, which all apply here unchanged.

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

As with the original, prefix inputs with query: or passage: .

Licence: MIT, as the original (see LICENSE).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mentisialog/multilingual-e5-small-ort

Finetuned
(226)
this model

Paper for mentisialog/multilingual-e5-small-ort