grepac-e5-small

intfloat/multilingual-e5-small fine-tuned on CPU to find the right article of the EU Common Agricultural Policy (CAP) regulations for a question. It powers grePAC. Code and training pipeline: github.com/stonarini/grepac.

Usage

Same query: / passage: prefixes as the base model.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("stonarini/grepac-e5-small")
q = model.encode(["query: When will I receive my CAP payments?"], normalize_embeddings=True)
p = model.encode(["passage: Article 44 Payment to beneficiaries\n..."], normalize_embeddings=True)
print(q @ p.T)

Training

The training data is generated, with no hand-written examples:

  • a local LLM (Mistral 7B Instruct v0.3) writes questions for each paragraph, plus rule-based queries from titles and definitions
  • each question gets a hard negative, a paragraph from another article that looks similar

~3000 training examples over +200 articles. MultipleNegativesRankingLoss, 2 epochs, batch 32, learning rate 2e-5, 73 minutes on a 4-core laptop CPU.

Evaluation

60 hand-written and hand-checked questions, each with the articles that answer it. Scored per article.

right article first in top 3 in top 5 MRR@10
base model 0.667 0.833 0.867 0.750
grepac-e5-small 0.717 0.900 0.900 0.797

The test is too small to be conclusive: 11 questions improved and 8 got worse. Real searches from the site will be the next test set.

License

MIT, like the base model. Regulation texts: EUR-Lex, ยฉ European Union, reused under the Commission's reuse policy.

Downloads last month
60
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for stonarini/grepac-e5-small

Finetuned
(229)
this model