Question about hard negative sample synthesis

#4
by majfu - opened

Hello!

Thank you for your work and the technical report. The report is very detailed, but while reading it, a few questions came to me about the hard negatives generation process. Since the report is so thorough, I was hoping you might be willing to answer these as well:

  1. Could you share what model was used for the data generation?
  2. The text mentions that for hard negative mining "top-10 to top-30 ranked results were selected". Does this mean that the top 10 samples were discarded? Or does it mean a range of top 10 to 30 samples was chosen - if so, what determines the number of chosen samples? And is a positive-aware negative mining strategy involved?
  3. It is stated that 4 hard negative samples are used in training - are they sampled from the bigger pool of negatives obtained from mining or was there additional ranking applied before selecting the final 4?
  4. The paper also says that you "filter out samples with low query-pos scores" - what is the threshold?

Any brief insights would be very helpful for understanding the pipeline.
Thank you! 😄

Kingsoft AI org
•
edited Oct 9

Sorry for the late reply. We only saw your issue about a month after you posted it. Thanks for the detailed questions. I’ll address them one by one and share a few things we’ve learned from our more recent embedding experiments.

  1. Which LLM we used

    We used OpenAI’s o1 series at the time, which is now an older generation of models. In our recent experiments, GPT-5.6 and GLM-5.2 have also worked well for these data construction tasks, so there’s no need to stick with the original model.

  2. How we mined hard negatives

    We first retrieved candidates using the teacher embedding model, then selected negatives from ranks 10–30. We skipped the highest-ranked results because they were more likely to be false negatives: some were actually relevant to the query but had not been labeled as positives.

    Based on our recent experiments, we now prefer a relative score threshold. We look for negatives with a high query–negative similarity score, while requiring that score to be no greater than 0.95 times the query–positive score. The 0.95 threshold is a starting point and needs to be tuned for the model and dataset.

    We did not generate negatives from positive passages in the original setup. More recently, though, we’ve had good results using an LLM to rewrite a positive passage so that it remains very similar to the original but no longer answers the query. This can produce useful hard negatives, provided the rewritten passage is checked to ensure it is no longer a valid answer.

  3. How the four hard negatives were selected

    Selection was random. For example, a training instance might have two negatives sampled from retrieval ranks 10–30 and two generated by the LLM. The split between the two sources was not fixed.

  4. How we filtered positive pairs

    We used a threshold of 0.75 at the time. The problem with an absolute threshold is that similarity scores vary considerably across embedding models. One model might assign a query–positive pair a score of 0.9, while another gives the same pair 0.6.

    We’ve recently been trying a different filtering procedure:

    • Randomly sample 15 passages from a large corpus.
    • Keep the positive if its similarity to the query is higher than that of all 15 random passages.
    • If any random passage scores at least as high as the positive, draw a fresh set of 15 and repeat the comparison.
    • Keep the positive if it passes the second comparison; otherwise, discard it.

    We use this as a heuristic for cleaning pretraining data, rather than as proof that a positive is valid. For fine-tuning data, we recommend more careful relevance checks with LLM assistance.

Beyond these specific questions, a few factors have stood out in our recent work:

  1. The quality and difficulty of hard negatives

    In our recent experiments, especially during fine-tuning data construction, LLM-generated hard negatives have worked better than negatives mined through teacher-model retrieval. The challenge is to make them difficult to distinguish from positives while ensuring they do not satisfy the query. An automated agent workflow that generates, reviews, and filters candidates can help make this process more consistent.

  2. The combination of learning rate and temperature

    For our 0.8B model, a learning rate of 1e-5 with a temperature of 0.05 worked better than 2e-5 with 0.02. With the latter combination, evaluation scores rose quickly early in training, then gradually declined. The former converged more slowly but finished about 4% higher on the average score across the C-MTEB subsets we evaluated. This was specific to our setup, but it was a useful reminder that faster early gains do not necessarily lead to a better final model.

  3. Using LLMs throughout data construction

    We’ve found LLMs useful for rewriting and expanding queries, writing positive passages, and creating hard negatives. They can help produce clear, coherent training examples, as well as negatives that are challenging for the right reasons.

    Some of the public datasets we’ve reviewed have uneven quality. With stronger LLMs now available, we think it makes sense to rely less on existing datasets and invest more in data tailored to the task. Approaches such as the synthetic data generation used in E5-Mistral have been around for a few years, but revisiting them with today’s models can still yield substantially better training data.

Thank you so much for such detailed response!

Sign up or log in to comment