How to use from the
Use from the
sentence-transformers library
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("hugginganonym/Encode_ICLR27")

sentences = [
    "That is a happy person",
    "That is a happy dog",
    "That is a very happy person",
    "Today is a sunny day"
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]

Fine-tuned answer encoder for semantic uncertainty quantification

This repository contains the answer encoder from our anonymous ICLR 2027 submission.

The model is sentence-t5-large fine-tuned so that answers with the same meaning are close, and answers with different meanings are pushed apart, even when they share most of their words.

Model

Subfolder finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB
Base model sentence-t5-large
Input format "{question} [SEP] {answer}"
Output Embedding; use cosine similarity after L2 normalization

Always include the question. The encoder was trained with the question in its input, so the same answer can have a different meaning depending on the question.

Toy example

This script compares a correct and an incorrect answer with the fine-tuned encoder and with the base encoder, then shows which tokens of the correct answer matter most to the fine-tuned encoder.

"""Toy example: cosine similarity between a correct and an incorrect answer with the
fine-tuned encoder (input "question [SEP] answer") and with the base encoder (answer
only), then the token importance of the correct answer under the fine-tuned encoder."""
import torch
from huggingface_hub import snapshot_download
from sentence_transformers import SentenceTransformer

REPO_ID = "hugginganonym/Encode_ICLR27"
MODEL_SUBFOLDER = "finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB"
BASE_MODEL_ID = "sentence-t5-large"

QUESTION = "What is the capital of France?"
ANSWER_1 = "Paris is the capital of France."
ANSWER_2 = "Madrid is the capital of France since 1922."
PREFIX = f"{QUESTION} [SEP] "  # the fine-tuned encoder always sees the question


def cosine(model, text_1, text_2):
    embedding_1, embedding_2 = model.encode([text_1, text_2], normalize_embeddings=True)
    return float(embedding_1 @ embedding_2)


def print_token_importance(model, answer):
    """L2 norm of each answer token's last hidden state, divided by their sum: the weight
    each token gets from mean pooling followed by L2 normalization."""
    tokenizer, encoder = model[0].tokenizer, model[0].auto_model
    features = tokenizer(PREFIX + answer, return_tensors="pt", return_offsets_mapping=True)
    in_answer = features.pop("offset_mapping")[0, :, 1] > len(PREFIX)  # drops question and special tokens
    with torch.inference_mode():
        hidden = encoder(**features.to(encoder.device)).last_hidden_state[0].cpu()

    norms = hidden[in_answer].norm(dim=-1)
    tokens = tokenizer.convert_ids_to_tokens(features["input_ids"][0].cpu()[in_answer])
    print(f'\nToken importance for "{answer}" (internal hidden states of the fine-tuned encoder):')
    for token, score in zip(tokens, (norms / norms.sum()).tolist()):
        print(f"  {token!r:20s} {score:.4f}")


def main():
    print(f"Question: {QUESTION}")
    print(f"Answer 1 (correct):   {ANSWER_1}")
    print(f"Answer 2 (incorrect): {ANSWER_2}\n")

    repo_path = snapshot_download(REPO_ID, allow_patterns=f"{MODEL_SUBFOLDER}/*")
    model = SentenceTransformer(f"{repo_path}/{MODEL_SUBFOLDER}")
    print(f"Fine-tuned encoder (question [SEP] answer):\nsim(A1, A2) = {cosine(model, PREFIX + ANSWER_1, PREFIX + ANSWER_2):.4f}")
    base_model = SentenceTransformer(BASE_MODEL_ID)
    print(f"Base encoder {BASE_MODEL_ID} (answer only):\nsim(A1, A2) = {cosine(base_model, ANSWER_1, ANSWER_2):.4f}")

    print_token_importance(model, ANSWER_1)


if __name__ == "__main__":
    main()

Expected output

Question: What is the capital of France?
Answer 1 (correct):   Paris is the capital of France.
Answer 2 (incorrect): Madrid is the capital of France since 1922.

Fine-tuned encoder (question [SEP] answer):
sim(A1, A2) = 0.4899
Base encoder sentence-t5-large (answer only):
sim(A1, A2) = 0.8199

Token importance for "Paris is the capital of France." (internal hidden states of the fine-tuned encoder):
  '▁Paris'             0.7174
  '▁is'                0.1585
  '▁the'               0.0150
  '▁capital'           0.0479
  '▁of'                0.0259
  '▁France'            0.0138
  '.'                  0.0215

The two answers share most of their words, so the base encoder rates them as very similar (0.82).

Intended use and limitations

  • Built for comparing sampled answers to the same question.
  • Trained on English question answering; behaviour on other languages or tasks is untested.
  • Cosine similarities are only meaningful between answers to the same question.
  • The encoder was trained with a deliberately minimal recipe and would likely benefit from larger-scale training. Its goal is to show that an encoder can focus on the answer itself, so the language model can keep producing verbose outputs without hurting the uncertainty quantification metrics.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for hugginganonym/Encode_ICLR27

Finetuned
(3)
this model