Instructions to use hugginganonym/Encode_ICLR27 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use hugginganonym/Encode_ICLR27 with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("hugginganonym/Encode_ICLR27") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
Fine-tuned answer encoder for semantic uncertainty quantification
This repository contains the answer encoder from our anonymous ICLR 2027 submission.
The model is sentence-t5-large fine-tuned so that answers with the same meaning are close, and answers with different meanings are pushed apart, even when they share most of their words.
Model
| Subfolder | finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB |
| Base model | sentence-t5-large |
| Input format | "{question} [SEP] {answer}" |
| Output | Embedding; use cosine similarity after L2 normalization |
Always include the question. The encoder was trained with the question in its input, so the same answer can have a different meaning depending on the question.
Toy example
This script compares a correct and an incorrect answer with the fine-tuned encoder and with the base encoder, then shows which tokens of the correct answer matter most to the fine-tuned encoder.
"""Toy example: cosine similarity between a correct and an incorrect answer with the
fine-tuned encoder (input "question [SEP] answer") and with the base encoder (answer
only), then the token importance of the correct answer under the fine-tuned encoder."""
import torch
from huggingface_hub import snapshot_download
from sentence_transformers import SentenceTransformer
REPO_ID = "hugginganonym/Encode_ICLR27"
MODEL_SUBFOLDER = "finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB"
BASE_MODEL_ID = "sentence-t5-large"
QUESTION = "What is the capital of France?"
ANSWER_1 = "Paris is the capital of France."
ANSWER_2 = "Madrid is the capital of France since 1922."
PREFIX = f"{QUESTION} [SEP] " # the fine-tuned encoder always sees the question
def cosine(model, text_1, text_2):
embedding_1, embedding_2 = model.encode([text_1, text_2], normalize_embeddings=True)
return float(embedding_1 @ embedding_2)
def print_token_importance(model, answer):
"""L2 norm of each answer token's last hidden state, divided by their sum: the weight
each token gets from mean pooling followed by L2 normalization."""
tokenizer, encoder = model[0].tokenizer, model[0].auto_model
features = tokenizer(PREFIX + answer, return_tensors="pt", return_offsets_mapping=True)
in_answer = features.pop("offset_mapping")[0, :, 1] > len(PREFIX) # drops question and special tokens
with torch.inference_mode():
hidden = encoder(**features.to(encoder.device)).last_hidden_state[0].cpu()
norms = hidden[in_answer].norm(dim=-1)
tokens = tokenizer.convert_ids_to_tokens(features["input_ids"][0].cpu()[in_answer])
print(f'\nToken importance for "{answer}" (internal hidden states of the fine-tuned encoder):')
for token, score in zip(tokens, (norms / norms.sum()).tolist()):
print(f" {token!r:20s} {score:.4f}")
def main():
print(f"Question: {QUESTION}")
print(f"Answer 1 (correct): {ANSWER_1}")
print(f"Answer 2 (incorrect): {ANSWER_2}\n")
repo_path = snapshot_download(REPO_ID, allow_patterns=f"{MODEL_SUBFOLDER}/*")
model = SentenceTransformer(f"{repo_path}/{MODEL_SUBFOLDER}")
print(f"Fine-tuned encoder (question [SEP] answer):\nsim(A1, A2) = {cosine(model, PREFIX + ANSWER_1, PREFIX + ANSWER_2):.4f}")
base_model = SentenceTransformer(BASE_MODEL_ID)
print(f"Base encoder {BASE_MODEL_ID} (answer only):\nsim(A1, A2) = {cosine(base_model, ANSWER_1, ANSWER_2):.4f}")
print_token_importance(model, ANSWER_1)
if __name__ == "__main__":
main()
Expected output
Question: What is the capital of France?
Answer 1 (correct): Paris is the capital of France.
Answer 2 (incorrect): Madrid is the capital of France since 1922.
Fine-tuned encoder (question [SEP] answer):
sim(A1, A2) = 0.4899
Base encoder sentence-t5-large (answer only):
sim(A1, A2) = 0.8199
Token importance for "Paris is the capital of France." (internal hidden states of the fine-tuned encoder):
'▁Paris' 0.7174
'▁is' 0.1585
'▁the' 0.0150
'▁capital' 0.0479
'▁of' 0.0259
'▁France' 0.0138
'.' 0.0215
The two answers share most of their words, so the base encoder rates them as very similar (0.82).
Intended use and limitations
- Built for comparing sampled answers to the same question.
- Trained on English question answering; behaviour on other languages or tasks is untested.
- Cosine similarities are only meaningful between answers to the same question.
- The encoder was trained with a deliberately minimal recipe and would likely benefit from larger-scale training. Its goal is to show that an encoder can focus on the answer itself, so the language model can keep producing verbose outputs without hurting the uncertainty quantification metrics.
Model tree for hugginganonym/Encode_ICLR27
Base model
sentence-transformers/sentence-t5-large
from sentence_transformers import SentenceTransformer model = SentenceTransformer("hugginganonym/Encode_ICLR27") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4]