hugginganonym commited on
Commit
ecf6414
·
verified ·
1 Parent(s): 320dda2

Delete mian

Browse files
Files changed (1) hide show
  1. mian +0 -122
mian DELETED
@@ -1,122 +0,0 @@
1
- ---
2
- library_name: sentence-transformers
3
- base_model: sentence-transformers/sentence-t5-large
4
- pipeline_tag: sentence-similarity
5
- tags:
6
- - sentence-transformers
7
- - sentence-similarity
8
- - uncertainty-quantification
9
- - semantic-entropy
10
- language:
11
- - en
12
- ---
13
-
14
- # Fine-tuned answer encoder for semantic uncertainty quantification
15
-
16
- This repository contains the answer encoder from our anonymous ICLR 2027 submission.
17
-
18
- The model is `sentence-t5-large` fine-tuned so that **answers with the same meaning are close, and answers with different meanings are pushed apart**, even when they share most of their words.
19
-
20
- ## Model
21
-
22
- | | |
23
- |---|---|
24
- | Subfolder | `finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB` |
25
- | Base model | `sentence-t5-large` |
26
- | Input format | `"{question} [SEP] {answer}"` |
27
- | Output | Embedding; use cosine similarity after L2 normalization |
28
-
29
- **Always include the question.** The encoder was trained with the question in its input, so the same answer can have a different meaning depending on the question.
30
-
31
-
32
- ## Toy example
33
-
34
- This script compares a correct and an incorrect answer with the fine-tuned encoder and with the base encoder, then shows which tokens of the correct answer matter most to the fine-tuned encoder.
35
-
36
- ```python
37
- """Toy example: cosine similarity between a correct and an incorrect answer with the
38
- fine-tuned encoder (input "question [SEP] answer") and with the base encoder (answer
39
- only), then the token importance of the correct answer under the fine-tuned encoder."""
40
- import torch
41
- from huggingface_hub import snapshot_download
42
- from sentence_transformers import SentenceTransformer
43
-
44
- REPO_ID = "hugginganonym/Encode_ICLR27"
45
- MODEL_SUBFOLDER = "finetuned_answer_embedder_correctness_sentence_32_m0.2_llama_nq_FB"
46
- BASE_MODEL_ID = "sentence-t5-large"
47
-
48
- QUESTION = "What is the capital of France?"
49
- ANSWER_1 = "Paris is the capital of France."
50
- ANSWER_2 = "Madrid is the capital of France since 1922."
51
- PREFIX = f"{QUESTION} [SEP] " # the fine-tuned encoder always sees the question
52
-
53
-
54
- def cosine(model, text_1, text_2):
55
- embedding_1, embedding_2 = model.encode([text_1, text_2], normalize_embeddings=True)
56
- return float(embedding_1 @ embedding_2)
57
-
58
-
59
- def print_token_importance(model, answer):
60
- """L2 norm of each answer token's last hidden state, divided by their sum: the weight
61
- each token gets from mean pooling followed by L2 normalization."""
62
- tokenizer, encoder = model[0].tokenizer, model[0].auto_model
63
- features = tokenizer(PREFIX + answer, return_tensors="pt", return_offsets_mapping=True)
64
- in_answer = features.pop("offset_mapping")[0, :, 1] > len(PREFIX) # drops question and special tokens
65
- with torch.inference_mode():
66
- hidden = encoder(**features.to(encoder.device)).last_hidden_state[0].cpu()
67
-
68
- norms = hidden[in_answer].norm(dim=-1)
69
- tokens = tokenizer.convert_ids_to_tokens(features["input_ids"][0].cpu()[in_answer])
70
- print(f'\nToken importance for "{answer}" (internal hidden states of the fine-tuned encoder):')
71
- for token, score in zip(tokens, (norms / norms.sum()).tolist()):
72
- print(f" {token!r:20s} {score:.4f}")
73
-
74
-
75
- def main():
76
- print(f"Question: {QUESTION}")
77
- print(f"Answer 1 (correct): {ANSWER_1}")
78
- print(f"Answer 2 (incorrect): {ANSWER_2}\n")
79
-
80
- repo_path = snapshot_download(REPO_ID, allow_patterns=f"{MODEL_SUBFOLDER}/*")
81
- model = SentenceTransformer(f"{repo_path}/{MODEL_SUBFOLDER}")
82
- print(f"Fine-tuned encoder (question [SEP] answer):\nsim(A1, A2) = {cosine(model, PREFIX + ANSWER_1, PREFIX + ANSWER_2):.4f}")
83
- base_model = SentenceTransformer(BASE_MODEL_ID)
84
- print(f"Base encoder {BASE_MODEL_ID} (answer only):\nsim(A1, A2) = {cosine(base_model, ANSWER_1, ANSWER_2):.4f}")
85
-
86
- print_token_importance(model, ANSWER_1)
87
-
88
-
89
- if __name__ == "__main__":
90
- main()
91
- ```
92
-
93
- ### Expected output
94
-
95
- ```
96
- Question: What is the capital of France?
97
- Answer 1 (correct): Paris is the capital of France.
98
- Answer 2 (incorrect): Madrid is the capital of France since 1922.
99
-
100
- Fine-tuned encoder (question [SEP] answer):
101
- sim(A1, A2) = 0.4899
102
- Base encoder sentence-t5-large (answer only):
103
- sim(A1, A2) = 0.8199
104
-
105
- Token importance for "Paris is the capital of France." (internal hidden states of the fine-tuned encoder):
106
- '▁Paris' 0.7174
107
- '▁is' 0.1585
108
- '▁the' 0.0150
109
- '▁capital' 0.0479
110
- '▁of' 0.0259
111
- '▁France' 0.0138
112
- '.' 0.0215
113
- ```
114
-
115
- The two answers share most of their words, so the base encoder rates them as very similar (0.82).
116
-
117
- ## Intended use and limitations
118
-
119
- - Built for comparing sampled answers to the same question.
120
- - Trained on English question answering; behaviour on other languages or tasks is untested.
121
- - Cosine similarities are only meaningful between answers to the same question.
122
- - The encoder was trained with a deliberately minimal recipe and would likely benefit from larger-scale training. Its goal is to show that an encoder can focus on the answer itself, so the language model can keep producing verbose outputs without hurting the uncertainty quantification metrics.