ikuyamada commited on
Commit
0bc52e3
·
verified ·
1 Parent(s): b675ad8

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +90 -90
README.md CHANGED
@@ -1,128 +1,128 @@
1
  ---
2
- tags:
3
- - transformers
4
- - sentence-transformers
5
- language:
6
- - en
7
- license: apache-2.0
8
- library_name: transformers
9
- base_model:
10
- - google-bert/bert-base-uncased
11
- model_index:
12
- - name: kpr-bert-base-uncased
13
- results:
14
- ---
15
-
16
- # Knowledgeable Embedding: kpr-bert-base-uncased
17
-
18
- ## Introduction
19
 
20
- **Injecting dynamically updatable entity knowledge into embeddings to enhance RAG**
21
 
22
- A key limitation of large language models (LLMs) is their inability to capture less-frequent or up-to-date entity knowledge, often leading to factual inaccuracies and hallucinations. Retrieval-augmented generation (RAG), which incorporates external knowledge through retrieval, is a common approach to mitigate this issue.
23
 
24
- Although RAG typically relies on embedding-based retrieval, the embedding models themselves are also based on language models and therefore struggle with queries involving less-frequent entities ([Sciavolino et al., 2021](https://arxiv.org/abs/2109.08535)), often failing to retrieve the crucial knowledge needed to overcome this limitation.
25
 
26
- **Knowledgeable Embedding** enhances performance on such queries by injecting real-world entity knowledge into embeddings, making them more *knowledgeable*.
27
 
28
- **The entity knowledge is pluggable and can be dynamically updated with ease.**
29
 
30
- For further details, refer to [our paper](https://arxiv.org/abs/2507.03922) or [GitHub repository](https://github.com/knowledgeable-embedding/knowledgeable-embedding).
31
 
32
- ## Model List
33
 
34
- | Model | Model Size | Base Model |
35
- | --- | --- | --- |
36
- | [knowledgeable-ai/kpr-bert-base-uncased](https://huggingface.co/knowledgeable-ai/kpr-bert-base-uncased) | 112M | [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) |
37
- | [knowledgeable-ai/kpr-retromae](https://huggingface.co/knowledgeable-ai/kpr-retromae) | 112M | [RetroMAE](https://huggingface.co/Shitao/RetroMAE) |
38
- | [knowledgeable-ai/kpr-bge-base-en](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en) | 112M | [bge-base-en](https://huggingface.co/BAAI/bge-base-en) |
39
- | [knowledgeable-ai/kpr-bge-base-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en-v1.5) | 112M | [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) |
40
- | [knowledgeable-ai/kpr-bge-large-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-large-en-v1.5) | 340M | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) |
41
 
42
- For practical use, we recommend `knowledgeable-ai/kpr-bge-*`, which significantly outperforms state-of-the-art models on queries involving less-frequent entities while performing comparably on other queries, as reported in [our paper](https://arxiv.org/abs/2507.03922).
43
 
44
- Regarding the model size, we do not count the entity embeddings since they are stored in CPU memory and have a negligible impact on runtime performance. See [this page](https://github.com/knowledgeable-embedding/knowledgeable-embedding/wiki/Internals-of-Knowledgeable-Embedding) for details.
 
 
 
 
 
 
45
 
46
- ## Model Details
47
 
48
- - Model Name: kpr-bert-base-uncased
49
- - Base Model: [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)
50
- - Maximum Sequence Length: 512
51
- - Embedding Dimension: 768
52
 
53
- ## Usage
54
 
55
- This model can be used via [Hugging Face Transformers](https://github.com/huggingface/transformers) or [Sentence Transformers](https://github.com/UKPLab/sentence-transformers):
 
 
 
56
 
57
- ### Hugging Face Transformers
58
 
59
- ```python
60
- from transformers import AutoTokenizer, AutoModel
61
- import torch
62
 
63
- MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-bert-base-uncased"
64
 
65
- input_texts = [
66
- "Who founded Dominican Liberation Party?",
67
- "Who owns Mompesson House?"
68
- ]
69
 
70
- # Load model and tokenizer from the Hugging Face Hub
71
- tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
72
- model = AutoModel.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
73
 
74
- # Preprocess the text
75
- preprocessed_inputs = tokenizer(input_texts, return_tensors="pt", padding=True)
 
 
76
 
77
- # Compute embeddings
78
- with torch.no_grad():
79
- embeddings = model.encode(**preprocessed_inputs)
80
 
81
- print("Embeddings:", embeddings)
82
- ```
83
 
84
- ### Sentence Transformers
 
 
85
 
86
- ```python
87
- from sentence_transformers import SentenceTransformer
88
 
89
- MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-bert-base-uncased"
90
 
91
- input_texts = [
92
- "Who founded Dominican Liberation Party?",
93
- "Who owns Mompesson House?"
94
- ]
95
 
96
- # Load model from the Hugging Face Hub
97
- model = SentenceTransformer(MODEL_NAME_OR_PATH, trust_remote_code=True)
98
 
99
- # Compute embeddings
100
- embeddings = model.encode(input_texts)
 
 
101
 
102
- print("Embeddings:", embeddings)
103
- ```
104
 
105
- **IMPORTANT:** This code will be supported in versions of Sentence Transformers later than v5.1.0, which have not yet been released at the time of writing. Until then, please install the library directly from GitHub:
 
106
 
107
- ```bash
108
- pip install git+https://github.com/UKPLab/sentence-transformers.git
109
- ```
110
 
111
- ## License
112
 
113
- This model is licensed under the Apache License, Version 2.0.
 
 
114
 
115
- ## Citation
116
 
117
- If you use this model in your research, please cite the following paper:
118
 
119
- [Dynamic Injection of Entity Knowledge into Dense Retrievers](https://arxiv.org/abs/2507.03922)
120
 
121
- ```bibtex
122
- @article{yamada2025kpr,
123
- title={Dynamic Injection of Entity Knowledge into Dense Retrievers},
124
- author={Ikuya Yamada and Ryokan Ri and Takeshi Kojima and Yusuke Iwasawa and Yutaka Matsuo},
125
- journal={arXiv preprint arXiv:2507.03922},
126
- year={2025}
127
- }
128
- ```
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - transformers
4
+ - sentence-transformers
5
+ language:
6
+ - en
7
+ license: apache-2.0
8
+ library_name: transformers
9
+ base_model:
10
+ - bert-base-uncased
11
+ model_index:
12
+ - name: kpr-bert-base-uncased
13
+ results:
14
+ ---
 
 
 
 
15
 
16
+ # Knowledgeable Embedding: kpr-bert-base-uncased
17
 
18
+ ## Introduction
19
 
20
+ **Injecting dynamically updatable entity knowledge into embeddings to enhance RAG**
21
 
22
+ A key limitation of large language models (LLMs) is their inability to capture less-frequent or up-to-date entity knowledge, often leading to factual inaccuracies and hallucinations. Retrieval-augmented generation (RAG), which incorporates external knowledge through retrieval, is a common approach to mitigate this issue.
23
 
24
+ Although RAG typically relies on embedding-based retrieval, the embedding models themselves are also based on language models and therefore struggle with queries involving less-frequent entities, often failing to retrieve the crucial knowledge needed to overcome this limitation.
25
 
26
+ **Knowledgeable Embedding** enhances performance on such queries by injecting real-world entity knowledge into embeddings, making them more *knowledgeable*.
27
 
28
+ **The entity knowledge is pluggable and can be dynamically updated with ease.**
29
 
30
+ For further details, refer to [our paper](https://arxiv.org/abs/2507.03922) or [GitHub repository](https://github.com/knowledgeable-embedding/knowledgeable-embedding).
 
 
 
 
 
 
31
 
32
+ ## Model List
33
 
34
+ | Model | Model Size | Base Model |
35
+ | --- | --- | --- |
36
+ | [knowledgeable-ai/kpr-bert-base-uncased](https://huggingface.co/knowledgeable-ai/kpr-bert-base-uncased) | 112M | [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) |
37
+ | [knowledgeable-ai/kpr-retromae](https://huggingface.co/knowledgeable-ai/kpr-retromae) | 112M | [RetroMAE](https://huggingface.co/Shitao/RetroMAE) |
38
+ | [knowledgeable-ai/kpr-bge-base-en](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en) | 112M | [bge-base-en](https://huggingface.co/BAAI/bge-base-en) |
39
+ | [knowledgeable-ai/kpr-bge-base-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en-v1.5) | 112M | [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) |
40
+ | [knowledgeable-ai/kpr-bge-large-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-large-en-v1.5) | 340M | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) |
41
 
42
+ For practical use, we recommend `knowledgeable-ai/kpr-bge-*`, which significantly outperforms state-of-the-art models on queries involving less-frequent entities while performing comparably on other queries, as reported in [our paper](https://arxiv.org/abs/2507.03922).
43
 
44
+ Regarding the model size, we do not count the entity embeddings since they are stored in CPU memory and have a negligible impact on runtime performance. See [this page](https://github.com/knowledgeable-embedding/knowledgeable-embedding/wiki/Internals-of-Knowledgeable-Embedding) for details.
 
 
 
45
 
46
+ ## Model Details
47
 
48
+ - Model Name: kpr-bert-base-uncased
49
+ - Base Model: [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)
50
+ - Maximum Sequence Length: 512
51
+ - Embedding Dimension: 768
52
 
53
+ ## How to use
54
 
55
+ This model can be used via [Hugging Face Transformers](https://github.com/huggingface/transformers) or [Sentence Transformers](https://github.com/UKPLab/sentence-transformers):
 
 
56
 
57
+ ### Hugging Face Transformers
58
 
59
+ ```python
60
+ from transformers import AutoTokenizer, AutoModel
61
+ import torch
 
62
 
63
+ MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-bert-base-uncased"
 
 
64
 
65
+ input_texts = [
66
+ "Who founded Dominican Liberation Party?",
67
+ "Who owns Mompesson House?"
68
+ ]
69
 
70
+ # Load model and tokenizer from the Hugging Face Hub
71
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
72
+ model = AutoModel.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
73
 
74
+ # Preprocess the text
75
+ preprocessed_inputs = tokenizer(input_texts, return_tensors="pt", padding=True)
76
 
77
+ # Compute embeddings
78
+ with torch.no_grad():
79
+ embeddings = model.encode(**preprocessed_inputs)
80
 
81
+ print("Embeddings:", embeddings)
82
+ ```
83
 
84
+ ### Sentence Transformers
85
 
86
+ ```python
87
+ from sentence_transformers import SentenceTransformer
 
 
88
 
89
+ MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-bert-base-uncased"
 
90
 
91
+ input_texts = [
92
+ "Who founded Dominican Liberation Party?",
93
+ "Who owns Mompesson House?"
94
+ ]
95
 
96
+ # Load model from the Hugging Face Hub
97
+ model = SentenceTransformer(MODEL_NAME_OR_PATH, trust_remote_code=True)
98
 
99
+ # Compute embeddings
100
+ embeddings = model.encode(input_texts)
101
 
102
+ print("Embeddings:", embeddings)
103
+ ```
 
104
 
105
+ **IMPORTANT:** This code will be supported in versions of Sentence Transformers later than v5.1.0, which have not yet been released at the time of writing. Until then, please install the library directly from GitHub:
106
 
107
+ ```bash
108
+ pip install git+https://github.com/UKPLab/sentence-transformers.git
109
+ ```
110
 
111
+ ## License
112
 
113
+ This model is licensed under the Apache License, Version 2.0.
114
 
115
+ ## Citation
116
 
117
+ If you use this model in your research, please cite the following paper:
118
+ [Dynamic Injection of Entity Knowledge into Dense Retrievers](https://arxiv.org/abs/2507.03922)
119
+
120
+ ```bibtex
121
+ @article{yamada2025kpr,
122
+ title={Dynamic Injection of Entity Knowledge into Dense Retrievers},
123
+ author={Ikuya Yamada and Ryokan Ri and Takeshi Kojima and Yusuke Iwasawa and Yutaka Matsuo},
124
+ journal={arXiv preprint arXiv:2507.03922},
125
+ year={2025}
126
+ }
127
+ ```
128
+