jealk commited on
Commit
542e4a1
·
verified ·
1 Parent(s): 84a08f1

Add quantization / model size section to README

Browse files
Files changed (1) hide show
  1. README.md +21 -0
README.md CHANGED
@@ -89,6 +89,27 @@ instruction shown above. Documents are encoded without an instruction.
89
  Pooling is mean pooling over the content tokens; the instruction and the
90
  beginning-of-text token are excluded — exactly as in LLM2Vec.
91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
  ### Notes
93
 
94
  - `trust_remote_code=True` is required: it loads the bidirectional Llama
 
89
  Pooling is mean pooling over the content tokens; the instruction and the
90
  beginning-of-text token are excluded — exactly as in LLM2Vec.
91
 
92
+ ### Model size and quantization
93
+
94
+ The model has ~8B parameters and loads in roughly 14 GB (bfloat16). To run on
95
+ smaller GPUs it can be loaded quantized via `bitsandbytes`:
96
+
97
+ ```python
98
+ from sentence_transformers import SentenceTransformer
99
+ from transformers import BitsAndBytesConfig
100
+
101
+ model = SentenceTransformer(
102
+ "jealk/TTC-L2V-supervised-2", trust_remote_code=True,
103
+ model_kwargs={"quantization_config": BitsAndBytesConfig(load_in_8bit=True)},
104
+ )
105
+ ```
106
+
107
+ Approximate memory footprint: **bfloat16 ≈ 14 GB**, **8-bit ≈ 8 GB**,
108
+ **4-bit ≈ 5 GB** (`load_in_4bit=True`).
109
+
110
+ > The quantized variants have not been evaluated on benchmarks — embedding
111
+ > quality may differ from the full-precision model.
112
+
113
  ### Notes
114
 
115
  - `trust_remote_code=True` is required: it loads the bidirectional Llama