- Libraries
- sentence-transformers
How to use thoddnn/all-MiniLM-L6-v2-4bit with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("thoddnn/all-MiniLM-L6-v2-4bit")
sentences = [
"That is a happy person",
"That is a happy dog",
"That is a very happy person",
"Today is a sunny day"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4] - Transformers
How to use thoddnn/all-MiniLM-L6-v2-4bit with Transformers:
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("thoddnn/all-MiniLM-L6-v2-4bit")
model = AutoModel.from_pretrained("thoddnn/all-MiniLM-L6-v2-4bit", device_map="auto") - MLX
How to use thoddnn/all-MiniLM-L6-v2-4bit with MLX:
# Download the model from the Hub
pip install huggingface_hub[hf_xet]
hf download thoddnn/all-MiniLM-L6-v2-4bit --local-dir all-MiniLM-L6-v2-4bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
