Instructions to use netprojects/Nemotron-3.5-Rerank-8B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use netprojects/Nemotron-3.5-Rerank-8B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible embeddings server: llama serve -hf netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M --embeddings
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible embeddings server: llama serve -hf netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible embeddings server: ./build/bin/llama serve -hf netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M --embeddings
- Llama new
- LM Studio
- Jan
- Ollama
How to use netprojects/Nemotron-3.5-Rerank-8B-GGUF with Ollama:
ollama run hf.co/netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use netprojects/Nemotron-3.5-Rerank-8B-GGUF with Docker Model Runner:
docker model run hf.co/netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M
- Lemonade
How to use netprojects/Nemotron-3.5-Rerank-8B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull netprojects/Nemotron-3.5-Rerank-8B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Nemotron-3.5-Rerank-8B-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
File size: 1,563 Bytes
50bfecf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | """GGUF rerank wrapper (Option A): llama-server embeddings (mean pool, unnormalized) . score_weight"""
import json, sys, csv, time, urllib.request, numpy as np
URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8099"
OUT = sys.argv[2] if len(sys.argv) > 2 else "scores.csv"
w = np.load(__file__.rsplit("\\", 1)[0] + "\\score_weight.npy").astype(np.float32).reshape(-1)
Z = "\u200c"
def esc(t): return t.replace("[", "[" + Z).replace("<", "<" + Z)
def fmt(q, d): return "query: " + esc(q) + "\n\npassage: " + esc(d) # chat_template.jinja, no BOS/EOS
def emb(text):
req = urllib.request.Request(URL + "/embedding", json.dumps({"content": text, "add_special": False}).encode(), {"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req))
e = r[0]["embedding"] if isinstance(r, list) else r["embedding"]
e = np.array(e, dtype=np.float32).reshape(-1, w.size); assert e.shape[0] == 1, e.shape; return e[0]
def score(q, d): return float(np.dot(w, emb(fmt(q, d))))
if __name__ == "__main__":
pairs = json.load(open(__file__.rsplit("\\", 1)[0] + "\\pairs.json", encoding="utf-8-sig"))
rows = []; t = time.time()
for i, p in enumerate(pairs):
rows.append((i, p["rel"], round(score(p["q"], p["d"]), 4), p["q"], p["d"][:60]))
with open(OUT, "w", newline="", encoding="utf-8") as f:
c = csv.writer(f); c.writerow(["idx", "relevant", "score", "query", "doc"]); c.writerows(rows)
for r in rows: print(r[0], r[1], r[2], r[3][:40])
print(f"{len(rows)} pairs in {time.time()-t:.1f}s")
|