prism-ml/Ternary-Bonsai-2-27B-gguf
Text Generation • 27B • Updated • 3.97M • 2.36k
Pablo
p-ferrando
AI & ML interests
None yet
Recent Activity
reacted to pavle-scalably's post with 🔥 12 days ago
14 days serving
https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 to production agents on 2x RTX 5090 (vLLM 0.27, TP=2, 262K context, FP8 KV):
28,097 requests, 860.6M prompt tokens, 82.6% prefix-cache hit rate, TTFT p50 0.61 s, 0 engine errors.
The observation: prefix cache, not throughput, decides whether a 27B model keeps up with agents.
Mean request is 30,100 tokens in, 983 out, because every turn resends the whole session. Two flags mattered most: --max-num-seqs 12 (queue p95 went 9.4 s to 233.6 s past that) and --watermark 0.08 (preemptions 29 to 2). And thinking off for tool loops: 917 tokens in 11.7 s vs 11,170 in 144 s, same answer.
Full config and counters: scalably.io/blog/qwen3-8-27b-nvfp4-rtx-5090-production
Next we are preparing an 8x B300 node in an EU data center for open-weight serving. Which models or workloads are underserved for you? liked a model 15 days ago
prism-ml/Ternary-Bonsai-2-27B-gguf liked a model 23 days ago
deepseek-ai/DeepSeek-V4.1-FlashOrganizations
None yet
liked a
model 15 days ago
liked a
model 23 days ago
liked 2
models about 2 months ago
liked a
model 2 months ago
liked 2
models 4 months ago
liked 2
models 5 months ago
liked 2
models 6 months ago
liked a
model 7 months ago
liked 8
models 9 months ago
baichuan-inc/Baichuan-M3-235B
Text Generation • 235B • Updated • 468 • 104
mistralai/Mistral-Large-3-675B-Instruct-2512
Updated • 1.69k • 252
mistralai/Ministral-3-14B-Instruct-2512
14B • Updated • 251k • 330
deepseek-ai/DeepSeek-V3.2-Speciale
Text Generation • 685B • Updated • 3.56k • 726
tencent/Youtu-LLM-2B
Text Generation • 2B • Updated • 11.6k • 231
LiquidAI/LFM2.5-1.2B-Instruct
Text Generation • 1B • Updated • 119k • 673
zai-org/GLM-4.7
Text Generation • 358B • Updated • 102k • • 2.06k
MiniMaxAI/MiniMax-M2.1
Text Generation • 229B • Updated • 13.8k • • 1.36k