Instructions to use tchbcb/samai-9b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use tchbcb/samai-9b with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf tchbcb/samai-9b:Q4_K_M # Run inference directly in the terminal: llama cli -hf tchbcb/samai-9b:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf tchbcb/samai-9b:Q4_K_M # Run inference directly in the terminal: llama cli -hf tchbcb/samai-9b:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf tchbcb/samai-9b:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf tchbcb/samai-9b:Q4_K_M
Use Docker
docker model run hf.co/tchbcb/samai-9b:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use tchbcb/samai-9b with Ollama:
ollama run hf.co/tchbcb/samai-9b:Q4_K_M
- Unsloth Desktop
- Pi
How to use tchbcb/samai-9b with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tchbcb/samai-9b:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "tchbcb/samai-9b:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use tchbcb/samai-9b with Docker Model Runner:
docker model run hf.co/tchbcb/samai-9b:Q4_K_M
- Lemonade
How to use tchbcb/samai-9b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull tchbcb/samai-9b:Q4_K_M
Run and chat with the model
lemonade run user.samai-9b-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use tchbcb/samai-9b with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tchbcb/samai-9b:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default tchbcb/samai-9b:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use tchbcb/samai-9b with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf tchbcb/samai-9b:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "tchbcb/samai-9b:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
R10 probe artifacts: mixbit_build.log
Browse files
artifacts/m15_reports/mixbit_build.log
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[00:23:56] === s8_mixbit_build start ===
|
| 2 |
+
[00:23:56] override-tensor support: False
|
| 3 |
+
[00:23:56] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G)
|
| 4 |
+
[00:23:56] WARN: no --override-tensor, 4x8 档 attn 保持 Q4 (退化为仅 embed/out 升档)
|
| 5 |
+
[00:23:56] WARN: no --override-tensor, attn 保持 Q6_K (仍远优于 Q4_K_M)
|
| 6 |
+
[00:23:56] [mixbit_4x8] quantize start: /tmp/lcfin/llama-quantize /tmp/k8b/m13_r8_f16.gguf /tmp/k8b/m13_r8_mixbit_4x8.gguf Q4_K_M --token-embedding-type Q8_0 --output-tensor-type Q8_0
|
| 7 |
+
[00:26:07] === s8_mixbit_build start ===
|
| 8 |
+
[00:26:07] override-tensor support: False
|
| 9 |
+
[00:26:07] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G)
|
| 10 |
+
[00:26:07] WARN: no --override-tensor, 4x8 档 attn 保持 Q4 (退化为仅 embed/out 升档)
|
| 11 |
+
[00:26:07] WARN: no --override-tensor, attn 保持 Q6_K (仍远优于 Q4_K_M)
|
| 12 |
+
[00:26:07] [mixbit_4x8] quantize start: /tmp/lcfin/llama-quantize /tmp/k8b/m13_r8_f16.gguf /tmp/k8b/m13_r8_mixbit_4x8.gguf Q4_K_M --token-embedding-type Q8_0 --output-tensor-type Q8_0
|
| 13 |
+
[00:46:23] === s8_mixbit_build v2 start ===
|
| 14 |
+
[00:46:23] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 15 |
+
[00:46:23] [mixbit_4x8] start
|
| 16 |
+
[00:46:23] [mixbit_4x8] done 1s ok=False size=N/A tail=init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 17 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 18 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 19 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 20 |
+
|
| 21 |
+
[00:46:23] [mixbit-high] start
|
| 22 |
+
[00:46:24] [mixbit-high] done 1s ok=False size=N/A tail=init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 23 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 24 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 25 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 26 |
+
|
| 27 |
+
[00:46:24] [q8km] start
|
| 28 |
+
[00:46:24] [q8km] done 1s ok=False size=N/A tail=ggml_cuda_init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 29 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 30 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 31 |
+
llama_quantize: invalid ftype 'Q8_K_M'
|
| 32 |
+
|
| 33 |
+
[00:46:24] MIXBIT_BUILD_DONE {"f16": "/tmp/k8b/m13_r8_f16.gguf", "products": {"mixbit_4x8": {"out": "/tmp/k8b/m13_r8_mixbit_4x8.gguf", "ok": false, "size_g": 0}, "mixbit-high": {"out": "/tmp/k8b/m13_r8_mixbit.gguf", "ok": false, "size_g": 0}, "q8km": {"out": "/tmp/k8b/m13_r8_q8_k_m.gguf", "ok": false, "size_g": 0}}, "done_at": "2026-09-23 00:46:24"}
|
| 34 |
+
[00:55:32] already done
|
| 35 |
+
[01:10:36] === s8_mixbit_build v2 start ===
|
| 36 |
+
[01:10:36] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 37 |
+
[01:10:36] [mixbit_4x8] start
|
| 38 |
+
[01:10:36] [mixbit_4x8] done 1s ok=False size=N/A tail=init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 39 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 40 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 41 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 42 |
+
|
| 43 |
+
[01:10:36] [mixbit-high] start
|
| 44 |
+
[01:10:37] [mixbit-high] done 1s ok=False size=N/A tail=init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 45 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 46 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 47 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 48 |
+
|
| 49 |
+
[01:10:37] [q8km] start
|
| 50 |
+
[01:10:37] [q8km] done 0s ok=False size=N/A tail=ggml_cuda_init: found 2 CUDA devices (Total VRAM: 29823 MiB):
|
| 51 |
+
Device 0: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 52 |
+
Device 1: Tesla T4, compute capability 7.5, VMM: yes, VRAM: 14911 MiB
|
| 53 |
+
llama_quantize: invalid ftype 'Q8_K_M'
|
| 54 |
+
|
| 55 |
+
[01:10:37] MIXBIT_BUILD_DONE {"f16": "/tmp/k8b/m13_r8_f16.gguf", "products": {"mixbit_4x8": {"out": "/tmp/k8b/m13_r8_mixbit_4x8.gguf", "ok": false, "size_g": 0}, "mixbit-high": {"out": "/tmp/k8b/m13_r8_mixbit.gguf", "ok": false, "size_g": 0}, "q8km": {"out": "/tmp/k8b/m13_r8_q8_k_m.gguf", "ok": false, "size_g": 0}}, "done_at": "2026-09-23 01:10:37"}
|
| 56 |
+
[01:18:33] === s8_mixbit_build v2 start ===
|
| 57 |
+
[01:18:33] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 58 |
+
[01:18:33] [mixbit_4x8] start
|
| 59 |
+
[01:18:33] [mixbit_4x8] done 0s ok=False size=N/A tail=ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
|
| 60 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 61 |
+
|
| 62 |
+
[01:18:33] [mixbit-high] start
|
| 63 |
+
[01:18:33] [mixbit-high] done 0s ok=False size=N/A tail=ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
|
| 64 |
+
llama_quantize: invalid nthread '--token-embedding-type' (stoi)
|
| 65 |
+
|
| 66 |
+
[01:18:33] [q8km] start
|
| 67 |
+
[01:18:33] [q8km] done 0s ok=False size=N/A tail=ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
|
| 68 |
+
llama_quantize: invalid ftype 'Q8_K_M'
|
| 69 |
+
|
| 70 |
+
[01:18:33] MIXBIT_BUILD_DONE {"f16": "/tmp/k8b/m13_r8_f16.gguf", "products": {"mixbit_4x8": {"out": "/tmp/k8b/m13_r8_mixbit_4x8.gguf", "ok": false, "size_g": 0}, "mixbit-high": {"out": "/tmp/k8b/m13_r8_mixbit.gguf", "ok": false, "size_g": 0}, "q8km": {"out": "/tmp/k8b/m13_r8_q8_k_m.gguf", "ok": false, "size_g": 0}}, "done_at": "2026-09-23 01:18:33"}
|
| 71 |
+
[01:27:57] === s8_mixbit_build v2 start ===
|
| 72 |
+
[01:27:57] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 73 |
+
[01:43:17] === s8_mixbit_build v2 start ===
|
| 74 |
+
[01:43:17] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 75 |
+
[01:43:17] [mixbit_4x8] start
|
| 76 |
+
[01:43:17] [mixbit_4x8] done 0s ok=False size=N/A tail=/bin/sh: 1: --token-embedding-type: not found
|
| 77 |
+
|
| 78 |
+
[01:43:17] [mixbit-high] start
|
| 79 |
+
[01:43:17] [mixbit-high] done 0s ok=False size=N/A tail=/bin/sh: 1: --token-embedding-type: not found
|
| 80 |
+
|
| 81 |
+
[01:43:17] [q8km] start
|
| 82 |
+
[01:44:51] [q8km] done 95s ok=True size=9.53G tail= 1], type = f32, size = 0.016 MiB
|
| 83 |
+
llama_model_quantize_impl: model size = 17080.05 MiB (16.00 BPW)
|
| 84 |
+
llama_model_quantize_impl: quant size = 9075.67 MiB (8.50 BPW)
|
| 85 |
+
|
| 86 |
+
llama_quantize: quantize time = 92055.53 ms
|
| 87 |
+
llama_quantize: total time = 92055.53 ms
|
| 88 |
+
|
| 89 |
+
[01:44:52] MIXBIT_BUILD_DONE {"f16": "/tmp/k8b/m13_r8_f16.gguf", "products": {"mixbit_4x8": {"out": "/tmp/k8b/m13_r8_mixbit_4x8.gguf", "ok": false, "size_g": 0}, "mixbit-high": {"out": "/tmp/k8b/m13_r8_mixbit.gguf", "ok": false, "size_g": 0}, "q8km": {"out": "/tmp/k8b/m13_r8_q8_k_m.gguf", "ok": true, "size_g": 9.53}}, "done_at": "2026-09-23 01:44:52"}
|
| 90 |
+
[01:48:23] === s8_mixbit_build v2 start ===
|
| 91 |
+
[01:48:23] f16 source: /tmp/k8b/m13_r8_f16.gguf (17.92G) surgery_bak=True
|
| 92 |
+
[01:48:23] [mixbit_4x8] start
|
| 93 |
+
[01:53:57] [mixbit_4x8] done 334s ok=True size=6.91G tail=], type = f32, size = 0.016 MiB
|
| 94 |
+
llama_model_quantize_impl: model size = 17080.05 MiB (16.00 BPW)
|
| 95 |
+
llama_model_quantize_impl: quant size = 6582.67 MiB (6.17 BPW)
|
| 96 |
+
|
| 97 |
+
llama_quantize: quantize time = 333528.82 ms
|
| 98 |
+
llama_quantize: total time = 333528.82 ms
|
| 99 |
+
|
| 100 |
+
[01:53:57] [mixbit-high] start
|
| 101 |
+
[01:56:57] [mixbit-high] done 180s ok=True size=8.16G tail=], type = f32, size = 0.016 MiB
|
| 102 |
+
llama_model_quantize_impl: model size = 17080.05 MiB (16.00 BPW)
|
| 103 |
+
llama_model_quantize_impl: quant size = 7772.22 MiB (7.28 BPW)
|
| 104 |
+
|
| 105 |
+
llama_quantize: quantize time = 179602.06 ms
|
| 106 |
+
llama_quantize: total time = 179602.06 ms
|
| 107 |
+
|
| 108 |
+
[01:56:57] MIXBIT_BUILD_DONE {"f16": "/tmp/k8b/m13_r8_f16.gguf", "products": {"mixbit_4x8": {"out": "/tmp/k8b/m13_r8_mixbit_4x8.gguf", "ok": true, "size_g": 6.91}, "mixbit-high": {"out": "/tmp/k8b/m13_r8_mixbit.gguf", "ok": true, "size_g": 8.16}}, "done_at": "2026-09-23 01:56:57"}
|