Instructions to use singularity-layer/Singularity-xMESH-0.8B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with Ollama:
ollama run hf.co/singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with Docker Model Runner:
docker model run hf.co/singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
- Lemonade
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Singularity-xMESH-0.8B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use singularity-layer/Singularity-xMESH-0.8B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "singularity-layer/Singularity-xMESH-0.8B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:# Run inference directly in the terminal:
llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:# Run inference directly in the terminal:
./build/bin/llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:Use Docker
docker model run hf.co/singularity-layer/Singularity-xMESH-0.8B-GGUF:
Singularity xMESH 0.8B GGUF
Original model ยท Release article ยท Singularity Layer
Official quantized GGUF builds of Singularity xMESH 0.8B, the x402 Multi-Endpoint Selection Head. Singularity Layer made every file here from its own BF16 release (Singularity-xMESH-0.8B-BF16.gguf, SHA-256 9f4c08f19c4d203059bc74ed2754e800ec472d7276d8ddb2321480e86d392fff) with llama.cpp b11719, then checked each one decision by decision against that BF16 file.
xMESH is a decision model, not a chat model. It answers typed questions (noul yes/no, choice, score) about a state with calibrated probabilities, through llama.cpp's native /v1/systemone endpoint.
At a glance
| File | Bits | Size | Same answer as BF16, x402 suite | Same answer as BF16, general | Peak RAM on CPU | Use |
|---|---|---|---|---|---|---|
Singularity-xMESH-0.8B-Q8_0.gguf |
8-bit | 787 MB | 99.9% | 99.6% | 1,455 MB | Recommended. Best fidelity: effectively the same answers as BF16 at half the size. |
Singularity-xMESH-0.8B-Q6_K.gguf |
6-bit | 667 MB | 99.6% | 98.1% | 1,308 MB | High quality. Changed 9 of 1,224 test decisions. |
Singularity-xMESH-0.8B-Q5_K_M.gguf |
5-bit | 613 MB | 99.3% | 98.1% | 1,179 MB | Balanced. Changed 12 of 1,224 test decisions. |
Singularity-xMESH-0.8B-Q4_K_M.gguf |
4-bit | 547 MB | 98.7% | 92.0% | 1,087 MB | Small. Changed 34 of 1,224; most drift on general decisions. |
Singularity-xMESH-0.8B-IQ4_XS.gguf |
4-bit (i-quant) | 487 MB | 98.4% | 94.7% | 919 MB | Smallest. Changed 29 of 1,224: smaller than Q4_K_M and closer to BF16. |
The full-precision BF16 file (1,572 MB) stays in the original repository and remains the reference.
Which file should I use?
- Most people:
Q8_0. Half the size of BF16, and it changed 2 of 1,224 test decisions. - Less disk and memory, still close:
Q6_KorQ5_K_M. They changed 9 and 12 of 1,224 test decisions. - Smallest files:
IQ4_XS, thenQ4_K_M. For small machines and edge boxes. They change noticeably more answers, so keep your own budget and approval checks in front of anything that moves money (you should anyway). If you want a 4-bit file, IQ4_XS is both smaller and closer to BF16 than Q4_K_M in our tests.
Quality against BF16
Two suites, run through /v1/systemone with the production profile (-c 2048 -b 2048 -ub 2048 --parallel 1):
- x402 suite (frozen): the 80-case, 960-decision held-out x402 routing suite used for the original release (12 decisions per case: which stages a flow needs, which service to pick for each stage, whether the flow is valid, whether to ask for clarification).
- General decisions: 76 requests, 264 decisions written for this check: routing support messages, yes/no checks on listings, logs and payments, planner service picks under budget and network limits, and ticket, language and sentiment classification (including 0 to 3 scores).
| File | Size | Same as BF16, x402 | Same as BF16, general | Probability drift (x402 / general) | Correct, x402 | Correct, general | Complete flows exact | ECE change | Brier change |
|---|---|---|---|---|---|---|---|---|---|
| BF16 (reference) | 1,572 MB | 960/960 | 264/264 | 0.0000 / 0.0000 | 854/960 | 178/264 | 22/80 | +0.000 / +0.000 | +0.000 / +0.000 |
| Q8_0 | 787 MB | 959/960 | 263/264 | 0.0034 / 0.0070 | 853/960 | 179/264 | 22/80 | +0.008 / +0.001 | +0.005 / +0.002 |
| Q6_K | 667 MB | 956/960 | 259/264 | 0.0054 / 0.0210 | 858/960 | 183/264 | 24/80 | +0.007 / -0.001 | -0.001 / +0.010 |
| Q5_K_M | 613 MB | 953/960 | 259/264 | 0.0085 / 0.0228 | 851/960 | 179/264 | 21/80 | +0.007 / +0.004 | +0.003 / +0.012 |
| Q4_K_M | 547 MB | 947/960 | 243/264 | 0.0240 / 0.0685 | 843/960 | 170/264 | 18/80 | +0.014 / +0.020 | +0.014 / +0.057 |
| IQ4_XS | 487 MB | 945/960 | 250/264 | 0.0229 / 0.0529 | 851/960 | 171/264 | 23/80 | +0.004 / +0.014 | +0.005 / +0.034 |
Probability drift is the mean absolute difference per option. ECE (10 bins, top answer) and Brier are measured against the labels; a negative change means better calibrated than BF16. The BF16 reference reproduced the published result exactly (854/960 correct, the same as ACCELERATOR_PARITY.json in the original repository), so this harness matches the release evaluation. Quality runs used Apple M4 Metal for every file including the reference, so the comparison isolates the effect of quantization.
Memory and speed on CPU
| File | Median per x402 flow | Median per general request | Prompt tokens/s | Peak RAM |
|---|---|---|---|---|
| BF16 (reference) | 2.8 s | 1.5 s | 521 | 1,925 MB |
| Q8_0 | 5.3 s | 2.5 s | 259 | 1,455 MB |
| Q6_K | 9.3 s | 3.9 s | 148 | 1,308 MB |
| Q5_K_M | 11.4 s | 5.3 s | 117 | 1,179 MB |
| Q4_K_M | 9.7 s | 4.6 s | 147 | 1,087 MB |
| IQ4_XS | 6.3 s | 3.3 s | 204 | 919 MB |
Measured with llama.cpp b11719 on an Apple M4 CPU only (-ngl 0, 4 threads), production profile, the first 10 requests of each suite after one warm-up, best of two passes per file. An x402 flow is about 1,400 prompt tokens with 12 decisions; xMESH reads, it does not generate. Peak RAM is the resident memory of the whole llama-server process, including the 2,048-token compute buffer.
Read the speed numbers as relative. On macOS, BF16 matrix maths runs through Apple's Accelerate library, which makes BF16 unusually fast there, so BF16 beats every quant on speed in this table. We have not measured x86 cloud CPUs yet. The test Mac was also running other work. The quants' main win on CPU is memory and download size.
Run with llama.cpp
Requires llama.cpp b11719 or newer (native /v1/systemone support for this architecture).
hf download singularity-layer/Singularity-xMESH-0.8B-GGUF Singularity-xMESH-0.8B-Q8_0.gguf --local-dir .
llama-server \
-m Singularity-xMESH-0.8B-Q8_0.gguf \
--host 127.0.0.1 --port 8080 \
-c 2048 -b 2048 -ub 2048 --parallel 1 -ngl 0
curl http://127.0.0.1:8080/v1/systemone \
-H 'Content-Type: application/json' \
-d '{
"state": "Translate a supplied French contract into English. Constraints: total_budget_usd=0.05; preferred_network=base.",
"questions": {
"pick_service": {"type": "choice", "instructions": "Choose the best candidate by capability, then I/O, constraints, and price.",
"criteria": {"translate_1": "cap=language translation; in=text; out=text; price=0.02", "ocr_1": "cap=image text extraction; in=image; out=text; price=0.01"}},
"fits_budget": {"type": "noul", "instructions": "Can the job be done within the total budget?"}
}
}'
- Keep
-c 2048 -b 2048 -ub 2048: the validated window is 2,048 tokens, and the default batch of 512 rejects normal multi-question requests. - One slot (
--parallel 1) is the validated default. -ngl 0is the portable CPU setting. Apple Metal (-ngl 99) is how the quality table was produced.
Ollama and LM Studio
Not supported yet. xMESH needs llama.cpp's /v1/systemone decision endpoint and its Clef decision head, which Ollama and LM Studio do not expose. In our test, Ollama 0.24.0 refused to load the file (unable to load model). Use llama-server from llama.cpp b11719 or newer, or run xMESH on Singularity Layer where it is already hosted.
Why these files differ from a stock quant
A stock llama-quantize of xMESH (for example the Q8_0 file in the original repository, or a community quant made the standard way) crashes on CPU in llama.cpp b11719 with a segmentation fault at start-up. When the token-embedding tensor uses a type that llama.cpp's CPU backend repacks for faster matrix multiplication (on ARM: Q4_0, Q4_K, Q5_K, Q6_K, Q8_0, IQ4_NL; on x86: Q4_0, Q4_K, IQ4_NL), the decision graph reads that repacked tensor and the scheduler hits a missing copy function.
These builds avoid the problem in the file itself, so no extra runtime flag is needed:
- Token embeddings use a type no CPU repacks: Q8_0
q5_1, Q6_Kq5_1, Q5_K_Mq5_1, Q4_K_Mq5_0, IQ4_XSiq4_xs. - The Clef decision head (
dec.*,decision.*,token_types, about 52 MB) stays in BF16 in every file. - Everything else uses the named quant type. No importance matrix was used.
Note: the Q8_0 here is a different file from the Q8_0 in the original repository, which is a stock quant kept there for Metal only. If you run a stock quant anyway, --no-repack avoids the crash at some speed cost.
Validation
- Every file loads and answers
/v1/systemoneon CPU (-ngl 0) and on Apple Metal (-ngl 99) with llama.cppb11719(official macOS arm64 release). A stock Q8_0 made from the same BF16 with the same tool is byte-identical to theQ8_0in the original repository and reproduces the CPU crash (exit 139); these files do not. - CPU quality against BF16 on CPU, full suites:
| File | Same as BF16 (CPU), x402 | Same as BF16 (CPU), general |
|---|---|---|
| Q8_0 | 959/960 | 264/264 |
| Q6_K | 955/960 | 255/264 |
| Q5_K_M | 952/960 | 257/264 |
| Q4_K_M | 946/960 | 241/264 |
| IQ4_XS | 944/960 | 250/264 |
- All numbers, per suite and per file, are in
QUANT_EVAL.json. - Not yet verified: x86 Linux CPUs and NVIDIA through Vulkan with these specific files. The fix applies there by construction (the embedding types used are not repacked on x86 either), but so far we have run them on Apple silicon only.
Checksums
| File | Bytes | SHA-256 |
|---|---|---|
Singularity-xMESH-0.8B-Q8_0.gguf |
787,166,080 | 62ecc4f4e65b6b3bdbe10ac11f3833fcbcfebb4a36568d2b0fcc82eae2f0d9a9 |
Singularity-xMESH-0.8B-Q6_K.gguf |
666,649,984 | 0604ad977cdc84f9afadcac1553114a88e64e66e77300c166201afcfece9d516 |
Singularity-xMESH-0.8B-Q5_K_M.gguf |
613,233,536 | 701045897a952e2572d9044e1537bd3f0bff02f69aa9ec1713a3e214b2bec213 |
Singularity-xMESH-0.8B-Q4_K_M.gguf |
547,066,752 | 0698d28944ab0928c1de8ef3cb0131e831f843c89b463bead641c7356f12d17b |
Singularity-xMESH-0.8B-IQ4_XS.gguf |
487,091,072 | ace80d58e858dc509d9933682c6d49592186b1b60323be71c1c29283285cfde6 |
Also in SHA256SUMS: shasum -a 256 -c SHA256SUMS.
Made with llama-quantize from llama.cpp b11719 (5aeac37a35b44fde4bdffaec9159f2fcb253b622). Recipe, for example Q8_0:
llama-quantize \
--tensor-type '^dec\.=bf16' --tensor-type '^decision\.=bf16' --tensor-type 'token_types=bf16' \
--token-embedding-type q5_1 \
Singularity-xMESH-0.8B-BF16.gguf Singularity-xMESH-0.8B-Q8_0.gguf Q8_0
Intended use and limits
Same as the original model: xMESH is an advisory router specialised for x402 service selection, with a 2,048-token contract. It must not receive wallet secrets, construct signed payments, bypass approval, invent endpoints, or execute arbitrary URLs. Restrict choices to retrieved candidate IDs and enforce budgets, networks, schemas, and approval rules in your own code. Quantization adds the drift shown above on top of the original model's limits; see the original model card.
License
Apache-2.0, the same as the original model. Qwen and Cloudflare Clef attribution is in ATTRIBUTION.md, THIRD_PARTY_NOTICES.md, NOTICE.cloudflare-clef, and LICENSE.cloudflare-clef.
- Downloads last month
- -
4-bit
5-bit
6-bit
8-bit
Model tree for singularity-layer/Singularity-xMESH-0.8B-GGUF
Base model
Qwen/Qwen3.5-0.8B-Base
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:# Run inference directly in the terminal: llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF: