How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
# Run inference directly in the terminal:
llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
# Run inference directly in the terminal:
llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
# Run inference directly in the terminal:
./build/bin/llama cli -hf singularity-layer/Singularity-xMESH-0.8B-GGUF:
Use Docker
docker model run hf.co/singularity-layer/Singularity-xMESH-0.8B-GGUF:
Quick Links
Singularity xMESH 0.8B: official GGUF quants

Singularity xMESH 0.8B GGUF

Original model ยท Release article ยท Singularity Layer

Official quantized GGUF builds of Singularity xMESH 0.8B, the x402 Multi-Endpoint Selection Head. Singularity Layer made every file here from its own BF16 release (Singularity-xMESH-0.8B-BF16.gguf, SHA-256 9f4c08f19c4d203059bc74ed2754e800ec472d7276d8ddb2321480e86d392fff) with llama.cpp b11719, then checked each one decision by decision against that BF16 file.

xMESH is a decision model, not a chat model. It answers typed questions (noul yes/no, choice, score) about a state with calibrated probabilities, through llama.cpp's native /v1/systemone endpoint.

At a glance

File Bits Size Same answer as BF16, x402 suite Same answer as BF16, general Peak RAM on CPU Use
Singularity-xMESH-0.8B-Q8_0.gguf 8-bit 787 MB 99.9% 99.6% 1,455 MB Recommended. Best fidelity: effectively the same answers as BF16 at half the size.
Singularity-xMESH-0.8B-Q6_K.gguf 6-bit 667 MB 99.6% 98.1% 1,308 MB High quality. Changed 9 of 1,224 test decisions.
Singularity-xMESH-0.8B-Q5_K_M.gguf 5-bit 613 MB 99.3% 98.1% 1,179 MB Balanced. Changed 12 of 1,224 test decisions.
Singularity-xMESH-0.8B-Q4_K_M.gguf 4-bit 547 MB 98.7% 92.0% 1,087 MB Small. Changed 34 of 1,224; most drift on general decisions.
Singularity-xMESH-0.8B-IQ4_XS.gguf 4-bit (i-quant) 487 MB 98.4% 94.7% 919 MB Smallest. Changed 29 of 1,224: smaller than Q4_K_M and closer to BF16.

The full-precision BF16 file (1,572 MB) stays in the original repository and remains the reference.

Which file should I use?

  • Most people: Q8_0. Half the size of BF16, and it changed 2 of 1,224 test decisions.
  • Less disk and memory, still close: Q6_K or Q5_K_M. They changed 9 and 12 of 1,224 test decisions.
  • Smallest files: IQ4_XS, then Q4_K_M. For small machines and edge boxes. They change noticeably more answers, so keep your own budget and approval checks in front of anything that moves money (you should anyway). If you want a 4-bit file, IQ4_XS is both smaller and closer to BF16 than Q4_K_M in our tests.

Quality against BF16

Agreement with BF16 for each quant File size against changed decisions for each quant

Two suites, run through /v1/systemone with the production profile (-c 2048 -b 2048 -ub 2048 --parallel 1):

  • x402 suite (frozen): the 80-case, 960-decision held-out x402 routing suite used for the original release (12 decisions per case: which stages a flow needs, which service to pick for each stage, whether the flow is valid, whether to ask for clarification).
  • General decisions: 76 requests, 264 decisions written for this check: routing support messages, yes/no checks on listings, logs and payments, planner service picks under budget and network limits, and ticket, language and sentiment classification (including 0 to 3 scores).
File Size Same as BF16, x402 Same as BF16, general Probability drift (x402 / general) Correct, x402 Correct, general Complete flows exact ECE change Brier change
BF16 (reference) 1,572 MB 960/960 264/264 0.0000 / 0.0000 854/960 178/264 22/80 +0.000 / +0.000 +0.000 / +0.000
Q8_0 787 MB 959/960 263/264 0.0034 / 0.0070 853/960 179/264 22/80 +0.008 / +0.001 +0.005 / +0.002
Q6_K 667 MB 956/960 259/264 0.0054 / 0.0210 858/960 183/264 24/80 +0.007 / -0.001 -0.001 / +0.010
Q5_K_M 613 MB 953/960 259/264 0.0085 / 0.0228 851/960 179/264 21/80 +0.007 / +0.004 +0.003 / +0.012
Q4_K_M 547 MB 947/960 243/264 0.0240 / 0.0685 843/960 170/264 18/80 +0.014 / +0.020 +0.014 / +0.057
IQ4_XS 487 MB 945/960 250/264 0.0229 / 0.0529 851/960 171/264 23/80 +0.004 / +0.014 +0.005 / +0.034

Probability drift is the mean absolute difference per option. ECE (10 bins, top answer) and Brier are measured against the labels; a negative change means better calibrated than BF16. The BF16 reference reproduced the published result exactly (854/960 correct, the same as ACCELERATOR_PARITY.json in the original repository), so this harness matches the release evaluation. Quality runs used Apple M4 Metal for every file including the reference, so the comparison isolates the effect of quantization.

Memory and speed on CPU

Peak memory on CPU for each file CPU latency per x402 flow for each file
File Median per x402 flow Median per general request Prompt tokens/s Peak RAM
BF16 (reference) 2.8 s 1.5 s 521 1,925 MB
Q8_0 5.3 s 2.5 s 259 1,455 MB
Q6_K 9.3 s 3.9 s 148 1,308 MB
Q5_K_M 11.4 s 5.3 s 117 1,179 MB
Q4_K_M 9.7 s 4.6 s 147 1,087 MB
IQ4_XS 6.3 s 3.3 s 204 919 MB

Measured with llama.cpp b11719 on an Apple M4 CPU only (-ngl 0, 4 threads), production profile, the first 10 requests of each suite after one warm-up, best of two passes per file. An x402 flow is about 1,400 prompt tokens with 12 decisions; xMESH reads, it does not generate. Peak RAM is the resident memory of the whole llama-server process, including the 2,048-token compute buffer.

Read the speed numbers as relative. On macOS, BF16 matrix maths runs through Apple's Accelerate library, which makes BF16 unusually fast there, so BF16 beats every quant on speed in this table. We have not measured x86 cloud CPUs yet. The test Mac was also running other work. The quants' main win on CPU is memory and download size.

Run with llama.cpp

Requires llama.cpp b11719 or newer (native /v1/systemone support for this architecture).

hf download singularity-layer/Singularity-xMESH-0.8B-GGUF Singularity-xMESH-0.8B-Q8_0.gguf --local-dir .

llama-server \
  -m Singularity-xMESH-0.8B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 2048 -b 2048 -ub 2048 --parallel 1 -ngl 0

curl http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "state": "Translate a supplied French contract into English. Constraints: total_budget_usd=0.05; preferred_network=base.",
    "questions": {
      "pick_service": {"type": "choice", "instructions": "Choose the best candidate by capability, then I/O, constraints, and price.",
        "criteria": {"translate_1": "cap=language translation; in=text; out=text; price=0.02", "ocr_1": "cap=image text extraction; in=image; out=text; price=0.01"}},
      "fits_budget": {"type": "noul", "instructions": "Can the job be done within the total budget?"}
    }
  }'
  • Keep -c 2048 -b 2048 -ub 2048: the validated window is 2,048 tokens, and the default batch of 512 rejects normal multi-question requests.
  • One slot (--parallel 1) is the validated default.
  • -ngl 0 is the portable CPU setting. Apple Metal (-ngl 99) is how the quality table was produced.

Ollama and LM Studio

Not supported yet. xMESH needs llama.cpp's /v1/systemone decision endpoint and its Clef decision head, which Ollama and LM Studio do not expose. In our test, Ollama 0.24.0 refused to load the file (unable to load model). Use llama-server from llama.cpp b11719 or newer, or run xMESH on Singularity Layer where it is already hosted.

Why these files differ from a stock quant

A stock llama-quantize of xMESH (for example the Q8_0 file in the original repository, or a community quant made the standard way) crashes on CPU in llama.cpp b11719 with a segmentation fault at start-up. When the token-embedding tensor uses a type that llama.cpp's CPU backend repacks for faster matrix multiplication (on ARM: Q4_0, Q4_K, Q5_K, Q6_K, Q8_0, IQ4_NL; on x86: Q4_0, Q4_K, IQ4_NL), the decision graph reads that repacked tensor and the scheduler hits a missing copy function.

These builds avoid the problem in the file itself, so no extra runtime flag is needed:

  • Token embeddings use a type no CPU repacks: Q8_0 q5_1, Q6_K q5_1, Q5_K_M q5_1, Q4_K_M q5_0, IQ4_XS iq4_xs.
  • The Clef decision head (dec.*, decision.*, token_types, about 52 MB) stays in BF16 in every file.
  • Everything else uses the named quant type. No importance matrix was used.

Note: the Q8_0 here is a different file from the Q8_0 in the original repository, which is a stock quant kept there for Metal only. If you run a stock quant anyway, --no-repack avoids the crash at some speed cost.

Validation

  • Every file loads and answers /v1/systemone on CPU (-ngl 0) and on Apple Metal (-ngl 99) with llama.cpp b11719 (official macOS arm64 release). A stock Q8_0 made from the same BF16 with the same tool is byte-identical to the Q8_0 in the original repository and reproduces the CPU crash (exit 139); these files do not.
  • CPU quality against BF16 on CPU, full suites:
File Same as BF16 (CPU), x402 Same as BF16 (CPU), general
Q8_0 959/960 264/264
Q6_K 955/960 255/264
Q5_K_M 952/960 257/264
Q4_K_M 946/960 241/264
IQ4_XS 944/960 250/264
  • All numbers, per suite and per file, are in QUANT_EVAL.json.
  • Not yet verified: x86 Linux CPUs and NVIDIA through Vulkan with these specific files. The fix applies there by construction (the embedding types used are not repacked on x86 either), but so far we have run them on Apple silicon only.

Checksums

File Bytes SHA-256
Singularity-xMESH-0.8B-Q8_0.gguf 787,166,080 62ecc4f4e65b6b3bdbe10ac11f3833fcbcfebb4a36568d2b0fcc82eae2f0d9a9
Singularity-xMESH-0.8B-Q6_K.gguf 666,649,984 0604ad977cdc84f9afadcac1553114a88e64e66e77300c166201afcfece9d516
Singularity-xMESH-0.8B-Q5_K_M.gguf 613,233,536 701045897a952e2572d9044e1537bd3f0bff02f69aa9ec1713a3e214b2bec213
Singularity-xMESH-0.8B-Q4_K_M.gguf 547,066,752 0698d28944ab0928c1de8ef3cb0131e831f843c89b463bead641c7356f12d17b
Singularity-xMESH-0.8B-IQ4_XS.gguf 487,091,072 ace80d58e858dc509d9933682c6d49592186b1b60323be71c1c29283285cfde6

Also in SHA256SUMS: shasum -a 256 -c SHA256SUMS.

Made with llama-quantize from llama.cpp b11719 (5aeac37a35b44fde4bdffaec9159f2fcb253b622). Recipe, for example Q8_0:

llama-quantize \
  --tensor-type '^dec\.=bf16' --tensor-type '^decision\.=bf16' --tensor-type 'token_types=bf16' \
  --token-embedding-type q5_1 \
  Singularity-xMESH-0.8B-BF16.gguf Singularity-xMESH-0.8B-Q8_0.gguf Q8_0

Intended use and limits

Same as the original model: xMESH is an advisory router specialised for x402 service selection, with a 2,048-token contract. It must not receive wallet secrets, construct signed payments, bypass approval, invent endpoints, or execute arbitrary URLs. Restrict choices to retrieved candidate IDs and enforce budgets, networks, schemas, and approval rules in your own code. Quantization adds the drift shown above on top of the original model's limits; see the original model card.

License

Apache-2.0, the same as the original model. Qwen and Cloudflare Clef attribution is in ATTRIBUTION.md, THIRD_PARTY_NOTICES.md, NOTICE.cloudflare-clef, and LICENSE.cloudflare-clef.

Downloads last month
-
GGUF
Model size
0.8B params
Architecture
clef
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for singularity-layer/Singularity-xMESH-0.8B-GGUF

Quantized
(2)
this model