Instructions to use Xylolabs/Cosmos3-Nano-Text-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama cli -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Use Docker

docker model run hf.co/Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Llama new
LM Studio
Jan

vLLM

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Xylolabs/Cosmos3-Nano-Text-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Xylolabs/Cosmos3-Nano-Text-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Ollama
How to use Xylolabs/Cosmos3-Nano-Text-GGUF with Ollama:
```
ollama run hf.co/Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS
```
Unsloth Desktop

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Xylolabs/Cosmos3-Nano-Text-GGUF with Docker Model Runner:
```
docker model run hf.co/Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS
```

Lemonade

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Run and chat with the model

lemonade run user.Cosmos3-Nano-Text-GGUF-IQ4_XS

List all available models

lemonade list

Hermes Agent

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Xylolabs/Cosmos3-Nano-Text-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Xylolabs/Cosmos3-Nano-Text-GGUF:IQ4_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Cosmos3-Nano text-understanding GGUF

gguf · llama-cpp · quantized · cosmos3 · text-generation

Five preserved GGUF quantizations of the text-understanding tower in NVIDIA Cosmos3-Nano, extracted into Qwen3-compatible tensor names and quantized by Xylolabs with llama.cpp. Choose IQ4_XS with or without the existing importance matrix, Q5_K_M, Q6_K, or Q8_0.

This export needs about 8.70 GB of device memory for its largest GGUF weight choice alone, plus room for the KV cache and buffers.

Formats

Each alternative contains all 399 tensors of the 36-layer text-understanding decoder, its tokenizer and original chat template. Generation experts and image/video/audio/action generation components are omitted; this archive covers text input and text output.

File	Download GB	Average bits/weight
`Cosmos3-Nano-Text-IQ4_XS-imat.gguf`	4.562	4.450
`Cosmos3-Nano-Text-IQ4_XS.gguf`	4.593	4.481
`Cosmos3-Nano-Text-Q5_K_M.gguf`	5.851	5.709
`Cosmos3-Nano-Text-Q6_K.gguf`	6.726	6.563
`Cosmos3-Nano-Text-Q8_0.gguf`	8.710	8.501

Average bits include embedding, output head, norms and every stored tensor. Actual tensor types are:

Choice	Tensor types and counts
`IQ4_XS-imat`	F32: 145, IQ4_XS: 217, Q5_K: 36, Q6_K: 1
`IQ4_XS`	F32: 145, IQ4_XS: 213, Q5_K: 40, Q6_K: 1
`Q5_K_M`	F32: 145, Q5_K: 217, Q6_K: 37
`Q6_K`	F32: 145, Q6_K: 254
`Q8_0`	F32: 145, Q8_0: 254

The -imat file preserves the existing importance-matrix result. Historical relative calibration filenames are metadata labels; no calibration dataset is included. No new training, calibration or quantization was performed for publication.

Quickstart

With a llama.cpp build, download one file and run:

llama-cli -m Cosmos3-Nano-Text-Q5_K_M.gguf -c 2048 -n 32 --temp 0 --jinja \
  --single-turn -p "Name the capital of France. Reply with only the city."

This command is checked with llama.cpp v0.9.0. Use one quantization at a time. Context is explicitly limited to 2,048 tokens in this example; original context metadata is unchanged. KV and working buffers need additional memory.

Evaluation

All five files pass the relative reference check. Stored BF16 and current Q4_K_M answer 31/33. Plain IQ4_XS answers 29/33 with one counted loss; the other four answer 31/33 with no counted loss. Broken answers: zero for all five.

Choice	Mean KL	KL / Q4_K_M	Right	Counted lost/broken
`IQ4_XS-imat`	0.022186	0.658×	31/33	0
`IQ4_XS`	0.038555	1.144×	29/33	1
`Q5_K_M`	0.009037	0.268×	31/33	0
`Q6_K`	0.004359	0.129×	31/33	0
`Q8_0`	0.001276	0.038×	31/33	0

The 16 matched 1,024-token sequences cover 16,272 scored positions across English/Korean plain/chat text groups. KL is measured against stored BF16 logits. Current Q4_K_M gives 0.033699. All ratios are below 1.5×, with at most one more counted lost/broken answer than Q4_K_M. Checks use an 8,192-token context and retain every original 33-prompt token sequence, stop ID and generation budget. Plain IQ4_XS loses ko-fact-hangul; BF16 near ties (top-two gap below 1.0 at the first divergent token) are excluded from counted losses. Broken answers count where BF16 is right.

Seven extended-budget reasoning rechecks were also run at 4,096 tokens. This original template has no thinking switch. Their BF16 reference reuses the corresponding completed same-input responses from the stored 33-prompt pass: all had already stopped before its 160-token limit. Under the thinking-format policy these rechecks are not applicable and pardon no losses.

The exact Q5_K_M CLI example loads the released file and completes one text query. These checks concern stored texts and short prompts; an integrated text application needs its own task evaluation. Speed is not reported here.

License

OpenMDW-1.1 is the NVIDIA Cosmos3-Nano license. NVIDIA identifies Qwen3-VL-8B as the initialization source in Cosmos 3, Table 2. The inherited Qwen Team / Alibaba Cloud Apache 2.0 terms are also retained in full, with origin attribution in NOTICE. The Qwen3-VL-8B model card declares Apache 2.0. Both sets of applicable notices accompany this derivative. Provided as is, without warranty or upstream endorsement.

Citation

Cite the original Cosmos3-Nano model card and its linked Cosmos 3 technical report.

Downloads last month: 300

GGUF

Model size

8B params

Architecture

qwen3

Hardware compatibility

4-bit

5-bit

6-bit

8-bit

Model tree for Xylolabs/Cosmos3-Nano-Text-GGUF

Base model

nvidia/Cosmos3-Nano

Quantized

(14)

this model

Collection including Xylolabs/Cosmos3-Nano-Text-GGUF

Cosmos 3 GGUF text references

Collection

Preserved Cosmos3 text-understanding GGUF references with matched checks. Made by Xylolabs. Not affiliated with or endorsed by Tenstorrent. • 1 item • Updated about 20 hours ago