Instructions to use NagusameCS/ultratensor-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use NagusameCS/ultratensor-models with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: llama cli -hf NagusameCS/ultratensor-models
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: llama cli -hf NagusameCS/ultratensor-models
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: ./build/bin/llama cli -hf NagusameCS/ultratensor-models
Use Docker
docker model run hf.co/NagusameCS/ultratensor-models
- Llama new
- LM Studio
- Jan
- Ollama
How to use NagusameCS/ultratensor-models with Ollama:
ollama run hf.co/NagusameCS/ultratensor-models
- Unsloth Desktop
- Docker Model Runner
How to use NagusameCS/ultratensor-models with Docker Model Runner:
docker model run hf.co/NagusameCS/ultratensor-models
- Lemonade
How to use NagusameCS/ultratensor-models with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull NagusameCS/ultratensor-models
Run and chat with the model
lemonade run user.ultratensor-models-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
Upload model_catalog.json with huggingface_hub
Browse files- model_catalog.json +11 -1
model_catalog.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"note": "Catalog of UltraTensor spliced DeepSeek-V4-Coder GGUFs. Built with ultratensor/gguf_keep.py from the public BatiAI DeepSeek-V4-Pro Q3_K_M GGUF.
|
| 3 |
"serve_flags_common": "--no-op-offload is mandatory on every Q3_K tier (the engine's default op-offload corrupts CPU-resident Q3_K dequants and crashes at >=36 prompt tokens).",
|
| 4 |
"models": [
|
| 5 |
{
|
|
@@ -71,6 +71,16 @@
|
|
| 71 |
"desc": "IQ2_XS requant of keep8u. Fastest per GiB on the ladder.",
|
| 72 |
"serve": "Same as keep16u-iq2xs.",
|
| 73 |
"tiers": ["gpu"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
}
|
| 75 |
]
|
| 76 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"note": "Catalog of UltraTensor spliced DeepSeek-V4-Coder GGUFs. Built with ultratensor/gguf_keep.py from the public BatiAI DeepSeek-V4-Pro Q3_K_M GGUF. Licensing: upstream deepseek-ai/DeepSeek-V4-Pro is MIT and the BatiAI GGUF inherits MIT (verified 2026-08-21); redistributable with attribution.",
|
| 3 |
"serve_flags_common": "--no-op-offload is mandatory on every Q3_K tier (the engine's default op-offload corrupts CPU-resident Q3_K dequants and crashes at >=36 prompt tokens).",
|
| 4 |
"models": [
|
| 5 |
{
|
|
|
|
| 71 |
"desc": "IQ2_XS requant of keep8u. Fastest per GiB on the ladder.",
|
| 72 |
"serve": "Same as keep16u-iq2xs.",
|
| 73 |
"tiers": ["gpu"]
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"id": "keep64-iq2xs",
|
| 77 |
+
"file": "DeepSeek-V4-Coder-keep64-iq2xxs.gguf",
|
| 78 |
+
"bytes": 102376842624,
|
| 79 |
+
"quant": "IQ2_XS",
|
| 80 |
+
"bpw": 2.36,
|
| 81 |
+
"desc": "IQ2_XS requant of keep64 (159858.29 -> 97628.97 MiB, 2.36 BPW, chunked expert-slice requant). MIXED-E: hash layers 0-2 keep 384 experts, dense layers 3+ keep 64 behind a full-width gate. Requires the UltraTensor llama.cpp fork (>= b4c34be): stock llama.cpp rejects the file via check_tensor_dims. Dense-layer selections beyond the kept stack route to fallback expert 0 (the hash-layer remap convention); the numpy reference silently drops them instead - both are documented approximations of the 2.3% out-of-stack routing tail.",
|
| 82 |
+
"serve": "CPU-only on 30+ GB RAM (mmap stream): llama-cli -m <file> -n 8 -t 8 -c 128. Load takes ~20 min over 1GbE NFS.",
|
| 83 |
+
"tiers": ["cpu"]
|
| 84 |
}
|
| 85 |
]
|
| 86 |
}
|