Instructions to use NagusameCS/ultratensor-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use NagusameCS/ultratensor-models with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: llama cli -hf NagusameCS/ultratensor-models
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: llama cli -hf NagusameCS/ultratensor-models
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf NagusameCS/ultratensor-models # Run inference directly in the terminal: ./build/bin/llama cli -hf NagusameCS/ultratensor-models
Use Docker
docker model run hf.co/NagusameCS/ultratensor-models
- Llama new
- LM Studio
- Jan
- Ollama
How to use NagusameCS/ultratensor-models with Ollama:
ollama run hf.co/NagusameCS/ultratensor-models
- Unsloth Desktop
- Docker Model Runner
How to use NagusameCS/ultratensor-models with Docker Model Runner:
docker model run hf.co/NagusameCS/ultratensor-models
- Lemonade
How to use NagusameCS/ultratensor-models with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull NagusameCS/ultratensor-models
Run and chat with the model
lemonade run user.ultratensor-models-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
model card: keep16u-iq2xs replaced; 2-bit cause documented
Browse files
README.md
CHANGED
|
@@ -24,9 +24,11 @@ defects. The previous files remain available in this repository's history.
|
|
| 24 |
files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512
|
| 25 |
tokens of mixed calibration text).
|
| 26 |
3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
|
| 27 |
-
4. **The IQ2_XS files emitted garbage** under upstream llama.cpp
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
| 30 |
|
| 31 |
## Measured quality: read this before using them
|
| 32 |
|
|
@@ -40,6 +42,7 @@ and the top-1 agreement with it.
|
|
| 40 |
|---|---|---|---|---|---|---|
|
| 41 |
| `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
|
| 42 |
| `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
|
|
|
|
| 43 |
|
| 44 |
Previous files, for comparison (same test):
|
| 45 |
|
|
@@ -57,7 +60,6 @@ Larger tiers are more faithful (see the table).
|
|
| 57 |
## Files not (yet) replaced
|
| 58 |
|
| 59 |
- `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
|
| 60 |
-
- `keep16u-iq2xs/DeepSeek-V4-Coder-keep16u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
|
| 61 |
- `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.
|
| 62 |
- `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.
|
| 63 |
- `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.
|
|
|
|
| 24 |
files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512
|
| 25 |
tokens of mixed calibration text).
|
| 26 |
3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
|
| 27 |
+
4. **The IQ2_XS files emitted garbage** under upstream llama.cpp: repeated tokens, top-1 agreement 0 %. The cause
|
| 28 |
+
is llama.cpp's IQ2_XS preset, which also puts attention, the indexer, the shared expert and the hyper-connection
|
| 29 |
+
mixers at 2 bits; a requant of our own with a correct imatrix failed the same way. In the new IQ2_XS files only
|
| 30 |
+
the routed-expert weights (most of the bytes) are IQ2_XS, with the full-model imatrix sliced to the kept
|
| 31 |
+
experts; every other tensor is Q4_K_M.
|
| 32 |
|
| 33 |
## Measured quality: read this before using them
|
| 34 |
|
|
|
|
| 42 |
|---|---|---|---|---|---|---|
|
| 43 |
| `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
|
| 44 |
| `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
|
| 45 |
+
| `keep16u-iq2xs` | IQ2_XS | 31.1 | 9.0× / 52% | 26.6× / 30% | 20.5× / 30% | 143.1× / 20% |
|
| 46 |
|
| 47 |
Previous files, for comparison (same test):
|
| 48 |
|
|
|
|
| 60 |
## Files not (yet) replaced
|
| 61 |
|
| 62 |
- `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
|
|
|
|
| 63 |
- `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.
|
| 64 |
- `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.
|
| 65 |
- `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.
|