Thinklite-27B-NVFP4-GGUF

Thinklite-27B is a fine-tune of Qwen3.8-27B (Apache-2.0, image-text-to-text) that is trained on the base model's own verified-correct answers so it reaches the same results with shorter reasoning. Only the thinking is shortened, and the final answer keeps its normal length and format. It is a drop-in replacement with the same chat template, tokenizer, and reasoning_effort control (low, medium, xhigh). On the authors' six-benchmark suite (GSM8K, AIME 2025, MMLU-Pro, IFEval, LiveCodeBench v6, BFCL v4), it uses about 47% fewer reasoning tokens at xhigh and about 28% fewer at low. At xhigh it keeps 99.6% of the base model's average accuracy (89.8 vs 90.2), and at low it keeps 97.5% (84.7 vs 86.9). At xhigh, time-to-final-output is roughly 2x faster, since decoding speed is unchanged. It also beats two comparable efficiency fine-tunes, ThinkingCap and Swift-1.5, on token savings, but those two score higher on average accuracy in the authors' own tables. Its vision encoder is unchanged and its training data was text-only. A small 88-question check suggests image understanding is preserved, with 24% fewer reasoning tokens. The card's limitations include lower accuracy on IFEval and MMLU-Pro at xhigh, a larger drop on AIME 2025 at low (81.7 vs 90.0), weaker reliability on long single-answer code generation, wide confidence intervals from single benchmark runs, and no safety tuning beyond the base model. It is served with vLLM or Transformers, and a 16.5 GB Q4_K_M GGUF is available for llama.cpp and Ollama.

MTP β€” Speculative Decoding: Yes | Accelerated FP4 Tensor Core Performance

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name Quant Type File Size File Link Description
Thinklite-27B-NVFP4.gguf NVFP4 33.1 GB Link NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 27B parameter model.
Thinklite-27B.mmproj.gguf mmproj 931 MB Link Multimodal projection file. Used for vision/language capabilities.

NVFP4 Test Run

Tested Thinklite-27B-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric Value
Average Prompt Processing Speed 267.66 tokens/s
Average Generation Speed 83.61 tokens/s
Total Wall Time 73.9 seconds
Context Length 16,384 tokens
KV Cache (K) q8_0
GPU NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity 96 GB
Peak GPU VRAM Usage 30.98 GiB
Peak GPU Utilization 97%
GPU Layers Offloaded 99 (-ngl 99)
Random Seed 42
Speculative Decoding MTP (draft_n_max=2)
Test Runs 5

Test Shots

Shot Test
Shot 1 C: Insert Node at End of Singly Linked List
Shot 2 Jogger and Train Passing Time
Shot 3 Two Pipes Filling a Tank
Shot 4 Students in Examination Rooms
Shot 5 Father and Son Ages

Run Inference

./build/bin/llama-cli \
  -m /path/to/models/Thinklite-27B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

Snapshots (5)

download (1) download (2) download (3) download (4) download (5) download

llama.cpp

LLM inference in C/C++ β€” https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods.

Downloads last month
339
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for prithivMLmods/Thinklite-27B-NVFP4-GGUF

Base model

Qwen/Qwen3.8-27B
Quantized
(5)
this model

Collections including prithivMLmods/Thinklite-27B-NVFP4-GGUF