capyctl/FrogNano-4B-2609-MLX-4bit

FrogNano-4B-2609, MLX affine 4-bit, text only

A 4-bit MLX affine quantization of microsoft/FrogNano-4B-2609 (revision b90468c1), made so TensorFold can serve it on NVIDIA GPUs. TensorFold's CUDA engine reads quantized weights only; the original checkpoint is BF16.

All credit for the model goes to its authors. Read the original model card for intended use, limitations and safety guidance; they apply unchanged. This repository changes only the storage format.

What changed from the original

Weights	MLX affine 4-bit, group size 64 (mlx-lm 0.32.0), about 2.6 GB
Output layer	lm_head written as an exact copy of embed_tokens before quantizing, tie_word_embeddings: false. TensorFold 0.6.3 refuses tied heads (TensorFold#306).
Vision tower	Removed (297 tensors and vision_config). FrogNano is text only in its intended use; the original card does not claim image or video support.
MTP layer	Removed (15 tensors).
Tokenizer, chat template	The original files, unchanged.
generation_config.json	Added, with eos_token_id: [248046, 248044] (<|im_end|>, <|endoftext|>), so replies stop at the end of a turn.

The conversion script is in the CapyCTL recipe linked below.

Serve it

TensorFold 0.6.3 or later, CUDA:

tensorfold serve capyctl/FrogNano-4B-2609-MLX-4bit --no-drafts --parallel 8

--no-drafts is required: there is no drafter for this model, and TensorFold's Qwen dense engine on CUDA needs either a drafter or --no-drafts. --parallel 8 decodes up to eight requests together; without it TensorFold on CUDA serves one request at a time.

Through CapyCTL: see the recipe tensorfold/frognano-4b-mlx-4bit-rtx4090. CapyCTL starts TensorFold with --parallel 8 by default (or the deployment's max_concurrent_requests).

Measured

One RTX 4090 Laptop GPU (16 GB), through CapyCTL, greedy, thinking on, max_tokens 512, median of 3 prompts:

	This checkpoint (TensorFold 0.6.3)	Original BF16 (vLLM 0.30.0)
Decode, one stream	52.4 tok/s	59.9 tok/s
Time to first token	0.045 s	0.054 s
GPU memory, whole card	3.4 GiB with short prompts, 7.3 GiB at most in the benchmark (11 GiB cap)	13.1 GiB

With several streams (TensorFold --parallel 8, 512 tokens a request, greedy, thinking on, median of 5 rounds):

Streams	Together	Each
1	50.4 tok/s	50.6 tok/s
2	93.4 tok/s	46.9 tok/s
4	176.5 tok/s	44.3 tok/s
8	319.1 tok/s	40.3 tok/s

One stream decodes 49 to 50 tok/s from 0.5k to 32k tokens of context. Each stream's memory grows with its request; eight 28k-token prompts sent at once all completed with the GPU at 9.9 GiB at most, under an 11 GiB cap (TENSORFOLD_CUDA_MEMORY_LIMIT_GB=11). Give TensorFold a cap that covers your context and streams.

Quality check

A small greedy comparison against the original BF16 weights on vLLM 0.30.0, thinking off, max_tokens 1024. Generated code was run against unit tests in a sandbox. This is a spot check, not SWE-bench or a perplexity measurement.

Prompts	This checkpoint	Original BF16
Python coding, 30	27	26
Tool calls, 30 (right tool and arguments)	26	28
Reasoning and math, 20	20	20
Total, 80	73 (91%)	74 (93%)

Every tool call from both was valid JSON. With thinking on (10 prompts, max_tokens 4096) this checkpoint passed 8 and BF16 9; the miss was one reply that kept repeating its reasoning until the token limit. Set a max_tokens limit when thinking is on.

License

The original repository is tagged MIT, and its card's license row names Apache 2.0. This derived checkpoint keeps the original's terms and attribution; check the original repository for them.