Spaces:

alphabell
/

Model-Observatory

Running

App Files Files Community

Model-Observatory / experiments /articles /02-blind-quantization.md

alphabell-research

Fix explicit experiment entry paths for static hosting (#4)

933c910 about 7 hours ago

preview code

Raw History Blame Contribute Delete

3.86 kB

# Can you spot the 4-bit model? Eleven of our twelve answer pairs were identical.

![Two measured quantization answers, with the expected arithmetic result.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/02-quantization.png)

We hid the precision labels and compared Q4_1 with Q8_0. In 11 of 12 prompt/model pairs, the captured answer text was exactly the same.

The one different pair did not produce a winner. Asked to calculate `(17 * 23) - 19`, Flint answered `374` at Q8_0 and `352` at Q4_1. The correct answer is `372`.

The blind test retains all twelve pairs, including identical answers and shared errors. [Spot the 4-bit model](https://alphabell-model-observatory.static.hf.space/experiments/index.html?experiment=quant) asks you to guess before revealing the precision, outputs, timings and checks. Answer order is deterministic for each case. Guesses stay in your browser, and the displayed tally is local rather than a community vote count.

We tested six prompts on each of two checkpoints: Flint and the alphaBell Qwen Open 27B derivative. The tasks cover arithmetic, exact JSON, a factual symbol, an in-context correction, a Python expression and a sentence with required words. Each prompt was run at Q8_0 and Q4_1.

Several comparisons are indistinguishable from the answer alone. Both Flint variants returned the same JSON object. Both answered `Au` for gold. Both returned the same photosynthesis sentence. All six Qwen Open answer pairs were identical, including its incorrect arithmetic answer of `362`.

That last example matters. Agreement between precisions is evidence of output stability on a case, not evidence that the answer is right. A blind preference can miss an error that both variants share. We keep the expected answer and check rule beside the captured text so you can inspect both questions separately.

The smaller checkpoint files did save space. Flint's files were approximately 42.35 GB at Q8_0 and 24.96 GB at Q4_1; Qwen Open's were approximately 28.60 GB and 16.83 GB. Those are decimal file sizes, not GPU memory requirements or a guarantee about the memory needed by a deployment.

Smaller also did not mean uniformly faster in this run. For the gold-symbol prompt, the Qwen Open requests took about 4.67 seconds at Q8_0 and 7.81 seconds at Q4_1. These are instrumented CPU measurements from individual requests, not repeated performance estimates. The downloadable data names the timing metrics and records process peak RSS, including loading and warm-up.

The controls were eight CPU threads, context 2,048, temperature zero, seed 42, reasoning disabled and a 64-token output limit. Flash Attention was disabled. The attention KV cache used the runtime's default F16 representation. No quality conclusion about an entire precision format follows from these twelve short pairs.

[Try the blind comparison](https://alphabell-model-observatory.static.hf.space/experiments/index.html?experiment=quant), then [inspect every result and its provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json).

Suggest an exact, short prompt where you expect Q4_1 and Q8_0 to diverge. Include the scoring rule before seeing the answers. We can add submitted cases to the same reproducible comparison.

Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and [llama.cpp](https://github.com/ggml-org/llama.cpp). The [public alphaBell Qwen Open FP8 sibling](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-FP8) is related to the tested derivative, but it is a different weight artifact from these GGUF checkpoints.

Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF).

	# Can you spot the 4-bit model? Eleven of our twelve answer pairs were identical.

	![Two measured quantization answers, with the expected arithmetic result.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/02-quantization.png)

	We hid the precision labels and compared Q4_1 with Q8_0. In 11 of 12 prompt/model pairs, the captured answer text was exactly the same.

	The one different pair did not produce a winner. Asked to calculate `(17 * 23) - 19`, Flint answered `374` at Q8_0 and `352` at Q4_1. The correct answer is `372`.

	The blind test retains all twelve pairs, including identical answers and shared errors. [Spot the 4-bit model](https://alphabell-model-observatory.static.hf.space/experiments/index.html?experiment=quant) asks you to guess before revealing the precision, outputs, timings and checks. Answer order is deterministic for each case. Guesses stay in your browser, and the displayed tally is local rather than a community vote count.

	We tested six prompts on each of two checkpoints: Flint and the alphaBell Qwen Open 27B derivative. The tasks cover arithmetic, exact JSON, a factual symbol, an in-context correction, a Python expression and a sentence with required words. Each prompt was run at Q8_0 and Q4_1.

	Several comparisons are indistinguishable from the answer alone. Both Flint variants returned the same JSON object. Both answered `Au` for gold. Both returned the same photosynthesis sentence. All six Qwen Open answer pairs were identical, including its incorrect arithmetic answer of `362`.

	That last example matters. Agreement between precisions is evidence of output stability on a case, not evidence that the answer is right. A blind preference can miss an error that both variants share. We keep the expected answer and check rule beside the captured text so you can inspect both questions separately.

	The smaller checkpoint files did save space. Flint's files were approximately 42.35 GB at Q8_0 and 24.96 GB at Q4_1; Qwen Open's were approximately 28.60 GB and 16.83 GB. Those are decimal file sizes, not GPU memory requirements or a guarantee about the memory needed by a deployment.

	Smaller also did not mean uniformly faster in this run. For the gold-symbol prompt, the Qwen Open requests took about 4.67 seconds at Q8_0 and 7.81 seconds at Q4_1. These are instrumented CPU measurements from individual requests, not repeated performance estimates. The downloadable data names the timing metrics and records process peak RSS, including loading and warm-up.

	The controls were eight CPU threads, context 2,048, temperature zero, seed 42, reasoning disabled and a 64-token output limit. Flash Attention was disabled. The attention KV cache used the runtime's default F16 representation. No quality conclusion about an entire precision format follows from these twelve short pairs.

	[Try the blind comparison](https://alphabell-model-observatory.static.hf.space/experiments/index.html?experiment=quant), then [inspect every result and its provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json).

	Suggest an exact, short prompt where you expect Q4_1 and Q8_0 to diverge. Include the scoring rule before seeing the answers. We can add submitted cases to the same reproducible comparison.

	Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and [llama.cpp](https://github.com/ggml-org/llama.cpp). The [public alphaBell Qwen Open FP8 sibling](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-FP8) is related to the tested derivative, but it is a different weight artifact from these GGUF checkpoints.

	Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF).