XHToken

/

Spark-X2.5-4B

Like 1.44k

Follow

SparkLLM 1.19k

Great evaluation! Could you summarize it in a score table? It would be much more intuitive for visualization.

Thanks for sharing the evaluation and documenting the setup. I ran a local comparison of Spark-X2.5 and Qwen3.5, and obtained substantially higher MATH results. The 512-token generation limit looks like a major factor worth investigating before interpreting these scores as capability limits.

My setup was:

• Hardware/runtime: Ryzen 9 5950X + Radeon RX 7900 XTX 24 GB, using the Spark-supporting XHToken/llama.cpp fork with ROCm.
• Precision: BF16 GGUF weights and separate Q4_K_M runs; Q8_0 KV cache.
• Context: 32,768 tokens per slot, four concurrent slots.
• Dataset: 100 questions from HuggingFaceH4/MATH-500, with 20 per difficulty level, using selection seed 20260811. Every model and precision used the same subset.
• Math decoding: native thinking enabled, temperature=1.0, top_p=0.95, top_k=0, min_p=0, repeat_penalty=1.0, inference seed 4242.
• Generation budget: 4,096 tokens, plus an additional 8,192-token experiment for Spark 4B Q4_K_M.
• Scoring: one generated response per question, scored with Math-Verify. Every response ending with finish_reason="length" was counted as incorrect.

The math prompt was:

Solve the following mathematics problem. Show concise working sufficient to support the result. Put the final answer in \boxed{} and do not put anything after it.

{problem}

My Spark results were:

In the BF16 runs, 61% of Spark 4B responses and 70% of Spark 1.7B responses exceeded 512 output tokens. Average output lengths were approximately 1,691 and 2,000 tokens respectively. Even at 4,096 tokens, truncation remained substantial.

The 8,192-token experiment is particularly relevant: Spark 4B Q4_K_M improved from 68% to 83% on the same questions, while truncations fell from 29 to 16.

For broader context, these were my BF16 results:

These used 285 MMLU questions, 140 MMLU-Pro questions for each mode, and all 164 HumanEval+ tasks. Native thinking was disabled outside MATH; MMLU-Pro CoT used explicit five-shot reasoning prompts.

The comparison was mixed: Qwen led on MMLU and MMLU-Pro, while Spark 4B led on HumanEval+ in this setup. The low Qwen math scores also need the same budget caveat: Qwen 4B and 2B hit the 4,096-token limit on 71 and 93 questions respectively. I would not interpret those scores as their unrestricted mathematical ability.

These are subset results from a different runtime, sampling policy, and scoring pipeline, so they are not a controlled reproduction of your evaluation. I also did not test GSM8K or AIME in this suite.

Would you consider rerunning the same questions at 4,096 and 8,192 tokens, reporting truncation and answer-extraction failure rates alongside accuracy? A greedy-versus-sampling comparison at each budget would help separate the effects. Your existing 512-versus-1,024-token observation already points toward budget sensitivity; my results suggest it remains significant at larger budgets too.

Thanks for the careful comparison and the important reminder - I agree with your core point. My numbers should be read as conservative floor estimates at a 512-token greedy bf16 budget, not as the model's capability ceiling.

A few notes:

1. My own data already pointed at the same effect: on GSM8K, the 1.7B improved from 0.35 -> 0.42 when I raised the budget from 512 to 1024 tokens. Your 4096 -> 8192 jump on Spark-4B (68% -> 83%) confirms the budget sensitivity is still significant at much larger budgets. So this is a real, large effect.

2. Cross-runtime caveat: my runs used transformers + trust_remote_code, greedy (temperature=0), bf16, 512 max_new_tokens, on an NVIDIA GB10. Yours used llama.cpp/ROCm, sampling (temperature=1.0, top_p=0.95), 4096-8192 tokens, Math-Verify scoring, and a different subset (100 items, 20 per level). The two are not directly comparable - but the directional conclusion (budget dominates) clearly holds, and your 74%-83% figures are far closer to Spark-4B's real MATH ability than my conservative 0.25.

3. On greedy vs. sampling: I only ran greedy. Your suggested protocol - a greedy-vs-sampling comparison at each budget, reporting truncation and answer-extraction failure rates - is the better design, and I'd expect the short-budget deficit to shrink further under sampling. I can't rerun right now (the eval environment has been cleaned), but I'll make the "512-token greedy" precondition more explicit so readers don't misread these as capability limits.

Your MMLU / HumanEval+ comparison is also valuable, and I agree the low Qwen MATH scores shouldn't be read as capability either - you've already flagged that correctly. Thanks again; this makes the evaluation more complete.

@Duonglv Thanks! Here's the results summary as a score table:

Note: pass@1 under greedy bf16 with a 512-token budget (conservative floor estimates). The full reasoning-integrity breakdown, contamination check and cross-hardware note are in the post above.