ThinkingCap vs Arex-2 Bench

#9
by hiutaky - opened

ThinkingCap vs AREX-2 β€” standard benchmark report

Date: 2026-11-17/18. Both models are Qwen3.8-27B-based, INT4 (W4A16), served by HyperQwen (batch mode), one RTX 3090 each.

  • ThinkingCap: ThinkingCap-Qwen3.8-27B-W4A16-Merlin (GPU 3, 64 seqs, 150k context)
  • AREX-2: BAAI/AREX-2 27B INT4-W4A16 (GPU 1, 16 seqs, 40k context)

Tools: EvalPlus 0.3.1 (HumanEval+), lm-eval 0.4.13 (IFEval), bfcl-eval 2026.3.23 (BFCL v4, non multi-turn).
Setup: greedy decoding (temperature 0 / 0.001), max 16,000 output tokens, 16 parallel requests (EvalPlus: 4 shards), one run.

1. Quality

Benchmark Metric ThinkingCap AREX-2
HumanEval+ pass@1 (base tests) 97.0% 97.0%
HumanEval+ pass@1 (base + extra tests) 91.5% 91.5%
IFEval prompt-level strict 86.5% 75.4%
IFEval prompt-level loose 89.1% 76.3%
IFEval instruction-level strict 89.0% 78.5%
IFEval instruction-level loose 90.6% 79.1%
IFEval empty answers (token cap hit) 27 / 541 (5.0%) 94 / 541 (17.4%)
BFCL v4 non-live overall (AST) 84.67% 87.67%
BFCL v4 live overall (AST) 82.68% 82.53%

BFCL non-live detail

Category ThinkingCap AREX-2
Simple (avg of Python/Java/JS) 77.17% 75.67%
Python simple 96.50% 95.00%
Java simple 63.00% 62.00%
JavaScript simple 72.00% 70.00%
Multiple 95.50% 97.00%
Parallel 81.00% 92.50%
Parallel multiple 85.00% 85.50%
Irrelevance detection 75.00% 70.83%

BFCL live detail

Category ThinkingCap AREX-2
Python simple 87.60% 87.21%
Python multiple 81.58% 81.67%
Python parallel 87.50% 75.00%
Python parallel multiple 75.00% 75.00%
Irrelevance detection 75.90% 49.10%
Relevance detection 81.25% 100.00%

2. Speed

Measure ThinkingCap AREX-2
IFEval wall time (541 prompts, 16 parallel) 29.6 min (1,776 s) 88.5 min (5,307 s)
BFCL wall time (3,641 requests, 16 threads) ~34 min ~35 min
Pagoda single-file 3D prompt, total time 1,581 s (26.4 min) 2,071 s (34.5 min)
Pagoda output tokens (reasoning + answer) 74,084 (63,758 reasoning) 87,145 (78,746 reasoning)
Pagoda single-stream speed 46.8 tok/s 42.1 tok/s
Pagoda time to first token 0.17 s 0.15 s
Aggregate throughput, IFEval, 16 parallel not logged ~545 tok/s

Pagoda: both finished with finish_reason=stop, ThinkingCap first (about 490 s earlier). Sampling there was temperature 0.6, top_p 0.95, max 32,000 tokens requested (streams ran to 74-87k, so see caveat below).
HumanEval+ wall time is not reported: the run was relaunched several times (timeouts, dataset download race), so the timings are not comparable.

3. Reading the results

  • Code generation: identical (HumanEval+ 97.0 / 91.5).
  • Instruction following: ThinkingCap leads by ~11-13 points. Part of the gap is AREX-2 reasoning past the 16k-token cap (17% empty answers vs 5%), so the real gap at a higher cap would likely be smaller.
  • Tool calling: AREX-2 is ahead on non-live (+3.0, mostly parallel calls: 92.5 vs 81.0). Live is a tie overall, but AREX-2 is much weaker at irrelevance detection (49.1% vs 75.9%: it calls a tool when it should not) and perfect on relevance (100%).
  • Speed: per-stream decoding is similar (47 vs 42 tok/s); AREX-2 is slower end-to-end because it produces ~18% more reasoning tokens, and ThinkingCap ran with a larger batch budget.

4. Limits and fairness notes

  • One run per benchmark, greedy decoding: no variance estimate. Differences of 1-2 points are noise (IFEval stderr about 1.8 points).
  • 16,000-token cap: empty answers (reasoning not finished) count as failures. This penalizes the model with longer reasoning.
  • Different server settings: ThinkingCap 64 seqs / 150k context, AREX-2 16 seqs / 40k context.
  • Local patches to EvalPlus: max_new_tokens 768 β†’ 16,000, request alarm 900 s, None content treated as empty string.
  • BFCL ran through the Qwen/Qwen3-32B-FC handler (hence the label "Qwen3-32B (FC)" in BFCL tables); the models actually tested are the ones above. Only non multi-turn categories were run (13 categories).
  • All GPUs report PCIe Gen1 x1; this affects load time, not generation speed measured here.

Sign up or log in to comment