Is it really better in real world task?
I have found the Qwen3 series very interesting in the sense of being open-weights and fast compared to other options in this category, while there's also not many open options and deciding to provide these models to common people who doesn't have thousand or millions of dollars to pretrain their own is awesome even before the project conception. I really appreciate that.
Despites the honored intentions, I didn't have good experience trying Qwen3 series for real world tasks, they struggle to understand the prompts or forgets the intructions before conclusion, and I'm almost sure it is because an aggressive censorship, even in the Base versions.
Looking on Benchmarks, Qwen3.6 looks much worse in knowledge and skills, but I don't trust entirely on benchmarks since the feeling from community is almost always differ, and there's also the common tendency of "benchmaxxing" in many AI projects.
The fact Benchmarks compares Qwen3.6 with Gemma4 worry me a lot, since I never saw anything positive on Gemma series, they seems like the most "lobotomized" LLMs, no matter the version, and some benchmarks points it very clearly. Qwen3.6 doesn't look better even against Qwen3.5 in most evaluations.
My question is: The model is really performing better at anything compared to previous versions?
If the model is able to strict follow the instructions, deal with contexts appended via RAG and call functions without hallucinations until tasks are completed, for me it's what really matters.
Expect LLMs to be an oracle is quite absurd, would worth to improve them to just help in basic but time wasting repetitive tasks.
hm
At least I find it extremely useful. It works perfectly with Hermes. Take off!
You probably hate hearing this all the time but, the best way is to just try it out yourself. Benchmarks are benchmaxxed. Word of mouth could just be astroturfing. There's seriously no authentic way of determining real-use cases other than by your own observations.
@BornSaint just passing by I can add that some benchmarks are specifically assessing hallucination tendency. Have a look at the AA-Omniscience Non-Hallucination Rate scores at https://artificialanalysis.ai/leaderboards/models (expand the Intelligence column to reveal the details)
And can't really be benchmaxxed:
In order to maintain evaluation integrity, AA-Omniscience-Public is a 10% subset of the full question set.
(https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public)
In reasoning mode, Qwen's ones are doing pretty well:
@owao Qwen3.6 is basically just Q3.5 with more fine-tuning, especially in coding, and that was accompanied by a drop in general performance, which I saw in my testing (e.g. more factual hallucinations).
There's simply no way it jumped from 20 to 52% on any meaningful test. If such a massive jump isn't due to benchmaxxing, as you suggested, then the test has little value.
Anyways, Artificial Analysis is nonsense, and it baffles me that so many people don't realize this. For example, Qwen3.5 9b and 4b (w/ reasoning) have scores around 27, which is up there with models like DeepSeek V1, but when I used Qwen3.5 9b sans reasoning it performed horrendously across almost every task, yet it's still vastly better than Qwen3.5 4b with reasoning. These models are utterly useless. What is the point of AA if incredibly weak models like Qwen3.5 4b can be benchmaxxed to the same score as models that are vastly larger and more powerful?
You probably hate hearing this all the time but, the best way is to just try it out yourself. Benchmarks are benchmaxxed. Word of mouth could just be astroturfing. There's seriously no authentic way of determining real-use cases other than by your own observations.
To be honest, I don't hear it too much, but I guess you're absolutely right on that.
About benchmaxxing, I really believe some results are very unintentional due to the very sensitive and somehow chaotic nature of large neural networks. A single new token can mess up the entire model.
@BornSaint just passing by I can add that some benchmarks are specifically assessing hallucination tendency. Have a look at the
AA-Omniscience Non-Hallucination Ratescores at https://artificialanalysis.ai/leaderboards/models (expand the Intelligence column to reveal the details)And can't really be benchmaxxed:
In order to maintain evaluation integrity, AA-Omniscience-Public is a 10% subset of the full question set.
(https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public)In reasoning mode, Qwen's ones are doing pretty well:
Thank you very much, this surely is very helpful.
@owao Qwen3.6 is basically just Q3.5 with more fine-tuning, especially in coding, and that was accompanied by a drop in general performance, which I saw in my testing (e.g. more factual hallucinations).
There's simply no way it jumped from 20 to 52% on any meaningful test. If such a massive jump isn't due to benchmaxxing, as you suggested, then the test has little value.
Anyways, Artificial Analysis is nonsense, and it baffles me that so many people don't realize this. For example, Qwen3.5 9b and 4b (w/ reasoning) have scores around 27, which is up there with models like DeepSeek V1, but when I used Qwen3.5 9b sans reasoning it performed horrendously across almost every task, yet it's still vastly better than Qwen3.5 4b with reasoning. These models are utterly useless. What is the point of AA if incredibly weak models like Qwen3.5 4b can be benchmaxxed to the same score as models that are vastly larger and more powerful?
4b to 9b models can be very good in some tasks if you don't care to provide a huge ruleset or just give them much simpler tasks, mainly because they have much less space in the embedding layer to similar or equal tokens with very different meanings. They require a lot more active guideness than larger ones, but are not useless at all.
This jump could be explained just by the large neural network nature, it's not necessarily a benchmaxxing.
Due to the difficulty process to map the internal activations of "AI mind", we can't, in most cases, measure precisely how much impact some dataset entries has in the model.
The finetune can have unpredictable side effects, catastrophic forgetting could happen to some data group, but also can improve the model responsiveness to a data group that wasn't the goal of this particular finetune.
Anyway, I should always follow the advice to try by myself to figure out if some new model has improvements to my particular task, despites the benchmark results.
Thank you everyone for the advices.
@phil111 you could be interested by this article https://kaitchup.substack.com/p/did-the-model-see-the-benchmark-during their articles access is subscription based and I didn't but even the free access part of the articles is always interesting and providing great insights. You'll find the paper link too
Here was the results for Qwen3.5-27B vs Gemma4-31B:
So you are clearly right to say Qwen's have affinity with the benchmarks.
But coming back to AA, I find curious your private factual testing shows such a different story than this precise benchmark.
@owao Thanks, I read the paper and the linked summary page, and the graph you provided says it all.
They also singled out Qwen2.5 in the paper on the GSM8K, and it was the transition from Qwen2 to Qwen2.5 that I first observed suspiciously high tests scores despite no improvement, and often a regression, in my testing.
Additionally, this domain and test over-fitting was accompanied by a sharp drop in broad knowledge (e.g. ~18 to ~10 on the English SimpleQA going from Qwen2 72b to Qwen2.5 72b, which was matched in my personal broad knowledge testing). And it was all downhill across the board from here since other companies needed to appear competitive, so they started testmaxing.
However, Qwen started adding broad knowledge back, and Qwen3.5 has notably more per parameter than the original Qwen3 version, and now has about the same amount as Gemma 4 / parameter because the Gemma series started slightly regressing in broad knowledge going from G2 to G3 to G4, so now they're approximately tied.
And in regards to AA, it doesn't really include factual testing. Tests like the GPQA diamond and MMLU-pro primarily focus on problem solving rather than knowledge, plus they're multiple choice. Multiple choice testing is not suited for knowledge testing because weakly held info might allow a model to pick the answer out of a lineup despite not being able to fully recover it, which is what's required in real-world use cases. Additionally, the included AA tests only cover about 10% of humanity's most popular domains of knowledge (e.g. mainly just STEM, and to a lesser degree, academia in general). Because of this AA rewards the gross over-fitting of select domains, leaving what most people care most about (e.g. popular culture) with orders of magnitude less coverage. In short, the AA focuses more on problem solving than knowledge, tests multiple-choice vs full recall performance, and is extremely lopsided (heavily focused on only ~10% of humanity's most popular knowledge).
Lastly, the largest Qwen models now have test scores that almost correlate with their real-world performance. However, at progressively smaller sizes the disparity between test scores and real-world performance keeps growing, and by Qwen3.5 4b the test scores are reliably ~10 points higher or more than their real-world performance (e.g. MMLU-Pro).
I ran this Qwen3.6-35B-A3B model on => AMD MI300x 1 GPU - 192 GB VRAM - 20 vCPU - 240 GB RAMBoot disk: 720 GB NVMe- Scratch disk: 5 TB NVMe Droplet, using vllm and connected with VSCode Cline, and must say it worked like pro.
i tried DeepSeek-R1-Distill-Llama-70B-FP8-dynamic, Qwen/Qwen2.5-72B-Instruct and some other , but all failed in Agentic task like CRUD operations on file, changed multiple IDE , but finally came back to VSCode Cline, i developed web app as of now, with attached reference images and code sample, it worked perfectly, ya it lagged in some task but it was good.
I have connected this model with VSCode Ollama, it is completely useless. It barely managed to create a "Hello world!" app, and seems to have no ability to actually do anything. It keeps referring to a FilePath error during reasoning, then it gives me a summary that concludes in "How can I help you?" (or similar) without having done anything. I think this is some sort of config error with the Ollama extension for VSCode
Running the model in ollama on the command line seems to work a whole lot better. I gave it the same instruction to create an app that calculates the number of days since a given date. It generated code that looks good, but it is trying to use private class methods which is giving compiler errors. I asked the model to fix the errors, but despite a lengthy summary of the fix, it still doesn't compile (it gives the same errors and new errors). My conclusion for now: something I could have done in one or two passes with Claude or Codex is now taking me at least 20 minutes and shows now signs of delivering working code in the foreseeable future. I won't be using it.
