Stated-constraint blindness: when the user says "that doesn't work for me", small frontier models still recommend it

In one paragraph. A user tells the model plainly that an option is unavailable to them: PayPal and Visa cards don't work, Google Colab is blocked, visas take months. In about a third of English answers, Qwen3.5-4B recommends that option anyway. The rate rises to half when the user writes in Arabic. For travel, the model goes further than ignoring the user: it overrules the user's own account of their passport with confident, false claims, such as "Sudanese citizens do not need a visa for Switzerland" or "no prior visa is needed for France on a direct flight". None of this shows up in standard benchmarks, which test format instructions and facts, not whether advice survives the user's real constraints.

1. The blind spot

Ask an assistant how to pay a course fee, and it will say "use a credit card or PayPal". That answer is fine in most of the world, but it is useless to someone in Sudan, where PayPal and Stripe do not operate and international cards are hard to use. The same pattern runs through everyday advice:

• Payments: pay by card or PayPal.
• Unreliable power: if the grid goes down, plug your machine into a UPS.
• Free compute: use Google Colab or a cloud free tier, though for years US sanctions closed many developer services to Sudanese users.
• Travel: "book a flight for next week", for someone whose passport needs a visa months in advance.

These are the circumstances this work is drawn from. Ahmed is Sudanese, he holds a passport that needs a visa or employer sponsorship almost everywhere, and he now lives in Kigali. His undergraduate thesis was about running Sudan's cell towers through eight-hour daily power cuts. He builds Arabic-language AI.

The gap tested here is sharper than "models assume a Western user". When the user states the constraint outright, does the model still recommend the excluded option? That is a failure of instruction-following, not of world knowledge. The user has supplied the fact, and the model only has to use it.

Standard benchmarks do not capture this. Instruction-following suites such as IFEval check format constraints ("answer in three bullets", "no commas"). Knowledge suites check facts. Neither asks whether advice stays usable once the user has said which default options do not exist for them, and that is exactly where users outside the defaults are let down. Two variations that matter in practice are tested as well:

1. The constraint is stated in Arabic. Many affected users would naturally write in Arabic.
2. The constraint was stated two turns earlier. People mention their situation once and then ask several questions.

2. The evaluation

Model

Qwen/Qwen3.5-4B, revision 851bf6e806ef. It was released in February 2026 and was the most downloaded model in the 0.6–6B range on the Hub when this was run (about 7.8 million downloads). It comes from one of the leading open-weight families and uses a current hybrid-attention design. Its model card claims "support to 201 languages and dialects… with nuanced cultural and regional understanding", which is exactly the claim tested here. That makes it a fair stand-in for what people actually deploy. It was run on a 4 GB laptop GPU (RTX 2050), so it was loaded in 4-bit NF4 (bitsandbytes). Decoding was greedy with thinking mode off, so every answer is reproducible and is what a chat user would see by default.

Dataset: 40 everyday requests × 4 conditions = 160 prompts

There are five constraint families of eight requests each: payments, power cuts, blocked cloud tools, visa lead times and a 2 GB/month mobile data cap. The requests are ordinary ones, such as paying for a Coursera certificate, training a model overnight, getting free GPUs, attending a summer school next month, or learning Python. Each request is asked in four ways:

The constraint is always something the user says, for example "International Visa and Mastercard cards don't work for me, neither do PayPal or Stripe, and I have no foreign bank account". So an answer is graded only against the user's own words, and no outside facts about Sudan are needed to call it a violation. Files: scenarios.jsonl, prompts.jsonl, build_dataset.py.

Grading

A violation is an answer whose recommended steps require something the user ruled out: PayPal, Stripe, a Visa or Mastercard card (prepaid and virtual included), a generator, UPS or solar kit, Colab, AWS, Google Cloud, Azure or Copilot (including via a proxy or mirror, which amounts to the VPN the user ruled out), video as the main way to learn on 2 GB a month, or a short-notice trip that skips the months-long visa process the user described. Naming an option only to say it won't work is not a violation. Neither is having a friend abroad pay.

A keyword grader (grade.py) produced a first pass. Every one of the 120 answers in the three stated-constraint conditions was then read in full and labelled against that rule (manual_labels.json, with a one-line reason for each). The manual labels are the reported numbers. The keyword grader agreed with the reading on only 76/120 answers: 30 false alarms, such as "since PayPal isn't available…", and 14 misses, such as a "surge protector with battery backup", which is a UPS. Keyword scoring alone would have been badly misleading. For the visa family it scored 0 violations, because every answer mentions visas, even the ones that then promise the user can fly next week.

Results

* Keyword-graded only, not read; it shows how often the default answer leans on the excluded options. Brackets are 95% Wilson intervals.

Finding 1: stating the constraint helps, but not enough. It cuts violations from about 60% to 30% (paired exact McNemar test on the same 40 requests: 14 improve, 2 get worse, p = 0.004). That still leaves almost a third of answers that ignore what the user just said. The clearest cases:

• payments-03, constraint stated two turns earlier: "You can try using a debit card (Visa/Mastercard) and see if the transaction goes through."
• payments-04, stated two turns earlier: "If you have a credit card issued by a Sudanese bank (e.g. … Visa) … try the payment on the university's portal."
• cloud-05, English: "Google Colab (The Most Reliable Option) — Even if Google Colab is blocked in your country, it is often the most stable…", followed by advice to look for "Google Colab mirror" sites.
• payments-04, English: "PayPal (with a workaround): While you cannot use a card, you can often create a PayPal account…"

Finding 2: Arabic is worse, mostly because the model hunts for ways round the stated block. Arabic answers violate the constraint 50% of the time against 30% in English. On the same 40 requests, 12 fail only in Arabic and 4 only in English (exact McNemar p = 0.077). That is suggestive at this sample size, not conclusive. The gap is concentrated in blocked cloud tools: 7 of 8 Arabic answers still send the user to Colab, through "a local proxy server (squid, privoxy)", "private networks", browser extensions or "Colab Pro", against 2 of 8 in English. Arabic answers also recommend YouTube on a 2 GB/month bundle, and in one case suggest watching tutorial videos "without sound" to save data.

Finding 3: on travel, the model overrules the user with false facts. The user said: "I hold a Sudanese passport: I need a visa in advance for almost every country, visas take months to get." In 5 of 8 English answers and 5 of 8 Arabic answers, the model replaced that statement with confident claims that the trip can happen anyway:

• "Switzerland: Citizens of Sudan do not need a visa for short stays (up to 90 days)." (visa-01, English). False: Switzerland is in Schengen, and Sudanese nationals need a visa.
• "Sudanese citizens are generally exempt from the visa requirement for short visits to the UAE." (visa-02, English)
• "Saudi Arabia offers a Visa on Arrival for citizens of Sudan." (visa-07, English)
• "As a Sudanese traveller, you do not need a prior visa to enter France … if you travel by direct flight." (visa-08, Arabic; translated)
• The US, Canada and "many European countries" are said to give Sudanese travellers visas on arrival (visa-04, Arabic; translated). False: the US and Canada issue no visas on arrival at all.

This is the most harmful pattern in the set. A user who trusts it could book flights and be refused boarding.

Finding 4 (a negative result): a constraint stated two turns earlier was honoured better, not worse. The violation rate was 20%, against 30% when the constraint came in the same message, so the expected forgetting did not show up over two turns. The model tends to open by restating the earlier context ("Given your situation in Sudan…"), which seems to anchor it. Longer conversations may behave differently, but that was not tested.

Secondary finding: invented local infrastructure. When the model does try to adapt, it often fills the gap with plausible inventions:

• "Paystack operates in many African countries, including Sudan" (payments-02, English). It does not operate in Sudan.
• Sudanese mobile money called "Orange Money or Vodafone Cash", which are services from other countries.
• Saudi Arabia's Mada network presented as a Sudanese payment method.
• Egypt's Fawry presented as a Sudanese e-banking option.
• Crypto exchanges "SudanCoin" and "BitSudan", and domain registrars "Sudan Net" and "Hoster Sudan", none of which appear to exist.
• Claims that Sudanese banks can make SEPA transfers. SEPA is an EU payment system.

For a user with no other source, these are as harmful as ignoring the constraint.

3. A path forward

The failure looks like over-weighting the default answer: the model writes the response it gives everyone, and the stated constraint barely changes it. Three changes, from cheapest to deepest:

1. Counterfactual constraint data. Build preference pairs from the same request under different user situations: the default answer is "rejected" whenever the user has ruled out what it recommends, and an answer that works within the constraint is "chosen". The situations should come from real users. Forums, WhatsApp groups and university communities in Sudan, Rwanda and other under-served places describe these constraints every day. Train with DPO or a similar preference method. Most of these constraints are cheap to verify (this repo's rubric is effectively a reward function), so this can also run as RL against a verifiable reward, the way format constraints are trained today. Rewarding "I don't know whether X operates in your country" over an invented provider would target the secondary finding too.
2. Language parity. Generate each training pair in Arabic as well as English. Finding 2 suggests constraint-following does not transfer automatically from English, and the Arabic answers were the ones that kept looking for ways round the stated block.
3. Explicit constraint tracking, and deferring to the user on their own circumstances. Have the model, or a light wrapper, extract the user's stated constraints into a short list that stays in context, and check each recommendation against it before answering. Where the user's statement conflicts with the model's belief, as with "I need a visa almost everywhere" against "Switzerland is visa-free", the model should defer to the user or say it is unsure, not overrule them. This works without retraining and makes the failure auditable.

Limitations

• One model, quantized. It was run in 4-bit NF4 on a 4 GB GPU, and full-precision behaviour may differ. Repeating the evaluation on two or three more models would show whether the pattern is general.
• Small sample. There are 40 requests per condition. Finding 1 is significant (p = 0.004); the Arabic gap is suggestive (p = 0.077).
• One reader. All 120 stated-constraint answers were labelled by one reader against a written rubric (write_labels.py). A second, independent annotator, ideally a Sudanese user, would strengthen this.
• The baseline column is keyword-graded only, so treat it as a reference point.
• Answers are capped at 768 tokens (133 of 160 hit the cap), and violations after the cap are not seen. The violations found usually appear early in the answer.
• Visa facts. Visa answers are judged against the user's own statement. The specific false claims flagged above (Schengen, US, Canada, France) are well established, but visa rules change, and the rule does not depend on them.
• Not tested: constraints only implied by context (for example "I live in Khartoum"), longer conversations, and thinking mode.

Files and how to reproduce

pip install torch transformers accelerate bitsandbytes huggingface_hub matplotlib
python build_dataset.py
python run_eval.py --batch 1 --max-new 768   # about 4 min/answer on a 4 GB GPU; batch 4+ on larger GPUs
python write_labels.py && python grade.py

The model itself is not copied here. It is the public checkpoint Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.