Excellent piece — the "morphological reconstruction bill" line is the one I keep coming back to, and the four-taxes framing is the right way to say this is a structural problem, not a data problem.
Two things from a Germanic angle that I think sharpen it. The penalty is genuinely language-dependent: in the Shelpuk Llama-2 → Gemma comparison German improves only ~5% while Greek jumps 47% and Arabic 59%, so Danish and German are the least penalized by English BPE — but "least" is doing a lot of work there, and the compounding argument still holds, just at a lower baseline. And the EMNLP 2024 "scale partially recovers the gap" finding is the one I'd underline for the efficiency crowd: it means the tax is paid disproportionately by smaller models, which is exactly the regime where linear-attention and other efficiency wins are supposed to matter. A 1B linear-attention model with an English BPE tokenizer is carrying deadweight that a 70B dense model can partly afford to buy back.
Your open question #1 — does morphological misalignment predict reasoning degradation at fixed parameter count — is the controlled experiment I'd most like to see, because it's the one that separates the tokenizer bill from the model-size bill cleanly. I work with Danish and German corpora (compounds, the ø/å/æ diacritic mess) and would be glad to help run a small fixed-parameter comparison if it's useful to the Sawaldi work. The diacritic brittleness you describe is very real here: one dropped diacritic in Danish changes the token set enough to be a different word.
