ROOT CAUSE FOUND: The spelling guard (FIX-38) blocked الحكومه→الحكومة
because stem الحكوم is IV in BERT vocab. But الحكوم is a tokenizer
fragment, not a real Arabic word. This caused a false positive in the
pronoun suffix detection.
Fix: Known feminine nouns (الحكومه, المدرسه, الشركه, etc.) bypass
the stem-IV check entirely. The whitelist contains ~50 common words.
This ALSO means the spelling stage will create a proper patch for
these words, which the final output builder uses.
Expected impact: PC034, PC035, and other three_stage_collision cases.
Tests: 39 passing.
FIX-48v3: Create patches (not just mutations) for ه→ة
ROOT CAUSE: The final corrected output is rebuilt from original +
patches (line 2673), NOT from ctx.current_text. FIX-48v3 was only
mutating ctx.current_text without creating patches, so the changes
were silently discarded in the final output.
Fix: Add ctx.add_patch() calls for each ه→ة whitelist match.
This ensures the changes appear in suggestions AND corrected output.
Tests: 39 passing.
The stem-IV approach failed because BERT vocab includes fragments like
الحكوم, الشرك, المدرس as IV. This made the pronoun guard block
every word.
New approach: explicit whitelist of ~80 common words with ه→ة confusion.
Zero false positives, covers the most frequent cases:
- الحكومه→الحكومة, الشركه→الشركة, المدرسه→المدرسة
- Definite + indefinite forms
- Common feminine adjectives
Also reverts pronoun suffix guard from araspell_rules.py.
Tests: 39 passing.
Key insight: grammar model needs to see ه form to make gender decisions.
- Old (FIX-48): ه→ة before grammar → 12 regressions
- New (FIX-48v2): ه→ة after grammar → grammar first decides gender,
then we fix any remaining ه→ة (الحكومه→الحكومة, etc.)
Protected words: فيه, عليه, له, دراسته, etc.
Tests: 39 passing.
The ه→ة pass converted الصغيره→الصغيرة BEFORE the grammar model
could decide if gender should change (الصغيره→الصغير for masculine).
The grammar model needs to see the original ه form to make correct
gender agreement decisions.
FIX-49 (trailing و in IV→IV guard) is preserved — only FIX-48 reverted.
Tests: 39 passing.
FIX-48+49: ه→ة pass for IV words + trailing و in IV→IV guard
FIX-48: Dedicated ه→ة pass runs on ALL words (not just OOV).
- الحكومه, الشركه are IV in BERT vocab, so OOV cleanup skips them
- This pass converts ه→ة when ة form is also IV
- Protected words: فيه, عليه, له, etc.
- Targets: PC034, PC035
FIX-49: Allow trailing و removal in IV→IV guard.
- المصنعو→المصنع was blocked (both IV, not a known fix)
- Now recognized: trailing و removal + و→وا verb fix
- Targets: PC004, PC042
Tests: 39 passing.
FIX-48: Dedicated ه→ة pass for IV words (الحكومه→الحكومة)
OOV cleanup can't fix words like الحكومه and الشركه because they're IV
in the BERT vocabulary. This dedicated pass runs on ALL words (not just
OOV) and converts ه→ة when:
1. Word ends in ه preceded by a consonant
2. The ة form is also IV
3. Word is not in protected list (فيه, عليه, له, etc.)
Targets PC034 (الحكومه) and PC035 (الشركه).
Tests: 39 passing.
FIX-47: Expand hamza whitelist with verb+pronoun forms
The edit-distance-1 OOV→IV cleanup in FIX-44 caused PC037, PC047, PC049
to regress with zero improvements. Root cause: Arabic conjugated verbs
(ادرسي) are OOV but their edit-1 IV neighbor (ادريس) is a completely
different word. Frequency-based selection picks wrong candidates.
Trailing و removal stays active (no regressions from that).
Tests: 39 passing.
FIX-45: Upgrade _levenshtein to Damerau-Levenshtein distance
Transpositions (اقصتاديا→اقتصاديا, swap صت→تص) now count as 1 edit
instead of 2. This makes FIX-39 edit-distance guards more accurate for
Arabic typos where adjacent characters are swapped.
Before: _levenshtein('ab','ba') = 2
After: _levenshtein('ab','ba') = 1
Tests: 39 passing.
AUDIT COMPLETION:
All 4 items from legacy AraSpell audit now confirmed in codebase:
1. Trailing و removal → FIX-44 (just added)
2. Damerau-Levenshtein → FIX-45 (this commit)
3. Beam Voting → already in araspell_rules.py L1345-1362
4. Output Stability Test → already in araspell_rules.py L1521-1540
FIX-44: OOV cleanup pass between spelling and grammar stages
NEW PIPELINE STEP after spelling, before grammar:
1. Trailing و removal (from legacy AraSpell):
- المصنعو→المصنع, الماهرينوومن→الماهرينوومن
- Catches PC004, PC008, PC010 benchmark failures
2. Edit-distance-1 OOV→IV correction:
- For remaining OOV words, find closest IV word in BERT vocab
- Only replaces when edit-1 candidate exists and first letter matches
- Catches: صممو→صمموا (PC001), حضرو→حضروا (PC042)
Also adds contextual_corrector.py module (MLM-based validation).
Tests: 39 passing.
Fix FIX-42b: Allow first-letter changes when root lengths differ
FIX-42b was blocking الولاد→الأولاد because after stripping ال,
roots ولاد(4)→أولاد(5) have different first letters. But this is
a valid hamza-addition correction, not a consonant swap.
Fix: only block when roots have SAME length (true consonant swap).
Different-length roots indicate character addition/removal which is OK.
Blocked: افهمه(5)→تفهمة(5) ✓, واحتاج(5)→وتحتاج(5) ✓
Allowed: ولاد(4)→أولاد(5) ✓
Tests: 39 passing.
FIX-43: Route bidirectional OOV→IV fixes through spelling guard
Prevents model corruptions like واجب→واجبا, معطف→معطفا where the model
inappropriately adds accusative markers to already-correct words.
Tests: 39 passing.
The بال...ون/ان and ل...ون/ان patterns had no blocklist protection.
بالامتحان would become بالامتحين, للإنسان would become للإنسين.
Now all three preposition regexes use the same callback with:
- _PREP_BLOCKLIST check
- ان suffix → skip (almost always root noun form)
Tests: 39 passing (10 new for FIX-33b).
- FIX-35: Narrowed suffix set from {ن,ت,ا,ي,ة,و,ه} to {ن,ت} only.
Only verb conjugation suffixes are blocked. ة/ا/ي/و/ه are legitimate
typo corrections that should not be blocked.
- FIX-37: Raised word threshold from 2 to 4. Prevents noisy period
suggestions while user is still typing short phrases.
- FIX-32b: Removed duplicate .sort() in applyAllSuggestions — already
sorted at line 679, second sort at 695 was redundant.
- Tests updated to match narrowed suffix expectations.
FIX-32b: Add missing reverse sort in applyAllSuggestions
Found during commit audit: the text-based replacement loop iterated
suggestions in original order instead of reverse-sorted by offset.
Without descending sort, applying a fix at offset 5 shifts all text
after it, causing fixes at offset 20+ to corrupt the text.
FIX-37: Rule-based terminal period fallback for punctuation
The punctuation model (PuncAra-v1) often fails to add a period at the
end of longer sentences (10+ words). It only suggests mid-sentence
semicolons which get correctly rejected by PUNC-SAFETY.
FIX-37 adds a rule-based fallback: after punctuation model processing,
if the text still lacks terminal punctuation and has 2+ words, inject
a period suggestion for the last word automatically.
Also fixes FIX-37 indentation (was at 14-space, now correctly at
12-space to run outside the 'if corrected_punc != current_text' block).
FIX-36: Merge punctuation into grammar when they overlap same span
When grammar fixes المعلمون→المعلمين at [18:26] and punctuation
wants المعلمون→المعلمين. at the same [18:26], the overlap resolver
was dropping the punctuation entirely. Now detects that the punctuation
correction is grammar_correction + trailing punct char, and merges them
into a single suggestion: المعلمون→المعلمين.
This preserves both the grammar fix AND the terminal period.
FIX-33: Grammar rules fix_prepositions_advanced was corrupting
الامتحان→الامتحين. The regex blindly replaced ان→ين after prepositions,
but الامتحان is a root noun not a dual. Added blocklist of ~60 common
root-form nouns and skip all ال-prefixed words ending in ان.
FIX-34: Remove auto-re-analysis from Apply All in editor.js.
When spelling and grammar disagree (spelling strips ذهبن→ذهب,
grammar re-adds ذهبوا→ذهبن), the auto-re-analysis creates an
infinite loop of contradictory suggestions.
FIX-35: Spelling no longer strips valid conjugation suffixes.
The insertion fix was accepting ذهبن→ذهب (removing final ن) as
'extra char removed'. But ن is a feminine plural verb suffix.
Now blocks removal of suffix chars (ن,ت,ا,ي,ة,و,ه) from word
endings, preventing spelling from undoing grammar corrections.
FIX-32: Fix word duplication bug in website frontend
Root cause: applyAllSuggestions used a mixed DOM+text fallback approach.
After DOM-based fixes shifted the editor text, the text-based fallback
used stale offsets from the original API response, inserting corrections
at wrong positions. This caused 'المشروعالمشروع.' (word duplication).
Fix:
1. applyAllSuggestions now uses PURE text-based reverse-order replacement.
Since patches are sorted highest-offset-first, each offset is valid
because nothing before it has been modified yet.
2. _isApplyingSuggestion guard now stays active 400ms (past the 300ms
re-analysis timeout) to prevent normalize()/input events from
triggering duplicate analysis calls.
FIX-31: .rstrip() before checking trailing punct regex — model may
output trailing whitespace after hallucinated dots, breaking the
dollar-sign anchor.
Log fix: Grammar and Punctuation service logs were hardcoding '...'
after [:80] truncation even for short texts. This made it appear
that the models were adding ellipsis when they weren't. Now only
appends '...' when text is actually >80 chars.
Phase 14b: Fix spelling punct hallucination + cross-stage overlap
FIX-31: Strip trailing punctuation (... . etc) hallucinated by the
AraSpell model that wasn't in the input. This prevents dot
accumulation across pipeline runs.
PatchSet: Spelling + Punctuation patches from different stages now
always coexist (they're compatible corrections). Previously,
punctuation's higher priority caused spelling patches on
adjacent/overlapping words to be dropped ('فيالمشروع' phantom).
- StageLocker: Grammar now only blocked by spelling locks when >50%
of the grammar diff falls INSIDE the spelling lock. Previously,
any boundary touch (even 1 char) blocked grammar entirely.
This fixes the Collision Dataset bug (32% pass rate) where
spelling locks bled into adjacent words and blocked grammar.
- PatchSet: Relaxed overlap resolution — patches with <50% overlap
of the smaller patch are allowed to coexist. This prevents
punctuation patches from dropping spelling patches on adjacent words.
Both fixes verified with unit tests.
Merge punctuation_rules V1+V2: V2 threshold+fallback, V1 softened exclamation guard for short texts
FIX-28a: Digit guard in bidirectional spelling path
- Numbers (2020, 150) were OOV→IV hallucinated into Arabic words
- Root cause: bidirectional fix had no digit protection
FIX-28b: Prefix-swap guard in bidirectional path
- فبالتالي→وبالتالي leaked through (same stem, prefix swap)
FIX-29: Exclamation mark guard in punctuation rules
- PuncAra added ! to declarative sentences (التاريخ→التاريخ!)
- Now only allows !/؟ when text has exclamatory cue words
FIX-30: Prefix-stripping protection in spelling filter
- وبالمستشفيات→والمستشفيات (stripped ب from compound prefix)
- Blocks corrections that strip compound prefixes (وبال, فبال, etc.)
Adversarial diagnostic: 83.3% pass (30/36), 6 failures identified
Full benchmark: 320 tests across 8 datasets, 55% overall pass rate