The بال...ون/ان and ل...ون/ان patterns had no blocklist protection.
بالامتحان would become بالامتحين, للإنسان would become للإنسين.
Now all three preposition regexes use the same callback with:
- _PREP_BLOCKLIST check
- ان suffix → skip (almost always root noun form)
Tests: 39 passing (10 new for FIX-33b).
- FIX-35: Narrowed suffix set from {ن,ت,ا,ي,ة,و,ه} to {ن,ت} only.
Only verb conjugation suffixes are blocked. ة/ا/ي/و/ه are legitimate
typo corrections that should not be blocked.
- FIX-37: Raised word threshold from 2 to 4. Prevents noisy period
suggestions while user is still typing short phrases.
- FIX-32b: Removed duplicate .sort() in applyAllSuggestions — already
sorted at line 679, second sort at 695 was redundant.
- Tests updated to match narrowed suffix expectations.
Add test suite for FIX-33 through FIX-37 (29 tests)
FIX-28a: Digit guard in bidirectional spelling path
- Numbers (2020, 150) were OOV→IV hallucinated into Arabic words
- Root cause: bidirectional fix had no digit protection
FIX-28b: Prefix-swap guard in bidirectional path
- فبالتالي→وبالتالي leaked through (same stem, prefix swap)
FIX-29: Exclamation mark guard in punctuation rules
- PuncAra added ! to declarative sentences (التاريخ→التاريخ!)
- Now only allows !/؟ when text has exclamatory cue words
FIX-30: Prefix-stripping protection in spelling filter
- وبالمستشفيات→والمستشفيات (stripped ب from compound prefix)
- Blocks corrections that strip compound prefixes (وبال, فبال, etc.)
Adversarial diagnostic: 83.3% pass (30/36), 6 failures identified
Full benchmark: 320 tests across 8 datasets, 55% overall pass rate
Real-world case: بالرفم→بالرغم (ف→غ) blocked by _is_small_spelling_change()
because ف↔غ is not in ORTHO_PAIRS.
The spelling model CORRECTLY detects the fix, but the filter
rejects it as a 'root change' instead of recognizing it as a
keyboard-adjacent typo.
Added Priority 6 recommendation: add keyboard-adjacent pairs
to ORTHO_PAIRS (Option A) or separate KEYBOARD_TYPO_PAIRS with
dampened confidence (Option B).
- Add _tel_events collector list in analyze_text()
- Append events to _tel_events alongside [FILTER-TEL] logger lines
- Include 'telemetry' key in API response JSON
- Update telemetry_capture.py to read from response instead of HF logs
- NO correction logic changes
Previously Rule 0c only rejected rearrangement (same punct chars in
different positions). Now also rejects substitution (comma→colon,
comma→semicolon) when the alphabetic content is unchanged.
Live tested: Fixes P013 (أحمد، كيف→أحمد كيف:) and H026
(الأمم، وبدونه→الأمم وبدونه؛). P012 and P015 already fixed.
FIX-25: Block grammar punct spacing (حالك؟→حالك ؟)
Grammar model inserts spaces around punctuation marks which is wrong
in Arabic. This affects P012-P015, H026, SC015-SC017 and more.
Traced from full failure analysis: 10+ failures share this pattern.
Added normalize-then-compare filter to catch spacing-only diffs.
FIX-24: Block grammar model punct stripping (البلاد.→البلاد)
Root cause traced from benchmark results: ALL 30 Hallucination FP samples
had the grammar model stripping the sentence-final period. The grammar
model diff was: 'البلاد.'→'البلاد' (type=grammar).
Added filter to block grammar diffs that only differ by punctuation marks
and/or tanween marks. This should restore Hallucination to ~97%.
FIX-23: Block grammar model tanween removal + HF log fetcher
1. Tanween blocker: grammar model strips tanween (جداً→جدا) from correct text.
Added filter to block diffs that only differ by tanween marks.
2. HF log fetcher script for automated tracing.
The regex patterns (ون→ين after prepositions, ون→وا after nasb particles)
were matching correct text everywhere. CamelTools-based rules handle these
with POS-tag awareness. Keep only narrow five-nouns patterns.
CRITICAL FIX: _DIRECTIONAL_BLOCKS was local scope → NameError in grammar diff filter
Root cause found from production logs:
NameError: name '_DIRECTIONAL_BLOCKS' is not defined (line 1633)
_DIRECTIONAL_BLOCKS was defined inside _compute_spelling_confidence() but
referenced in analyze_text()'s grammar diff processing section. This caused
ALL grammar corrections to silently crash and be dropped.
Fix: Move _DIRECTIONAL_BLOCKS to module level so both sections can access it.
- Expanded grammar pattern whitelist for SV suffix additions (ذهب→ذهبوا, يذهب→يذهبون)
- Added gender agreement suffix (جميل→جميلة) to whitelist
- Added preposition case (ون→ين after في/من) and nasb (ون→وا after لن/لم) regex fallbacks
- Smart pronoun-ه guard: uses IV check instead of hardcoded consonant list