Fetching metadata from the HF Docker repository...
-
Fix: Preserve input punctuation during grammar correction
Bayan AI
-
Phase 11: Hierarchical StageLocker — grammar overrides spelling locks
- Refactored StageLocker with STAGE_PRIORITY hierarchy:
protection(99) > grammar(3) > spelling(2) > punctuation(1)
- Added is_locked_for() and is_locked_by_for() hierarchy-aware methods
- Grammar stage now uses is_locked_for('grammar') to override spelling locks
- Punctuation stage uses is_locked_by_for('punctuation') to stay blocked
- Deduplicated pipeline_collision.json (PC021-PC050 were identical copies)
- Expanded to 50 unique collision test cases across 7 categories
- Fixed test_collisions.py hardcoded path, runs all 50 samples
- Added run_collision_benchmark.py with structured reporting
- Registered collision as 8th dataset in benchmark_runner.py
- All 13 StageLocker hierarchy unit tests passed -
inspecting and auditing
Mohamed Atef
-
Phase 12: Fix 6 batches — grammar pipeline bypass, religious/structured protection, punct rearrangement, SV/gender agreement
Batch 1: Grammar IV->OOV bypass for case/nasb/five_nouns/dual patterns (14 FN)
Batch 2: Religious text now skips ALL stages including spelling (9 FP)
Batch 3: Expanded HAMZA_WHITELIST with 30+ new entries
Batch 4: Reject punct rearrangement + mid-sentence insertion (4 FP)
Batch 5: Skip spelling for URLs/emails/Latin/hashtags (7+4 FP)
Batch 6: Fix present-tense SV agreement + noun-adj gender (15 FN)
Also: Added directional blocks for كانت→كأنت (H016)
