Grammar service:
- Add word count validation (reject if ratio < 0.5 or > 2.0)
- Add repetition detection (reject if 2+ consecutive repeated words)
- Add length check (reject if output > 2x input)
- Add character overlap check (reject if < 50% chars preserved)
Grammar diff in app.py:
- Break large replace blocks into individual word suggestions
- When word counts match, process EACH word pair separately
- Skip large blocks (>3 words) to prevent hallucination suggestions
- Each grammar suggestion now points to a single word, not entire sentence
- Keep spelling suggestions as-is (word-by-word, always correct)
- Build word-index map from corrected text back to original text
- Grammar diffs: spelling-corrected vs grammar-corrected (word-by-word)
- Map grammar positions back to original text via word-index map
- Skip grammar suggestions that overlap with spelling suggestions
- Remove broken unified diff approach that caused wrong alignments
fix: Rewrite suggestion system — unified diff eliminates text duplication - Instead of mapping grammar offsets back through OffsetMapper (broken), now: - Run full pipeline: spelling -> grammar -> punctuation - Do ONE clean diff: original_text vs final_text - Classify each change as spelling/grammar based on what spelling changed - Guarantees non-overlapping suggestions with correct offsets
revert: Restore ORIGINAL grammar params that were working (num_beams=4, max_length=512) - Remove all hallucination checks that were blocking valid corrections - The model WAS correcting ذهب→ذهبت before, only highlight was wrong - Keep word-level diff fix (in app.py) + raw output logging
fix: Increase analyze debounce to 1.5s to prevent text overwrite while typing - Debounce 500ms -> 1500ms - Abort in-flight requests when user starts typing again - Double-check no new input before running analysis
fix: Handle N-to-M word replacements in analyze post-filter - Process each original word individually against corrected segment - Accept 1-to-1 small edits and 1-to-N word splits within multi-word blocks - Previously all multi-word blocks were silently rejected
fix: Relax analyze post-filters to show AraSpell corrections in editor - Allow 3 char edits (was 2), 50% ratio (was 40%) - Lower word-split threshold from 12 to 5 chars
feat: Add AraSpell spelling correction endpoint (POST /api/spelling) with lazy model loading - Extract AraSpell pipeline into src/nlp/spelling/ (araspell_rules.py + araspell_service.py) - Pre-download AraBERT encoder-decoder + checkpoint in Dockerfile - Lazy load on first request, contextual=False to save RAM - Update health check to report spelling status
- HF Spaces free tier blocks ALL outbound DNS - no external API calls possible
- Load summarization model locally with float16 (280MB vs 560MB)
- Install CPU-only PyTorch (~200MB vs ~2GB)
- Spelling/punctuation/autocomplete gracefully degrade (return input unchanged)
- Health check accurately reports model availability
fix: DNS endpoint discovery - test multiple HF hostnames to find reachable one
- Add src/hf_inference.py: wrapper for HF Inference API calls
- Update app.py: USE_HF_API flag enables remote inference when HF_API_TOKEN is set
- All routes (summarize, spelling, grammar, punctuation, autocomplete, analyze) support HF API mode
- Health check returns HTTP 200 with all models=true in HF API mode
- No local model loading needed - saves 2GB+ RAM
- Backwards compatible: local models still work when HF_API_TOKEN is not set
Fix: Google link failure now falls back to full sign-in instead of error toast
