Spaces:
Sleeping
Sleeping
Commit History
-
feat: cross-detector language policy + tidy structure & file naming
Language routing (core/language.py + pipeline): English text is now processed only by PII, NER, and Profanity; Indonesian-only detectors (Word Quality, Risky Content, Filler, Special Char, Syntax, Field-Fit) are skipped. Coverage of the three cross-language detectors is expanded. Detectors & coverage - Filler: drop English greetings (hi/hello/hallo/helo); keep halo/hai/hei. - Profanity: add English lexicon (high_en/medium_en.txt), active cross-language. - PII: international patterns SSN_US (+validator), NIN_UK, IBAN, MAC_ADDRESS, TELEPON_US + UI placeholders; no collision with Indonesian numbers. - NER: English titles (Mr/Mrs/Ms/Miss/Sir), 23 org suffixes & 14 global acronyms; span trimming reworked to "longest connector-aware capitalized run" ("Mr. John Smith from Apple Inc" -> "Mr. John Smith", "Dia bekerja di OpenAI Inc" -> "OpenAI Inc", "Bank of America" kept intact). Refactor & cleanup - Unify main detector file names to *_detector.py (filler/profanity/ risky_content/s…5ddfd1f
-
refactor: curate lexicons to Indonesia-only
Lexicons & detectors (curate at the source, no allow/deny patches): - Remove the English profanity dictionaries (external_en.txt, blocklist_en.txt) and their subtraction logic; strip English words from high.txt/medium.txt at the source so the profanity lexicon is purely Indonesian. - NER: replace the hand-made common_non_entities list with the full stopwords-iso Indonesian set; expand names_id from an online list, then filter it against KBBI so dictionary words (bunga, mawar, cinta) no longer trigger person false positives. - Word Quality: drop the non_id_short_tokens skip-list (rely on the KBBI + wordfreq gate). - import_lexicons.py: remove the English-profanity importers, add a KBBI-filtered names importer; update provenance in sources.json.85020b5
-
feat: add Field-Fit & Completeness detectors, rework ML layers, expand Indonesian-only lexicons
New detectors - Field-Fit: detect content in the wrong field (persona in Task → suggest moving to Role), rule-based - Completeness: prompt-level coverage note (LOW_COVERAGE) delivered via prompt_notes/banner - Pipeline 8 → 10 detectors; add sub-servers, test pages, and unit tests ML layers - Responsible: REMOVED the ML injection layer — the English-trained model caused false positives on neutral Indonesian text; now regex-only - Profanity: add Layer 2 IndoBERTweet toxicity classifier (optional, advice-only, graceful fallback) - Word Quality: add Layer 2 IndoBERT fill-mask (suppress-only — drops typo false positives) - Remove library-availability checks (find_spec transformers/torch) across all detectors Lexicons & data - Import more complete online sources: okky new_kamusalay (slang 4k→15k), stopwords-iso (758), KBBI wordlist (66k formal words) — with English filtering and curation-preserving merge - Externalize inline NER lists (location/title/non-entity cues) & English blocklis…794bedb
-
Initial commit
ea3fcbe