Spaces:
Sleeping
Sleeping
Commit History
-
feat: cross-detector language policy + tidy structure & file naming
Language routing (core/language.py + pipeline): English text is now processed only by PII, NER, and Profanity; Indonesian-only detectors (Word Quality, Risky Content, Filler, Special Char, Syntax, Field-Fit) are skipped. Coverage of the three cross-language detectors is expanded. Detectors & coverage - Filler: drop English greetings (hi/hello/hallo/helo); keep halo/hai/hei. - Profanity: add English lexicon (high_en/medium_en.txt), active cross-language. - PII: international patterns SSN_US (+validator), NIN_UK, IBAN, MAC_ADDRESS, TELEPON_US + UI placeholders; no collision with Indonesian numbers. - NER: English titles (Mr/Mrs/Ms/Miss/Sir), 23 org suffixes & 14 global acronyms; span trimming reworked to "longest connector-aware capitalized run" ("Mr. John Smith from Apple Inc" -> "Mr. John Smith", "Dia bekerja di OpenAI Inc" -> "OpenAI Inc", "Bank of America" kept intact). Refactor & cleanup - Unify main detector file names to *_detector.py (filler/profanity/ risky_content/s…5ddfd1f
-
feat: word-quality-aware detectors + NER, Filler, Field-Fit fixes
Word Quality corrections (slang/typo) are now reused across detectors so garbled/typo'd text is still caught. WQ runs first in the pipeline and the resulting norm_text is passed to Risky Content, Syntax, Field-Fit, and Filler. Badge offsets & text still point to the original via word-index alignment. - pipeline: compute norm_text once, pass to risky/syntax/fieldfit/filler; bump NER above word_quality in dedup priority so proper nouns aren't overridden by false typo badges - risky_content: accept norm_text as a check variant + _word_align_map for accurate spans (e.g. "bgmn cr mmbuwat bom" -> detected as HARMFUL) - syntax: score PLL on normalized sentence, display the original sentence - field_fit: match rules & embeddings on norm_text; dynamic Layer 1 reason; fix wrong UI EXPLAIN text for cross-field ML findings - filler: add "hi/hello" greetings; match patterns on norm_text; THANKS pattern now consumes the full politeness tail ("terima kasih sebelumnya ya atas bantuannya")…d607ac1
-
refactor: curate lexicons to Indonesia-only
Lexicons & detectors (curate at the source, no allow/deny patches): - Remove the English profanity dictionaries (external_en.txt, blocklist_en.txt) and their subtraction logic; strip English words from high.txt/medium.txt at the source so the profanity lexicon is purely Indonesian. - NER: replace the hand-made common_non_entities list with the full stopwords-iso Indonesian set; expand names_id from an online list, then filter it against KBBI so dictionary words (bunga, mawar, cinta) no longer trigger person false positives. - Word Quality: drop the non_id_short_tokens skip-list (rely on the KBBI + wordfreq gate). - import_lexicons.py: remove the English-profanity importers, add a KBBI-filtered names importer; update provenance in sources.json.85020b5
-
feat: add Field-Fit & Completeness detectors, rework ML layers, expand Indonesian-only lexicons
New detectors - Field-Fit: detect content in the wrong field (persona in Task → suggest moving to Role), rule-based - Completeness: prompt-level coverage note (LOW_COVERAGE) delivered via prompt_notes/banner - Pipeline 8 → 10 detectors; add sub-servers, test pages, and unit tests ML layers - Responsible: REMOVED the ML injection layer — the English-trained model caused false positives on neutral Indonesian text; now regex-only - Profanity: add Layer 2 IndoBERTweet toxicity classifier (optional, advice-only, graceful fallback) - Word Quality: add Layer 2 IndoBERT fill-mask (suppress-only — drops typo false positives) - Remove library-availability checks (find_spec transformers/torch) across all detectors Lexicons & data - Import more complete online sources: okky new_kamusalay (slang 4k→15k), stopwords-iso (758), KBBI wordlist (66k formal words) — with English filtering and curation-preserving merge - Externalize inline NER lists (location/title/non-entity cues) & English blocklis…794bedb
-
refactor: improve detectors logic, clean fallback lexicons, and polish UI layout/labels
a6a6b96 -
feat: strengthen detectors, slim API schema, and improve UX
Continues the Indonesia-only effort with detector correctness fixes, a leaner response schema, one-click suggestions and expanded hidden-character detection. Detector fixes: - PII: require a context keyword for passport; add a word boundary to "rek" so it no longer matches inside other words (e.g. "merek") - Responsible: carry evidence start/end offsets from the checker instead of a fragile string re-search, so highlights land correctly - Profanity: move medical/religious terms (payudara, setan, iblis) to the whitelist so educational prompts are not flagged - NER: guard single-token lexicon names at sentence start, wire the global org/place patterns into the rule set, and chunk long text before inference - Word Quality: move the tech whitelist to resources/lexicons; keep the English wordfreq guard (it prevents mis-correcting English words, it is not detection) Indonesia-only cleanup: - Remove the dead English regex from responsible and filler and the fragile substring fil…170ad43
-
feat: strengthen all detectors and simplify to Indonesia-only
Word Quality - Add _TECH_WHITELIST with 100+ technical/library/algorithm terms to avoid typo false positives - Expand l33tspeak cap from 15 to 25 characters; add SymSpell fallback for long words like "m3ng1mpl3mntasikan" - Pattern 5: detect Jakarta/Betawi -in suffix forms ("nemenin", "bacain") as SLANG, not TYPO - Context-aware reclassification: TYPO confidence < 0.82 becomes SLANG when the sentence contains >= 2 other informal words - Fix _language_for_token: use cross-language frequency comparison with a 10x threshold instead of immediately falling back to the default - Add 16 -in suffix entries to slang_id.tsv - Remove English SymSpell dictionary (_SYMSPELL_DICT_CACHE_EN) Profanity & Responsible - Move English profanity from _BUILTIN_HIGH/_BUILTIN_MEDIUM to _ENGLISH_PROFANITY_BLOCKLIST so it is not shown as a badge - Add "speda" (informal "sepeda") to the whitelist so it is not flagged as profanity - Responsible K1: add 7 self-harm patterns (Indonesian + English) as H…aa16b4b
-
feat: add syntax checker and improve multilingual prompt analysis
- add Indonesian word-order syntax checker with optional IndoBERT scoring - integrate syntax findings into pipeline status, analysis, hot reload, and sub-server startup - add syntax issue highlighting and explanation in the web UI - improve English/Indonesian handling across PII, filler, NER, profanity, responsible, and word quality detectors - tune lexicons to reduce false positives for Indonesian termsc85625f
-
feat: prepare pipeline for HF Spaces deployment
- add Docker Space config and GitHub Actions auto deploy - externalize detector lexicons and add import script - add lightweight language routing for Indonesian and English rules - improve fast pipeline startup and frontend highlight behavior - update README with production and lexicon workflow713c68e -
Initial commit
ea3fcbe