Spaces:
Running
Running
Commit History
-
feat: cross-detector language policy + tidy structure & file naming
Language routing (core/language.py + pipeline): English text is now processed only by PII, NER, and Profanity; Indonesian-only detectors (Word Quality, Risky Content, Filler, Special Char, Syntax, Field-Fit) are skipped. Coverage of the three cross-language detectors is expanded. Detectors & coverage - Filler: drop English greetings (hi/hello/hallo/helo); keep halo/hai/hei. - Profanity: add English lexicon (high_en/medium_en.txt), active cross-language. - PII: international patterns SSN_US (+validator), NIN_UK, IBAN, MAC_ADDRESS, TELEPON_US + UI placeholders; no collision with Indonesian numbers. - NER: English titles (Mr/Mrs/Ms/Miss/Sir), 23 org suffixes & 14 global acronyms; span trimming reworked to "longest connector-aware capitalized run" ("Mr. John Smith from Apple Inc" -> "Mr. John Smith", "Dia bekerja di OpenAI Inc" -> "OpenAI Inc", "Bank of America" kept intact). Refactor & cleanup - Unify main detector file names to *_detector.py (filler/profanity/ risky_content/s…5ddfd1f
-
refactor: rename detektor Responsible → Konten Berisiko (risky_content
d15e0b1
-
refactor: curate lexicons to Indonesia-only
Lexicons & detectors (curate at the source, no allow/deny patches): - Remove the English profanity dictionaries (external_en.txt, blocklist_en.txt) and their subtraction logic; strip English words from high.txt/medium.txt at the source so the profanity lexicon is purely Indonesian. - NER: replace the hand-made common_non_entities list with the full stopwords-iso Indonesian set; expand names_id from an online list, then filter it against KBBI so dictionary words (bunga, mawar, cinta) no longer trigger person false positives. - Word Quality: drop the non_id_short_tokens skip-list (rely on the KBBI + wordfreq gate). - import_lexicons.py: remove the English-profanity importers, add a KBBI-filtered names importer; update provenance in sources.json.85020b5
-
fix: reduce evaluator false positives in prompt analysis
- filter NER output to only surface relevant privacy entities - trim punctuation from PII and NER entity spans - prioritize rule-based NER over fragmented ML entities - skip mixed-script tokens in word quality checks - whitelist common prompt term "sarkas" from typo correction - make responsible ML fallback run only when regex finds no issue - add Indonesian-only warning and refresh demo/evaluator UI labels - add regression tests for entity spans, NER filtering, homoglyphs, and ML fallback97f3a53 -
feat: strengthen detectors, slim API schema, and improve UX
Continues the Indonesia-only effort with detector correctness fixes, a leaner response schema, one-click suggestions and expanded hidden-character detection. Detector fixes: - PII: require a context keyword for passport; add a word boundary to "rek" so it no longer matches inside other words (e.g. "merek") - Responsible: carry evidence start/end offsets from the checker instead of a fragile string re-search, so highlights land correctly - Profanity: move medical/religious terms (payudara, setan, iblis) to the whitelist so educational prompts are not flagged - NER: guard single-token lexicon names at sentence start, wire the global org/place patterns into the rule set, and chunk long text before inference - Word Quality: move the tech whitelist to resources/lexicons; keep the English wordfreq guard (it prevents mis-correcting English words, it is not detection) Indonesia-only cleanup: - Remove the dead English regex from responsible and filler and the fragile substring fil…170ad43
-
feat: strengthen all detectors and simplify to Indonesia-only
Word Quality - Add _TECH_WHITELIST with 100+ technical/library/algorithm terms to avoid typo false positives - Expand l33tspeak cap from 15 to 25 characters; add SymSpell fallback for long words like "m3ng1mpl3mntasikan" - Pattern 5: detect Jakarta/Betawi -in suffix forms ("nemenin", "bacain") as SLANG, not TYPO - Context-aware reclassification: TYPO confidence < 0.82 becomes SLANG when the sentence contains >= 2 other informal words - Fix _language_for_token: use cross-language frequency comparison with a 10x threshold instead of immediately falling back to the default - Add 16 -in suffix entries to slang_id.tsv - Remove English SymSpell dictionary (_SYMSPELL_DICT_CACHE_EN) Profanity & Responsible - Move English profanity from _BUILTIN_HIGH/_BUILTIN_MEDIUM to _ENGLISH_PROFANITY_BLOCKLIST so it is not shown as a badge - Add "speda" (informal "sepeda") to the whitelist so it is not flagged as profanity - Responsible K1: add 7 self-harm patterns (Indonesian + English) as H…aa16b4b
-
feat: add syntax checker and improve multilingual prompt analysis
- add Indonesian word-order syntax checker with optional IndoBERT scoring - integrate syntax findings into pipeline status, analysis, hot reload, and sub-server startup - add syntax issue highlighting and explanation in the web UI - improve English/Indonesian handling across PII, filler, NER, profanity, responsible, and word quality detectors - tune lexicons to reduce false positives for Indonesian termsc85625f
-
fix: improve detector accuracy based on QA findings
- profanity: add leet substitutions 9→g and 8→b to catch newer evasion variants - responsible: add 10 injection patterns for roleplay, hypothetical framing, and indirect requests (both Indonesian and English) - ner: exclude known idioms (Ibu Kota, Ibu Pertiwi, Bapak Bangsa) from person entity rule via negative lookahead; fix PT/CV rule to only consume capitalized words, preventing greedy match into surrounding sentence - pii: tighten address pattern so Desa/Kelurahan/Kecamatan only trigger when followed by additional address components (RT/RW/Kab/Kec), reducing false positives in narrative text5b02fa7 -
feat: prepare pipeline for HF Spaces deployment
- add Docker Space config and GitHub Actions auto deploy - externalize detector lexicons and add import script - add lightweight language routing for Indonesian and English rules - improve fast pipeline startup and frontend highlight behavior - update README with production and lexicon workflow713c68e -
Initial commit
ea3fcbe