Spaces:
Sleeping
Sleeping
Commit History
-
feat: cross-detector language policy + tidy structure & file naming
Language routing (core/language.py + pipeline): English text is now processed only by PII, NER, and Profanity; Indonesian-only detectors (Word Quality, Risky Content, Filler, Special Char, Syntax, Field-Fit) are skipped. Coverage of the three cross-language detectors is expanded. Detectors & coverage - Filler: drop English greetings (hi/hello/hallo/helo); keep halo/hai/hei. - Profanity: add English lexicon (high_en/medium_en.txt), active cross-language. - PII: international patterns SSN_US (+validator), NIN_UK, IBAN, MAC_ADDRESS, TELEPON_US + UI placeholders; no collision with Indonesian numbers. - NER: English titles (Mr/Mrs/Ms/Miss/Sir), 23 org suffixes & 14 global acronyms; span trimming reworked to "longest connector-aware capitalized run" ("Mr. John Smith from Apple Inc" -> "Mr. John Smith", "Dia bekerja di OpenAI Inc" -> "OpenAI Inc", "Bank of America" kept intact). Refactor & cleanup - Unify main detector file names to *_detector.py (filler/profanity/ risky_content/s…5ddfd1f
-
feat: add flowchart diagrams (proses evaluasi, use case, PII) + simplify field-fit ML layer
- Add three new flowchart diagrams: Proses Evaluasi, Use Case, dan PII detector flow - Remove Layer-1 rule-based field-fit (peran check); keep Layer-2 ML embedding only - Improve Filler detector: better l33t normalization & offset tracking across text variants - Refine Special Char labels (smart quote, full-width consolidation) - Update Field-Fit test page and detector implementation to match simplified flow - Add background color white to diagram renderingdd30e82
-
refactor: rename detektor Responsible → Konten Berisiko (risky_content
d15e0b1
-
feat: cross-field ML field-fit + context-aware typo reranking; drop Completeness
Field-Fit — add ML Layer 2 (embedding similarity, on by default): - Check EVERY field, not just Task. Each field's content is embedded and compared (cosine) to per-field prototype centroids; if another field outranks the current one by >= FIELDFIT_ML_MARGIN and cos >= FIELDFIT_ML_COS_MIN, suggest moving it there. Advice-only, LOW severity. - Keep the rule layer (persona at start of Task -> Role) and dedup overlaps. - Model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (FIELDFIT_ML_MODEL). Graceful fallback to rules; off with --fast/--no-fieldfit-ml. - Chosen empirically: embedding ~93% acc / ~31 ms-per-field vs zero-shot NLI ~40% / ~11 s-per-field. Word Quality — fill-mask Layer 2 now reranks, not just suppresses: - The IndoBERT fill-mask pass also reranks the same-edit-distance SymSpell candidates by contextual probability, so "searang" in "...adalah ___ guru" corrects to "seorang" (not "sekarang"), while "...ini ___ juga" still corrects to "sekarang". …f2f9311
-
feat: add Field-Fit & Completeness detectors, rework ML layers, expand Indonesian-only lexicons
New detectors - Field-Fit: detect content in the wrong field (persona in Task → suggest moving to Role), rule-based - Completeness: prompt-level coverage note (LOW_COVERAGE) delivered via prompt_notes/banner - Pipeline 8 → 10 detectors; add sub-servers, test pages, and unit tests ML layers - Responsible: REMOVED the ML injection layer — the English-trained model caused false positives on neutral Indonesian text; now regex-only - Profanity: add Layer 2 IndoBERTweet toxicity classifier (optional, advice-only, graceful fallback) - Word Quality: add Layer 2 IndoBERT fill-mask (suppress-only — drops typo false positives) - Remove library-availability checks (find_spec transformers/torch) across all detectors Lexicons & data - Import more complete online sources: okky new_kamusalay (slang 4k→15k), stopwords-iso (758), KBBI wordlist (66k formal words) — with English filtering and curation-preserving merge - Externalize inline NER lists (location/title/non-entity cues) & English blocklis…794bedb
-
refactor: improve detectors logic, clean fallback lexicons, and polish UI layout/labels
a6a6b96 -
fix: reduce evaluator false positives in prompt analysis
- filter NER output to only surface relevant privacy entities - trim punctuation from PII and NER entity spans - prioritize rule-based NER over fragmented ML entities - skip mixed-script tokens in word quality checks - whitelist common prompt term "sarkas" from typo correction - make responsible ML fallback run only when regex finds no issue - add Indonesian-only warning and refresh demo/evaluator UI labels - add regression tests for entity spans, NER filtering, homoglyphs, and ML fallback97f3a53 -
feat: strengthen detectors, slim API schema, and improve UX
Continues the Indonesia-only effort with detector correctness fixes, a leaner response schema, one-click suggestions and expanded hidden-character detection. Detector fixes: - PII: require a context keyword for passport; add a word boundary to "rek" so it no longer matches inside other words (e.g. "merek") - Responsible: carry evidence start/end offsets from the checker instead of a fragile string re-search, so highlights land correctly - Profanity: move medical/religious terms (payudara, setan, iblis) to the whitelist so educational prompts are not flagged - NER: guard single-token lexicon names at sentence start, wire the global org/place patterns into the rule set, and chunk long text before inference - Word Quality: move the tech whitelist to resources/lexicons; keep the English wordfreq guard (it prevents mis-correcting English words, it is not detection) Indonesia-only cleanup: - Remove the dead English regex from responsible and filler and the fragile substring fil…170ad43