Spaces:
Sleeping
Sleeping
Commit History
-
feat: cross-detector language policy + tidy structure & file naming
Language routing (core/language.py + pipeline): English text is now processed only by PII, NER, and Profanity; Indonesian-only detectors (Word Quality, Risky Content, Filler, Special Char, Syntax, Field-Fit) are skipped. Coverage of the three cross-language detectors is expanded. Detectors & coverage - Filler: drop English greetings (hi/hello/hallo/helo); keep halo/hai/hei. - Profanity: add English lexicon (high_en/medium_en.txt), active cross-language. - PII: international patterns SSN_US (+validator), NIN_UK, IBAN, MAC_ADDRESS, TELEPON_US + UI placeholders; no collision with Indonesian numbers. - NER: English titles (Mr/Mrs/Ms/Miss/Sir), 23 org suffixes & 14 global acronyms; span trimming reworked to "longest connector-aware capitalized run" ("Mr. John Smith from Apple Inc" -> "Mr. John Smith", "Dia bekerja di OpenAI Inc" -> "OpenAI Inc", "Bank of America" kept intact). Refactor & cleanup - Unify main detector file names to *_detector.py (filler/profanity/ risky_content/s…5ddfd1f
-
feat: add flowchart diagrams (proses evaluasi, use case, PII) + simplify field-fit ML layer
- Add three new flowchart diagrams: Proses Evaluasi, Use Case, dan PII detector flow - Remove Layer-1 rule-based field-fit (peran check); keep Layer-2 ML embedding only - Improve Filler detector: better l33t normalization & offset tracking across text variants - Refine Special Char labels (smart quote, full-width consolidation) - Update Field-Fit test page and detector implementation to match simplified flow - Add background color white to diagram renderingdd30e82
-
feat: word-quality-aware detectors + NER, Filler, Field-Fit fixes
Word Quality corrections (slang/typo) are now reused across detectors so garbled/typo'd text is still caught. WQ runs first in the pipeline and the resulting norm_text is passed to Risky Content, Syntax, Field-Fit, and Filler. Badge offsets & text still point to the original via word-index alignment. - pipeline: compute norm_text once, pass to risky/syntax/fieldfit/filler; bump NER above word_quality in dedup priority so proper nouns aren't overridden by false typo badges - risky_content: accept norm_text as a check variant + _word_align_map for accurate spans (e.g. "bgmn cr mmbuwat bom" -> detected as HARMFUL) - syntax: score PLL on normalized sentence, display the original sentence - field_fit: match rules & embeddings on norm_text; dynamic Layer 1 reason; fix wrong UI EXPLAIN text for cross-field ML findings - filler: add "hi/hello" greetings; match patterns on norm_text; THANKS pattern now consumes the full politeness tail ("terima kasih sebelumnya ya atas bantuannya")…d607ac1 -
feat: add syntax test page (web/syntax-test.html)
6316c87
-
refactor: rename detektor Responsible → Konten Berisiko (risky_content
d15e0b1
-
feat: detect excessive blank lines + readable whitespace-fix labels
Special Char detector: - Add MULTIPLE_NEWLINE: flag 3+ consecutive line breaks (optionally interspersed with spaces/tabs) and collapse them to a single blank line (\n\n). A normal paragraph break (two newlines) is left untouched. Also wired into normalize_text so the one-click autofix cleans it. Frontend (suggestion popover): - The suggestion row showed nothing after the arrow when the replacement was a single space or a newline (invisible). Whitespace/empty replacements now render a readable label: "(jadi 1 spasi)", "(jadi 1 baris kosong)", or "(hapus)". The underlying replacement value is unchanged, so Apply still works.915188f -
feat: cross-field ML field-fit + context-aware typo reranking; drop Completeness
Field-Fit — add ML Layer 2 (embedding similarity, on by default): - Check EVERY field, not just Task. Each field's content is embedded and compared (cosine) to per-field prototype centroids; if another field outranks the current one by >= FIELDFIT_ML_MARGIN and cos >= FIELDFIT_ML_COS_MIN, suggest moving it there. Advice-only, LOW severity. - Keep the rule layer (persona at start of Task -> Role) and dedup overlaps. - Model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (FIELDFIT_ML_MODEL). Graceful fallback to rules; off with --fast/--no-fieldfit-ml. - Chosen empirically: embedding ~93% acc / ~31 ms-per-field vs zero-shot NLI ~40% / ~11 s-per-field. Word Quality — fill-mask Layer 2 now reranks, not just suppresses: - The IndoBERT fill-mask pass also reranks the same-edit-distance SymSpell candidates by contextual probability, so "searang" in "...adalah ___ guru" corrects to "seorang" (not "sekarang"), while "...ini ___ juga" still corrects to "sekarang". …f2f9311
-
refactor: curate lexicons to Indonesia-only
Lexicons & detectors (curate at the source, no allow/deny patches): - Remove the English profanity dictionaries (external_en.txt, blocklist_en.txt) and their subtraction logic; strip English words from high.txt/medium.txt at the source so the profanity lexicon is purely Indonesian. - NER: replace the hand-made common_non_entities list with the full stopwords-iso Indonesian set; expand names_id from an online list, then filter it against KBBI so dictionary words (bunga, mawar, cinta) no longer trigger person false positives. - Word Quality: drop the non_id_short_tokens skip-list (rely on the KBBI + wordfreq gate). - import_lexicons.py: remove the English-profanity importers, add a KBBI-filtered names importer; update provenance in sources.json.85020b5
-
feat: add Field-Fit & Completeness detectors, rework ML layers, expand Indonesian-only lexicons
New detectors - Field-Fit: detect content in the wrong field (persona in Task → suggest moving to Role), rule-based - Completeness: prompt-level coverage note (LOW_COVERAGE) delivered via prompt_notes/banner - Pipeline 8 → 10 detectors; add sub-servers, test pages, and unit tests ML layers - Responsible: REMOVED the ML injection layer — the English-trained model caused false positives on neutral Indonesian text; now regex-only - Profanity: add Layer 2 IndoBERTweet toxicity classifier (optional, advice-only, graceful fallback) - Word Quality: add Layer 2 IndoBERT fill-mask (suppress-only — drops typo false positives) - Remove library-availability checks (find_spec transformers/torch) across all detectors Lexicons & data - Import more complete online sources: okky new_kamusalay (slang 4k→15k), stopwords-iso (758), KBBI wordlist (66k formal words) — with English filtering and curation-preserving merge - Externalize inline NER lists (location/title/non-entity cues) & English blocklis…794bedb
-
refactor: improve detectors logic, clean fallback lexicons, and polish UI layout/labels
a6a6b96 -
fix: reduce evaluator false positives in prompt analysis
- filter NER output to only surface relevant privacy entities - trim punctuation from PII and NER entity spans - prioritize rule-based NER over fragmented ML entities - skip mixed-script tokens in word quality checks - whitelist common prompt term "sarkas" from typo correction - make responsible ML fallback run only when regex finds no issue - add Indonesian-only warning and refresh demo/evaluator UI labels - add regression tests for entity spans, NER filtering, homoglyphs, and ML fallback97f3a53 -
feat(web): improve evaluation progress and final prompt layout
9326ee9 -
feat(web): improve responsive evaluation progress and field caching
2cf5552 -
fix: README.md HF configuration
d0c6bfd -
feat: strengthen detectors, slim API schema, and improve UX
Continues the Indonesia-only effort with detector correctness fixes, a leaner response schema, one-click suggestions and expanded hidden-character detection. Detector fixes: - PII: require a context keyword for passport; add a word boundary to "rek" so it no longer matches inside other words (e.g. "merek") - Responsible: carry evidence start/end offsets from the checker instead of a fragile string re-search, so highlights land correctly - Profanity: move medical/religious terms (payudara, setan, iblis) to the whitelist so educational prompts are not flagged - NER: guard single-token lexicon names at sentence start, wire the global org/place patterns into the rule set, and chunk long text before inference - Word Quality: move the tech whitelist to resources/lexicons; keep the English wordfreq guard (it prevents mis-correcting English words, it is not detection) Indonesia-only cleanup: - Remove the dead English regex from responsible and filler and the fragile substring fil…170ad43
-
feat: strengthen all detectors and simplify to Indonesia-only
Word Quality - Add _TECH_WHITELIST with 100+ technical/library/algorithm terms to avoid typo false positives - Expand l33tspeak cap from 15 to 25 characters; add SymSpell fallback for long words like "m3ng1mpl3mntasikan" - Pattern 5: detect Jakarta/Betawi -in suffix forms ("nemenin", "bacain") as SLANG, not TYPO - Context-aware reclassification: TYPO confidence < 0.82 becomes SLANG when the sentence contains >= 2 other informal words - Fix _language_for_token: use cross-language frequency comparison with a 10x threshold instead of immediately falling back to the default - Add 16 -in suffix entries to slang_id.tsv - Remove English SymSpell dictionary (_SYMSPELL_DICT_CACHE_EN) Profanity & Responsible - Move English profanity from _BUILTIN_HIGH/_BUILTIN_MEDIUM to _ENGLISH_PROFANITY_BLOCKLIST so it is not shown as a badge - Add "speda" (informal "sepeda") to the whitelist so it is not flagged as profanity - Responsible K1: add 7 self-harm patterns (Indonesian + English) as H…aa16b4b
-
feat: add syntax checker and improve multilingual prompt analysis
- add Indonesian word-order syntax checker with optional IndoBERT scoring - integrate syntax findings into pipeline status, analysis, hot reload, and sub-server startup - add syntax issue highlighting and explanation in the web UI - improve English/Indonesian handling across PII, filler, NER, profanity, responsible, and word quality detectors - tune lexicons to reduce false positives for Indonesian termsc85625f
-
style: widen desktop app container
98c54d4 -
feat: prepare pipeline for HF Spaces deployment
- add Docker Space config and GitHub Actions auto deploy - externalize detector lexicons and add import script - add lightweight language routing for Indonesian and English rules - improve fast pipeline startup and frontend highlight behavior - update README with production and lexicon workflow713c68e -
Initial commit
ea3fcbe