Spaces:
Running on Zero
Running on Zero
Commit History
14 days ago
-
feat: MCP knowledge-base PII protocol (mask at boundary, unmask on demand)
Make the MCP server safely attachable anywhere (Claude Desktop, any MCP client): - services/mcp_pii.py: dependency-light boundary helpers - stable [TYPE_N] placeholders per real value in a shared TTL namespace; masked_source_dicts masks snippet + metadata strings before any search result crosses the tool boundary. - mcp_server: semantic_search/answer_question now return MASKED sources (raw to_dict leak closed; pii_placeholders count returned); generate_tags feeds the LLM a masked view; new tool pii_restore_placeholders unmasks the agent's final user-facing reply from memory (1h TTL, never persisted). - End-to-end verified offline: search output zero raw PII, agent composes with placeholders, restore returns real values. - core/chunker: killed the 'aggressive cleaning' that stripped @/+ from stored chunks - it silently corrupted emails, phone country codes and checksum ids, hurting both retrieval and PII masking. Chunks now keep natural text (whitespace-collapsed …e763ab9
16 days ago
-
fix: catch checksum-failed Aadhaar digits near context keywords
Aadhaar detection silently skipped any 12-digit run failing the Verhoeff checksum - even next to 'aadhaar/aadhar/uid/uidai' - leaving raw digits exposed to the agent. Fail direction was wrong. - core/pii_patterns: IN_AADHAAR_UNVERIFIED safety net (same regex, no validator, context-gated; verified detector outranks it) - services/pii_service: _pattern_priority replaces the inverted mapping (validated 3.5 > validator-less tier 3.0 with India-first registry order preserved for identical spans > validator+weak 1.0) - app.py: unverified type in PII-tab choices + India region switch - tests: checksum-failure fallback + unverified mask/restore roundtrip (15 passing); SCALING.md documents the design and disclaimer20734d5 -
Fix HF crash: asyncio compat shim (nest_asyncio + uvicorn loop_factory) and UTF-8 stdio
ee81207 -
Scale to 1M+ docs: SQLite core, FAISS auto-tiering, hybrid retrieval, PII protection, ingestion worker queue
- Storage: single SQLite (WAL) core (documents/chunks/chunk_vectors/tombstones); BLOB embeddings as source of truth - Retrieval: hybrid dense FAISS + bm25s sparse with Reciprocal Rank Fusion; optional cross-encoder rerank; auto tiering flat -> IVF/SQ8 with compaction via tombstones - Ingestion: streaming batched embed->index pipeline, O(N) chunker fix, persistent SQLite-backed worker queue with job status (survives restarts) - PII protection: India/USA/global detectors with checksum validators (Aadhaar Verhoeff, card Luhn), reversible placeholder masking at every LLM boundary (mappings memory-only), custom keyword/regex rules persisted, log masking, optional NER (presidio) - UI: new PII Protection and Ingestion Queue Gradio tabs; MCP tools scan_pii, pii_custom_rule_add/remove/list, ingest_directory_queue, ingest_queued, job_status, list_ingest_jobs - Ops: SCALING.md (architecture, capacity math, env knobs, ops playbook), migrate + benchmark scripts, 13 offline tests (pipeline, RRF, IVF…106c7b9
-
Resolved Dropdown issue And MCP Server
4a0fab5
-
Gradio mcp
9145e48
-
delated storage
ed42a41 -
Server initialization
8ba2581
-
First vesrion
3e772ec