Spaces:
Running on Zero
Running on Zero
Commit History
4 days ago
-
Free-tier LLM provider chain: Gemini primary, Groq backup, OpenRouter optional; auto-failover on quota/auth errors Running
6e74004 -
Fix voice QA + podcast pipeline (llama_index workflow API, PII boundary, honest failures)
953fef2 -
feat: MCP knowledge-base PII protocol (mask at boundary, unmask on demand)
Make the MCP server safely attachable anywhere (Claude Desktop, any MCP client): - services/mcp_pii.py: dependency-light boundary helpers - stable [TYPE_N] placeholders per real value in a shared TTL namespace; masked_source_dicts masks snippet + metadata strings before any search result crosses the tool boundary. - mcp_server: semantic_search/answer_question now return MASKED sources (raw to_dict leak closed; pii_placeholders count returned); generate_tags feeds the LLM a masked view; new tool pii_restore_placeholders unmasks the agent's final user-facing reply from memory (1h TTL, never persisted). - End-to-end verified offline: search output zero raw PII, agent composes with placeholders, restore returns real values. - core/chunker: killed the 'aggressive cleaning' that stripped @/+ from stored chunks - it silently corrupted emails, phone country codes and checksum ids, hurting both retrieval and PII masking. Chunks now keep natural text (whitespace-collapsed …e763ab9 -
feat: persistence across Space restarts (snapshot backup + /data autodetect)
- services/backup_service.py: tar.gz snapshot of the whole data tree (WAL-checkpointed SQLite + FAISS + BM25 + uploads) pushed to a private HF dataset repo; restore on boot only into an empty dir so newer local data is never clobbered; traversal-guarded extraction; oversize files (podcast audio, PERSIST_BACKUP_MAX_FILE_MB) excluded. - config.py: auto-redirect DATA_DIR/DB_PATH to the /data mount when HF Persistent Storage is attached; PERSIST_BACKUP_* knobs. - bootstrap: restore() before stores open in __init__ (never fails boot) and debounced flush_if_due() in the maintenance loop whenever library.db changed - vectors/deletes/uploads survive restarts within ~2 min of any change. - tests/test_backup_service.py: round-trip + traversal-refusal + disabled-without-secrets + max-MB trim (offline, stubbed network). - SCALING.md: persistence section + honest '100 docs in 1 min on GPU?' stage table (GPU speeds encode; parse/OCR stays CPU-bound).23c41aa -
feat: ZeroGPU support (RTX Pro 6000 Blackwell) for the local embedding leg
Spaces hardware upgrade detected. ZeroGPU contract handled end to end: - app.py imports spaces as the very first import (before torch/ transformers chain) so CUDA init is routed by the ZeroGPU runtime - embedding_service: local encode goes through a module-level @spaces.GPU(duration=60) encoder (model moved .to('cuda') inside the decorated call; SPACES_ZERO_GPU 1/true detection; no-op elsewhere) - background queue workers drive encode outside a request context, where ZeroGPU can refuse allocation -> per-batch CPU fallback keeps ingestion alive (warns once per batch) - SCALING.md documents the ZeroGPU realities. 18/18 testse4ee5fc -
feat: multi-file upload to queue; honest 1M/1min throughput notes; crash recovery
- Upload widget accepts multiple files; each is archived + enqueued as its own background job and the request returns instantly (verified 3 at once locally). Message lists job ids per file. - SCALING.md: 'Ingestion speed reality' section with the actual math - 1M docs/1min would need ~500k-1M chunk-embeddings/s (hundreds of GPUs or precomputed snapshots); table comparing CPU/API/GPU/cluster paths and the wired levers. - bootstrap: register the queue worker task on the service so worker_alive reports truthfully (was always 'stopped'). - ingestion_queue_service: crash-recovery sweep requeues stale 'running' jobs at worker boot (claim_next only picked 'pending' before), with idempotency test (18/18).4021ac2
6 days ago
-
feat: DoChat - ChatGPT/Claude-style chat tab with visible PII masking
- 'Ask Questions' tab renamed to '🤖 DoChat': gr.Chatbot (messages type) with user/assistant bubbles, avatars, copy buttons, Send button + Enter-to-send, 'New chat' reset, and a Sources-accordion under the thread (answer footer cites confidence + source count). - New 🛡️ PII masking checkbox (default ON). Per answer the UI shows a visible status banner: masking ON prints how many identifiers were hidden and the exact placeholder view the agent received; OFF warns that the agent saw raw text. mcp answer_question_async returns pii metadata; generative_tool.answer_question gained mask_pii=False opt-out path (skips mask/restore for that call only). - 17/17 tests; chat flows verified locally for both banner statesbdcf37e -
perf: uploads go to background queue; faster local embedding default
User-visible pain: a 2.1MB PDF blocked the Upload button for 115s+ (Space CPU embeddings at 0.4-2 chunks/s plus the old OpenAI retry tax). - upload_and_process_file now archives the upload under data/documents/uploads and enqueues a 'document' job; the request returns instantly and progress/failures show in the Ingestion Queue tab (mcp_server.enqueue_document_async added). - EMBEDDING_LOCAL_MODEL default flipped to harrier-oss-v1-270m: measured ~4x faster than 0.6b (~1.7 chunks/s warm), 640-dim vectors, near-equal MTEB Multilingual quality for its size. 0.6b stays one env var away. - SCALING.md updated; 17/17 tests; UI rebuild verified364c2a2 -
fix: PII tab library-document dropdown live-sync + region/type toggles honored
- 'Scan a Library Document' dropdown was built once at boot on an empty library and never refreshed. All document dropdowns (library, content, tags, delete, PII scan, podcast) now update through one shared State broadcast fed by refresh/upload/delete/queue-job flows. - pii_scan_document now passes the Detection Options toggles through (was hardcoded True/True). - BUG: mcp_server.pii_scan_async parsed disabled_types then dropped them (mask_for_agent never received them) - 'Disable specific types' had no effect anywhere. Wire-through fixed. - vector_store: cold rebuild in auto mode hard-coded IVF and crashed faiss ('nx >= k') on tiny libraries; now stays flat below the IVF switch threshold, and _train_ivf falls back to flat under 64 vectors. Verified locally: ingest -> dropdown refresh -> scan with PAN/card disabled (4 masked, correct entities), both regions off (globals only); 17/17 tests.6cd589c -
feat: auto-fallback to local embeddings on permanent OpenAI quota/auth errors
Log showed the retry storm (5 attempts x backoff per batch) against a dead OPENAI_API_KEY ('credit_balance_exhausted'). _openai_request_with_retry now classifies auth/quota failures (401/403, insufficient_quota, credit_balance_exhausted and friends) as permanent: no retries, and generate_embeddings flips the service to the free local model mid-request (_disable_openai_leg) with a loud warning that old OpenAI-indexed libraries need scripts/reembed_library.py once. Transient 429s/5xx keep the existing backoff-retry behavior. tests/test_embedding_fallback.py covers both paths (17 passing).2ad0d13 -
fix: catch checksum-failed Aadhaar digits near context keywords
Aadhaar detection silently skipped any 12-digit run failing the Verhoeff checksum - even next to 'aadhaar/aadhar/uid/uidai' - leaving raw digits exposed to the agent. Fail direction was wrong. - core/pii_patterns: IN_AADHAAR_UNVERIFIED safety net (same regex, no validator, context-gated; verified detector outranks it) - services/pii_service: _pattern_priority replaces the inverted mapping (validated 3.5 > validator-less tier 3.0 with India-first registry order preserved for identical spans > validator+weak 1.0) - app.py: unverified type in PII-tab choices + India region switch - tests: checksum-failure fallback + unverified mask/restore roundtrip (15 passing); SCALING.md documents the design and disclaimer20734d5 -
fix: disable gradio SSR mode on launch
Space logs showed a Node SSR 405 ('POST method not allowed. No actions exist for this page') whenever non-API clients POST to the page URL - harmless but noisy. ssr_mode=False drops the Node SSR layer entirely; the page renders client-side. Space health verified online before the change (RUNNING, GET / + MCP SSE + config all 200).0f186f8 -
chore: untrack legacy AiDigitalLibraryAssistant copy
851b887 -
feat: free local embedding default (Microsoft harrier-oss-v1-0.6b)
- config: EMBEDDING_LOCAL_MODEL + EMBEDDING_QUERY_PROMPT knobs - embedding_service: local fallback model configurable; query-side instruction prompt applied only for query encodes (harrier web_search_query); fp32 on CPU (bf16 measured ~10x slower), bf16 on CUDA - search_tool: query encodes flagged is_query=True - llamaindex_service: voice/podcast fallback uses the same local model - scripts/reembed_library.py: one-time migration (backup, re-encode, BLOB rewrite, FAISS compact+persist) for model switches - tests: FakeEmbedder accepts is_query; SCALING.md env table + free embedding section with measured throughput4ce11da -
Fix HF crash: asyncio compat shim (nest_asyncio + uvicorn loop_factory) and UTF-8 stdio
ee81207 -
Fix HF runtime: pin uvicorn<0.35 (loop_factory kwarg breaks nest_asyncio patch)
b82fca8 -
Fix HF build: pin mistralai to 1.x API and add requests
3531d28 -
ci: retrigger HF Space deploy after token fix
165ab1b -
ci: retrigger HF Space deploy
bcbb565 -
Fix HF deploy: target the real Space name and align gradio pin to 5.38.0
8060e14 -
ci: trigger HF Space deploy (secrets now configured)
9306f7d -
CI: auto-deploy main to Hugging Face Space on push
f61ad1b -
Scale to 1M+ docs: SQLite core, FAISS auto-tiering, hybrid retrieval, PII protection, ingestion worker queue
- Storage: single SQLite (WAL) core (documents/chunks/chunk_vectors/tombstones); BLOB embeddings as source of truth - Retrieval: hybrid dense FAISS + bm25s sparse with Reciprocal Rank Fusion; optional cross-encoder rerank; auto tiering flat -> IVF/SQ8 with compaction via tombstones - Ingestion: streaming batched embed->index pipeline, O(N) chunker fix, persistent SQLite-backed worker queue with job status (survives restarts) - PII protection: India/USA/global detectors with checksum validators (Aadhaar Verhoeff, card Luhn), reversible placeholder masking at every LLM boundary (mappings memory-only), custom keyword/regex rules persisted, log masking, optional NER (presidio) - UI: new PII Protection and Ingestion Queue Gradio tabs; MCP tools scan_pii, pii_custom_rule_add/remove/list, ingest_directory_queue, ingest_queued, job_status, list_ingest_jobs - Ops: SCALING.md (architecture, capacity math, env knobs, ops playbook), migrate + benchmark scripts, 13 offline tests (pipeline, RRF, IVF…106c7b9
-
Migrate to Gradio native MCP integration - fixes HF Spaces deployment
a59fc6f
-
Built eleven Ai agent
51faba9
-
updated app.py
24e491c -
updated project name
902bca6 -
Fix config.py conflict markers
7abdd18 -
Merge branch 'main' of https://github.com/nihald2000/intelligent-content-organizer
d6ffa40 -
Merge remote changes and resolve config conflict
ed65b20 -
Update Intelligent Content Organizer with new features (Voice, Podcast, LlamaIndex)
e9f7396 -
Implement voice assistant and podcast generation
nihald2000Unverified signatureAdded voice and podcast generation features with associated services.2c10c54 -
updated gradio layout for new tools
5a9d2b2 -
feat: Add LlamaIndex RAG, ElevenLabs Voice AI, Podcast Generation, and Blaxel deployment
- LlamaIndexService with ReAct agent for agentic RAG - ElevenLabsService for conversational AI - PodcastGeneratorService for document-to-podcast conversion - Enhanced MCP server with voice_qa and generate_podcast tools - Blaxel cloud deployment configuration - Updated dependencies and configuration21ad0c0
-
Update services/llm_service.py
Verified signature30025c3 -
Update config.py
Verified signature1992efa
-
Update README.md
Verified signaturebefe9e4
-
Update services/llm_service.py
Verified signature4228dd5
-
Update README.md
Verified signaturecc72373 -
Update README.md
Verified signature70b7429 -
Update README.md
Verified signature45c4909 -
Update README.md
Verified signature15131ea -
Update app.py
Verified signaturea5d8df6
-
Update mcp_server.py
Verified signatured72be5c -
Update requirements.txt
Verified signatureae5f974 -
Resolved Dropdown issue And MCP Server
4a0fab5
-
Update services/ocr_service.py
Verified signature16ca714
-
Update app.py
Verified signature74d5794 -
Update requirements.txt
Verified signature809023c -
Update app.py
Verified signature1080be3