rishik1111 commited on
Commit
b384948
·
1 Parent(s): 3d1e510

refactor: update branding, titles, and architecture to VECTOR Multilingual RAG Engine

Browse files
Files changed (6) hide show
  1. README.md +22 -32
  2. api/main.py +1 -1
  3. app.py +5 -5
  4. app/__init__.py +1 -1
  5. demo/index.html +2 -2
  6. rag_eval_report.md +1 -1
README.md CHANGED
@@ -1,16 +1,16 @@
1
  ---
2
- title: Hacker House Goa 2026 - Voice Indic RAG
3
- emoji: 🌴
4
  colorFrom: green
5
- colorTo: yellow
6
  sdk: docker
7
  app_port: 7860
8
  pinned: false
9
  license: mit
10
- short_description: Voice Indic RAG with Sub-10ms FAISS Retrieval
11
  ---
12
 
13
- # 🌴 Hacker House Goa 2026: Voice-Enabled Multilingual Indic RAG
14
 
15
  <div align="center">
16
 
@@ -31,7 +31,7 @@ short_description: Voice Indic RAG with Sub-10ms FAISS Retrieval
31
 
32
  ## 📌 Executive Summary
33
 
34
- Active runtime deployment is optimized for **3 core languages** (**English [`en`]**, **Hindi [`hi`]**, and **Marathi [`mr`]**) loading **148,854 in-memory vectors** (148,545 native passage vectors + 309 semantic longdoc vectors), achieving **~7.04 ms retrieval latency** (p95: 7.97 ms vs 50.0 ms budget).
35
 
36
  The system provides zero-code extensibility across **all 14 Indic languages** (*Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu*) plus **English** (15 languages total, **~743,000 deduplicated passages**) via a single configuration entry (`config.LANGUAGES`).
37
 
@@ -40,7 +40,7 @@ The system provides zero-code extensibility across **all 14 Indic languages** (*
40
  - 🛡️ **Cascaded 4-Tier Guardrails**: Stem regex, Meta Prompt-Guard 86M neural DPI/IPI shield, 6-class intent filter, and own-language centroid weighting.
41
  - 🔀 **Script-Aware BM25 + Dense Fusion**: Automatic cross-script detection bypassing BM25 penalties for cross-lingual queries.
42
  - 🧮 **Deterministic Synthesis**: TextRank graph centrality + SVD singular energy context ranking (zero LLM API latency or cost).
43
- - 🌴 **Command Center UI**: Retro-tropical Web Audio waveform interface with stage-by-stage telemetry waterfall.
44
 
45
  ---
46
 
@@ -48,31 +48,31 @@ The system provides zero-code extensibility across **all 14 Indic languages** (*
48
 
49
  ```mermaid
50
  graph TD
51
- A[Spoken Voice Audio / Text Bypass] --> B[Sarvam Saaras v3 STT + ffmpeg 16kHz Normalizer]
52
- B --> C[Language Resolution: config.LANGUAGES Router]
53
- C --> D[Guardrail 1: Tier-1 Fast Regex + Safety Patterns]
54
- D -- Safe --> PG[Guardrail 2: Meta Prompt-Guard 86M Neural DPI Shield]
55
  D -- Blocked --> X[Declined Response: Safety Violation]
56
- PG -- Safe --> IF[Guardrail 3: Pre-Retrieval Query Intent Filter]
57
  PG -- Injected --> X
58
  IF -- Factual --> E[Query Embedding: 'query: ' Prefix multilingual-e5-small INT8]
59
  IF -- Non-Factual Intent --> X
60
- E --> F[Guardrail 4: Centroid Distance Off-Topic Filter]
61
  F -- Off-Topic --> X
62
  F -- On-Topic --> CACHE{Dynamic Vector & Gold QA Cache}
63
  CACHE -- Cache Hit <0.5ms --> N[Grounded Response + Zero-Latency Fast Path]
64
- CACHE -- Cache Miss --> G[Parallel Multi-Strategy FAISS HNSW Retrieval]
65
  G --> H1[Passage Native Index: 148,545 Vectors]
66
  G --> H2[Semantic LongDoc Index: 309 Vectors]
67
  H1 --> I[Candidate Merge & Reciprocal Rank Fusion RRF k=60]
68
  H2 --> I
69
- I --> J[Adaptive Script-Aware BM25 Score Fusion]
70
  J --> K[Relevance & Disqualification Gate: Dense / CE Threshold]
71
  K -- Score < Threshold --> Y[Declined Response: No Relevant Info in Corpus]
72
  K -- High Relevance --> CS[Context Chunk Safety: Batched Prompt-Guard 86M IPI Scan]
73
  CS -- Poisoned Chunks --> X
74
- CS -- Clean Chunks --> L[Deterministic Non-LLM Context Synthesis: TextRank + SVD Energy]
75
- L --> M[Post-Generation Grounding & Hallucination Guardrail]
76
  M -- Grounded --> N[Grounded JSON Response + Full 9-Stage Telemetry]
77
  M -- Insufficient Info --> Y
78
  ```
@@ -244,19 +244,19 @@ pytest tests/ -v
244
 
245
  ```
246
  VECTOR/
247
- ├── api/ # FastAPI web server (/query, /health, /languages)
248
  ├── app/ # Fast ONNX retriever & 50ms benchmark runner
249
  ├── benchmark/ # Latency, cold-start, & throughput evaluation scripts
250
  ├── chunking/ # Native, sentence-window, semantic, & RRF splitters
251
  ├── data/ # FAISS HNSW indexes, centroids, JSONL corpora scripts
252
- ├── demo/ # Hacker House Goa 2026 Web Audio UI & assets
253
  ├── generation/ # TextRank + SVD non-LLM synthesis & LLM fallback
254
  ├── guardrails/ # 4-tier cascaded pre-retrieval & post-gen grounding
255
  ├── pipeline/ # Async 9-stage pipeline state machine & schemas
256
  ├── retrieval/ # multilingual-e5-small INT8 ONNX & FAISS engine
257
- ├── stt/ # Sarvam Saaras v3 STT & ffmpeg 16kHz audio pipeline
258
  ├── tests/ # 50/50 unit & integration test suite
259
- ├── app.py # Space entrypoint application
260
  ├── config.py # Single source of truth configuration
261
  ├── Dockerfile # Container definition for Hugging Face Spaces
262
  └── requirements.txt # Python dependencies
@@ -264,15 +264,5 @@ VECTOR/
264
 
265
  ---
266
 
267
- ## 🚀 Hugging Face Space Deployment
268
-
269
- Deployed via **Docker SDK** on Hugging Face Spaces:
270
- - **Live Space URL**: [https://ansh123456789-ragingoa.hf.space](https://ansh123456789-ragingoa.hf.space)
271
-
272
- > [!TIP]
273
- > Free `cpu-basic` Spaces sleep after 48h inactivity. Initial container spin-up takes **30–90s**. Warm runtime operates at **~7–16 ms**.
274
-
275
- ---
276
-
277
  ## 📜 License
278
- MIT License. Created for **Hacker House Goa 2026**.
 
1
  ---
2
+ title: VECTOR - Voice Indic RAG
3
+ emoji: ⚡
4
  colorFrom: green
5
+ colorTo: blue
6
  sdk: docker
7
  app_port: 7860
8
  pinned: false
9
  license: mit
10
+ short_description: VECTOR Voice Indic RAG with Sub-10ms FAISS Retrieval
11
  ---
12
 
13
+ # ⚡ VECTOR: Voice-Enabled Multilingual Indic RAG Engine
14
 
15
  <div align="center">
16
 
 
31
 
32
  ## 📌 Executive Summary
33
 
34
+ **VECTOR** is optimized for low-latency, voice-enabled Retrieval-Augmented Generation across Indian languages. Active runtime deployment is optimized for **3 core languages** (**English [`en`]**, **Hindi [`hi`]**, and **Marathi [`mr`]**) loading **148,854 in-memory vectors** (148,545 native passage vectors + 309 semantic longdoc vectors), achieving **~7.04 ms retrieval latency** (p95: 7.97 ms vs 50.0 ms budget).
35
 
36
  The system provides zero-code extensibility across **all 14 Indic languages** (*Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu*) plus **English** (15 languages total, **~743,000 deduplicated passages**) via a single configuration entry (`config.LANGUAGES`).
37
 
 
40
  - 🛡️ **Cascaded 4-Tier Guardrails**: Stem regex, Meta Prompt-Guard 86M neural DPI/IPI shield, 6-class intent filter, and own-language centroid weighting.
41
  - 🔀 **Script-Aware BM25 + Dense Fusion**: Automatic cross-script detection bypassing BM25 penalties for cross-lingual queries.
42
  - 🧮 **Deterministic Synthesis**: TextRank graph centrality + SVD singular energy context ranking (zero LLM API latency or cost).
43
+ - 🌴 **VECTOR Command Center UI**: Retro-tropical Web Audio waveform interface with stage-by-stage telemetry waterfall.
44
 
45
  ---
46
 
 
48
 
49
  ```mermaid
50
  graph TD
51
+ A[Spoken Voice Audio / Text Bypass] --> B[Sarvam Saaras STT Engine + ffmpeg 16kHz Normalizer]
52
+ B --> C[VECTOR Language Resolution Router: config.LANGUAGES]
53
+ C --> D[VECTOR Guardrail 1: Tier-1 Fast Stem Regex + Safety Patterns]
54
+ D -- Safe --> PG[VECTOR Guardrail 2: Meta Prompt-Guard 86M Neural DPI Shield]
55
  D -- Blocked --> X[Declined Response: Safety Violation]
56
+ PG -- Safe --> IF[VECTOR Guardrail 3: Pre-Retrieval Query Intent Filter]
57
  PG -- Injected --> X
58
  IF -- Factual --> E[Query Embedding: 'query: ' Prefix multilingual-e5-small INT8]
59
  IF -- Non-Factual Intent --> X
60
+ E --> F[VECTOR Guardrail 4: Centroid Distance Off-Topic Filter]
61
  F -- Off-Topic --> X
62
  F -- On-Topic --> CACHE{Dynamic Vector & Gold QA Cache}
63
  CACHE -- Cache Hit <0.5ms --> N[Grounded Response + Zero-Latency Fast Path]
64
+ CACHE -- Cache Miss --> G[VECTOR Parallel Multi-Strategy FAISS HNSW Engine]
65
  G --> H1[Passage Native Index: 148,545 Vectors]
66
  G --> H2[Semantic LongDoc Index: 309 Vectors]
67
  H1 --> I[Candidate Merge & Reciprocal Rank Fusion RRF k=60]
68
  H2 --> I
69
+ I --> J[VECTOR Adaptive Script-Aware BM25 Score Fusion]
70
  J --> K[Relevance & Disqualification Gate: Dense / CE Threshold]
71
  K -- Score < Threshold --> Y[Declined Response: No Relevant Info in Corpus]
72
  K -- High Relevance --> CS[Context Chunk Safety: Batched Prompt-Guard 86M IPI Scan]
73
  CS -- Poisoned Chunks --> X
74
+ CS -- Clean Chunks --> L[VECTOR Deterministic Context Synthesis: TextRank + SVD Energy]
75
+ L --> M[VECTOR Post-Generation Grounding & Hallucination Guardrail]
76
  M -- Grounded --> N[Grounded JSON Response + Full 9-Stage Telemetry]
77
  M -- Insufficient Info --> Y
78
  ```
 
244
 
245
  ```
246
  VECTOR/
247
+ ├── api/ # VECTOR FastAPI web server (/query, /health, /languages)
248
  ├── app/ # Fast ONNX retriever & 50ms benchmark runner
249
  ├── benchmark/ # Latency, cold-start, & throughput evaluation scripts
250
  ├── chunking/ # Native, sentence-window, semantic, & RRF splitters
251
  ├── data/ # FAISS HNSW indexes, centroids, JSONL corpora scripts
252
+ ├── demo/ # VECTOR Web Audio UI & visual assets
253
  ├── generation/ # TextRank + SVD non-LLM synthesis & LLM fallback
254
  ├── guardrails/ # 4-tier cascaded pre-retrieval & post-gen grounding
255
  ├── pipeline/ # Async 9-stage pipeline state machine & schemas
256
  ├── retrieval/ # multilingual-e5-small INT8 ONNX & FAISS engine
257
+ ├── stt/ # Sarvam Saaras STT & ffmpeg 16kHz audio pipeline
258
  ├── tests/ # 50/50 unit & integration test suite
259
+ ├── app.py # VECTOR Space entrypoint application
260
  ├── config.py # Single source of truth configuration
261
  ├── Dockerfile # Container definition for Hugging Face Spaces
262
  └── requirements.txt # Python dependencies
 
264
 
265
  ---
266
 
 
 
 
 
 
 
 
 
 
 
267
  ## 📜 License
268
+ MIT License. VECTOR Multilingual RAG Engine.
api/main.py CHANGED
@@ -62,7 +62,7 @@ async def lifespan(app: FastAPI):
62
 
63
 
64
  app = FastAPI(
65
- title="Voice-Enabled Indic RAG API",
66
  description="Instrumented low-latency Voice RAG pipeline for Indic languages (English, Hindi, Marathi)",
67
  version="1.0.0",
68
  lifespan=lifespan,
 
62
 
63
 
64
  app = FastAPI(
65
+ title="⚡ VECTOR Voice Indic RAG API",
66
  description="Instrumented low-latency Voice RAG pipeline for Indic languages (English, Hindi, Marathi)",
67
  version="1.0.0",
68
  lifespan=lifespan,
app.py CHANGED
@@ -1,5 +1,5 @@
1
  """
2
- Hugging Face Space Application for Hacker House Goa 2026: Voice-Enabled Indic RAG.
3
  Renders the full retro-tropical Command Center UI and exposes FastAPI endpoints.
4
  ZeroGPU compatible.
5
  """
@@ -74,7 +74,7 @@ def _dummy_zerogpu():
74
 
75
 
76
  # Create core FastAPI application
77
- app = FastAPI(title="🌴 Hacker House Goa 2026 — Voice-Enabled Indic RAG")
78
 
79
  app.add_middleware(
80
  CORSMiddleware,
@@ -104,7 +104,7 @@ async def serve_index() -> HTMLResponse:
104
  if demo_file.exists():
105
  with open(demo_file, "r", encoding="utf-8") as f:
106
  return HTMLResponse(content=f.read())
107
- return HTMLResponse(content="<h1>Hacker House Goa 2026 Command Center</h1>")
108
 
109
 
110
  @app.get("/health", response_class=JSONResponse)
@@ -204,7 +204,7 @@ async def query_pipeline(
204
 
205
 
206
  # Create Gradio interface block and mount on FastAPI app
207
- with gr.Blocks(title="🌴 Hacker House Goa 2026 — Voice Indic RAG") as demo:
208
  gr.HTML(get_custom_html())
209
  dummy_btn = gr.Button("zero_gpu_anchor", visible=False)
210
  dummy_btn.click(fn=_dummy_zerogpu)
@@ -215,5 +215,5 @@ app = gr.mount_gradio_app(app, demo, path="/gradio")
215
  if __name__ == "__main__":
216
  port = int(os.getenv("PORT", "7860"))
217
  host = os.getenv("HOST", "0.0.0.0")
218
- print(f"[Space Startup] Starting Hacker House Goa Command Center UI on http://{host}:{port}")
219
  uvicorn.run(app, host=host, port=port)
 
1
  """
2
+ Hugging Face Space Application for VECTOR: Voice-Enabled Indic RAG.
3
  Renders the full retro-tropical Command Center UI and exposes FastAPI endpoints.
4
  ZeroGPU compatible.
5
  """
 
74
 
75
 
76
  # Create core FastAPI application
77
+ app = FastAPI(title="⚡ VECTOR — Voice-Enabled Indic RAG")
78
 
79
  app.add_middleware(
80
  CORSMiddleware,
 
104
  if demo_file.exists():
105
  with open(demo_file, "r", encoding="utf-8") as f:
106
  return HTMLResponse(content=f.read())
107
+ return HTMLResponse(content="<h1>VECTOR 2026 Command Center</h1>")
108
 
109
 
110
  @app.get("/health", response_class=JSONResponse)
 
204
 
205
 
206
  # Create Gradio interface block and mount on FastAPI app
207
+ with gr.Blocks(title="⚡ VECTOR — Voice Indic RAG") as demo:
208
  gr.HTML(get_custom_html())
209
  dummy_btn = gr.Button("zero_gpu_anchor", visible=False)
210
  dummy_btn.click(fn=_dummy_zerogpu)
 
215
  if __name__ == "__main__":
216
  port = int(os.getenv("PORT", "7860"))
217
  host = os.getenv("HOST", "0.0.0.0")
218
+ print(f"[Space Startup] Starting VECTOR Command Center UI on http://{host}:{port}")
219
  uvicorn.run(app, host=host, port=port)
app/__init__.py CHANGED
@@ -1,5 +1,5 @@
1
  """
2
- Hacker House Goa 2026 Indic Voice RAG Application Package.
3
  """
4
 
5
  from pathlib import Path
 
1
  """
2
+ VECTOR Indic Voice RAG Application Package.
3
  """
4
 
5
  from pathlib import Path
demo/index.html CHANGED
@@ -3,7 +3,7 @@
3
  <head>
4
  <meta charset="UTF-8" />
5
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6
- <title>Hacker House Goa 2026 — Voice-Enabled Multilingual RAG</title>
7
 
8
  <!-- Fonts (Creative & Stylish Beach Pop Art & Logo Typography) -->
9
  <link rel="preconnect" href="https://fonts.googleapis.com">
@@ -488,7 +488,7 @@
488
 
489
  <div>
490
  <div class="flex items-center gap-2">
491
- <span class="action-burst text-xs">HACKER HOUSE GOA 2026</span>
492
  <span class="px-2 py-0.5 bg-black text-beachcyan font-comic text-xs border-2 border-black rounded shadow-brutal-sm">POP ART EDITION</span>
493
  </div>
494
  <h1 class="font-pop-art text-3xl md:text-4xl tracking-wide mt-0.5">VOICE INDIC RAG 🌊 SURF THE VECTORS! 🏄‍♂️</h1>
 
3
  <head>
4
  <meta charset="UTF-8" />
5
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6
+ <title>VECTOR 2026 — Voice-Enabled Multilingual RAG Engine</title>
7
 
8
  <!-- Fonts (Creative & Stylish Beach Pop Art & Logo Typography) -->
9
  <link rel="preconnect" href="https://fonts.googleapis.com">
 
488
 
489
  <div>
490
  <div class="flex items-center gap-2">
491
+ <span class="action-burst text-xs">VECTOR 2026</span>
492
  <span class="px-2 py-0.5 bg-black text-beachcyan font-comic text-xs border-2 border-black rounded shadow-brutal-sm">POP ART EDITION</span>
493
  </div>
494
  <h1 class="font-pop-art text-3xl md:text-4xl tracking-wide mt-0.5">VOICE INDIC RAG 🌊 SURF THE VECTORS! 🏄‍♂️</h1>
rag_eval_report.md CHANGED
@@ -1,7 +1,7 @@
1
  # RAG Evaluation Report
2
 
3
  **Author:** Manus AI
4
- **Target:** [Hacker House Goa 2026 — Voice-Enabled Multilingual RAG](https://ansh123456789-ragingoa.hf.space/) [1]
5
  **Evaluation date:** 16 August 2026
6
  **Evaluation type:** Black-box VIGOURLS-style RAG evaluation battery
7
 
 
1
  # RAG Evaluation Report
2
 
3
  **Author:** Manus AI
4
+ **Target:** VECTOR — Voice-Enabled Multilingual Indic RAG [1]
5
  **Evaluation date:** 16 August 2026
6
  **Evaluation type:** Black-box VIGOURLS-style RAG evaluation battery
7