Upload 59 files

.env

@@ -11,9 +11,6 @@ POSTGRES_READONLY_URL=postgresql://chat_ro_user:strong_password@hayabusa.proxy.r
 NEO4J_URI=neo4j+s://b55eb74c.databases.neo4j.io
 NEO4J_USER=b55eb74c
 NEO4J_PASSWORD=ba7mRDw67kwIDDquVXp_e10msUFvaOz0ipH7_nE-a4Y
-NEO4J_DATABASE=b55eb74c
-AURA_INSTANCEID=b55eb74c
-AURA_INSTANCENAME=KG
 
 # --- Redis (reuse existing instance, separate DB index) ---
 REDIS_URL=redis://default:UzTsAxeHPgOQUKPjDSvlxkdMcRVoJalU@hayabusa.proxy.rlwy.net:56307

README.md

@@ -1,11 +1 @@
----
-title: Chat Service
-emoji: 🔥
-colorFrom: pink
-colorTo: pink
-sdk: docker
-pinned: false
-short_description: Chat Service
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+chat-service

TASKS.md

@@ -0,0 +1,182 @@
+# TASKS — Conversational Analytics Assistant (chat-service)
+
+Step-by-step build checklist, derived from the implementation plan. Check items off as you go. Order matters — each milestone assumes the previous one works.
+
+---
+
+## Milestone 0 — Repo & environment setup
+
+- [x] Create `chat-service/` folder structure
+- [x] `git init`, set `origin` remote
+- [x] Create virtual environment (`python -m venv venv`)
+- [x] Activate venv
+- [x] Create `.env.example` with all vars from plan §10 (placeholders)
+- [x] Create local `.env` (real values, gitignored)
+- [x] Add `.gitignore` (venv, `.env`, `__pycache__`, `*.pyc`)
+- [x] Initial commit ("chore: project skeleton")
+
+---
+
+## Milestone 1 — Service skeleton (boots + Groq round-trip)
+
+- [x] Add core deps to `requirements.txt`: `fastapi`, `uvicorn[standard]`, `pydantic-settings`, `groq`, `python-dotenv`
+- [x] `pip install -r requirements.txt`
+- [x] `app/config.py` — `Settings(BaseSettings)` class reading env vars
+- [x] `app/agent/llm.py` — thin Groq client wrapper (single `ask(prompt: str) -> str` function)
+- [x] `app/main.py`:
+  - [x] `GET /health` → `{"status": "ok"}`
+  - [x] Temporary `POST /chat` → calls Groq wrapper directly, no LangGraph yet, just to prove the chain works
+- [x] Run locally: `uvicorn app.main:app --reload --port 8002`
+- [x] Test `/health` with curl/Postman
+- [x] Test `/chat` with curl/Postman — confirm real Groq response comes back
+- [x] Write `Dockerfile` (Python 3.12 base, matches existing services)
+- [x] Build + run container locally, re-test `/health` and `/chat` inside Docker
+- [x] Commit ("feat: service skeleton with Groq round-trip")
+
+**Done when:** `/health` and `/chat` both work locally and in Docker.
+
+---
+
+## Milestone 2 — First real tool + LangGraph agent (`sql_query_tool`)
+
+- [x] Add deps: `langgraph`, `langchain-groq`, `sqlalchemy`, `psycopg[binary]` (or `asyncpg`)
+- [x] Create **read-only Postgres role** (`chat_ro_user`) — DB side, not app code
+- [x] `app/db/postgres.py` — read-only connection pool using `POSTGRES_READONLY_URL`
+- [x] Confirm exact schema fields needed (resolve plan §12.1: `ArticleFakeNews`, `ArticleTopic`, etc. field names)
+- [x] `app/tools/sql_tool.py`:
+  - [x] Define `metric` enum: `fake_news_count`, `articles_by_topic`, `sentiment_breakdown`, `propaganda_count`, `hate_speech_count`, `article_lookup`, `dialect_breakdown`
+  - [x] Write one parametrized SQL query per metric
+  - [x] Return shape: `{"rows": [...], "sql_used": "...", "source_refs": [...]}`
+- [x] Write unit tests for each metric in `tests/test_sql_tool.py` (against a test/staging DB)
+- [x] `app/agent/state.py` — `AgentState` TypedDict (`messages`, `sources`, `session_id`, `user_id`)
+- [x] `app/agent/prompts.py` — system prompt (identity, tool-preference rule, citation rule, language-matching rule, no-fabrication rule)
+- [x] `app/agent/graph.py` — LangGraph ReAct-style agent wired to `sql_query_tool` only
+- [x] Wire `/chat` in `main.py` to call the LangGraph agent instead of the raw Groq wrapper
+- [x] `app/schemas/request.py` / `app/schemas/response.py` — formalize `ChatRequest` / `ChatResponse` models per plan §4
+- [x] Manually test: "How many fake news articles were detected last month?" → correct tool call + correct answer
+- [x] Commit ("feat: sql_query_tool + LangGraph agent")
+
+**Done when:** agent correctly answers at least 3 different stat questions using real DB data, citing article IDs.
+
+---
+
+## Milestone 3 — Django proxy endpoint
+
+- [x] In Django: add `CHAT_SERVICE_URL` and `CHAT_SERVICE_INTERNAL_TOKEN` to settings/env
+- [x] New view: `POST /api/v1/chat/`
+  - [x] Verify JWT (reuse existing auth)
+  - [ ] Apply DRF throttling/rate limit (no need for now)
+  - [x] Forward `{session_id, message, user_id}` to chat service with internal token header
+  - [x] Return chat service's JSON response unchanged
+- [x] Confirm chat service rejects requests without the internal token
+- [x] Test end-to-end: frontend-style request → Django → chat service → Groq → back
+- [x] Commit on Django repo ("feat: chat proxy endpoint")
+
+**Done when:** a JWT-authenticated request through Django reaches the chat service and gets a real answer.
+
+---
+
+## Milestone 4 — `graph_query_tool` (Neo4j)
+
+- [x] Confirm current Neo4j label/relationship conventions used by `kg_sync` (resolve plan §12.2)
+- [x] ~~Create read-only Neo4j role/user~~ — N/A, Aura Free has no RBAC; enforcement moved to app-level (read-only transactions + no write Cypher in tool code)
+- [x] `app/db/neo4j.py` — driver wrapper using `NEO4J_*` vars, enforcing read-only via explicit read transactions
+- [x] `app/tools/graph_tool.py`:
+  - [x] Define `query_type` enum (7 total): `entity_connections`, `entity_mentions`, `shared_entities_between_articles`, `most_connected_entities`, `article_verdict`, `claims_for_article`, `analysis_for_article`
+  - [x] Write one parametrized Cypher template per query_type
+  - [x] Return shape: `{"rows": [...], "source_refs": [...]}`
+- [x] Add `graph_query_tool` to the LangGraph agent's tool list
+- [x] Unit tests in `tests/test_graph_tool.py`
+- [x] Manually test: "Who is connected to [entity]?" → correct Cypher template used, correct answer
+- [x] Commit ("feat: graph_query_tool")
+
+**Done when:** agent correctly answers at least 2 relationship questions using real graph data.
+
+---
+
+## Milestone 5 — Memory (multi-turn)
+
+- [x] Add dep: Redis client (`redis` / `redis[hiredis]`)
+- [x] `app/memory/redis_checkpointer.py` — LangGraph checkpointer backed by Redis, key prefix `chat:checkpoint:{session_id}`
+- [x] Wire checkpointer into `app/agent/graph.py` compilation
+- [x] Set `CHAT_SESSION_TTL_SECONDS` TTL on session keys
+- [x] Implement history trimming: keep last `CHAT_HISTORY_MAX_TURNS` verbatim, summarize older turns beyond that
+- [x] Manual test: ask a question, then a follow-up ("and what about last month?") using the same `session_id` — confirm context carries over
+- [x] Manual test: new `session_id` → confirm no memory leaks across sessions
+- [x] Commit ("feat: Redis-backed multi-turn memory")
+
+**Done when:** follow-up questions correctly resolve using prior turn context, and TTL/trimming work as expected.
+
+---
+
+## Milestone 6 — Citations end-to-end
+
+- [x] `app/tools/article_tool.py` — `get_article_detail(article_id)` for resolving full title/snippet
+- [x] Confirm every tool (`sql_query_tool`, `graph_query_tool`) consistently returns `source_refs`
+- [x] Agent post-processing step: merge all `source_refs` collected during the turn into final `sources[]` in the response
+- [x] Update system prompt to explicitly require citing every ID it used
+- [x] Manual test: ask a question, verify `sources[]` in the JSON response matches what was actually used
+- [x] Commit ("feat: citation resolution")
+
+**Done when:** every factual answer includes a non-empty, accurate `sources[]` array.
+
+---
+
+## Milestone 7 — `hybrid_search_tool`
+
+- [x] Confirm reuse path for `_embed` (call existing Django/analysis embedding function - the function is in app/services/nlp_client.py)
+- [x] `app/tools/hybrid_tool.py`:
+  - [x] Step 1: embed query text
+  - [x] Step 2: pgvector cosine similarity search over `search_vector`
+  - [x] Step 3: for each top article, pull mentioned entities from Neo4j
+  - [x] Step 4: merge into single ranked result set with combined `source_refs`
+- [x] Add tool to agent's tool list
+- [ ] Manual test: "Articles about the election, and who's mentioned" → correct combined results
+- [x] Commit ("feat: hybrid_search_tool")
+
+**Done when:** at least one combined semantic+graph question is answered correctly with merged citations.
+
+---
+
+## Milestone 8 — Frontend chat widget
+
+- [ ] Generate/persist `session_id` client-side (e.g. `localStorage`, created on first load)
+- [ ] Chat UI component: message list, input box, send button
+- [ ] Call `POST /api/v1/chat/` with `{session_id, message}`
+- [ ] Render `answer` text
+- [ ] Render `sources[]` as clickable links (article/entity)
+- [ ] Loading state while waiting for response
+- [ ] Basic error state (service down / timeout)
+- [ ] Commit ("feat: chat widget")
+
+**Done when:** a real user can open the widget, ask a question, get an answer with clickable citations, and ask a follow-up.
+
+---
+
+## Milestone 9 — Tests, docs, hardening
+
+- [ ] Unit tests for all tools (`sql_tool`, `graph_tool`, `hybrid_tool`) with mocked/test DB data
+- [ ] `tests/test_agent_e2e.py` — at least 3 end-to-end scenarios (stat question, relationship question, follow-up question)
+- [ ] `chat-service/README.md` — setup instructions, env vars, how to run locally + in Docker
+- [ ] Verify `.env.example` is complete and matches plan §10
+- [ ] Security checklist review (plan §8) — confirm every box is actually true in the running system:
+  - [ ] Read-only DB roles confirmed (test that a write attempt fails)
+  - [ ] No raw SQL/Cypher ever comes from the LLM — code review confirms only enum params reach tools
+  - [ ] Internal token required and enforced
+  - [ ] Rate limiting active on Django proxy
+  - [ ] Groq key not logged anywhere
+  - [ ] Tool calls logged (name + params, not raw rows)
+- [ ] Add `chat_service` to `docker-compose.yml` (no published host port in prod config)
+- [ ] Final walkthrough / demo run-through for defense
+- [ ] Tag/commit ("chore: v1 complete")
+
+**Done when:** everything above is checked and you can demo the full flow live without surprises.
+
+---
+
+## Open items to resolve before/while coding (carry over from plan §12)
+
+- [ ] Confirm exact Postgres field names per metric
+- [ ] Confirm Neo4j label/relationship naming conventions
+- [ ] Decide: authenticated-only sessions, or also support anonymous/demo sessions for defense
+- [ ] Decide: citation links deep-link into real frontend routes, or raw IDs for now

app/db/neo4j.py

@@ -1,5 +1,4 @@
 from __future__ import annotations
-import os
 
 from neo4j import AsyncGraphDatabase, AsyncDriver, AsyncTransaction
 
@@ -47,4 +46,4 @@ async def run_read_query(query: str, **params) -> list[dict]:
         return await result.data()
 
     async with driver.session() as session:
-        return await session.execute_read(_tx_func)
+        return await session.execute_read(_tx_func)