chore: format README for Hugging Face [skip ci]

README.md

@@ -1,363 +1,12 @@
-# Joule Dynamics Real Estate Intelligence Server (Pulse AI)
-
-[![FastAPI](https://img.shields.io/badge/FastAPI-005571?style=for-the-badge&logo=fastapi)](https://fastapi.tiangolo.com)
-[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org)
-[![Groq LPU](https://img.shields.io/badge/Groq-LPU_Inference-F05A28?style=for-the-badge)](https://groq.com)
-[![Supabase](https://img.shields.io/badge/Supabase-PostgreSQL_%26_pgvector-3ECF8E?style=for-the-badge&logo=supabase&logoColor=white)](https://supabase.com)
-[![Resend](https://img.shields.io/badge/Resend-HTTPS_Email-000000?style=for-the-badge&logo=resend&logoColor=white)](https://resend.com)
-[![Langfuse](https://img.shields.io/badge/Langfuse-Observability-000000?style=for-the-badge&logo=langfuse&logoColor=white)](https://langfuse.com)
-[![Hugging Face Spaces](https://img.shields.io/badge/Hugging_Face-Spaces_Docker-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/spaces)
-
----
-
-## 1. Executive Summary
-
-**Joule Dynamics Server** is a high-throughput, agentic inference backend engineered specifically to power the **Real Estate Telemetry & Rate Intelligence Dashboard (Pulse AI)**.
-
-Operating far beyond standard CRUD paradigms, this server functions as an autonomous reasoning engine that compiles unstructured, natural-language investor queries into deterministic, parameterized **PostgreSQL PL/pgSQL stored procedures**, **window function aggregations**, and **pgvector semantic scans**. By combining ultra-low-latency LPU inference (Groq) with hierarchical schema gating, Server-Sent Events (SSE), and an autonomous double opt-in alert dispatch engine, the system delivers real-time market snapshots, rate change notifications, and geographic proximity analytics with sub-second time-to-first-token.
-
----
-
-## 2. System Architecture (The AI Pipeline)
-
-The telemetry backend utilizes an asynchronous, non-blocking pipeline spanning intent classification, localized dynamic tool retrieval, database execution, and streaming synthesis.
-
-```
-                           ┌─────────────────────────────────────────────────────────────┐
-                           │               Client Query via SSE Protocol                 │
-                           │            POST /api/v1/real-estate/chat/stream             │
-                           └──────────────────────────────┬──────────────────────────────┘
-                                                          │
-                                                          ▼
-                           ┌─────────────────────────────────────────────────────────────┐
-                           │          1. INTENT ROUTING & CATEGORY GATING                │
-                           │          Model: openai/gpt-oss-20b (~150ms)                 │
-                           │          • Classifies: PATH_A | PATH_B | BOTH | etc.        │
-                           │          • Tags Domain: ["MARKET", "ANOMALY", "GEO"...]     │
-                           │          • Alert Shortcut: Direct Gating to Subscription    │
-                           └──────────────────────────────┬──────────────────────────────┘
-                                                          │
-                                                          ▼
-                           ┌─────────────────────────────────────────────────────────────┐
-                           │          2. DYNAMIC TOOL DISCOVERY (Embedding Engine)       │
-                           │          Model: all-MiniLM-L6-v2 (Local, ~20ms latency)     │
-                           │          • Filters registry by domain categories            │
-                           │          • Cosine similarity ranking on tool descriptions   │
-                           │          • Injects Top-3 Tools + Universal Action Tools     │
-                           └──────────────────────────────┬──────────────────────────────┘
-                                                          │
-                                                          ▼
-                           ┌───────────────────────────��─────────────────────────────────┐
-                           │          3. BRAIN & AGENTIC SYNTHESIS LOOP                  │
-                           │          Model: openai/gpt-oss-120b                         │
-                           │          • Multi-step tool orchestration (max 2 rounds)     │
-                           │          • Single-turn terminal tool break & directive      │
-                           │          • Parameter clamping & alias normalization         │
-                           │          • Execution via PostgREST RPC & pgvector RAG       │
-                           └──────────────────────────────┬──────────────────────────────┘
-                                                          │
-                                                          ▼
-                           ┌─────────────────────────────────────────────────────────────┐
-                           │          4. SERVER-SENT EVENTS (SSE) STREAMING              │
-                           │          • event: status       (Routing state)              │
-                           │          • event: tool_call    (Active DB RPC payload)      │
-                           │          • event: token        (Live Markdown tokens)       │
-                           │          • event: done         (Final + Action Chips)       │
-                           └──────────────────────────────┬──────────────────────────────┘
-                                                          │
-                                                          ▼
-                           ┌─────────────────────────────────────────────────────────────┐
-                           │          5. ACTION RESOLVER ENGINE (Modular Fallback)       │
-                           │          • Dynamically infers relevant action button chips  │
-                           │          • Contextual follow-up suggestions per domain      │
-                           └─────────────────────────────────────────────────────────────┘
-```
-
-### Pipeline Lifecycle:
-1. **Routing & Intent Disambiguation:** Fast-path classifier (`openai/gpt-oss-20b`) categorizes the query into one of 6 pathways (`PATH_A` Live Data, `PATH_B` Methodology/RAG, `BOTH` Hybrid, `GREETING`, `COMMERCIAL_HANDOFF`, `OUT_OF_SCOPE`) and tags domain categories. When a user provides alert monitoring parameters (criteria + email), the router bypasses general retrieval and gates directly to `create_alert_subscription`.
-2. **Context-Aware Dynamic Schema Gating:** To prevent token bloat (saving >75% of prompt tokens), a local sentence-transformer (`all-MiniLM-L6-v2`) ranks tool schemas against the user prompt (enriched with conversational history) and exposes strictly the Top-3 relevant tools + universal actions.
-3. **Autonomous Execution Loop (Max 2 Rounds):** The synthesis engine (`openai/gpt-oss-120b`) generates JSON function call payloads and dispatches them against Supabase PostgREST RPC endpoints. For terminal tools (`create_alert_subscription`, `generate_data_export`, `generate_contact_buttons`, `suggest_actions`), the loop executes immediately in 1 turn, injecting an `_ACTION_DIRECTIVE` to deliver a clean confirmation without regenerating past tables.
-4. **Action Resolver Engine:** Decoupled in `services/action_resolver.py`, the system guarantees that interactive follow-up chips (`suggest_actions`) are always dispatched to the UI even if the LLM omits them in its generation.
-
----
-
-## 3. Tool Registry & Agentic Framework
-
-The agent has direct access to specialized tools mapping to PostgreSQL PL/pgSQL stored procedures, geospatial calculations, background file exporters, and subscription provisioning:
-
-| Category | Tool Name | Database / Execution Target | Technical Function & Capabilities |
-|---|---|---|---|
-| **Market & KPIs** | `get_real_estate_kpis` | `public.get_real_estate_kpis` | Aggregates portfolio KPIs (active properties, 7d delta, 25%+ price surge count, scrape health). |
-| **Market & KPIs** | `get_market_averages` | `public.get_market_averages` | Calculates nightly rates and 7-day trailing baseline averages using SQL window functions. |
-| **Market & KPIs** | `get_market_snapshot` | `public.get_market_snapshot` | Single-day comprehensive summary: active inventory, min/max rates, and occupancy rate %. |
-| **Market & KPIs** | `get_market_trend` | `public.get_market_trend` | Multi-day historical rate trajectory (up to 90 days) across tracked market regions. |
-| **Market & KPIs** | `get_market_rate_changes` | `public.get_market_rate_changes` | Counts properties with ANY nightly rate increase or decrease over lookback window (not just spikes). |
-| **Market & KPIs** | `get_tracked_markets` | `public.get_tracked_markets` | Returns platform coverage, listing volumes, and operational regions in the database. |
-| **Market & KPIs** | `get_recently_changed_tracking` | `public.get_recently_changed_tracking` | Audit log of listings newly activated or paused in the scraping registry. |
-| **Anomalies** | `get_spike_alerts` | `public.get_spike_alerts` | Scans for sudden rate spikes exceeding configurable threshold (default 25% over 7d baseline). |
-| **Anomalies** | `get_rate_anomaly_report` | `public.get_rate_anomaly_report` | Deep pricing anomaly investigation: lists previous rates, baseline delta, and deviation %. |
-| **Anomalies** | `get_most_volatile_properties` | `public.get_most_volatile_properties` | Ranks properties with the highest dynamic pricing frequency and variance over lookback window. |
-| **Listings & Rates** | `get_property_snapshot` | `public.get_property_snapshot` | Single-listing telemetry: bedrooms, coordinates, current rate, 7d baseline, and availability. |
-| **Listings & Rates** | `get_property_detail` | `public.get_property_detail` | Comprehensive listing metadata, room configurations, amenities, host details, and current pricing. |
-| **Listings & Rates** | `get_property_rate_changes` | `public.get_property_rate_changes` | Chronological price revision log tracking exact host rate adjustments over time. |
-| **Listings & Rates** | `compare_properties` | `public.compare_properties` | Multi-property side-by-side comparison across pricing, occupancy, and bedroom configurations. |
-| **Listings & Rates** | `search_properties` | `public.search_properties` | Multi-parameter search filtered by market, bedroom count, platform, availability, and search terms. |
-| **Listings & Rates** | `get_availability_rate` | `public.get_availability_rate` | Computes calendar booking ratios, vacancy percentage, and active inventory metrics. |
-| **Geospatial** | `geocode_address` | *Mapbox Geocoding Worker* | Translates natural language addresses/landmarks into decimal latitude/longitude coordinates. |
-| **Geospatial** | `get_nearby_properties` | `public.get_nearby_properties` | Executes Haversine radius queries on geographic coordinates to find nearest listings. |
-| **Geospatial** | `get_distance_km` | `public.get_distance_km` | PostGIS / mathematical straight-line distance computation between two property UUIDs. |
-| **Universal Actions** | `generate_data_export` | *Appwrite Storage Worker* | Compiles structured real estate analysis into downloadable Markdown (`.md`) or CSV reports. |
-| **Universal Actions** | `suggest_actions` | *SSE UI Protocol* | Dispatches native interactive action chips/buttons to guide user follow-ups and next steps. |
-| **Universal Actions** | `generate_contact_buttons` | *Local Worker* | Generates interactive Email and WhatsApp contact buttons for bespoke engineering inquiries. |
-| **Alerts & Monitor** | `create_alert_subscription` | *Alert Service Worker* | Provisions double opt-in subscriptions for market anomalies, price thresholds, availability shifts, or scheduled digests. |
-
----
-
-## 4. Alert Subscriptions & Background Workers Subsystem
-
-Pulse AI features an autonomous alert monitoring and notification engine that monitors live market telemetry and dispatches emails via double opt-in verification.
-
-```
-                    ┌────────────────────────────────────────────────────────┐
-                    │            User Requests Alert via Chat or API         │
-                    │               "Alert me when Miami rates spike"        │
-                    └───────────────────────────┬────────────────────────────┘
-                                                │
-                                                ▼
-                    ┌────────────────────────────────────────────────────────┐
-                    │         1. PROVISION PENDING SUBSCRIPTION              │
-                    │         • Status: pending_confirmation                 │
-                    │         • Secure Token: secrets.token_urlsafe(32)      │
-                    │         • Expiry: 48 hours                             │
-                    └───────────────────────────┬────────────────────────────┘
-                                                │
-                                                ▼
-                    ┌────────────────────────────────────────────────────────┐
-                    │         2. VERIFICATION EMAIL DISPATCH                 │
-                    │         • Primary: Resend REST API (HTTPS Port 443)    │
-                    │         • Fallback: Gmail SMTP (Port 587/465)          │
-                    └───────────────────────────┬────────────────────────────┘
-                                                │ User clicks link:
-                                                │ GET /api/v1/alerts/confirm?token=...
-                                                ▼
-                    ┌────────────────────────────────────────────────────────┐
-                    │         3. ACTIVATION & WELCOME PREVIEW                │
-                    │         • Status: active                               │
-                    │         • Sends Instant Welcome & Diagnostic Email     │
-                    └───────────────────────────┬────────────────────────────┘
-                                                │
-                                                ▼
-                    ┌────────────────────────────────────────────────────────┐
-                    │         4. AUTONOMOUS WORKER EVALUATION                │
-                    │         • POST /api/v1/internal/evaluate-alerts        │
-                    │         • POST /api/v1/internal/send-digests           │
-                    │         • POST /api/v1/internal/cleanup-alerts         │
-                    │         • Cooldown Safeguard: 24h per subscription     │
-                    │         • Max Cap: 5 subscriptions per email           │
-                    └────────────────────────────────────────────────────────┘
-```
-
-### Supported Criteria Types (10 Monitors)
-- `spike`: Rate surges exceeding configurable % threshold over 7-day baseline.
-- `rate_change`: Any nightly rate adjustment (increases or decreases) in target market.
-- `price_threshold`: Nightly rate crossing above or below a specific numerical target.
-- `availability_change`: Specific listing transitions to available, booked, or any state change.
-- `new_listing`: New properties indexed in target market region.
-- `tracking_removed`: Active properties paused or removed from telemetry.
-- `anomaly`: Statistical pricing deviations beyond standard deviations.
-- `volatility`: High-frequency dynamic rate fluctuations.
-- `trend_reversal`: Trajectory shift (e.g., market transitioning from falling to rising).
-- `digest`: Scheduled daily or weekly market intelligence summary.
-
-### Dual-Transport Email Dispatch Subsystem
-Cloud container environments (such as Hugging Face Spaces, AWS, and GCP) frequently block outbound TCP traffic on traditional SMTP ports (`587` and `465`). To guarantee 100% email deliverability:
-1. **Primary Transport — Resend REST API:** Operates strictly over **HTTPS port 443**, bypassing all firewall restrictions.
-2. **Fallback Transport — SMTP:** Standard TLS/SSL dispatch for environments with open SMTP ports.
-
----
-
-## 5. Observability & Diagnostics
-
-Every request and worker job passing through `joule-dynamics-server` is instrumented with **Langfuse** and **OpenTelemetry**:
-
-```
-[Trace: chat-session-001]
- ├── [Span: router-classification]      Model: openai/gpt-oss-20b  Tokens: 210  Latency: 148ms
- ├── [Span: dynamic-tool-discovery]    Model: all-MiniLM-L6-v2    Ranked: 19   Latency: 22ms
- ├── [Generation: agent-round-1]       Model: openai/gpt-oss-120b Tokens: 480  Latency: 610ms
- │    └── [Tool: get_market_snapshot]  Status: success            Rows: 1      Latency: 45ms
- └── [Generation: agent-synthesis]     Model: openai/gpt-oss-120b Tokens: 320  Latency: 410ms
-```
-
-- **Step-by-Step Trajectory Tracking:** Captures multi-round LLM tool calls, arguments, database query responses, and final streaming text.
-- **Worker & Email Telemetry:** Traces background alert evaluations, digest dispatches, and token cleanups.
-- **Diagnostic Network Probe:** Real-time endpoint (`GET /api/v1/internal/diagnose-smtp`) verifies Resend API key status, HTTP 443 connectivity, and socket reachability on ports 587 and 465.
-
----
-
-## 6. API Contracts & Streaming Protocol
-
-### Chat Endpoints
-
-#### 1. Streaming Chat: `POST /api/v1/real-estate/chat/stream`
-- **Headers:** `Content-Type: application/json`, `Accept: text/event-stream`
-- **Request Body:**
-```json
-{
-  "message": "What is the average nightly rate in Miami for 2-bedroom listings?",
-  "session_id": "usr_session_8f92a",
-  "context": {
-    "market": "Miami"
-  }
-}
-```
-
-##### Event Stream Chunks:
-- `event: status` -> `{"classification": "PATH_A"}`
-- `event: tool_call` -> `{"tool": "get_market_averages", "args": {"market_param": "Miami"}}`
-- `event: token` -> `{"text": "In Miami, the average rate..."}`
-- `event: done` -> Final payload including `reply`, `tools_called`, and `suggested_actions`.
-
-#### 2. Non-Streaming Chat: `POST /api/v1/real-estate/chat`
-Returns complete JSON payload for non-streaming consumers.
-
-#### 3. Starter Suggestions: `GET /api/v1/real-estate/chat/starters`
-Returns curated prompt starter buttons tailored to current market conditions.
-
----
-
-### Alert & Subscription Endpoints
-
-#### 1. Confirm Subscription: `GET /api/v1/alerts/confirm?token={token}`
-Activates a pending subscription when clicked by the user from their confirmation email. Returns styled HTML confirmation and dispatches the welcome preview email.
-
-#### 2. Unsubscribe: `GET /api/v1/alerts/unsubscribe?token={token}`
-Immediately cancels an active subscription.
-
 ---
-
-### Internal Worker & Cron Endpoints (Secured)
-All internal endpoints require header: `X-Internal-Key: <INTERNAL_API_KEY>`.
-
-| Method | Endpoint | Description |
-|---|---|---|
-| `POST` | `/api/v1/internal/evaluate-alerts` | Evaluates active subscriptions against current database telemetry. |
-| `POST` | `/api/v1/internal/send-digests` | Compiles and dispatches pending daily/weekly market digests. |
-| `POST` | `/api/v1/internal/cleanup-alerts` | Deletes unconfirmed subscriptions older than 48 hours. |
-| `POST` | `/api/v1/internal/run-all-workers` | Master trigger executing evaluate, digest, and cleanup sequentially. |
-| `GET` | `/api/v1/internal/diagnose-smtp` | Network connectivity probe for Resend HTTPS and SMTP port status. |
-
+title: SentimentScope
+emoji: 📊
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_port: 7860
+pinned: false
 ---
 
-### System & Health Endpoints
-- `GET /health`: Liveness probe for Docker / load balancers.
-- `GET /api/v1/health`: Versioned health status check.
-- `POST /api/v1/rag/chat`: Amara Home & Kitchen customer support RAG chat.
-
----
-
-## 7. Local Development & Environment Setup
-
-### 1. Prerequisites
-- Python 3.10+
-- Git & Virtualenv
-
-### 2. Installation
-```bash
-# Clone repository
-git clone https://github.com/JohnJodinho/joule-dynamics-server.git
-cd joule-dynamics-server
-
-# Create and activate virtual environment
-python -m venv venv
-
-# On Linux/macOS:
-source venv/bin/activate
-
-# On Windows (PowerShell):
-.\venv\Scripts\Activate.ps1
-```
-
-```bash
-# Install dependencies
-pip install -r requirements.txt
-```
-
-### 3. Environment Configuration (`.env`)
-Create a `.env` file in the project root with the following parameters:
-
-```env
-# ── Core Server Configuration ──
-PORT=7860
-HOST=0.0.0.0
-ALLOWED_ORIGINS=*
-
-# ── Groq LPU Inference ──
-GROQ_API_KEY=gsk_your_groq_api_key_here
-GROQ_ROUTE_MODEL=openai/gpt-oss-20b
-GROQ_FALLBACK_ROUTE_MODEL=groq/compound-mini
-GROQ_SYNTHESIS_MODEL=openai/gpt-oss-120b
-GROQ_FALLBACK_SYNTHESIS_MODEL=qwen/qwen3.6-27b
-
-# ── Supabase Database & Vectors ──
-SUPABASE_URL=https://your-project.supabase.co
-SUPABASE_KEY=your_supabase_anon_key
-SUPABASE_SERVICE_ROLE_KEY=your_supabase_service_role_key
-
-# ── Resend Email Dispatch (Primary - HTTPS 443) ──
-RESEND_API_KEY=re_your_resend_api_key_here
-RESEND_FROM_ADDRESS=Pulse AI <onboarding@resend.dev>
-
-# ── Alert Subscriptions Configuration ──
-ALERT_BASE_URL=https://johnalbarkaibrahim-sentimentscope.hf.space
-INTERNAL_API_KEY=your_secure_internal_cron_key
-ALERT_COOLDOWN_HOURS=24
-MAX_SUBSCRIPTIONS_PER_EMAIL=5
-CONFIRMATION_EXPIRY_HOURS=48
-
-# ── Fallback SMTP Configuration (Optional) ──
-EMAIL_SMTP_HOST=smtp.gmail.com
-EMAIL_SMTP_PORT=587
-EMAIL_FROM_ADDRESS=jouledynamicscto@gmail.com
-EMAIL_SMTP_PASSWORD=your_app_specific_password
-
-# ── Langfuse Observability ──
-LANGFUSE_PUBLIC_KEY=pk-lf-your_public_key
-LANGFUSE_SECRET_KEY=sk-lf-your_secret_key
-LANGFUSE_HOST=https://cloud.langfuse.com
-
-# ── Appwrite Storage (Report Exports) ──
-APP_WRITE_PROJECT_ID=your_project_id
-APP_WRITE_API_ENDPOINT=https://cloud.appwrite.io/v1
-APP_WRITE_API_KEY=your_appwrite_api_key
-APP_WRITE_BUCKET_ID=your_bucket_id
-
-# ── Geospatial & Contacts ──
-MAPBOX_ACCESS_TOKEN=your_mapbox_token
-CONTACT_EMAIL=hello@jouledynamics.com
-CONTACT_WHATSAPP=1234567890
-```
-
-### 4. Run Development Server
-```bash
-uvicorn main:app --host 0.0.0.0 --port 7860 --reload
-```
-
-Interactive Swagger documentation will be accessible at `http://localhost:7860/docs`.
-
----
-
-## 8. Deployment & CI/CD Workflow
-
-### Multi-Target Repository Architecture
-This project utilizes a dual-README CI/CD strategy to maintain rich GitHub documentation while strictly satisfying Hugging Face Space metadata requirements:
-
-1. **GitHub (`origin/main`):** Hosts the complete source code and this comprehensive, enterprise-grade `README.md`.
-2. **Hugging Face Spaces (`hf/main`):** Requires a concise YAML frontmatter header to initialize its Docker SDK container.
-3. **Automated CI/CD Workflow (`.github/workflows/sync_to_hf.yml`):**
-   - Triggers automatically upon push to `main` on GitHub.
-   - Replaces the Space root `README.md` with [`README_HF.md`](file:///c:/Users/user/Desktop/python_projects/AI_Engineering_And_Automation/WhatsApp%20Sentiment/whatsapp_sentiment_backend_HF/SentimentScope/README_HF.md) strictly during the deployment sync step.
-   - Pushes the build artifacts to Hugging Face Spaces without overwriting or dirtying the GitHub repository.
-
-> [!NOTE]
-> Never merge `hf/main` into your local `main` branch. All development and documentation updates occur on GitHub (`origin/main`), with automated downstream deployment to Hugging Face Spaces.
+# Joule Dynamics RAG Demo
+Backend API for the Amara Home & Kitchen RAG chatbot.