docs: complete enterprise technical README rewrite for joule-dynamics-server
Browse files
README.md
CHANGED
|
@@ -1,100 +1,261 @@
|
|
| 1 |
-
# Joule Dynamics Real Estate Intelligence Server (Pulse AI)
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
---
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
│
|
| 27 |
-
│
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
│
|
| 34 |
-
│
|
| 35 |
-
│
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
---
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Joule Dynamics Real Estate Intelligence Server (Pulse AI)
|
| 2 |
+
|
| 3 |
+
[](https://fastapi.tiangolo.com)
|
| 4 |
+
[](https://www.python.org)
|
| 5 |
+
[](https://groq.com)
|
| 6 |
+
[](https://supabase.com)
|
| 7 |
+
[](https://langfuse.com)
|
| 8 |
+
[](https://huggingface.co/spaces)
|
| 9 |
+
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
## 1. Executive Summary
|
| 13 |
+
|
| 14 |
+
**Joule Dynamics Server** is a high-throughput, agentic inference backend engineered specifically to power the **Real Estate Telemetry & Rate Intelligence Dashboard**.
|
| 15 |
+
|
| 16 |
+
Operating far beyond standard CRUD paradigms, this server functions as an autonomous reasoning engine that compiles unstructured, natural-language investor queries into deterministic, parameterized **PostgreSQL PL/pgSQL stored procedures**, **window function aggregations**, and **pgvector semantic scans**. By combining ultra-low-latency LPU inference (Groq) with hierarchical schema gating and Server-Sent Events (SSE), the engine delivers real-time market snapshots, anomaly alerts (25%+ rate deviations), and geographic proximity analytics with sub-second time-to-first-token.
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
## 2. System Architecture (The AI Pipeline)
|
| 21 |
+
|
| 22 |
+
The telemetry backend utilizes an asynchronous, non-blocking pipeline spanning intent classification, localized dynamic tool retrieval, database execution, and streaming synthesis.
|
| 23 |
+
|
| 24 |
+
```
|
| 25 |
+
┌─────────────────────────────────────────────────────────────┐
|
| 26 |
+
│ Client Query via SSE Protocol │
|
| 27 |
+
│ POST /api/v1/real-estate/chat/stream │
|
| 28 |
+
└──────────────────────────────┬──────────────────────────────┘
|
| 29 |
+
│
|
| 30 |
+
▼
|
| 31 |
+
┌─────────────────────────────────────────────────────────────┐
|
| 32 |
+
│ 1. INTENT ROUTING & CATEGORY GATING │
|
| 33 |
+
│ Model: openai/gpt-oss-20b (~150ms) │
|
| 34 |
+
│ • Classifies: PATH_A | PATH_B | BOTH | etc. │
|
| 35 |
+
│ • Tags Domain: ["MARKET", "ANOMALY", "GEO"...] │
|
| 36 |
+
└──────────────────────────────┬──────────────────────────────┘
|
| 37 |
+
│
|
| 38 |
+
▼
|
| 39 |
+
┌─────────────────────────────────────────────────────────────┐
|
| 40 |
+
│ 2. DYNAMIC TOOL DISCOVERY (Embedding Engine) │
|
| 41 |
+
│ Model: all-MiniLM-L6-v2 (Local, ~20ms latency) │
|
| 42 |
+
│ • Filters registry by domain categories │
|
| 43 |
+
│ • Cosine similarity ranking on tool descriptions │
|
| 44 |
+
│ • Injects Top-3 Tools + Universal Action Tools │
|
| 45 |
+
└──────────────────────────────┬──────────────────────────────┘
|
| 46 |
+
│
|
| 47 |
+
▼
|
| 48 |
+
┌─────────────────────────────────────────────────────────────┐
|
| 49 |
+
│ 3. BRAIN & AGENTIC SYNTHESIS LOOP │
|
| 50 |
+
│ Model: openai/gpt-oss-120b │
|
| 51 |
+
│ • Multi-step tool orchestration (max 4 rounds) │
|
| 52 |
+
│ • Parameter clamping & alias normalization │
|
| 53 |
+
│ • Execution via PostgREST RPC & pgvector RAG │
|
| 54 |
+
└──────────────────────────────┬──────────────────────────────┘
|
| 55 |
+
│
|
| 56 |
+
▼
|
| 57 |
+
┌─────────────────────────────────────────────────────────────┐
|
| 58 |
+
│ 4. SERVER-SENT EVENTS (SSE) STREAMING │
|
| 59 |
+
│ • event: status (Routing state) │
|
| 60 |
+
│ • event: tool_call (Active DB RPC payload) │
|
| 61 |
+
│ • event: token (Live Markdown tokens) │
|
| 62 |
+
│ • event: done (Final + Action Chips) │
|
| 63 |
+
└─────────────────────────────────────────────────────────────┘
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
### Pipeline Lifecycle:
|
| 67 |
+
1. **Routing & Intent Disambiguation:** Fast-path classifier (`openai/gpt-oss-20b`) categorizes the query into one of 6 pathways (`PATH_A` Live Data, `PATH_B` Methodology/RAG, `BOTH` Hybrid, `GREETING`, `COMMERCIAL_HANDOFF`, `OUT_OF_SCOPE`) and tags domain categories.
|
| 68 |
+
2. **Context-Aware Dynamic Schema Gating:** To prevent token bloat (saving >75% of prompt tokens), a local sentence-transformer (`all-MiniLM-L6-v2`) ranks tool schemas against the user prompt (enriched with conversational history) and exposes strictly the Top-3 relevant tools + universal tools.
|
| 69 |
+
3. **Autonomous Execution Loop:** The synthesis engine (`openai/gpt-oss-120b`) generates JSON function call payloads, dispatches them against Supabase PostgREST RPC endpoints, and streams real-time execution badges to the client.
|
| 70 |
+
4. **Resilient Loop Carryover:** If a complex, multi-intent prompt hits the 4-round loop ceiling before completing all operations (e.g. data fetch + export generation), the engine synthesizes findings and automatically emits interactive action chips (`suggest_actions`) to carry over work into subsequent turns.
|
| 71 |
+
|
| 72 |
+
---
|
| 73 |
+
|
| 74 |
+
## 3. The 18-Tool Agentic Framework
|
| 75 |
+
|
| 76 |
+
The agent has direct access to 18 specialized tools mapping to PostgreSQL PL/pgSQL stored procedures, geospatial calculations, and export workers:
|
| 77 |
+
|
| 78 |
+
| Category | Tool Name | Database / RPC Target | Technical Function & Capabilities |
|
| 79 |
+
|---|---|---|---|
|
| 80 |
+
| **Market & KPIs** | `get_dashboard_kpis` | `public.get_dashboard_kpis` | Aggregates portfolio KPIs (active properties, 7d delta, 25%+ price surge count, scrape health). |
|
| 81 |
+
| **Market & KPIs** | `get_market_averages` | `public.get_market_averages` | Calculates nightly rates and 7-day trailing baseline averages using SQL window functions. |
|
| 82 |
+
| **Market & KPIs** | `get_market_snapshot` | `public.get_market_snapshot` | Single-day comprehensive summary: active inventory, min/max rates, and occupancy rate %. |
|
| 83 |
+
| **Market & KPIs** | `get_market_trend` | `public.get_market_trend` | Multi-day historical rate trajectory (up to 90 days) across Miami and NYC/NJ Metro. |
|
| 84 |
+
| **Market & KPIs** | `get_tracked_markets` | `public.get_tracked_markets` | Returns platform coverage, listing volumes, and operational regions in the database. |
|
| 85 |
+
| **Market & KPIs** | `get_recently_changed_tracking` | `public.get_recently_changed_tracking` | Audit log of listings newly activated or paused in the scraping registry. |
|
| 86 |
+
| **Anomalies** | `get_spike_alerts` | `public.get_spike_alerts` | Scans for sudden rate spikes exceeding configurable threshold (default 25% over 7d baseline). |
|
| 87 |
+
| **Anomalies** | `get_rate_anomaly_report` | `public.get_rate_anomaly_report` | Deep pricing anomaly investigation: lists previous rates, baseline delta, and deviation %. |
|
| 88 |
+
| **Anomalies** | `get_most_volatile_properties` | `public.get_most_volatile_properties` | Ranks properties with the highest dynamic pricing frequency and variance over lookback window. |
|
| 89 |
+
| **Listings & Rates** | `get_property_snapshot` | `public.get_property_snapshot` | Single-listing telemetry: bedrooms, coordinates, current rate, 7d baseline, and availability. |
|
| 90 |
+
| **Listings & Rates** | `get_property_rate_changes` | `public.get_property_rate_changes` | Chronological price revision log tracking exact host rate adjustments over time. |
|
| 91 |
+
| **Listings & Rates** | `compare_properties` | `public.compare_properties` | Multi-property side-by-side comparison across pricing, occupancy, and bedroom configurations. |
|
| 92 |
+
| **Listings & Rates** | `search_properties` | `public.search_properties` | Multi-parameter search filtered by market, bedroom count, platform, availability, and search terms. |
|
| 93 |
+
| **Listings & Rates** | `get_availability_rate` | `public.get_availability_rate` | Computes calendar booking ratios, vacancy percentage, and active inventory metrics. |
|
| 94 |
+
| **Geospatial** | `geocode_address` | *Python Geocoding Worker* | Translates natural language addresses/landmarks into decimal latitude/longitude coordinates. |
|
| 95 |
+
| **Geospatial** | `get_nearby_properties` | `public.get_nearby_properties` | Executes Haversine radius queries on geographic coordinates to find nearest listings. |
|
| 96 |
+
| **Geospatial** | `get_distance_km` | `public.get_distance_km` | PostGIS / mathematical straight-line distance computation between two property UUIDs. |
|
| 97 |
+
| **Universal Actions** | `generate_data_export` | *Appwrite Storage Worker* | Compiles structured real estate analysis into downloadable Markdown (`.md`) reports. |
|
| 98 |
+
| **Universal Actions** | `suggest_actions` | *SSE UI Protocol* | Dispatches native interactive action chips/buttons to guide user follow-ups and next steps. |
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
+
## 4. Observability & Telemetry
|
| 103 |
+
|
| 104 |
+
Every request passing through `joule-dynamics-server` is deeply instrumented with **Langfuse** and **OpenTelemetry** for end-to-end tracing:
|
| 105 |
+
|
| 106 |
+
```
|
| 107 |
+
[Trace: chat-session-001]
|
| 108 |
+
├── [Span: router-classification] Model: openai/gpt-oss-20b Tokens: 210 Latency: 148ms
|
| 109 |
+
├── [Span: dynamic-tool-discovery] Model: all-MiniLM-L6-v2 Ranked: 18 Latency: 22ms
|
| 110 |
+
├── [Generation: agent-round-1] Model: openai/gpt-oss-120b Tokens: 480 Latency: 610ms
|
| 111 |
+
│ └── [Tool: get_market_snapshot] Status: success Rows: 1 Latency: 45ms
|
| 112 |
+
└── [Generation: agent-synthesis] Model: openai/gpt-oss-120b Tokens: 320 Latency: 410ms
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
- **Step-by-Step Trajectory Tracking:** Captures multi-round LLM tool calls, arguments, database query responses, and final streaming text.
|
| 116 |
+
- **Latency & Error Profiling:** Automatically detects and isolates PostgREST `PGRST202` schema errors, Groq rate-limiting retries, and tool execution bottlenecks.
|
| 117 |
+
- **Token & Cost Attribution:** Live calculation of input, completion, and reasoning tokens across both routing and synthesis model tiers.
|
| 118 |
+
|
| 119 |
+
---
|
| 120 |
+
|
| 121 |
+
## 5. API Contracts & Streaming Protocol
|
| 122 |
+
|
| 123 |
+
### Endpoint: `POST /api/v1/real-estate/chat/stream`
|
| 124 |
+
- **Request Headers:** `Content-Type: application/json`, `Accept: text/event-stream`
|
| 125 |
+
- **Request Body:**
|
| 126 |
+
```json
|
| 127 |
+
{
|
| 128 |
+
"message": "What is the average nightly rate in Miami for 2-bedroom listings?",
|
| 129 |
+
"session_id": "usr_session_8f92a",
|
| 130 |
+
"context": {
|
| 131 |
+
"market": "Miami"
|
| 132 |
+
}
|
| 133 |
+
}
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
### Event Stream Chunks:
|
| 137 |
+
|
| 138 |
+
#### 1. Route Classification
|
| 139 |
+
```http
|
| 140 |
+
event: status
|
| 141 |
+
data: {"classification": "PATH_A"}
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
#### 2. Live Tool Execution Badge
|
| 145 |
+
```http
|
| 146 |
+
event: tool_call
|
| 147 |
+
data: {"tool": "search_properties", "args": {"p_market": "Miami", "p_bedrooms": 2}}
|
| 148 |
+
```
|
| 149 |
+
|
| 150 |
+
#### 3. Token Stream (Live Markdown)
|
| 151 |
+
```http
|
| 152 |
+
event: token
|
| 153 |
+
data: {"text": "In "}
|
| 154 |
+
|
| 155 |
+
event: token
|
| 156 |
+
data: {"text": "Miami, "}
|
| 157 |
+
|
| 158 |
+
event: token
|
| 159 |
+
data: {"text": "the average rate for 2-bedroom properties is **$285/night**."}
|
| 160 |
+
```
|
| 161 |
+
|
| 162 |
+
#### 4. Final Payload & Action Chips
|
| 163 |
+
```http
|
| 164 |
+
event: done
|
| 165 |
+
data: {
|
| 166 |
+
"reply": "In Miami, the average rate for 2-bedroom properties is **$285/night**...",
|
| 167 |
+
"suggested_actions": ["Compare with NYC", "Generate Download Report", "View Price Volatility"],
|
| 168 |
+
"tools_called": [{"tool": "search_properties", "args": {"p_market": "Miami", "p_bedrooms": 2}}]
|
| 169 |
+
}
|
| 170 |
+
```
|
| 171 |
+
|
| 172 |
+
---
|
| 173 |
+
|
| 174 |
+
## 6. Local Development & Environment Setup
|
| 175 |
+
|
| 176 |
+
### 1. Prerequisites
|
| 177 |
+
- Python 3.10+
|
| 178 |
+
- Git & Virtualenv
|
| 179 |
+
|
| 180 |
+
### 2. Installation
|
| 181 |
+
```bash
|
| 182 |
+
# Clone repository
|
| 183 |
+
git clone https://github.com/JohnJodinho/joule-dynamics-server.git
|
| 184 |
+
cd joule-dynamics-server
|
| 185 |
+
|
| 186 |
+
# Create and activate virtual environment
|
| 187 |
+
python -m venv venv
|
| 188 |
+
# On Linux/macOS:
|
| 189 |
+
source venv/bin/activate
|
| 190 |
+
# On Windows:
|
| 191 |
+
. env\Scriptsctivate
|
| 192 |
+
|
| 193 |
+
# Install dependencies
|
| 194 |
+
pip install -r requirements.txt
|
| 195 |
+
```
|
| 196 |
+
|
| 197 |
+
### 3. Environment Configuration (`.env`)
|
| 198 |
+
Create a `.env` file in the project root with the following parameters:
|
| 199 |
+
|
| 200 |
+
```env
|
| 201 |
+
# ── Core Server Configuration ──
|
| 202 |
+
PORT=7860
|
| 203 |
+
HOST=0.0.0.0
|
| 204 |
+
ALLOWED_ORIGINS=*
|
| 205 |
+
|
| 206 |
+
# ── Groq LPU Inference ──
|
| 207 |
+
GROQ_API_KEY=gsk_your_groq_api_key_here
|
| 208 |
+
GROQ_ROUTE_MODEL=openai/gpt-oss-20b
|
| 209 |
+
GROQ_FALLBACK_ROUTE_MODEL=groq/compound-mini
|
| 210 |
+
GROQ_SYNTHESIS_MODEL=openai/gpt-oss-120b
|
| 211 |
+
GROQ_FALLBACK_SYNTHESIS_MODEL=qwen/qwen3.6-27b
|
| 212 |
+
|
| 213 |
+
# ── Supabase Database & Vectors ──
|
| 214 |
+
SUPABASE_URL=https://your-project.supabase.co
|
| 215 |
+
SUPABASE_KEY=your_supabase_service_role_or_anon_key
|
| 216 |
+
|
| 217 |
+
# ── Langfuse Observability ──
|
| 218 |
+
LANGFUSE_PUBLIC_KEY=pk-lf-your_public_key
|
| 219 |
+
LANGFUSE_SECRET_KEY=sk-lf-your_secret_key
|
| 220 |
+
LANGFUSE_HOST=https://cloud.langfuse.com
|
| 221 |
+
|
| 222 |
+
# ── Appwrite Storage (Report Exports) ──
|
| 223 |
+
APPWRITE_ENDPOINT=https://cloud.appwrite.io/v1
|
| 224 |
+
APPWRITE_PROJECT_ID=your_project_id
|
| 225 |
+
APPWRITE_API_KEY=your_appwrite_api_key
|
| 226 |
+
APPWRITE_BUCKET_ID=your_bucket_id
|
| 227 |
+
```
|
| 228 |
+
|
| 229 |
+
### 4. Run Development Server
|
| 230 |
+
```bash
|
| 231 |
+
uvicorn app:app --host 0.0.0.0 --port 7860 --reload
|
| 232 |
+
```
|
| 233 |
+
|
| 234 |
+
Server will be active at `http://localhost:7860`. Interactive Swagger OpenAPI docs available at `http://localhost:7860/docs`.
|
| 235 |
+
|
| 236 |
+
---
|
| 237 |
+
|
| 238 |
+
## 7. Production Deployment (Hugging Face Spaces)
|
| 239 |
+
|
| 240 |
+
This server is packaged as a Docker container designed for seamless deployment on **Hugging Face Spaces**.
|
| 241 |
+
|
| 242 |
+
### Deployment Steps:
|
| 243 |
+
1. Create a new Space on [Hugging Face](https://huggingface.co/new-space) and select **Docker** as the SDK.
|
| 244 |
+
2. In your Space's root `README.md`, ensure the standard YAML metadata block is present at the very top:
|
| 245 |
+
```yaml
|
| 246 |
+
---
|
| 247 |
+
title: Joule Dynamics Server
|
| 248 |
+
emoji: 📊
|
| 249 |
+
colorFrom: blue
|
| 250 |
+
colorTo: indigo
|
| 251 |
+
sdk: docker
|
| 252 |
+
app_port: 7860
|
| 253 |
+
pinned: false
|
| 254 |
+
---
|
| 255 |
+
```
|
| 256 |
+
3. In your Space **Settings** $
|
| 257 |
+
4. Push code to the Space repository:
|
| 258 |
+
```bash
|
| 259 |
+
git push hf main
|
| 260 |
+
```
|
| 261 |
+
The Docker container will automatically build, install requirements, pre-load the local embedding model cache, and expose the FastAPI SSE server on port `7860`.
|