Blame for context/4_production.md

# SolarWine — Production Architecture

## Overview

Migrate from a single-process Streamlit prototype to a 3-layer production system
deployed on free-tier infrastructure. The monorepo stays — no new repositories.

```
Baseline/                          (monorepo)
├── frontend/                      NEW — Lovable-generated React app
│   ├── src/
│   │   ├── pages/
│   │   │   ├── Dashboard.tsx      Overview + energy + weather
│   │   │   ├── Advisor.tsx        Chatbot + advisory cards
│   │   │   ├── Control.tsx        System status + tracker angles
│   │   │   ├── Photosynthesis.tsx FvCB/ML results + forecasts
│   │   │   ├── Shading.tsx        Interactive simulator
│   │   │   └── Docs.tsx           Documentation
│   │   ├── components/
│   │   │   ├── EnergyCard.tsx
│   │   │   ├── WeatherCard.tsx
│   │   │   ├── Chatbot.tsx
│   │   │   ├── TrackerStatus.tsx
│   │   │   └── Charts.tsx         Recharts / Plotly wrappers
│   │   ├── hooks/
│   │   │   ├── useWeather.ts      fetch /api/weather/*
│   │   │   ├── useEnergy.ts       fetch /api/energy/*
│   │   │   ├── useSensors.ts      fetch /api/sensors/*
│   │   │   ├── useChatbot.ts      POST /api/chatbot/message
│   │   │   └── useControl.ts      fetch /api/control/*
│   │   ├── lib/
│   │   │   └── api.ts             API base URL + fetch wrapper
│   │   └── App.tsx                Router + layout
│   ├── package.json
│   └── tailwind.config.js
│
├── backend/                       NEW — FastAPI middleware + background services
│   ├── api/
│   │   ├── __init__.py
│   │   ├── main.py                FastAPI app, CORS, lifespan
│   │   ├── routes/
│   │   │   ├── weather.py         GET /api/weather/*
│   │   │   ├── sensors.py         GET /api/sensors/*
│   │   │   ├── energy.py          GET /api/energy/*
│   │   │   ├── photosynthesis.py  GET /api/photosynthesis/*
│   │   │   ├── control.py         GET /api/control/*
│   │   │   ├── chatbot.py         POST /api/chatbot/message
│   │   │   └── health.py          GET /api/health
│   │   ├── deps.py                Shared dependencies (DataHub, Redis)
│   │   └── auth.py                JWT middleware (Phase 11.6)
│   ├── workers/
│   │   ├── control_tick.py        ControlLoop 15-min cron entry point
│   │   └── daily_planner.py       DayAheadPlanner daily cron entry point
│   ├── Dockerfile
│   └── requirements.txt
│
├── .github/
│   └── workflows/
│       ├── control-tick.yml       GitHub Actions: 15-min cron
│       └── daily-planner.yml      GitHub Actions: daily 05:00 IST cron
│
├── src/                           UNCHANGED — shared Python core
│   ├── control_loop.py
│   ├── command_arbiter.py
│   ├── tracker_dispatcher.py
│   ├── day_ahead_planner.py
│   ├── energy_budget.py
│   ├── data/
│   │   ├── data_providers.py      DataHub + 6 services
│   │   ├── thingsboard_client.py
│   │   └── ims_client.py
│   ├── chatbot/
│   │   ├── vineyard_chatbot.py
│   │   ├── guardrails.py
│   │   └── routing_agent.py
│   ├── models/
│   │   ├── farquhar_model.py
│   │   └── phenology.py
│   ├── forecasting/
│   │   ├── predictor.py
│   │   └── ts_predictor.py
│   ├── shading/
│   │   ├── solar_geometry.py
│   │   └── tradeoff_engine.py
│   └── ...
│
├── config/settings.py             UNCHANGED — shared configuration
├── app.py                         KEPT temporarily (Streamlit, deprecated)
├── ui/                            KEPT temporarily (Streamlit tabs, deprecated)
├── docker-compose.yml             NEW — local dev orchestration
└── requirements.txt               Shared Python deps
```

---

## Layer 1 — The Brain (Backend Services)

### What it is

All business logic, models, data access, and control algorithms. This layer
already exists in `src/`. The only change is extracting the runtime entry points
into `backend/workers/`.

### Components

| Component | Module | Runtime | Schedule |
|-----------|--------|---------|----------|
| ControlLoop | `src/control_loop.py` | GitHub Actions cron | Every 15 min |
| DayAheadPlanner | `src/day_ahead_planner.py` | GitHub Actions cron | Daily 05:00 IST + re-plan events |
| DataHub | `src/data/data_providers.py` | Imported by API + workers | Always available |
| ThingsBoard client | `src/data/thingsboard_client.py` | Imported by DataHub | On-demand |
| IMS client | `src/data/ims_client.py` | Imported by DataHub | On-demand (30-min TTL) |
| FvCB model | `src/models/farquhar_model.py` | Imported by PhotosynthesisService | On-demand (~10ms) |
| ML ensemble | `src/forecasting/predictor.py` | Imported by PhotosynthesisService | On-demand (~500ms) |
| Chronos forecaster | `src/forecasting/ts_predictor.py` | Imported by PhotosynthesisService | On-demand (~2-5s) |
| CommandArbiter | `src/command_arbiter.py` | Imported by ControlLoop | Per tick |
| TrackerDispatcher | `src/tracker_dispatcher.py` | Imported by ControlLoop | Per tick |
| EnergyBudget | `src/energy_budget.py` | Imported by ControlLoop | Per tick |
| Chatbot + guardrails | `src/chatbot/*.py` | Imported by API route | Per message |

### Caching strategy

| Data | Current | Production |
|------|---------|------------|
| IMS weather | In-memory TTLCache (30 min) | **Upstash Redis** (30 min TTL) |
| Vine sensors | In-memory TTLCache (5 min) | **Upstash Redis** (5 min TTL) |
| Energy daily | No cache | **Upstash Redis** (15 min TTL) |
| ML trained model | Pickle file | Pickle in `/tmp` + Redis metadata |
| Day-ahead plan | JSON file | Redis hash + JSON file backup |
| Chatbot history | Streamlit session_state | Redis list per session |

### External dependencies

| Service | URL | Auth | Purpose |
|---------|-----|------|---------|
| ThingsBoard | `web.seymouragri.com` | Username/password → JWT | 21 sensors, 4 trackers, energy |
| IMS | `api.ims.gov.il` | API token | Weather station 43 (Sde Boker) |
| Google Gemini | `generativelanguage.googleapis.com` | API key | Chatbot (2.5-flash), routing |
| Upstash Redis | `*.upstash.io` | Token | Shared cache |

---

## Layer 2 — Middleware (FastAPI on HuggingFace Spaces)

### What it is

A thin Python API layer that wraps `DataHub` services as REST endpoints. Hosted
on **HuggingFace Spaces** (Docker SDK, free tier) — Python-native, no cold starts
on the free tier's persistent container.

### Tech stack

| Component | Technology | Why |
|-----------|-----------|-----|
| Framework | **FastAPI** | Same Python team, async, auto-docs, WebSocket |
| Serialization | **Pydantic v2** | Already a FastAPI dependency, typed responses |
| Auth | **JWT** (PyJWT) | Simple, stateless, no external IdP needed initially |
| Rate limiting | **slowapi** | Token bucket per IP, protects Gemini quota |
| CORS | FastAPI `CORSMiddleware` | Allow Cloudflare Pages frontend origin |
| WebSocket | FastAPI WebSocket | Live tracker angles, energy push (future) |
| Process manager | **uvicorn** | ASGI server, production-ready |
| Hosting | **HuggingFace Spaces** (Docker SDK) | Free, Python-native, persistent container |

### HuggingFace Space configuration

The API Space uses Docker SDK. A `Dockerfile` at `backend/` builds the FastAPI
server. The Space README metadata:

```yaml
---
title: SolarWine API
emoji: 🌿
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
private: true
---
```

Note: HF Spaces exposes port 7860, so FastAPI binds to `0.0.0.0:7860`.

### API endpoints

```
Health
  GET  /api/health                         → { status, uptime, redis_ok, tb_ok }

Weather
  GET  /api/weather/current                → WeatherService.get_current()
  GET  /api/weather/history?days=7         → WeatherService.get_history()
  GET  /api/weather/forecast               → WeatherService.get_forecast()

Vine Sensors
  GET  /api/sensors/snapshot               → VineSensorService.get_snapshot()
  GET  /api/sensors/history?type=crop&hours=24  → VineSensorService.get_history()

Energy
  GET  /api/energy/current                 → EnergyService.get_current()
  GET  /api/energy/daily/{date}            → EnergyService.get_daily_production()
  GET  /api/energy/history?hours=24        → EnergyService.get_history()
  GET  /api/energy/predict/{date}          → EnergyService.predict()

Photosynthesis
  GET  /api/photosynthesis/current         → PhotosynthesisService.get_current()
  GET  /api/photosynthesis/forecast        → PhotosynthesisService.forecast_day_ahead()

Control
  GET  /api/control/status                 → Last TickResult from Redis
  GET  /api/control/plan                   → Current DayAheadPlan from Redis
  GET  /api/control/budget                 → EnergyBudget current state
  GET  /api/control/trackers               → Live tracker angles from TB

Chatbot
  POST /api/chatbot/message                → VineyardChatbot.chat()
       Body: { message, session_id }
       Response: { message, confidence, sources, caveats, rule_violations }

  POST /api/chatbot/feedback               → feedback.log_feedback()
       Body: { session_id, message_id, rating, comment }

Biology
  GET  /api/biology/phenology              → BiologyService current stage
  GET  /api/biology/rules                  → All biology rules
```

### Middleware stack (order matters)

1. **CORS** — allow `*.pages.dev` + localhost
2. **Rate limiter** — 60 req/min general, 10 req/min chatbot (Gemini costs)
3. **Auth** — JWT validation (initially optional, enforced in Phase 11.6)
4. **Request logging** — structured JSON logs
5. **Error handler** — consistent error response format

---

## Layer 3 — Frontend (Lovable → Cloudflare Pages)

### What it is

Production-grade React frontend generated via **Lovable** (AI-powered React
builder). Produces a standard Vite + React + TypeScript + Tailwind project.
Deployed to **Cloudflare Pages** (free tier, unlimited bandwidth, private org repos supported).

Lovable generates the initial scaffold and components rapidly. After generation,
the code is committed to the monorepo under `frontend/` and maintained manually
or with further Lovable iterations.

### Tech stack

| Component | Technology | Why |
|-----------|-----------|-----|
| Generator | **Lovable** | AI-generated React UI, rapid prototyping |
| Framework | **React 19 + Vite** | Lovable default output, fast build |
| Language | **TypeScript** | Type safety, IDE support |
| Styling | **Tailwind CSS + shadcn/ui** | Lovable default, beautiful components |
| Charts | **Recharts** (default) or **Plotly.js** | Data visualization |
| HTTP | **TanStack Query** (React Query) | Auto-refresh, caching, loading states |
| Routing | **React Router** | Page navigation |
| Hosting | **Cloudflare Pages** (free) | Unlimited bandwidth, private org repos, global CDN, auto-deploy from GitHub |

### Pages (mapped from current Streamlit tabs)

| Streamlit Tab | React Page | Route | Key Components |
|---------------|------------|-------|----------------|
| Vineyard Advisor | `Advisor.tsx` | `/` | Chatbot, weather card, energy card, shading card, irrigation card |
| System Status | `Control.tsx` | `/control` | Tracker angles, energy budget gauge, control replay, ROI |
| Overview | `Dashboard.tsx` | `/dashboard` | Energy history, crop comparisons |
| Photosynthesis & Data | `Photosynthesis.tsx` | `/photosynthesis` | FvCB/ML results, sensor explorer |
| Forecasting | — | merged into `/dashboard` | Weather + PS forecast cards |
| Shading Simulator | `Shading.tsx` | `/shading` | Interactive offset tester |
| Documentation | `Docs.tsx` | `/docs` | Help, API reference |

### Data fetching pattern

Using TanStack Query for all API calls. Auto-refetch on window focus and
configurable polling intervals.

```typescript
// hooks/useWeather.ts
import { useQuery } from "@tanstack/react-query";
import { api } from "@/lib/api";

export function useCurrentWeather() {
  return useQuery({
    queryKey: ["weather", "current"],
    queryFn: () => api.get("/api/weather/current"),
    refetchInterval: 60_000, // 60s auto-refresh
  });
}
```

```typescript
// lib/api.ts
const API_URL = import.meta.env.VITE_API_URL;

export const api = {
  get: async (path: string) => {
    const res = await fetch(`${API_URL}${path}`);
    if (!res.ok) throw new Error(res.statusText);
    return res.json();
  },
  post: async (path: string, body: unknown) => {
    const res = await fetch(`${API_URL}${path}`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(body),
    });
    if (!res.ok) throw new Error(res.statusText);
    return res.json();
  },
};
```

### Frontend ↔ Backend communication

| Pattern | Use Case | Protocol |
|---------|----------|----------|
| REST | All data reads, chatbot messages | HTTPS (fetch via TanStack Query) |
| Polling | Dashboard auto-refresh | 60s interval via `refetchInterval` |
| WebSocket | Live tracker angles, energy (future) | WSS (native WebSocket, future) |

---

## Infrastructure & Deployment

### Hosting (all free tier)

| Layer | Platform | Tier | Limits |
|-------|----------|------|--------|
| Frontend | **Cloudflare Pages** | Free | Unlimited bandwidth, 500 builds/month, auto-deploy from GitHub |
| API server | **HuggingFace Spaces** | Free (Docker SDK) | 2 vCPU, 16GB RAM, persistent container, private spaces |
| Cron workers | **GitHub Actions** | Free (private repos) | 2,000 min/month, plenty for 15-min cron + daily planner |
| Cache | **Upstash Redis** | Free | 10K commands/day, 256MB, REST API |
| Secrets | **HF Space secrets** + **Cloudflare env vars** + **GitHub Actions secrets** | Free | Per-service |
| Repo | **GitHub** | Free private | Unlimited |

### Deployment topology

```
GitHub (private repo, main branch)
    │
    ├──── push ────► Cloudflare Pages (auto-deploy)
    │                ├── frontend/ → Vite build → static React app
    │                ├── URL: baseline.pages.dev (or custom domain)
    │                └── Env: VITE_API_URL=https://<user>-solarwine-api.hf.space
    │
    ├──── push ────► HuggingFace Space (auto-deploy via HF GitHub sync)
    │                ├── backend/ → Dockerfile → FastAPI (uvicorn, port 7860)
    │                ├── URL: https://<user>-solarwine-api.hf.space
    │                └── Secrets: TB creds, IMS token, Gemini key, Redis URL
    │
    └──── cron ────► GitHub Actions (scheduled workflows)
                     ├── control-tick.yml: */15 * * * * → pip install + python control_tick.py
                     ├── daily-planner.yml: 0 2 * * * (02:00 UTC = 05:00 IST)
                     └── Secrets: TB creds, IMS token, Redis URL

Upstash Redis
    └── URL: solarwine-cache-*.upstash.io (shared by API + workers)
```

### GitHub Actions cron workflows

The ControlLoop and DayAheadPlanner run as scheduled GitHub Actions. This
replaces the need for a separate worker host.

```yaml
# .github/workflows/control-tick.yml
name: Control Loop Tick
on:
  schedule:
    - cron: "*/15 * * * *"    # every 15 min
  workflow_dispatch:           # manual trigger

jobs:
  tick:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt -r backend/requirements.txt
      - run: python -m backend.workers.control_tick
        env:
          THINGSBOARD_HOST: ${{ secrets.THINGSBOARD_HOST }}
          THINGSBOARD_USERNAME: ${{ secrets.THINGSBOARD_USERNAME }}
          THINGSBOARD_PASSWORD: ${{ secrets.THINGSBOARD_PASSWORD }}
          IMS_API_TOKEN: ${{ secrets.IMS_API_TOKEN }}
          UPSTASH_REDIS_URL: ${{ secrets.UPSTASH_REDIS_URL }}
          UPSTASH_REDIS_TOKEN: ${{ secrets.UPSTASH_REDIS_TOKEN }}
```

```yaml
# .github/workflows/daily-planner.yml
name: Day-Ahead Planner
on:
  schedule:
    - cron: "0 2 * * *"       # 02:00 UTC = 05:00 IST
  workflow_dispatch:

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt -r backend/requirements.txt
      - run: python -m backend.workers.daily_planner
        env:
          THINGSBOARD_HOST: ${{ secrets.THINGSBOARD_HOST }}
          THINGSBOARD_USERNAME: ${{ secrets.THINGSBOARD_USERNAME }}
          THINGSBOARD_PASSWORD: ${{ secrets.THINGSBOARD_PASSWORD }}
          IMS_API_TOKEN: ${{ secrets.IMS_API_TOKEN }}
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
          UPSTASH_REDIS_URL: ${{ secrets.UPSTASH_REDIS_URL }}
          UPSTASH_REDIS_TOKEN: ${{ secrets.UPSTASH_REDIS_TOKEN }}
```

### Docker setup (HuggingFace Space)

```dockerfile
# backend/Dockerfile
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
COPY backend/requirements.txt backend/
RUN pip install -r requirements.txt -r backend/requirements.txt

COPY src/ src/
COPY config/ config/
COPY backend/ backend/
COPY Data/ Data/

ENV PYTHONPATH=/app

# HuggingFace Spaces requires port 7860
EXPOSE 7860
CMD ["uvicorn", "backend.api.main:app", "--host", "0.0.0.0", "--port", "7860"]
```

### Environment variables

| Variable | Where | Source |
|----------|-------|--------|
| `THINGSBOARD_HOST` | HF Space, GH Actions | HF secrets, GH secrets |
| `THINGSBOARD_USERNAME` | HF Space, GH Actions | HF secrets, GH secrets |
| `THINGSBOARD_PASSWORD` | HF Space, GH Actions | HF secrets, GH secrets |
| `IMS_API_TOKEN` | HF Space, GH Actions | HF secrets, GH secrets |
| `GOOGLE_API_KEY` | HF Space, GH Actions | HF secrets, GH secrets |
| `UPSTASH_REDIS_URL` | HF Space, GH Actions | HF secrets, GH secrets |
| `UPSTASH_REDIS_TOKEN` | HF Space, GH Actions | HF secrets, GH secrets |
| `JWT_SECRET` | HF Space | HF secrets |
| `ALLOWED_ORIGINS` | HF Space | HF secrets (Cloudflare Pages URL) |
| `VITE_API_URL` | Cloudflare Pages | Cloudflare env vars (HF Space URL) |

---

## Data Flow

### Request flow (user views dashboard)

```
Browser → Cloudflare CDN → React SPA (static)
    │
    └── useQuery → fetch https://<user>-solarwine-api.hf.space/api/weather/current
                        │
                        ▼
                    FastAPI route (HuggingFace Spaces)
                        │
                        ▼
                    DataHub.weather.get_current()
                        │
                        ├── Redis cache hit? → return cached
                        └── Redis miss → IMS API → cache → return
```

### Control flow (15-min tick)

```
GitHub Actions cron (*/15 * * * *)
    │
    └── python -m backend.workers.control_tick
            │
            ▼
        ControlLoop.tick()
            │
            ├── DataHub.weather.get_current() → IMS (via Redis cache)
            ├── DataHub.vine_sensors.get_snapshot() → ThingsBoard
            ├── RoutingAgent → FvCB or ML model
            ├── DayAheadPlan lookup → Redis
            ├── InterventionGate check
            ├── BudgetGuard check
            ├── CommandArbiter → decide angle
            ├── TrackerDispatcher → ThingsBoard shared attributes
            ├── EnergyBudget.spend_slot()
            └── Save TickResult → Redis (for API /control/status)
```

### Chatbot flow

```
Browser → React → POST /api/chatbot/message { message, session_id }
    │
    ▼
FastAPI route (HF Space) → VineyardChatbot.chat(message, hub=DataHub)
    │
    ├── QueryClassifier → data / knowledge / greeting
    ├── Gemini Pass 1 → tool calls (if data query)
    ├── DataHub.dispatch_tool() → real data
    ├── tag_tool_result() → source metadata
    ├── Gemini Pass 2 → grounded response
    ├── ResponseValidator → rule checks
    ├── estimate_confidence()
    └── Return ChatResponse → JSON → Browser
```

---

## Security

| Concern | Mitigation |
|---------|-----------|
| API keys in repo | `.env` in `.gitignore`; secrets in HF/Cloudflare/GH settings |
| TB credentials | Server-side only; frontend never sees TB creds |
| Gemini API abuse | Rate limit 10 req/min on `/api/chatbot/message` |
| CORS | Whitelist Cloudflare Pages domain only |
| Auth (Phase 11.6) | JWT tokens; initially optional for research use |
| Tracker commands | Only GH Actions worker can write to TB; API is read-only |
| HF Space | Set to **private** — not publicly accessible without auth |

---

## Migration from Streamlit

The existing `app.py` and `ui/` folder remain functional throughout migration.
They are deprecated once the Lovable frontend reaches feature parity.

| Phase | Streamlit | Lovable/React | Notes |
|-------|-----------|---------------|-------|
| 11.1–11.3 | Primary | Not started | Backend being built |
| 11.4 | Primary | Scaffold | Pages generated via Lovable |
| 11.5 | Deprecated | Primary | Feature parity reached |
| 11.6+ | Removed | Primary | Auth, monitoring, polish |

---

## Scaling path (when free tier is outgrown)

| Bottleneck | Upgrade | Cost |
|-----------|---------|------|
| HF Space resources | HF Space upgrade (2× CPU, 16GB) | $0 (already free) → $9/mo for GPU |
| GitHub Actions minutes | GitHub Team (3,000 min) | $4/user/mo |
| Upstash 10K commands/day | Upstash Pay-as-you-go ($0.20/100K) | ~$1/mo |
| Cloudflare Pages | Already unlimited on free tier | $0 |
| Heavy ML inference | Add GPU worker (Modal/RunPod) | $5–10/mo |
| Multi-site (>1 vineyard) | Add Postgres (Neon free) for multi-tenant state | Free → $19/mo |

Total cost to run a real commercial deployment: **~$25/month**.