Spaces:

SolarWine
/

api

Sleeping

App Files Files Community

api / context /4_production.md

safraeli

Deploy: 2026 sensor migration + redesign + bucket B endpoints

13fc29d verified 6 months ago

preview code

Raw History Blame Contribute Delete

22.5 kB

SolarWine — Production Architecture

Overview

Migrate from a single-process Streamlit prototype to a 3-layer production system deployed on free-tier infrastructure. The monorepo stays — no new repositories.

Baseline/                          (monorepo)
├── frontend/                      NEW — Lovable-generated React app
│   ├── src/
│   │   ├── pages/
│   │   │   ├── Dashboard.tsx      Overview + energy + weather
│   │   │   ├── Advisor.tsx        Chatbot + advisory cards
│   │   │   ├── Control.tsx        System status + tracker angles
│   │   │   ├── Photosynthesis.tsx FvCB/ML results + forecasts
│   │   │   ├── Shading.tsx        Interactive simulator
│   │   │   └── Docs.tsx           Documentation
│   │   ├── components/
│   │   │   ├── EnergyCard.tsx
│   │   │   ├── WeatherCard.tsx
│   │   │   ├── Chatbot.tsx
│   │   │   ├── TrackerStatus.tsx
│   │   │   └── Charts.tsx         Recharts / Plotly wrappers
│   │   ├── hooks/
│   │   │   ├── useWeather.ts      fetch /api/weather/*
│   │   │   ├── useEnergy.ts       fetch /api/energy/*
│   │   │   ├── useSensors.ts      fetch /api/sensors/*
│   │   │   ├── useChatbot.ts      POST /api/chatbot/message
│   │   │   └── useControl.ts      fetch /api/control/*
│   │   ├── lib/
│   │   │   └── api.ts             API base URL + fetch wrapper
│   │   └── App.tsx                Router + layout
│   ├── package.json
│   └── tailwind.config.js
│
├── backend/                       NEW — FastAPI middleware + background services
│   ├── api/
│   │   ├── __init__.py
│   │   ├── main.py                FastAPI app, CORS, lifespan
│   │   ├── routes/
│   │   │   ├── weather.py         GET /api/weather/*
│   │   │   ├── sensors.py         GET /api/sensors/*
│   │   │   ├── energy.py          GET /api/energy/*
│   │   │   ├── photosynthesis.py  GET /api/photosynthesis/*
│   │   │   ├── control.py         GET /api/control/*
│   │   │   ├── chatbot.py         POST /api/chatbot/message
│   │   │   └── health.py          GET /api/health
│   │   ├── deps.py                Shared dependencies (DataHub, Redis)
│   │   └── auth.py                JWT middleware (Phase 11.6)
│   ├── workers/
│   │   ├── control_tick.py        ControlLoop 15-min cron entry point
│   │   └── daily_planner.py       DayAheadPlanner daily cron entry point
│   ├── Dockerfile
│   └── requirements.txt
│
├── .github/
│   └── workflows/
│       ├── control-tick.yml       GitHub Actions: 15-min cron
│       └── daily-planner.yml      GitHub Actions: daily 05:00 IST cron
│
├── src/                           UNCHANGED — shared Python core
│   ├── control_loop.py
│   ├── command_arbiter.py
│   ├── tracker_dispatcher.py
│   ├── day_ahead_planner.py
│   ├── energy_budget.py
│   ├── data/
│   │   ├── data_providers.py      DataHub + 6 services
│   │   ├── thingsboard_client.py
│   │   └── ims_client.py
│   ├── chatbot/
│   │   ├── vineyard_chatbot.py
│   │   ├── guardrails.py
│   │   └── routing_agent.py
│   ├── models/
│   │   ├── farquhar_model.py
│   │   └── phenology.py
│   ├── forecasting/
│   │   ├── predictor.py
│   │   └── ts_predictor.py
│   ├── shading/
│   │   ├── solar_geometry.py
│   │   └── tradeoff_engine.py
│   └── ...
│
├── config/settings.py             UNCHANGED — shared configuration
├── app.py                         KEPT temporarily (Streamlit, deprecated)
├── ui/                            KEPT temporarily (Streamlit tabs, deprecated)
├── docker-compose.yml             NEW — local dev orchestration
└── requirements.txt               Shared Python deps

Layer 1 — The Brain (Backend Services)

What it is

All business logic, models, data access, and control algorithms. This layer already exists in src/. The only change is extracting the runtime entry points into backend/workers/.

Components

Component	Module	Runtime	Schedule
ControlLoop	`src/control_loop.py`	GitHub Actions cron	Every 15 min
DayAheadPlanner	`src/day_ahead_planner.py`	GitHub Actions cron	Daily 05:00 IST + re-plan events
DataHub	`src/data/data_providers.py`	Imported by API + workers	Always available
ThingsBoard client	`src/data/thingsboard_client.py`	Imported by DataHub	On-demand
IMS client	`src/data/ims_client.py`	Imported by DataHub	On-demand (30-min TTL)
FvCB model	`src/models/farquhar_model.py`	Imported by PhotosynthesisService	On-demand (~10ms)
ML ensemble	`src/forecasting/predictor.py`	Imported by PhotosynthesisService	On-demand (~500ms)
Chronos forecaster	`src/forecasting/ts_predictor.py`	Imported by PhotosynthesisService	On-demand (~2-5s)
CommandArbiter	`src/command_arbiter.py`	Imported by ControlLoop	Per tick
TrackerDispatcher	`src/tracker_dispatcher.py`	Imported by ControlLoop	Per tick
EnergyBudget	`src/energy_budget.py`	Imported by ControlLoop	Per tick
Chatbot + guardrails	`src/chatbot/*.py`	Imported by API route	Per message

Caching strategy

Data	Current	Production
IMS weather	In-memory TTLCache (30 min)	Upstash Redis (30 min TTL)
Vine sensors	In-memory TTLCache (5 min)	Upstash Redis (5 min TTL)
Energy daily	No cache	Upstash Redis (15 min TTL)
ML trained model	Pickle file	Pickle in `/tmp` + Redis metadata
Day-ahead plan	JSON file	Redis hash + JSON file backup
Chatbot history	Streamlit session_state	Redis list per session

External dependencies

Service	URL	Auth	Purpose
ThingsBoard	`web.seymouragri.com`	Username/password → JWT	21 sensors, 4 trackers, energy
IMS	`api.ims.gov.il`	API token	Weather station 43 (Sde Boker)
Google Gemini	`generativelanguage.googleapis.com`	API key	Chatbot (2.5-flash), routing
Upstash Redis	`*.upstash.io`	Token	Shared cache

Layer 2 — Middleware (FastAPI on HuggingFace Spaces)

What it is

A thin Python API layer that wraps DataHub services as REST endpoints. Hosted on HuggingFace Spaces (Docker SDK, free tier) — Python-native, no cold starts on the free tier's persistent container.

Tech stack

Component	Technology	Why
Framework	FastAPI	Same Python team, async, auto-docs, WebSocket
Serialization	Pydantic v2	Already a FastAPI dependency, typed responses
Auth	JWT (PyJWT)	Simple, stateless, no external IdP needed initially
Rate limiting	slowapi	Token bucket per IP, protects Gemini quota
CORS	FastAPI `CORSMiddleware`	Allow Cloudflare Pages frontend origin
WebSocket	FastAPI WebSocket	Live tracker angles, energy push (future)
Process manager	uvicorn	ASGI server, production-ready
Hosting	HuggingFace Spaces (Docker SDK)	Free, Python-native, persistent container

HuggingFace Space configuration

The API Space uses Docker SDK. A Dockerfile at backend/ builds the FastAPI server. The Space README metadata:

---
title: SolarWine API
emoji: 🌿
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
private: true
---

Note: HF Spaces exposes port 7860, so FastAPI binds to 0.0.0.0:7860.

API endpoints

Health
  GET  /api/health                         → { status, uptime, redis_ok, tb_ok }

Weather
  GET  /api/weather/current                → WeatherService.get_current()
  GET  /api/weather/history?days=7         → WeatherService.get_history()
  GET  /api/weather/forecast               → WeatherService.get_forecast()

Vine Sensors
  GET  /api/sensors/snapshot               → VineSensorService.get_snapshot()
  GET  /api/sensors/history?type=crop&hours=24  → VineSensorService.get_history()

Energy
  GET  /api/energy/current                 → EnergyService.get_current()
  GET  /api/energy/daily/{date}            → EnergyService.get_daily_production()
  GET  /api/energy/history?hours=24        → EnergyService.get_history()
  GET  /api/energy/predict/{date}          → EnergyService.predict()

Photosynthesis
  GET  /api/photosynthesis/current         → PhotosynthesisService.get_current()
  GET  /api/photosynthesis/forecast        → PhotosynthesisService.forecast_day_ahead()

Control
  GET  /api/control/status                 → Last TickResult from Redis
  GET  /api/control/plan                   → Current DayAheadPlan from Redis
  GET  /api/control/budget                 → EnergyBudget current state
  GET  /api/control/trackers               → Live tracker angles from TB

Chatbot
  POST /api/chatbot/message                → VineyardChatbot.chat()
       Body: { message, session_id }
       Response: { message, confidence, sources, caveats, rule_violations }

  POST /api/chatbot/feedback               → feedback.log_feedback()
       Body: { session_id, message_id, rating, comment }

Biology
  GET  /api/biology/phenology              → BiologyService current stage
  GET  /api/biology/rules                  → All biology rules

Middleware stack (order matters)

CORS — allow *.pages.dev + localhost
Rate limiter — 60 req/min general, 10 req/min chatbot (Gemini costs)
Auth — JWT validation (initially optional, enforced in Phase 11.6)
Request logging — structured JSON logs
Error handler — consistent error response format

Layer 3 — Frontend (Lovable → Cloudflare Pages)

What it is

Production-grade React frontend generated via Lovable (AI-powered React builder). Produces a standard Vite + React + TypeScript + Tailwind project. Deployed to Cloudflare Pages (free tier, unlimited bandwidth, private org repos supported).

Lovable generates the initial scaffold and components rapidly. After generation, the code is committed to the monorepo under frontend/ and maintained manually or with further Lovable iterations.

Tech stack

Component	Technology	Why
Generator	Lovable	AI-generated React UI, rapid prototyping
Framework	React 19 + Vite	Lovable default output, fast build
Language	TypeScript	Type safety, IDE support
Styling	Tailwind CSS + shadcn/ui	Lovable default, beautiful components
Charts	Recharts (default) or Plotly.js	Data visualization
HTTP	TanStack Query (React Query)	Auto-refresh, caching, loading states
Routing	React Router	Page navigation
Hosting	Cloudflare Pages (free)	Unlimited bandwidth, private org repos, global CDN, auto-deploy from GitHub

Pages (mapped from current Streamlit tabs)

Streamlit Tab	React Page	Route	Key Components
Vineyard Advisor	`Advisor.tsx`	`/`	Chatbot, weather card, energy card, shading card, irrigation card
System Status	`Control.tsx`	`/control`	Tracker angles, energy budget gauge, control replay, ROI
Overview	`Dashboard.tsx`	`/dashboard`	Energy history, crop comparisons
Photosynthesis & Data	`Photosynthesis.tsx`	`/photosynthesis`	FvCB/ML results, sensor explorer
Forecasting	—	merged into `/dashboard`	Weather + PS forecast cards
Shading Simulator	`Shading.tsx`	`/shading`	Interactive offset tester
Documentation	`Docs.tsx`	`/docs`	Help, API reference

Data fetching pattern

Using TanStack Query for all API calls. Auto-refetch on window focus and configurable polling intervals.

// hooks/useWeather.ts
import { useQuery } from "@tanstack/react-query";
import { api } from "@/lib/api";

export function useCurrentWeather() {
  return useQuery({
    queryKey: ["weather", "current"],
    queryFn: () => api.get("/api/weather/current"),
    refetchInterval: 60_000, // 60s auto-refresh
  });
}

// lib/api.ts
const API_URL = import.meta.env.VITE_API_URL;

export const api = {
  get: async (path: string) => {
    const res = await fetch(`${API_URL}${path}`);
    if (!res.ok) throw new Error(res.statusText);
    return res.json();
  },
  post: async (path: string, body: unknown) => {
    const res = await fetch(`${API_URL}${path}`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(body),
    });
    if (!res.ok) throw new Error(res.statusText);
    return res.json();
  },
};

Frontend ↔ Backend communication

Pattern	Use Case	Protocol
REST	All data reads, chatbot messages	HTTPS (fetch via TanStack Query)
Polling	Dashboard auto-refresh	60s interval via `refetchInterval`
WebSocket	Live tracker angles, energy (future)	WSS (native WebSocket, future)

Infrastructure & Deployment

Hosting (all free tier)

Layer	Platform	Tier	Limits
Frontend	Cloudflare Pages	Free	Unlimited bandwidth, 500 builds/month, auto-deploy from GitHub
API server	HuggingFace Spaces	Free (Docker SDK)	2 vCPU, 16GB RAM, persistent container, private spaces
Cron workers	GitHub Actions	Free (private repos)	2,000 min/month, plenty for 15-min cron + daily planner
Cache	Upstash Redis	Free	10K commands/day, 256MB, REST API
Secrets	HF Space secrets + Cloudflare env vars + GitHub Actions secrets	Free	Per-service
Repo	GitHub	Free private	Unlimited

Deployment topology

GitHub (private repo, main branch)
    │
    ├──── push ────► Cloudflare Pages (auto-deploy)
    │                ├── frontend/ → Vite build → static React app
    │                ├── URL: baseline.pages.dev (or custom domain)
    │                └── Env: VITE_API_URL=https://<user>-solarwine-api.hf.space
    │
    ├──── push ────► HuggingFace Space (auto-deploy via HF GitHub sync)
    │                ├── backend/ → Dockerfile → FastAPI (uvicorn, port 7860)
    │                ├── URL: https://<user>-solarwine-api.hf.space
    │                └── Secrets: TB creds, IMS token, Gemini key, Redis URL
    │
    └──── cron ────► GitHub Actions (scheduled workflows)
                     ├── control-tick.yml: */15 * * * * → pip install + python control_tick.py
                     ├── daily-planner.yml: 0 2 * * * (02:00 UTC = 05:00 IST)
                     └── Secrets: TB creds, IMS token, Redis URL

Upstash Redis
    └── URL: solarwine-cache-*.upstash.io (shared by API + workers)

GitHub Actions cron workflows

The ControlLoop and DayAheadPlanner run as scheduled GitHub Actions. This replaces the need for a separate worker host.

# .github/workflows/control-tick.yml
name: Control Loop Tick
on:
  schedule:
    - cron: "*/15 * * * *"    # every 15 min
  workflow_dispatch:           # manual trigger

jobs:
  tick:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt -r backend/requirements.txt
      - run: python -m backend.workers.control_tick
        env:
          THINGSBOARD_HOST: ${{ secrets.THINGSBOARD_HOST }}
          THINGSBOARD_USERNAME: ${{ secrets.THINGSBOARD_USERNAME }}
          THINGSBOARD_PASSWORD: ${{ secrets.THINGSBOARD_PASSWORD }}
          IMS_API_TOKEN: ${{ secrets.IMS_API_TOKEN }}
          UPSTASH_REDIS_URL: ${{ secrets.UPSTASH_REDIS_URL }}
          UPSTASH_REDIS_TOKEN: ${{ secrets.UPSTASH_REDIS_TOKEN }}

# .github/workflows/daily-planner.yml
name: Day-Ahead Planner
on:
  schedule:
    - cron: "0 2 * * *"       # 02:00 UTC = 05:00 IST
  workflow_dispatch:

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt -r backend/requirements.txt
      - run: python -m backend.workers.daily_planner
        env:
          THINGSBOARD_HOST: ${{ secrets.THINGSBOARD_HOST }}
          THINGSBOARD_USERNAME: ${{ secrets.THINGSBOARD_USERNAME }}
          THINGSBOARD_PASSWORD: ${{ secrets.THINGSBOARD_PASSWORD }}
          IMS_API_TOKEN: ${{ secrets.IMS_API_TOKEN }}
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
          UPSTASH_REDIS_URL: ${{ secrets.UPSTASH_REDIS_URL }}
          UPSTASH_REDIS_TOKEN: ${{ secrets.UPSTASH_REDIS_TOKEN }}

Docker setup (HuggingFace Space)

# backend/Dockerfile
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
COPY backend/requirements.txt backend/
RUN pip install -r requirements.txt -r backend/requirements.txt

COPY src/ src/
COPY config/ config/
COPY backend/ backend/
COPY Data/ Data/

ENV PYTHONPATH=/app

# HuggingFace Spaces requires port 7860
EXPOSE 7860
CMD ["uvicorn", "backend.api.main:app", "--host", "0.0.0.0", "--port", "7860"]

Environment variables

Variable	Where	Source
`THINGSBOARD_HOST`	HF Space, GH Actions	HF secrets, GH secrets
`THINGSBOARD_USERNAME`	HF Space, GH Actions	HF secrets, GH secrets
`THINGSBOARD_PASSWORD`	HF Space, GH Actions	HF secrets, GH secrets
`IMS_API_TOKEN`	HF Space, GH Actions	HF secrets, GH secrets
`GOOGLE_API_KEY`	HF Space, GH Actions	HF secrets, GH secrets
`UPSTASH_REDIS_URL`	HF Space, GH Actions	HF secrets, GH secrets
`UPSTASH_REDIS_TOKEN`	HF Space, GH Actions	HF secrets, GH secrets
`JWT_SECRET`	HF Space	HF secrets
`ALLOWED_ORIGINS`	HF Space	HF secrets (Cloudflare Pages URL)
`VITE_API_URL`	Cloudflare Pages	Cloudflare env vars (HF Space URL)

Data Flow

Request flow (user views dashboard)

Browser → Cloudflare CDN → React SPA (static)
    │
    └── useQuery → fetch https://<user>-solarwine-api.hf.space/api/weather/current
                        │
                        ▼
                    FastAPI route (HuggingFace Spaces)
                        │
                        ▼
                    DataHub.weather.get_current()
                        │
                        ├── Redis cache hit? → return cached
                        └── Redis miss → IMS API → cache → return

Control flow (15-min tick)

GitHub Actions cron (*/15 * * * *)
    │
    └── python -m backend.workers.control_tick
            │
            ▼
        ControlLoop.tick()
            │
            ├── DataHub.weather.get_current() → IMS (via Redis cache)
            ├── DataHub.vine_sensors.get_snapshot() → ThingsBoard
            ├── RoutingAgent → FvCB or ML model
            ├── DayAheadPlan lookup → Redis
            ├── InterventionGate check
            ├── BudgetGuard check
            ├── CommandArbiter → decide angle
            ├── TrackerDispatcher → ThingsBoard shared attributes
            ├── EnergyBudget.spend_slot()
            └── Save TickResult → Redis (for API /control/status)

Chatbot flow

Browser → React → POST /api/chatbot/message { message, session_id }
    │
    ▼
FastAPI route (HF Space) → VineyardChatbot.chat(message, hub=DataHub)
    │
    ├── QueryClassifier → data / knowledge / greeting
    ├── Gemini Pass 1 → tool calls (if data query)
    ├── DataHub.dispatch_tool() → real data
    ├── tag_tool_result() → source metadata
    ├── Gemini Pass 2 → grounded response
    ├── ResponseValidator → rule checks
    ├── estimate_confidence()
    └── Return ChatResponse → JSON → Browser

Security

Concern	Mitigation
API keys in repo	`.env` in `.gitignore`; secrets in HF/Cloudflare/GH settings
TB credentials	Server-side only; frontend never sees TB creds
Gemini API abuse	Rate limit 10 req/min on `/api/chatbot/message`
CORS	Whitelist Cloudflare Pages domain only
Auth (Phase 11.6)	JWT tokens; initially optional for research use
Tracker commands	Only GH Actions worker can write to TB; API is read-only
HF Space	Set to private — not publicly accessible without auth

Migration from Streamlit

The existing app.py and ui/ folder remain functional throughout migration. They are deprecated once the Lovable frontend reaches feature parity.

Phase	Streamlit	Lovable/React	Notes
11.1–11.3	Primary	Not started	Backend being built
11.4	Primary	Scaffold	Pages generated via Lovable
11.5	Deprecated	Primary	Feature parity reached
11.6+	Removed	Primary	Auth, monitoring, polish

Scaling path (when free tier is outgrown)

Bottleneck	Upgrade	Cost
HF Space resources	HF Space upgrade (2× CPU, 16GB)	$0 (already free) → $9/mo for GPU
GitHub Actions minutes	GitHub Team (3,000 min)	$4/user/mo
Upstash 10K commands/day	Upstash Pay-as-you-go ($0.20/100K)	~$1/mo
Cloudflare Pages	Already unlimited on free tier	$0
Heavy ML inference	Add GPU worker (Modal/RunPod)	$5–10/mo
Multi-site (>1 vineyard)	Add Postgres (Neon free) for multi-tenant state	Free → $19/mo

Total cost to run a real commercial deployment: ~$25/month.