Hugging Face's logo

Spaces:
SolarWine
/
api
Sleeping

App Files Community
api / context /3_todo.md
safraeli's picture
safraeli
Deploy: 2026 sensor migration + redesign + bucket B endpoints
13fc29d verified
preview code
|
Raw History Blame Contribute Delete
61.8 kB

SolarWine – Todo Tracker

This file mirrors the high-level plan in 2_plan.md and tracks day-to-day progress.

Baseline v1 (Complete)

  • scaffold – project directory structure, .env, .gitignore, requirements.txt
  • context-files – create context/1_purpose.md, context/2_plan.md, context/3_todo.md
  • git-init – create git repository and make initial commit
  • oop-architecture – implement OOP modules in src/
  • ims-fetch – implement IMSClient and cache IMS data
  • stage1-farquhar – implement mechanistic Farquhar/Greer–Weedon model
  • stage1-validate – validate Stage 1 A outputs and plots
  • stage2-preprocess – preprocessing and temporal split
  • stage2-models – train and evaluate ML models
  • stage2-visualize – build evaluation visualizations
  • streamlit-app – build Streamlit UI (6 tabs)

SolarWine 2.0

Phase 1: Configuration & Data Infrastructure

  • config-extend – add energy budget, no-shade windows, fruiting zone, and tradeoff params to config/settings.py (most params added in earlier session; no-shade windows NO_SHADE_BEFORE_HOUR, NO_SHADE_MONTHS, NO_SHADE_GHI_BELOW, NO_SHADE_TLEAF_BELOW and SHADE_ELIGIBLE_HOURS added 2026-03-10; guardrails wired to reference settings)
  • data-schema – src/data/data_schema.py (telemetry dataclasses: SensorRaw, BiologicalState, TrackerKinematics, SimulationLog). Re-exported via src/data_schema.py
  • import-layout – scripts/import_layout.py generates Data/layout.json from config geometry + TB device registry (4 rows, 22 devices with 3D positions). Added 2026-03-10

Phase 2: Biological Digital Twin Upgrades

  • fvcb-semillon – upgrade src/farquhar_model.py with calc_photosynthesis_semillon() β†’ returns (A, limiting_state, shading_helps). Added 2026-03-10: uses configurable transition_temp (default SEMILLON_TRANSITION_TEMP_C=30Β°C); shading_helps is True only when Rubisco-limited AND Aj > Ac (light abundant relative to enzyme capacity)
  • shadow-fruiting – upgrade src/solar_geometry.py with fruiting_zone_shadow() and evaluate_candidate_offsets() (added 2026-03-10)
  • canopy-expose – compute_vine_A() now returns fruiting_zone_A, fruiting_zone_par, top_canopy_A, top_canopy_par alongside existing fields. Added 2026-03-10
  • safety-rails – src/advisor/safety_rails.py (FvCB vs ML divergence check, 12% threshold β†’ fallback). Re-exported via src/safety_rails.py
  • spectral-agg – src/spectral_aggregator.py with aggregate_spectral() (single-timestep) and aggregate_spectral_df() (batch DataFrame). CWSI from explicit value β†’ delta-T proxy β†’ VPD fallback. Physical bounds clipping with quality flags. Added 2026-03-10

Phase 3: The Control Algorithm (Core)

  • energy-budget – create src/energy_budget.py (hierarchical Yearβ†’Monthβ†’Weekβ†’Dayβ†’Slot budget planner). Added 2026-03-10: EnergyBudgetPlanner with compute_annual_plan(), compute_weekly_plan(), compute_daily_plan(), spend_slot(), emergency_draw(), compute_daily_rollover(). Analytical fallback when no ShadowModel. Budget: ~920 kWh potential β†’ 46 kWh budget (5%), July gets 17.6 kWh (45%)
  • intervention-gate – InterventionGate in src/shading/tradeoff_engine.py (physiology-only pass/fail: GHI, Tleaf, CWSI, shading_helps; no time/month hard-coding β€” geometry decides). Re-exported via src/tradeoff_engine.py
  • min-dose-engine – TradeoffEngine in src/shading/tradeoff_engine.py (smallest offset satisfying: fruiting PAR < 400 Β΅mol, top canopy β‰₯ 85% baseline, sacrifice ≀ budget; geometric feasibility pre-check; face-PAR sun-side selection)
  • command-arbiter – src/command_arbiter.py CommandArbiter (5-level priority stack: weatherβ†’harvestβ†’safetyβ†’timeoutβ†’engine; hysteresis filter with configurable window/tolerance; arbitrate() combines selection + filter; weather/harvest bypass hysteresis). Added 2026-03-10
  • astro-tracker – AstronomicalTracker in src/command_arbiter.py (pure sun-following via ShadowModel; get_angle(timestamp) interface). Added 2026-03-10

Phase 3.5: Agronomic Weighting & Day-Ahead Planner

  • zone-crop-weights – spatial zone weighting implemented in DayAheadPlanner._get_crop_multiplier() using STAGE_CROP_MULTIPLIER + ZONE_CROP_WEIGHTS from settings. Added 2026-03-10
  • day-ahead-dp – created src/day_ahead_planner.py with DayAheadPlanner (DP trajectory optimizer: forward pass over daylight slots Γ— candidate offsets, budget constraint via discretised DP, movement cost penalty for smooth trajectories, simplified InterventionGate using forecast data). Returns DayAheadPlan with SlotPlan list + explainability tags. Added 2026-03-10
  • daily-plan-output – DayAheadPlan.to_dict() serialises plan to JSON-ready dict with per-slot offset, energy cost, gate status, and tags. Added 2026-03-10
  • phenology-tracker – src/models/phenology.py upgraded with three estimation methods: (1) GDD-based via compute_gdd_from_ims() + estimate_stage_by_gdd() using IMS tdmax/tdmin and PHENOLOGY_GDD_THRESHOLDS, (2) camera-based via detect_stage_from_camera() using Gemini Vision on live vineyard feed, (3) calendar fallback. estimate_stage_combined() merges all three (camera high-confidence > GDD > calendar). Backward-compatible API. Added 2026-03-10
  • baseline-ps – created src/baseline_predictor.py (hybrid FvCB+ML with RoutingAgent rule-based selection per slot); wired into DayAheadPlanner as optional baseline_predictor param, replaces temperature heuristic with actual A predictions for crop value
  • arbiter-plan-integration – ControlLoop now tracks plan divergence (cumulative kWh + consecutive slots), triggers re-plan when thresholds exceeded, with cooldown to avoid thrashing
  • tracker-rpc – added send_rpc_command() and set_device_attributes() to ThingsBoardClient (two-way RPC with one-way fallback, shared attribute writes)
  • tracker-verify – TrackerDispatcher verifies |actual - target| < tolerance after dispatch via _verify_all()
  • tracker-dispatch – created src/tracker_dispatcher.py: takes ArbiterDecision β†’ sends to all 4 trackers via RPC/attribute fallback β†’ verifies execution β†’ logs result
  • control-loop – created src/control_loop.py: 15-min main loop (fetch sensors β†’ plan lookup β†’ live gate check β†’ budget guard β†’ arbitrate β†’ fleet overrides β†’ dispatch β†’ budget spend β†’ divergence check β†’ re-plan β†’ log). One-shot, continuous, and dry_run modes. TrackerFleet/Scheduler wired for per-tracker overrides (default = all same). EnergyBudgetPlanner wired for real-time budget consumption. Plan divergence tracking with auto re-plan

Phase 4: Operational Modes

  • ops-modes – created src/operational_modes.py (WindStow, HailStow, HeatShield, HarvestMode + composite OperationalModeChecker; integrated into ControlLoop)

Phase 5: ROI Reporting

  • roi-service – created src/roi_service.py (BudgetStatus, InterventionStats, LERResult, ROIService with budget tracking + LER computation)

Phase 6: Dashboard v2

  • tab-system-status – unified System Status tab (ui/tab_system_status.py) with 5 sub-sections:
    • Tracker Status: live angles from TB, coherence check, bar chart
    • Energy Budget: annual/monthly budget allocation, gauge, weekly plan
    • Control Replay: load simulation logs, timeline charts, slot-by-slot detail
    • Fruiting Zone: live canopy temp/PAR/VPD, sunburn risk assessment, thresholds
    • ROI Dashboard: budget utilisation gauge, intervention stats, LER calculator

Phase 7: Integration Testing

  • sim-script – created scripts/run_control_simulation.py (historical replay through full control loop with per-day summary, CLI args, JSON output)
  • validate-budget – scripts/validate_control_system.py verifies cumulative sacrifice never exceeds max_energy_reduction_pct (tested with real sim log + synthetic violations)
  • validate-gates – validates no interventions in May, before 10:00, below transition temp, or below GHI threshold (3/3 violation types caught in self-test)
  • validate-dose – validates average offset < 10Β°, max offset < 20Β°, intervention rate < 30% (catches excessive dose in self-test)

Phase 8: AI-Enhanced Forecasting & Routing

8A β€” Chronos-2 Multivariate Day-Ahead Forecasting

  • install-chronos – install chronos-forecasting>=2.0 and verify compatibility with existing env (done: conda env solarwine with Python 3.12, chronos 2.2.2, torch 2.10, llvm-openmp for XGBoost)
  • chronos-data-prep – build multivariate input pipeline: combine historical A_n (target) with past IMS station 43 measurements + on-site sensors (PAR, T_canopy, VPD, CWSI) as historical context, plus IMS forecast data (predicted GHI, air temperature, wind speed, humidity) as known-future covariates for the prediction horizon; IMS forecasts are available at inference time so this is not a data leak β€” only actual future sensor measurements would be
  • chronos-predict – execute zero-shot 96-step (24h @ 15-min) day-ahead A_n prediction using Chronos-2 (amazon/chronos-t5-small or -base) with multivariate context (past A_n + past sensor measurements as history, IMS forecasts as known-future covariates); integrate into src/day_ahead_planner.py baseline prediction
  • chronos-benchmark – benchmark Chronos-2 MAE against existing ML baseline (MAE β‰ˆ 2.7) on held-out test set; compare univariate (A_n only) vs multivariate (A_n + IMS covariates) performance; document results and decide on integration path

8B β€” Gemini LLM Data Engineering

  • install-gemini – google-genai>=1.0 added to requirements.txt; already installed in solarwine conda env (v1.65.0)
  • llm-data-cleaning – src/llm_data_engineer.py LLMDataEngineer.analyze_anomalies(): sends per-column descriptive stats + domain context (physical ranges, Negev site notes) to Gemini; receives Z-score/IQR/bound thresholds as JSON; apply_cleaning() applies clip/nan/drop strategies; graceful fallback to physics-based bounds when API key unavailable
  • llm-feature-eng – LLMDataEngineer.engineer_features(): adds hour_sin, hour_cos, doy_sin, doy_cos (cyclical time encodings) and stress_risk_score (normalised weighted VPD + CWSI, weights confirmed by Gemini); run_pipeline() orchestrates clean β†’ feature-engineer in one call
  • dashboard-8b – Data Explorer: added AI Data Engineering as third radio option (Gemini anomaly thresholds, cleaning summary, before/after charts, engineered features, daytime stress profile). Presentation tab: added Gemini LLM data engineering to β€œWhat’s Next” completed list, Key takeaways, and β€œWhat each tab shows” (Data Explorer bullet). All data precomputed in-tab (no cache loader).

8C β€” Gemini Intelligent Routing Layer

  • routing-prompt – define supervisory system prompt for Gemini model router: given real-time telemetry (Temp, Irradiance, CWSI), route to FvCB (Model A, accurate under standard conditions) or ML (Model B, handles non-linear stress) β€” reply only MODEL_A or MODEL_B
  • routing-agent – build RoutingAgent function: executes Gemini API call at start of 15-min control loop, constructs prompt from live telemetry, parses response to determine model path
  • arbiter-routing – RoutingAgent wired into ControlLoop tick (step 2b): routes each slot to FvCB or ML via rule-based fast path (>90% of cases) with Gemini fallback for transition zone; model_route field added to TickResult; safety_rails divergence check remains as independent safety layer

8E β€” Gemini Day-Ahead Advisor

  • day-ahead-advisor – src/day_ahead_advisor.py DayAheadAdvisor: Gemini analyzes IMS forecast + vine state β†’ structured stress profile, budget recommendations, model routing, Chronos sanity check; dashboard display in Predictions tab

8D β€” Shadow Mode Validation

  • shadow-mode-test – created scripts/shadow_mode_test.py: 4 validation tests β€” (1) LLM data cleaning filters catch anomalies, (2) BaselinePredictor generates valid A profiles (peak=17.0 Β΅mol, daylight mean=6.1), (3) RoutingAgent routes coolβ†’fvcb and hotβ†’ml correctly, (4) full pipeline shadow simulation over date range

Phase 9: Vineyard Advisor Chatbot

  • chatbot-todo – add Phase 9 to context/3_todo.md
  • chatbot-scaffold – create src/vineyard_chatbot.py scaffold + Gemini client (lazy init, same pattern as DayAheadAdvisor)
  • chatbot-system-prompt – write CHATBOT_SYSTEM_PROMPT (persona, site facts, 8 biology rules, tool definitions, response rules)
  • chatbot-tool-defs – implement TOOL_DEFINITIONS list + _dispatch_tool() router
  • chatbot-tools-weather – implement _tool_current_weather(), _tool_calc_photosynthesis(), _tool_explain_rule()
  • chatbot-tools-shading – implement _tool_simulate_shading(), _tool_compare_angles(), _tool_daily_schedule(), _tool_day_ahead_advisory()
  • chatbot-chat-flow – implement two-pass chat flow (_build_messages(), _call_gemini(), chat())
  • chatbot-tab – add Vineyard Advisor tab to app.py (Vineyard Advisor tab, chat UI, session state)
  • chatbot-quick-actions – add quick-action buttons + _render_tool_data() helper + clear chat
  • chatbot-test – syntax check, import check, tabs render, tool call test, fallback test

Phase 9.5: Chatbot Data Integration

Wire all five data sources into the chatbot (IMS weather, TB sensors, PS prediction, energy generation, energy prediction) via a loosely-coupled provider layer.

Architecture (src/data_providers.py) 

VineyardChatbot  ──▢  DataHub  ──┬── WeatherService       (IMS, 30-min TTL cache)
  (tool dispatch)     (registry)  β”œβ”€β”€ VineSensorService    (TB, 5-min TTL snapshot)
                                  β”œβ”€β”€ PhotosynthesisService (FvCB + ML + forecast)
                                  β”œβ”€β”€ EnergyService        (TB generation + prediction)
                                  β”œβ”€β”€ AdvisoryService      (Gemini day-ahead)
                                  └── BiologyService       (rules lookup)
  • Chatbot never imports data clients directly β€” only talks to self.hub.
  • Each Service owns its caching (TTLCache), error handling, lazy init.
  • Adding a new source = write a Service subclass + hub.register().
  • summarise_dataframe() auto-compresses large results for LLM context.

Data strategy

Source Service Strategy Detail
IMS weather WeatherService Cache (preload) ims_merged_15min.csv, 30-min TTL
TB sensor snapshot VineSensorService On-demand + 5-min TTL get_vine_snapshot()
TB sensor time-series VineSensorService On-demand get_timeseries() β†’ hourly downsample
PS prediction (FvCB/ML) PhotosynthesisService On-demand (compute) Lazy-load models
Energy generation (TB) EnergyService On-demand Fetch from inverter/meter devices
Energy prediction EnergyService On-demand (compute) GHI Γ— panel Γ— efficiency Γ— cos(ΞΈ)

Steps

  • chatbot-provider-scaffold – Create src/data_providers.py with BaseService ABC, TTLCache, summarise_dataframe, DataHub registry, and 6 service classes (Weather, VineSensor, Photosynthesis, Energy, Advisory, Biology)
  • chatbot-hub-wiring – Refactored VineyardChatbot.__init__ to accept a DataHub (default: DataHub.default()); rewrote _dispatch_tool() to delegate to hub services (15 tools); removed all direct imports of IMSClient/ThingsBoardClient/FarquharModel from chatbot
  • chatbot-ims-history – Implemented WeatherService.get_history() with hourly resample + summarise_dataframe(); added get_weather_history tool to system prompt and dispatch
  • chatbot-tb-timeseries – Implemented VineSensorService.get_history() (device_type, area, hours_back β†’ hourly averages via get_timeseries()); added get_sensor_history tool to system prompt and dispatch
  • chatbot-ml-predictor – Implemented PhotosynthesisService.predict_ml() with train-once-cache pattern: lazy loads sensor data + IMS, runs Preprocessor + PhotosynthesisPredictor, picks best model by MAE, auto-fills features from latest IMS if not provided. Note: requires IMS cache with temperature/GHI overlapping sensor data dates
  • chatbot-ps-forecast – Implemented PhotosynthesisService.forecast_day_ahead(): FvCB-based projection over IMS hourly weather (PAR2Γ—GHI, TleafTair+2Β°C, VPD from T+RH). Returns hourly A profile + peak/stress summary
  • chatbot-energy-devices – Discovered Plant asset "Yeruham Vineyard" (ASSET, not DEVICE) with power (W) + production (Wh/5min) keys. Added ASSET_REGISTRY, get_asset_timeseries(), get_asset_latest() to TB client. Also registered 4 Tracker devices (501-509) with angle/mode telemetry
  • chatbot-energy-tool – EnergyService rewritten: past/today β†’ real TB Plant asset data (daily ~270-290 kWh for 48 kW system); future β†’ analytical estimate (persistence forecast Γ— system capacity). Hourly profiles in Israel local time
  • chatbot-energy-predict – EnergyService.predict() now routes: past/today β†’ get_daily_production() (real TB), future β†’ _predict_analytical() (IMS GHI Γ— 48 kW)
  • chatbot-prompt-update – Updated system prompt with all 15 tools (grouped: weather, vine sensors, photosynthesis, shading, energy, advisory, biology); added energy/prediction/irrigation/fertiliser keywords to fallback matcher
  • chatbot-integration-test – All 15 tools dispatch correctly; verified imports + instantiation (no circular imports); tested fallback keyword matching; summarise_dataframe() compresses >48-row results. Remaining data gaps: IMS cache lacks GHI/temperature (only wind_speed_ms), no date overlap between IMS and sensors for ML training

Phase 10: Reliable Advisor β€” Anti-Hallucination Guardrails

Deterministic guardrails to prevent the chatbot from hallucinating data, violating biology rules, or answering data questions without tool grounding.

Step 1: Structured Response Schema

  • structured-response – Extended ChatResponse with confidence, sources, caveats, rule_violations fields; all responses carry grounding metadata

Step 2: Mandatory Tool Grounding

  • query-classifier – classify_query() in src/chatbot/guardrails.py: regex-based classification (data / knowledge / greeting / ambiguous); data queries force tool calls
  • tool-enforcement – If LLM answers a data query without calling a tool, chatbot re-prompts: "You MUST call a tool"

Step 3: Post-Response Rule Validator

  • response-validator – validate_response() in src/chatbot/guardrails.py: deterministic checks for no_shade_before_10 (block), no_shade_in_may (block), temperature_transition (warn), no_leaves_no_shade_problem (warn), no_shading_must_explain (warn)
  • validator-integration – Validator runs on every response in chat(); block-severity overrides response text; warn-severity appends caveats

Step 4: Source-Tagged Context Injection

  • source-tagging – tag_tool_result() adds _source, _data_age_minutes, _freshness_warning to tool results before Gemini Pass 2
  • citation-prompt – Pass 2 prompt instructs Gemini to cite source and timestamp when quoting numbers

Step 5: RAG Knowledge Base

  • rag-knowledge-base – Keyword-indexed rule retrieval with pinned rules (no_shade_before_10, energy_budget, temperature_transition) + top-scoring rules per query; retrieve_relevant_rules() + build_contextual_prompt() in vineyard_chatbot.py

Step 6: Conversation Memory

  • sliding-context – Summarize older messages into topics, keep recent 6 verbatim, pin critical context; _summarize_history() in vineyard_chatbot.py

Step 7: Confidence-Gated Responses

  • confidence-estimator – estimate_confidence() maps data freshness to high/medium/low/insufficient_data
  • confidence-ui – _render_grounding_metadata() in ui/tab_advisor.py shows colored confidence badge, sources, caveats, rule warnings

Step 8: Guardrail Test Suite

  • qa-tests-5 – 5 adversarial QA tests in tests/test_advisor_chatbot.py: shade-before-10 blocked, data-question forces tool, stale-data warns, no-shade-without-reason warns, tool-failure returns insufficient_data
  • guardrail-unit-tests – Unit tests for QueryClassifier, ResponseValidator, ConfidenceEstimation, SourceTagging (42 tests total, all passing)
  • qa-tests-expand – Expanded to 21 adversarial prompts (QA1-QA21) covering May shading, dormancy, night queries, prompt injection, boundary conditions, multi-rule violations, energy/irrigation classification, ambiguous queries. 58 tests total, all passing

Step 9: Dual-Channel Advisory

  • dual-channel – classify_response_mode() separates info mode (blue, factual) from advisory mode (amber, recommendations); response_mode field on ChatResponse; UI renders mode indicator

Step 10: Feedback Loop

  • feedback-ui – Thumbs up/down/flag buttons on each response in ui/tab_advisor.py; _render_feedback_buttons() + _submit_feedback()
  • feedback-storage – src/chatbot/feedback.py logs feedback to JSONL (query, response, tool_calls, rules, confidence, comment)

Phase 11: Production Migration

Full architecture documented in context/4_production.md. 3-layer architecture: Lovable/React frontend (Cloudflare Pages) β†’ FastAPI middleware (HuggingFace Spaces) β†’ existing Python core. Cron workers via GitHub Actions (free for private repos).

Phase 11.1: Redis Caching Layer

Replace in-memory TTLCache with Upstash Redis so API and workers share state.

  • upstash-setup – Upstash Redis instance created (many-mammal-76248.upstash.io), credentials in .env, connectivity verified with dual-layer TTLCache round-trip
  • redis-client – Create src/data/redis_cache.py: thin wrapper around Upstash Redis REST API with get_json(), set_json(ttl), delete(), exists(), ping(). Singleton via get_redis(), returns None when not configured
  • datahub-redis – Refactored TTLCache in data_providers.py: tries Redis first (via redis_prefix), falls back to in-memory. WeatherService prefix weather:, VineSensorService prefix vine:. Streamlit works unchanged (no Redis = pure in-memory)
  • tick-result-redis – backend/workers/control_tick.py writes TickResult to Redis control:last_tick (20-min TTL) after each tick
  • plan-redis – backend/workers/daily_planner.py writes plan to Redis control:plan (24h TTL) + backup Data/daily_plan.json
  • budget-redis – ControlLoop _ensure_daily_budget() restores from Redis control:budget on startup; _persist_budget() saves after every spend_slot(). Falls back to recompute if Redis unavailable

Phase 11.2: FastAPI Middleware (HuggingFace Spaces)

Create the API gateway in backend/api/, deploy as HF Space (Docker SDK).

  • fastapi-scaffold – Created backend/api/main.py with FastAPI app, CORS (configurable via ALLOWED_ORIGINS), lifespan, all route includes
  • fastapi-deps – Created backend/api/deps.py with get_datahub() (lru_cache singleton) and get_redis_client()
  • route-weather – backend/api/routes/weather.py: GET /api/weather/current, /api/weather/history?start_date&end_date, /api/weather/forecast
  • route-sensors – backend/api/routes/sensors.py: GET /api/sensors/snapshot, /api/sensors/history?type&area&hours
  • route-energy – backend/api/routes/energy.py: GET /api/energy/current, /api/energy/daily/{date}, /api/energy/history, /api/energy/predict/{date}
  • route-photosynthesis – backend/api/routes/photosynthesis.py: GET /api/photosynthesis/current?model, /api/photosynthesis/forecast
  • route-control – backend/api/routes/control.py: GET /api/control/status, /api/control/plan, /api/control/budget, /api/control/trackers
  • route-chatbot – backend/api/routes/chatbot.py: POST /api/chatbot/message, POST /api/chatbot/feedback with Pydantic request models
  • route-biology – backend/api/routes/biology.py: GET /api/biology/phenology, /api/biology/rules, /api/biology/rules/{name}
  • rate-limiting – Added slowapi: 60 req/min global default, 10 req/min on /api/chatbot/message
  • fastapi-dockerfile – Created backend/Dockerfile (Python 3.12 slim, port 7860, copies src/config/backend/Data)
  • hf-space-readme – Created backend/HF_README.md (sdk: docker, app_port: 7860, private: true, endpoint summary)
  • fastapi-test – Smoke tested: /api/health returns OK (uptime + redis status), /api/weather/current returns live IMS data

Phase 11.3: Background Workers (GitHub Actions)

Extract ControlLoop and DayAheadPlanner into standalone entry points, run as scheduled GitHub Actions.

  • worker-control – Created backend/workers/control_tick.py: loads ControlLoop, runs single tick, writes TickResult to Redis, supports --dry-run
  • worker-planner – Created backend/workers/daily_planner.py: runs DayAheadPlanner, writes plan to Redis + JSON backup
  • gh-action-tick – Created .github/workflows/control-tick.yml: schedule */15 * * * *, Python 3.12 + pip cache, all secrets
  • gh-action-planner – Created .github/workflows/daily-planner.yml: schedule 0 2 * * *, Python 3.12 + pip cache, all secrets
  • gh-secrets – All 7 secrets added to GitHub repo Settings β†’ Actions secrets
  • worker-logging – Structured JSON logging (asctime + level + name + message) in both workers
  • worker-test – Dry-run tested: python -m backend.workers.control_tick --dry-run loads plan (43 slots), computes angle (-10.5Β°), dispatches to 4 trackers (DRY RUN), logs tick result

Phase 11.4: Lovable Frontend (React + Cloudflare Pages)

Generate and customize the production frontend via Lovable.

  • lovable-project – Created via Lovable, exported to solarwine-ai/solarvine-dashboard repo
  • lovable-dashboard – Dashboard page: weather card, energy (yesterday/today/tomorrow), phenology, quick stats
  • lovable-advisor – Advisor page: chat panel with bubbles, advisory cards, confidence/channel badges, feedback
  • lovable-control – Control page: tracker bars, energy budget, last tick, plan timeline
  • lovable-photosynthesis – Photosynthesis page: FvCB vs ML, forecast chart, sensor snapshot
  • lovable-shading – Shading page: offset slider, 3-zone canopy, stress heatmap
  • lovable-docs – Docs page: biology rules accordion, architecture cards, API reference (18 endpoints)
  • lovable-export – Imported into frontend/ directory in main repo
  • api-hooks – 7 hooks (weather, energy, sensors/PS, control, chatbot, auth) + api.ts + types/api.ts
  • api-integration – All 6 pages wired to real API: mock data replaced with live TanStack Query hooks, field names mapped to backend snake_case responses
  • chatbot-component – Real mutations (useSendMessage/useSendFeedback), confidence badges, source citations, caveats, dual-channel (INFO/ADVISORY), auto-scroll
  • responsive-polish – Sidebar hamburger already implemented (lg breakpoint). Fixed: Power chart responsive height, Shading heatmap mobile min-width, Home grid breakpoints, camera/map sizing

Phase 11.5: Deployment & Integration

Wire everything together and deploy.

  • frontend-hosting – Switched from Vercel (requires Pro for private org repos) to Cloudflare Pages (free, unlimited bandwidth, private org repos supported). Connect GitHub repo β†’ root frontend/, build npm run build, output dist, set VITE_API_URL env var
  • hf-deploy – HF Space SolarWine/api deployed (Docker SDK, public), secrets injected via huggingface_hub, /api/health returns OK with Redis connected. 14/17 endpoints passing; 3 fixed (phenology missing arg, photosynthesis missing method, sensors null area)
  • upstash-connect – Verify Redis connectivity from HF Space and GitHub Actions, test cache read/write round-trip
  • e2e-smoke – Created scripts/smoke_test.py: tests 12 endpoints (health, weather, sensors, energy, PS, control, biology, chatbot). Run: python scripts/smoke_test.py [url]
  • streamlit-deprecate – Added migration banner to app.py; shows only when SOLARWINE_FRONTEND_URL env var is set (invisible until new frontend is live)
  • dns-custom – (Optional) Configure custom domain on Cloudflare Pages if available

Phase 11.6: Security & Monitoring (Post-Launch)

  • auth-jwt – Created backend/api/auth.py (create_token, require_auth, optional_auth) + backend/api/routes/login.py (POST /api/auth/login). Auth returns role: "guest" when JWT_SECRET not set (read-only). PyJWT added to backend/requirements.txt
  • api-hardening – Standardized error handling (HTTPException everywhere, no JSONResponse for errors), data provider error dict detection, startup path validation, Sentry fail-loud, chatbot session_id validation + blank message guard + init failure circuit breaker, Redis singleton retry-storm fix, CORS defaults include HF Space + Cloudflare Pages URLs
  • auth-frontend – Created useAuth.ts hook (login/logout/token restore from localStorage) + updated api.ts to auto-attach JWT Bearer header + auto-clear on 401
  • monitoring-health – UptimeRobot or similar free monitor on HF Space /api/health (5-min interval, email alerts)
  • monitoring-logs – Structured logging (asctime/level/name/message) in main.py + request logging middleware (method, path, status, duration). Health endpoint extended with ThingsBoard/IMS/Gemini connectivity checks
  • error-tracking – Added Sentry integration to backend/api/main.py (optional, enabled via SENTRY_DSN env var). sentry-sdk[fastapi]>=2.0 in backend/requirements.txt
  • budget-alerts – control_tick.py checks budget after each tick; logs warning if >80% spent before 14:00 IST; sends webhook to BUDGET_ALERT_WEBHOOK if configured

Phase 12: Robustness & Efficiency

12.1: Immediate (Week 1)

  • tb-health-fix – Fixed ThingsBoard health check: /api/noauth/health returns 404, switched to root URL. Needs HF Space redeploy
  • hf-redeploy – Redeployed HF Space via huggingface_hub API (git push blocked by binary file policy). Uploaded 8 files, space auto-rebuilt
  • redis-verify – Smoke test against live HF URL: 11/12 pass (control plan 404 expected β€” no cron run yet). Redis connected, IMS/Gemini configured
  • weather-fix – Fixed WeatherService crash: Redis deserialised DataFrame as dict/list β†’ .empty AttributeError. Added type guard + skip DataFrame Redis serialisation
  • uptime-monitor – UptimeRobot configured on /api/health (5-min interval). Added HEAD method support to health endpoint for monitoring compatibility
  • monitoring-page – Redesigned Alarms page β†’ Monitoring & Alarms: overall status banner, 3 data source cards (IMS/TB/Energy), backend services panel (Redis/TB/IMS/Gemini), data flow issues surface as system alarms. Sidebar renamed Alarms β†’ Monitoring
  • code-review-fixes – Fixed email alerter string formatting bug (CRITICAL), duplicate className in Home.tsx, DataFlowStatus null safety, energy price consistency (0.162 USD/kWh), table accessibility (scope attrs), negative age logging
  • data-flow-monitor – New /api/health/data-sources endpoint returning per-source green/yellow/red status (IMS weather, TB sensors, TB energy) with age tracking and configurable thresholds. DataFlowMonitor in backend/services/data_flow_monitor.py. Thresholds in config/settings.py: IMS yellow 60m/red 180m, TB yellow 15m/red 60m
  • data-flow-alerts – EmailAlerter in backend/services/email_alerter.py: SMTP email alerts when sources go red, per-source cooldown (60 min). Activated by SMTP_HOST + ALERT_EMAIL_TO env vars. Background loop in main.py checks every 5 min, pushes SSE health event
  • data-flow-frontend – DataFlowStatus component on Home page: colored dots (green/yellow/red) for each source, hover tooltips, auto-refresh via SSE. Hook: useDataFlowStatus, types: DataSourceStatus / DataFlowStatusResponse
  • chatbot-range-validation – validate_numeric_ranges() in guardrails.py: checks tool results against physical bounds (temp, GHI, PAR, VPD, CO2, power, etc.), flags sensor faults as caveats
  • chatbot-cross-source – check_cross_source_consistency() in guardrails.py: compares IMS vs TB temperature readings, warns when they diverge by >5Β°C
  • chatbot-rag-scoring – Improved retrieve_relevant_rules(): weighted scoring (exact match +2, partial word overlap +0.5), increased max_rules to 6

12.2: Short-term (Week 2–4)

  • responsive-polish – Sidebar hamburger already implemented (lg breakpoint). Fixed: Power chart responsive height (280/350/450px), Shading heatmap mobile min-width (400px), Home grid md breakpoint for map/camera/power row, camera/map responsive min-heights
  • load-test – scripts/load_test.py (Locust): 16 weighted endpoints, 10 users Γ— 1 min = 157 requests, p50=250ms, p95=1700ms, 0 errors. All endpoints healthy except /api/health/data-sources (not yet deployed at test time)
  • circuit-breaker – Added CircuitBreaker class (3 failures in 60s β†’ open for 5 min) to VineSensorService and EnergyService. Prevents retry storms when TB is down
  • tracker-timeout – Added exponential backoff retry (3 attempts: 1s, 2s) for TB shared attribute writes in TrackerDispatcher

12.2b: Code quality improvements (done 2026-03-29)

  • code-review-critical – Fixed: Redis health try-catch, chatbot init race condition (lock before null check), background task try-finally, daily planner IST timezone, JWT login parse safety, error boundary on Outlet
  • error-standardization – Extracted check_service_error() to backend/api/utils.py, used by energy + sensors routes. Fixed energy route ignoring errors when daily_kwh present
  • control-auth – Added optional_auth to all /control/* endpoints (status, plan, budget)
  • farquhar-sigmoid – Replaced hard 30Β°C RuBPβ†’Rubisco cutoff with smooth sigmoid (28–32Β°C). New "Transition" state. Prevents flip-flopping near threshold
  • divergence-fix – Fixed inverted logic in control_loop.py:704 divergence check
  • frontend-extract – Extracted StatCard to /components/StatCard.tsx, date formatters to /lib/dateUtils.ts, energy price to /config/constants.ts. Home.tsx reduced by ~50 lines
  • strict-types – Added TypeScript interfaces: ChillUnitResponse, SoilMoistureRow, RainRow, TrackerInfo, ControlTrackers, ControlBudget. Replaced inline any types
  • chart-memoize – useMemo on tempData chart transformation in Home.tsx

12.2c: Architecture refactors (done 2026-03-29)

  • vectorize-farquhar – compute_all() rewritten with vectorized pandas/numpy (no iterrows). 10K rows: 7.5ms (was ~750ms). Numerically identical output (max diff=0)
  • control-loop-di – ControlLoop.__init__() accepts all 8 dependencies as keyword args. Pass None (default) for lazy auto-create; pass mocks for testing
  • gate-pipeline – InterventionGate.evaluate() split into 5 methods (_check_meaningful_sun, _check_heat_stress, _check_water_stress, _check_radiation_load, _check_biology) composed as pipeline. First rejection short-circuits
  • budget-audit – src/budget_audit.py: BudgetAuditLog appends one parquet row per control tick. daily_summary(date) and weekly_report() for compliance verification. Wired into control_loop.py tick step 12. CLI: python -m src.budget_audit --report
  • chatbot-hebrew – Added 18 Hebrew keywords to query classifier. System prompt: language matching, phenology-first, conciseness. IMS stale β†’ auto-supplement with TB sensors. Confidence rule override for dormancy/blocks
  • refactor-tests – 23 new tests in tests/test_refactors.py: vectorized vs scalar (4), sigmoid transition (2), gate pipeline (6), ControlLoop DI (3), budget audit (3), Hebrew classification (3), confidence override (2)

12.2e: Data + energy-model refresh β€” done 2026-05-18

  • ims-refresh – Pulled IMS station 43 from 2025-12-01 β†’ 2026-05-10 (10,748 rows, 15-min). API has ~8-day lag; data ends 2026-05-10.
  • scripts-refresh-energy-data – New scripts/refresh_energy_data.py: joins TB Plant production (1h SUM agg) with IMS hourly weather, reindexes to contiguous hours (nighttime = 0 kWh), adds time + solar-geometry features, writes Data/energy_weather_merged.csv (3,477 rows Γ— 18 cols).
  • scripts-train-energy – New scripts/train_energy_predictor.py: temporal hold-out (last 14 days), sklearn GradientBoostingRegressor + LinearRegression fallback (xgboost requires libomp on macOS; not installed). Bundle slot named xgb_model for backward compat with EnergyPredictor.__init__.
  • energy-backtest-fix – EnergyPredictor.backtest() now reads weather from IMSClient instead of the dead Air1 device.
  • energy-model-validation – 1,343 train rows / 337 hold-out rows. Hourly MAE = 4.5 kWh. Daily mean signed error = +50% β€” model systematically over-predicts in late April/May. Cause: model has no tracker_angle feature, so it can't see the agrivoltaic interventions that actively detune trackers as growing season starts. See 12.5 below for the planned fix.
  • energy-tracker-feature-attempted – Wired tracker_angle_mean + tracker_angle_std + abs_tracker_angle from TB into the refresh pipeline (scripts/refresh_energy_data.py:fetch_tb_tracker_angles). Two variants tried: (a) with imputation of pre-Feb history via hour-of-day medians β†’ bias worsened to +57%; (b) without imputation + dropped sin_elevation β†’ bias worsened to +75%. Root cause is distribution shift, not features: training tracker_angle avg = -15Β° (mild), hold-out avg = -51Β° (aggressive interventions). The model can't extrapolate operational regimes it didn't see in training. Reverted features to the original best config. Tracker telemetry is still collected into the CSV (columns tracker_angle_mean/_std) β€” ready to be consumed once 2026 growing-season data accumulates. See 12.6 / bigml-tracker-feature for the proper fix path.

12.2d: 2026 sensor migration β€” done 2026-05-18

  • tb-2026-registry – Clean-replaced DEVICE_REGISTRY in src/data/thingsboard_client.py with the 2026 fleet (35 entries: 19 Crop_2Soil + 12 Thermocouples + 4 Trackers). Legacy Air1-4, Crop1-7, Soil*, Irrigation1, Thermocouples1-2 removed. Added DeviceInfo.position field (north/south/center/etc.). Convenience subsets TREATMENT_DEVICES/REFERENCE_DEVICES/CROP_2SOIL_DEVICES/THERMOCOUPLE_DEVICES derived at import. Treatment = rows 501/502/503/504/509; Reference = row 202.
  • vinesnapshot-2026 – Rewrote VineSnapshot with 61 fields. Removed PAR/DLI/VPD/CO2/Irrigation (no 2026 sensor); added NDVI/PRI/PSRI/SIPI/GCI/LCI/DUVI ratios, dual soil profile (shallow + deep moisture/temp/pore-water EC), panel_temp_active_count. Legacy field names kept as None-valued compat shims so downstream consumers don't crash. Per-position keys use {row}-{cardinal} form (e.g. "502-south-east").
  • vinesnapshot-aggregation – get_vine_snapshot() no longer hard-codes device names; iterates DEVICE_REGISTRY filtered by area. Full fleet fetched in 1.1 s via ThreadPoolExecutor.
  • bounds-fixes – Added physical bounds for PSRI/SIPI/GCI/LCI/DUVI/leaf-wetness in _BOUNDS. Critical fix: GCI was being clipped to [-1, 1] (real range 0..30), causing all daytime values to be discarded. Split treatment_soil_pore_water_ec into shallow + _deep. De-duped the treatment_pri computation.

12.3: Medium-term (Month 1–2)

  • chatbot-feedback – Review first 20 thumbs-down/flag submissions, identify top 3 guardrail failures, tune rules
  • model-versioning – Create model_registry/ to track Chronos/ML/Farquhar parameter versions, enable A/B testing
  • phenology-validation – Cross-reference GDD-based stage estimates vs camera observations, refine thresholds for Sde Boker

12.5: 2026 sensor follow-ups (queued after the foundational registry rewrite)

Bugs and design debt surfaced during the May 2026 sensor migration. Foundational work landed in 12.2d above; these are the deferred items.

Remaining bugs

  • panel-tc-semantics – panel_temp_active_count is per-device but the panel-temp avg is per-channel β€” inconsistent. Pick one. (src/data/thingsboard_client.py:get_vine_snapshot)
  • tab-system-status-2026 – ui/tab_system_status.py _render_fruiting_zone rewritten to use NDVI / PRI / leaf-air Ξ”T / PSRI; risk assessment now triggers on Ξ”T > 3Β°C and PSRI > 0.2 in addition to canopy temp. Done 2026-05-18.
  • frontend-types-2026 – frontend/src/types/api.ts and the React hooks (useSensors, etc.) still declare treatment_par_umol: number | null and similar. After the migration these are always null β€” type narrows to null. Audit frontend/src/lib/ + frontend/src/hooks/ and remove or mark deprecated.

Architectural concerns (numbered to match the post-migration code review)

  • 2026-A-auto-sync-registry – Hardcoded DEVICE_REGISTRY is brittle: each TB change requires a code edit + redeploy. Build scripts/sync_devices.py that reads the Crop 2Soil 2026 + Thermocouples 2026 entity groups, writes Data/devices.json, and have thingsboard_client.py prefer that file with the hardcoded dict as offline fallback. Position decoding belongs in this sync script (regex over the Hebrew TB labels: Χ¦Χ€Χ•ΧŸβ†’north, דרום→south, ΧžΧ–Χ¨Χ—β†’east, ΧžΧ’Χ¨Χ‘β†’west, ΧžΧ¨Χ›Χ–β†’center).
  • 2026-B-vinesnapshot-nest – VineSnapshot is 61 flat fields and growing. Nest as snapshot.treatment.{air,canopy,soil_shallow,soil_deep} and snapshot.panel, expose flat to_dict() only at the JSON boundary. Defer until you're already breaking the API for another reason.
  • 2026-C-schema-version – Compat shims (treatment_par_umol=None etc.) silently say "sensor down" when the truth is "this metric no longer exists." Add schema_version: "2026.1" to the snapshot and let consumers branch / error explicitly; remove the shims one release later.
  • 2026-D-split-fetch-aggregate – get_vine_snapshot() mixes ~80 lines of I/O with ~70 lines of transformation. Split into _fetch_all_latest(plan) β†’ raw_dict + _aggregate_snapshot(raw, now) β†’ VineSnapshot. Then unit-test the aggregate step with fixture dicts (currently no test covers it).
  • 2026-E-position-source-of-truth – Position labels were decoded by hand from Hebrew TB labels β€” not reproducible. After 2026-A lands, decode at sync time. Until then, store the original TB label as DeviceInfo.tb_label_hebrew so the human-readable source survives.
  • 2026-F-per-device-freshness – DataFlowMonitor checks aggregate "newest TB timestamp" β€” one fast-reporting device makes the fleet look healthy even if 11 thermocouples are silent. Move to per-device staleness with grace per device-type (Crop_2Soil ~15min, Thermocouples up to ~60min due to deep-sleep cycles).
  • 2026-G-layout-regen – scripts/import_layout.py updated to handle 2026 rows (202 reference; 501/502/503/504/509 treatment) and the DeviceInfo.position field via _POSITION_OFFSETS. Data/layout.json regenerated: 6 rows, 35 devices, each with position_m: [x, y, z]. Done 2026-05-18.
  • 2026-H-position-taxonomy – Position strings ("south-east", "center-east", …) are free-form. Convert to a CardinalPosition enum in src/data/data_schema.py (or document the canonical form in CLAUDE.md) so future labels match.
  • 2026-I-aggregation-tests – No unit test covers the new aggregation logic. Add at least 3 fixture-based tests in tests/test_thingsboard_client.py: full fleet, 1 device per area, all-empty. Depends on 2026-D.

Follow-on integration work

  • 2026-J-ims-ambient-bridge – Wire IMS Sde Boker station 43 β†’ VineSnapshot.ambient_* fields so they stop returning None. Currently Alarms.tsx and Home.tsx fall through to treatment_air_temp_c, which is misleading (it's not ambient).
  • 2026-K-chatbot-spectral-tools – Add chatbot tools get_vegetation_indices() and get_canopy_health() exposing the new 2026-only data (NDVI/PRI/PSRI/SIPI per row + position).
  • 2026-L-spectrometer-page – New frontend page (or extend Agro.tsx): vegetation-index time series, NDVI/SIPI heatmap across the 17 active Crop_2Soil devices, panel surface-temp 4Γ—12 grid from the thermocouples.

12.6: "Big" β€” Stage 2 photosynthesis ML retrain on 2026 schema

Plan, not yet executed. The Stage 2 ML predictor (src/forecasting/predictor.py) forecasts A_n (photosynthesis rate) from weather features. Currently trained on Data/Seymour/sensors_wide.csv (frozen at 2026-02-01) β€” features include PAR, airTemperature, CO2, VPD, leafTemperature from the legacy Air1/Crop fleet. The 2026 sensor refresh deprecated all of those sensors. The new fleet emits NDVI/PRI/PSRI/spectrometer bands + dual-soil profile + IRT ambient β€” a fundamentally different feature space. This is a redesign, not a refresh.

Phase 1 β€” Data collection pipeline (2-3 days work)

  • bigml-collection – scripts/collect_2026_training_data.py written and smoke-tested 2026-05-18. Pulls 15-min telemetry from the 10 treatment + 7 reference Crop_2Soil devices via TB, averages per area per timestep, inner-joins with IMS 15-min weather, writes to Data/2026/sensor_history.parquet. Append-mode dedupes on (timestamp, area). May 1–10 sample: 1,896 rows (948 per area).
  • bigml-stage1-2026 – Farquhar inputs derived from the 2026 fleet: PAR β‰ˆ 2.0 Γ— GHI (Akitsu et al. 2017, Β±10 %), VPD via Tetens from IMS RH + Tair, CO2 = 420 ppm assumed, CWSI = (Tleaf βˆ’ Tair)/15 clipped to [0, 1]. compute_farquhar_a() runs FarquharModel.calc_photosynthesis() row-by-row; result column a_farquhar_umol. Validation: daytime mean 13.1 Β΅mol COβ‚‚/mΒ²/s, peak 19.6 Β΅mol β€” squarely in Greer & Weedon's empirical Semillon range for the leaf-temp window seen.
  • bigml-collection-shading-correction – apply_shading_correction() added to scripts/collect_2026_training_data.py. For each treatment-area timestep: pulls tracker_angle_mean (or astronomical fallback when telemetry missing), runs ShadowModel.project_shadow() + compute_par_distribution(), takes the fruiting-zone mean PAR, and multiplies the IMS-derived par_umol_derived by that factor. Reference rows untouched. May 1–10 sample: treatment PAR factor distribution is 0.10–1.0 (median 0.82, mean 0.73) β€” panels meaningfully reduce fruiting-zone PAR. Farquhar A label is recomputed after the correction. Done 2026-05-18.
  • bigml-baseline – Until enough 2026 data accumulates (β‰₯3 months growing-season hours), run Farquhar-only as the photosynthesis estimate (skip Stage 2 ML for 2026). The chatbot's photosynthesis tool already supports this via PhotosynthesisService.compute_fvcb().
  • bigml-collection-cron – Schedule scripts.collect_2026_training_data to run weekly via GitHub Actions (similar pattern to control-tick.yml / daily-planner.yml). Window = last 8 days (overlaps to be safe against IMS's ~8-day publication lag).

Phase 2 β€” Feature engineering (1-2 days)

  • bigml-features – Define the 2026 ML feature set. Candidates: IMS GHI/Tair/RH/wind (forecastable), Crop_2Soil NDVI/PRI/PSRI/SIPI (now), soilMoisture (shallow + deep), leafTemperature, tracker angle (current), hour, doy_sin/cos. Decide whether to keep the per-device dimension (input = 17 device readings per timestep) or aggregate to area-level (~5 features).
  • bigml-spectral-pca – With 17 spectrometer bands per device Γ— 17 devices = 289 columns, dimensionality reduction is needed. Run PCA on the spectral subset and keep the top 5-8 components.

Phase 3 β€” Training + validation (1 day)

  • bigml-train – Train RF / GBR / Linear regressors (analogous to Stage 2 v1). Use temporal hold-out from the latest 14 days of the collected 2026 data. Compare MAE vs Farquhar-only baseline.
  • bigml-tracker-feature – Add tracker_angle and intervention_active as features to the energy predictor too (fixes the +50% over-prediction bias seen in 12.2e). Requires pulling tracker telemetry alongside production into the energy refresh script.
  • bigml-deploy – Wire the trained model into PhotosynthesisService.predict_ml(), replacing the current Stage 2 v1 model. Update model_versioning in model_registry/ (Phase 12.3 β€” model-versioning).

Dependencies / open questions

  • Need β‰₯3 months of 2026 growing-season data before Phase 3 is statistically meaningful. With data collection starting May 2026, retraining is realistic for August–September 2026.
  • CO2 atmospheric assumption (420 ppm) is conservative but the lack of a real sensor is a real degradation vs the legacy fleet. Consider sourcing hourly CO2 from a nearby Mauna Loa proxy if precision matters.
  • Tracker-angle feature needs ThingsBoard telemetry alignment β€” trackers emit angle as shared-attribute, not telemetry by default. Verify the control loop is writing telemetry to TB.

12.7: Frontend redesign β€” Claude Design handoff (done 2026-05-18)

  • redesign-foundation β€” Replaced :root tokens with Negev sand + vine green + sun gold; loaded Source Serif 4 + IBM Plex Sans + IBM Plex Mono; added brand stripe motif + masked-PNG wordmark; re-targeted shadcn aliases. tailwind.config.ts extended with serif/mono families and brand color tokens.
  • redesign-layout β€” Rewrote AppSidebar to a 240 px sand sidebar with 4-section nav (Vineyard / Energy / Operations / Knowledge), masked logo, brand stripes, vine left-bar on active. New Topbar.tsx (breadcrumb Β· ⌘K search Β· refresh Β· bell Β· avatar). New PageHeader.tsx (eyebrow + serif H1 with optional italic emphasis + deck + 3-row meta panel). AppLayout adds Topbar + footer hairline strip.
  • redesign-shared-components β€” New Sparkline, CardHead, DataFlowRail, TrackerArc, PlanTimeline (96-cell ribbon), StressHeatmap. StatCard rewritten with eyebrow + delta + serif tabular number + caption + Sparkline. New chart wrappers charts/SwAreaChart.tsx and charts/SwLineChart.tsx with shared SolarWine defaults (hairline grid, mono ticks, brand palette, dashed reference lines).
  • redesign-home β€” Editorial header, data-flow rail, 4-stat row, 96-slot plan ribbon + power chart, tracker fleet with arc dials + status pills, FvCB-vs-ML photosynthesis card, stress heatmap, dark Advisor card with sun-gold "Apply at next tick" CTA, 7-day climate w/ 32Β°C dashed line, chill-accumulation bars.
  • redesign-pages-all β€” All 10 remaining pages rewritten:
    • Agronomy β€” 4-stat row + A1–A4 block table w/ vigor bars + soil profile + agronomy log + water budget
    • Photosynthesis β€” model-comparison big numbers + Chronos forecast + sensor explorer
    • Shading β€” 64 px serif offset slider w/ vineβ†’sun gradient, 3-zone canopy diagram, stressΓ—offset map, 8-rule policy grid
    • Power β€” 4-cell KPI strip + today-vs-yesterday-vs-astro line, inverter cards, weekly bars, revenue ledger
    • Trackers β€” fleet table w/ arc dials + status pills + 24h sparklines, dispatch log, reliability fields
    • Control loop β€” 5-stage pipeline with active highlight, L0..L4 priority stack, hierarchical energy budget, plan ribbon
    • Advisor β€” chat surface w/ confidence pills + citations + feedback, dark day-ahead brief, active recommendations, quick-asks
    • Monitoring (was Alarms) β€” alert stream w/ severity pills + ack, 6-source health column
    • Documentation β€” numbered serif accordion (10 biology rules), API ref table (21 endpoints), 12-term glossary
    • Research (new) β€” vineyard-photo hero band, 3 paper cards, datasets table, 7-person crew strip. /research route added.
  • redesign-build β€” TypeScript clean, vite build 2.5 s, 240 KB gzipped JS. Dev server at http://localhost:8080/.

TODO markers inline (would benefit from new backend endpoints)

  • redesign-stress-grid β€” /api/photosynthesis/stress-grid?date=... for live 4Γ—24 CWSI grid (used by Home + Shading; currently synthetic).
  • redesign-inverters β€” /api/energy/inverters for per-inverter live state on Power.
  • redesign-dispatch-stats β€” /api/control/dispatches summary for p95 ack latency on Trackers.
  • redesign-events-dispatch β€” surface dispatch + alert events through /api/events/* so the Trackers log + Monitoring stream are live, not seeded.
  • redesign-frontend-types-purge β€” frontend/src/types/api.ts and hooks still declare treatment_par_umol: number | null etc. After the 2026 fleet migration these are always null; trim or mark deprecated.

12.8: Streamlit staleness audit (done 2026-05-18)

  • streamlit-control-replay-freshness β€” ui/tab_system_status.py:_render_control_replay: empty-state message now uses today's date; loaded log shows a freshness warning when older than 30 days; caption surfaces the simulated date range (sim_2026-05-01 β†’ 2026-05-10).
  • scripts-sim-defaults β€” scripts/run_control_simulation.py defaults changed from 2025-07-01 / 2025-07-07 to today βˆ’ 7 d / today so future runs are fresh.
  • fresh-sim-log β€” Generated Data/simulation_logs/sim_2026-05-01_2026-05-10.json (960 ticks). Required installing astral into the venv (was missing; fixed 960/960 errors on the first attempt).
  • streamlit-shading-caption β€” Added "historical analysis Β· sensors_wide.csv frozen at Feb 2026" caption to the Shading tab so the date picker isn't read as live.
  • camera-caption β€” Camera image caption now says "if the view shows only sky, the on-site camera needs realignment". Physical site issue; can't be fixed from code.

12.4: Long-term (3+ months)

  • multi-vineyard – Refactor config/settings.py to multi-tenant pattern (vineyard ID β†’ settings), add Postgres (Neon free tier)
  • licor-calibration – Ingest first real LiCor measurements, retrain Farquhar Vcmax/Jmax via src/calibration_pipeline.py
  • chronos-vs-ml – 3-month live comparison of Chronos-2 vs ML ensemble, promote winner to primary predictor

Session notes

2026-02-27 β€” Phase 8B: Gemini LLM Data Engineering

  • Created src/llm_data_engineer.py with LLMDataEngineer class.
  • analyze_anomalies(): constructs a structured prompt from per-column descriptive stats + site-specific domain context (physical ranges, Negev notes) β†’ Gemini returns a JSON with lower_bound, upper_bound, zscore_threshold, iqr_multiplier, rationale per column; falls back to physics-based bounds when API unavailable.
  • apply_cleaning(): applies union of bound violations + Z-score + IQR anomaly flags; three strategies: clip (default), nan, drop. Detected 14 PAR anomalies (0.05%), 9 VPD anomalies (0.03%), 168 COβ‚‚ anomalies (0.58%) on sensors_wide_sample.csv.
  • engineer_features(): adds hour_sin, hour_cos, doy_sin, doy_cos (cyclical encodings); stress_risk_score = vpd_weight Γ— norm(VPD) + cwsi_weight Γ— norm(CWSI) in [0, 1]; weights sourced from Gemini (default: 0.6/0.4, falls back gracefully without CWSI).
  • run_pipeline(): single-call orchestration of anomaly detection β†’ cleaning β†’ feature engineering.
  • Added google-genai>=1.0 to requirements.txt.

2026-02-27 β€” Dashboard & pvlib

  • Data Explorer: Intro text now lists all three data sources (Vineyard sensors, Weather station data, AI Data Engineering). Third option runs full LLM pipeline in-tab (thresholds, cleaning summary, before/after PAR/VPD, feature spec, stress profile); no cache loader, all precomputed on selection.
  • Presentation tab: β€œWhat’s Next” completed list includes Gemini LLM data engineering with pointer to Data Explorer β†’ AI Data Engineering. β€œWhat each tab shows” updated so Data Explorer bullet mentions AI Data Engineering. Key takeaways add Gemini data engineering (anomaly cleaning + stress/time features).
  • pvlib: src/solar_geometry.py β€” compute_tracker_tilt() now calls pvlib.tracking.singleaxis() with first two args positional (zenith, azimuth) so it works with both pvlib < 0.13.1 (apparent_azimuth) and β‰₯ 0.13.1 (solar_azimuth). Fixes β€œShadow model demo unavailable: singleaxis() got an unexpected keyword argument 'solar_azimuth'”.

2026-03-05 β€” ThingsBoard live telemetry integration (Step 1)

  • Rewrote src/thingsboard_client.py: full device registry (22 devices, UUIDs from devices.csv), area classification (treatment rows 501–502 / reference rows 503–504 / ambient Air1), parallel fetch with ThreadPoolExecutor, JWT auth with token refresh.
  • Added VineSnapshot dataclass: aggregated treatment vs reference readings β€” Air, Crop (per panel position: west-bottom/east-upper/east-bottom/west-upper), Soil, Irrigation, Thermocouples. Includes to_advisor_text() and to_dict().
  • Added _bounded_avg() with _BOUNDS dict: physical plausibility filter rejects sensor faults (e.g. Soil5 thermocouple reading 653Β°C). Soil, Air, PAR, NDVI, VPD, COβ‚‚ all filtered by domain bounds.
  • Wired into src/vineyard_chatbot.py: new _tool_get_vine_state() tool + get_vine_state in system prompt + dispatch router; _tool_day_ahead_advisory() now passes VineSnapshot to advisor.
  • Wired into src/day_ahead_advisor.py: optional vine_snapshot param in advise(); to_advisor_text() inserted before weather forecast in Gemini prompt.
  • TB credentials (TB_USERNAME, TB_PASSWORD) added to .env; config also accepts THINGSBOARD_USERNAME/THINGSBOARD_PASSWORD.
  • Tested live: all 22 devices fetched in 1.1 s, PAR shading ratio 0.33 (67% reduction), soil temp 15.5Β°C (outlier-filtered).
  • Synced Phase 9 checklist: all chatbot tasks marked done (implemented in a prior session, todo file not updated).

2026-03-09 β€” Phase 10: Anti-Hallucination Guardrails

  • Created src/chatbot/guardrails.py with 4 components: QueryClassifier (regex data/knowledge/greeting routing), ResponseValidator (deterministic rule checks with block/warn severity), estimate_confidence (data freshness β†’ confidence level), tag_tool_result (source metadata injection).
  • Rewrote src/chatbot/vineyard_chatbot.py: integrated all guardrails into chat() flow β€” query classification β†’ forced tool re-prompting β†’ source-tagged tool results β†’ response validation β†’ confidence estimation. Extended ChatResponse with confidence/sources/caveats/rule_violations. Strengthened system prompt: "NEVER invent sensor readings", "MUST call a tool for data questions", "cite source and timestamp".
  • Updated ui/tab_advisor.py: new _render_grounding_metadata() renders colored confidence badge, source citations, caveats, and rule violation warnings under each chat response. Metadata persisted in chat history.
  • Added 23 new tests to tests/test_advisor_chatbot.py: 5 adversarial QA scenarios (shade-before-10 blocked, data-question forces tool call, stale-data warns, no-shade-without-reason warns, tool-failure returns insufficient_data) + unit tests for all guardrail components. Total: 42 tests, all passing.
  • Updated context/2_plan.md with Phase 10 architecture, validation rules, confidence mapping, and remaining steps. Updated dependency order.

2026-02-17 β€” Code quality fixes

  • Fixed Preprocessor.merge_ims_with_labels position-based fallback: now raises ValueError instead of silently misaligning rows when timestamp_index_labels=False and labels have no named index.
  • Removed all ~50 deprecated use_container_width=True calls in app.py (Streamlit 1.54): st.image β†’ width='stretch'; st.plotly_chart / st.dataframe β†’ argument removed (default is already 'stretch').
  • Fixed follow-up SyntaxError: regex removal of use_container_width=True in the middle of argument lists left double commas (,,); collapsed all ,, β†’ ,.
  • Updated context/CODE_REVIEW.md sections 16–17 with applied fixes and remaining work list.
  • Updated "What's next" sections in app.py (tab_photosynthesis section 6 and About tab) to reflect Chronos-2 and Gemini routing as completed.