Spaces:
Running
Running
Commit History
-
Cache-bust CSS/JS on every container start
A browser cache miss between deploys served the old app.js with the new /api/bess shape, producing nonsense state-dropdown options (fetched_at, source, states, warnings). Render index.html at startup with the script and stylesheet URLs carrying a version query string derived from int(time.time()). Each container restart produces a fresh value, so the browser always refetches assets that paired with this build of the API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9f09117 -
Drop XLSX mirror layer to satisfy HuggingFace binary-file policy
The previous commit was rejected by HuggingFace ("push was rejected because it contains binary files. Please use Xet to store binary files. Offending files: app/data/aemo_gen_info.xlsx") so the sync_to_hf workflow failed and the Space stayed on the old code. The mirror layer (live -> raw.githubusercontent.com XLSX -> snapshot) was redundant: the daily CI commits the parsed JSON snapshot in the same step as the XLSX, so they always carried identical freshness. Collapsing to a two-tier fallback (live -> snapshot) keeps the user-visible behaviour identical while removing the binary file that HF rejects. - gen_info_fetcher.py: remove XLSX_MIRROR_URL and the mirror branch in fetch_bess_list. BessListResult.source is now "live" | "snapshot". - scripts/refresh_bess_list.py: stop writing app/data/aemo_gen_info.xlsx. Only bess_list.json and bess_list_meta.json are produced. - .github/workflows/refresh_bess_list.yml: only watch bess_list.json for changes. - app/data/aemo_gen_info.xlsx: r…4424ac7 -
Fix HF deployment serving stale BESS list with fictional DUIDs
The HuggingFace Space was returning a hand-written fallback bess_list.json on every /api/bess call because AEMO's WAF on www.aemo.com.au returns 403 to requests from the HF egress IP range (verified in the HF container logs — UA-spoofing alone doesn't help, the block is IP-based). The fallback had stale fictional DUIDs (NESBESS1, HORNSDALE_PWR1, etc.) and no capacity_mwh field, which caused every BESS to show "? MWh" in the dropdown and every SCADA/dispatch lookup to 404. Five fixes shipped together: 1. Send proper browser headers on the AEMO XLSX request (low-confidence shot — the block is IP-based, but no reason not to try). 2. Bake a fresh bess_list.json snapshot into the repo daily via a new GitHub Action (.github/workflows/refresh_bess_list.yml) running scripts/refresh_bess_list.py from GH runners (egress NOT on AEMO's blocklist). The bundled snapshot now has 38 real BESS units with real DUIDs and capacity_mwh values. 3. Mirror the raw XLSX through raw.githubuse…0d542b0
-
Fix bundle selection: try multiple candidates when best bundle misses
AEMO archive bundle dates don't always align with their contents — e.g. bundle PUBLIC_NEXT_DAY_FPPMW_20250727.zip starts at inner entry 20250731, so July 26 data (search string 20250727) isn't found there but IS in the previous bundle (20250630). Now returns up to 3 candidate bundles per format (sorted best-first) and tries each until one contains the target date. Also changed _fetch_fppmw_monthly_csv to return empty list instead of raising when no match is found, enabling graceful fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5a939d7 -
Fix OOM on Era 1 dates: chunk CSV output and pre-filter by DUID
Era 1 files are ~1 GB each (9.3M rows for all units). Loading two and parsing into DataFrames before filtering peaked at 6.8 GB RAM, causing 500 errors on HuggingFace Spaces. Two changes: - _fetch_fppmw_monthly_csv now returns list[bytes] (one per inner ZIP) instead of concatenating everything, so each chunk is processed and freed independently - _prefilter_bytes_by_duid scans raw bytes keeping only rows containing the target DUID before any DataFrame parsing, reducing 1 GB → ~2 MB Peak memory: 6.8 GB → 3.5 GB. Result unchanged (21,600 rows, 24h). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6250d00 -
Fix era-aware file matching for pre-Apr 2025 SCADA data and add warnings system
Era 1 (before Apr 29 2025) FPPMW files use the settlement date (D) in filenames, not the publication date (D+1). The previous code always searched with D+1, finding incorrect or no data for March/April 2025. Now searches with [D, D+1] for Era 1 to cover the full 04:00–04:00 trading day from midnight-to-midnight files. Also adds a warnings system: FetchResult dataclass carries user-facing warnings (e.g. Era 2 dates with only 12h of data), surfaced as amber boxes in the UI and X-Data-Warnings headers on downloads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c6457f4
-
Fix MW capacity column name to match AEMO xlsx header
ClaudeChange _COL_CAPACITY_MW from 'Unit Capacity (MW AC)' to 'Agg Nameplate Capacity (MW AC)' (column S, row 4) to correctly read the aggregated nameplate MW capacity from the AEMO Generation Information spreadsheet.6fd40d3 -
Replace inline expand with modal for data-type About panels
Claude- Remove click-to-expand inline panels that pushed page content down - Hover over ▶ About still shows a brief one-line tooltip (CSS only) - Click opens a centred modal with the full column definitions; closes via ✕ button, backdrop click, or Escape key - Modal content defined in JS (ABOUT map) to keep HTML clean https://claude.ai/code/session_01TrCrzZmWbQscJiAQaW7Rg843cb5cc -
Add hoverable/expandable About panels to data-type checkboxes
Claude- Remove sub-labels (Power, MW) and ⓘ info icon from checkbox rows - Add ▶ About toggle button next to each checkbox label; hover shows a compact tooltip, click expands an inline column-definition panel - Stack same-section result panels flush (card-group, no gap) while keeping clear spacing between top-level sections (Select Data, SCADA results, Energy results, About this app) https://claude.ai/code/session_01TrCrzZmWbQscJiAQaW7Rg8134bb77
-
Add SCADA data source & column reference (same component as 5-min energy)
ClaudeAdds a collapsible <details class="dataset-info"> accordion inside the SCADA Summary Statistics panel, matching the existing 5-minute energy component in structure and CSS classes (.col-defs / .col-def / .col-name / .col-desc — no new styles needed). The accordion is labelled "About this dataset — PUBLIC_NEXT_DAY_FPPMW" and expands to show: Column Descriptions: INTERVAL_DATETIME — target 4-second grid timestamp (YYYY/MM/DD HH:MM:SS) MEASUREMENT_DATETIME — actual AEMO telemetry recording time (may differ due to jitter) MEASURED_MW — instantaneous power output (MW); signed +/- for BESS MW_QUALITY_FLAG — numeric data-health code; live vs substituted reading MW_QUALITY_FLAG Values (from AEMO FPP Factor Calculation Guide): 0 Good — valid live telemetry (green) 1 Substituted/Estimated — previous known-good value held or interpolated (orange) 2 Bad — communication/hardware fault; exclude from analysis (red) 3 Manual …3af727f -
Fix chart width (Plotly hidden-container) and add section spacing
ClaudeChart width fix (app.js): Plotly.react() was being called while results-box, scada-results, and energy-results all had display:none. Plotly measured a zero-width container and locked in that dimension even after the divs were revealed. Fix: unhide resultsBox, then the section div, then call the render function — in that order — so Plotly measures the true container width. Section spacing (style.css): #results-box: display:flex / flex-direction:column / gap:3rem gives a clear visual break between the SCADA section and the Energy section when both are loaded. Hidden (.hidden = display:none) children produce no gap, so there is no unwanted whitespace when only one section is visible. #scada-results, #energy-results: same flex-column / gap:1.5rem so the panels within each section (stats, chart, download) are properly spaced. https://claude.ai/code/session_01TrCrzZmWbQscJiAQaW7Rg89845df1 -
Add inline help for dispatch energy data (tooltip icons + collapsible panel)
ClaudeOption 1 — ⓘ tooltip icons: - Added to the '5-minute Dispatch' checkbox label (tooltip appears below since it's near the top of the page, using .tip-below variant) - Added inline next to INITIALMW, INITIAL_ENERGY_STORAGE, and ENERGY_STORAGE in the Download Energy section; each tooltip carries a concise one-sentence description of the column's meaning Option 2 — collapsible 'About this dataset' panel: - <details>/<summary> block placed inside the Dispatch Energy Summary Statistics panel, above the stats grid - Expanded content includes: · Data source paragraph with hyperlinks to both NEMWEB Current and Archive directories for Next_Day_Dispatch · Three-row column definition list (col-name + col-desc in a CSS grid) for INITIALMW, INITIAL_ENERGY_STORAGE, ENERGY_STORAGE, using the full descriptions from AEMO's data model documentation - Collapsed by default; animated ▶ arrow rotates on open; summary text brightens on hover/open CSS additions (style.css)…1b5231e -
Add 5-minute dispatch energy data, data-type checkboxes, remove preview table
ClaudeNew data source — BESS Initial Energy (MWh): - config.py: add DISPATCH_CURRENT_URL, DISPATCH_ARCHIVE_URL, DISPATCH_START_DATE (2025-02-11), DISPATCH_COLUMNS, and TIMING_DEFAULT_DISPATCH_CURRENT/ARCHIVE_SEC constants - dispatch_fetcher.py: new service that fetches Next_Day_Dispatch ZIPs from NEMWEB Current (daily files: YYYYMMDD = settlement date) or Archive (monthly bundles, selected by largest start-date <= target); uses remotezip for archives; returns (csv_bytes, source_type) - dispatch_processor.py: new service that parses the AEMO C/I/D CSV format, extracts only the UNIT_SOLUTION table, filters by DUID and NEM trading-day window, and exposes compute_dispatch_summary(), to_csv_bytes(), to_parquet_bytes(), to_json_records() - analytics.py: extend _TIMING_DEFAULTS dict to cover dispatch_current and dispatch_archive; get_timing_estimate() now handles all 4 types - api.py: add GET /api/energy-data, /api/download/energy-csv, /api/download/energy-parquet; update /api/i…2f1129f -
Add learned wait-time estimates and stretch chart to fill frame
ClaudeTiming estimates (backend): - config.py: add FPPMW_CUTOVER_DATE, TIMING_MIN_SAMPLES, TIMING_DEFAULT_CURRENT_SEC / ARCHIVE_SEC as named constants - analytics.py: migrate schema to add duration_ms + source_type columns; extend log_request() to accept them; add get_timing_estimate() which returns p75 of the last 100 successful view durations per source_type, falling back to hardcoded defaults until >= TIMING_MIN_SAMPLES records exist - api.py: time fetch + process pipeline in all three data handlers; classify each request as 'current' (>= 2026-01-11) or 'archive'; pass duration_ms + source_type to log_request; expose get_timing_estimate results and cutover_date in /api/info Dynamic loading message (frontend): - app.js: store cutover_date and estimates from /api/info on boot; add formatEstimate() helper that converts seconds to human-readable bands and appends sample count when the estimate is learned (not default); replace hardcoded loading strings with the dynamic es…e6a4103 -
Fix FPPMW file lookup to use publication date (D+1) instead of settlement date
ClaudeFPPMW filenames embed the publication date (settlement date + 1), not the settlement date itself. The previous code searched for the settlement date and found nothing, or found the wrong day's files. Key changes: - Replace _is_month_end with _is_fppmw_bundle to distinguish archive bundles (end with _YYYYMMDD.zip, no suffix) from individual daily files (have a seq/timestamp suffix after the date) - _find_files_in: exact match now searches pub_date_str (D+1) and filters out bundles; bundle selection uses pub_date as the upper bound and finds the best Format-1 and Format-2 bundle independently (supports the Sep 2025+ split-format weekly archive) - _find_files: computes pub_date_str internally, returns it alongside results - _fetch_fppmw_monthly_csv: now receives pub_date_str so inner ZIP lookup searches for the publication date, not the settlement date - fetch_csv_for_date: passes pub_date_str to _fetch_fppmw_monthly_csv Covers all archive formats: Apr–Jul 2025: one large…7986823 -
Fix multi-segment CSV parsing so both half-day files use their own schema
ClaudeRoot cause of "only one row returned" for all dates after Jan 11 2026 ===================================================================== When two FPPMW ZIP files are downloaded (first half and second half of a NEM market day), their CSV bytes are concatenated and fed to _parse_aemo_csv as a single byte string. The old parser iterated over all lines and OVERWROTE the column header (`header = parts[3:]`) every time it saw a new "I," line. All data rows from BOTH files were collected under the single (last) header. If FPPMW and FPPMW_2 have even slightly different column schemas (e.g. different column count, or columns in a different order — which is the case for the new two-half-day format introduced Jan 11 2026), every row from File 1 is silently re-indexed under File 2's column structure: • datetime strings land in numeric columns → cast to Float64 → null • MEASUREMENT_DATETIME column is nulled for all File 1 rows • The date-window filter (on MEASUREMENT_DATETIME) drops t…c38e61e -
Fix silent 500: add global exception handler and NaN safety
ClaudeTwo issues caused 500 responses with no application-level log output: 1. Unhandled exception types Any exception that is not AEMOFetchError or DataProcessingError (e.g. polars.exceptions.PolarsError, KeyError, unexpected ValueError) bypassed both except clauses in every API endpoint and was caught only by Starlette's ServerErrorMiddleware, which logs to the 'uvicorn.error' logger rather than our application logger. The result was a 500 with no visible traceback. Fix: add a global @app.exception_handler(Exception) in main.py that logs the full traceback via our own logger.error() call before returning a JSON 500 response. This surfaces the real root cause in the application log for every future unhandled exception. 2. float('nan') in JSON serialization Polars cast(pl.Float64, strict=False) can produce a real float NaN (distinct from null/None) when the source CSV contains the literal string "nan" or certain edge-case numeric strings. float('n…36d7e96 -
Fix memory error: revert incorrect D+1 filename assumption
ClaudeLive AEMO directory data (from application logs) shows the second-half ZIP uses the SETTLEMENT date D, not D+1: PUBLIC_NEXT_DAY_FPPMW_2_20260310070050_XXXX.zip ← D=20260310 The YYYYMMDDHHMMSS is the *publication* timestamp (07:00 UTC = 17:00 AEST), and its date component is D. My previous commit incorrectly assumed D+1, adding an "_2_" + next_date_str condition that matched the FOLLOWING day's second-half file as well, causing three ZIPs to be downloaded for every request (first half D + second half D + second half D+1) and an OOM crash. Changes in this commit ----------------------- _find_files_in: Revert the fppmw_exact search to the simple `date_str in f` condition. This is sufficient: both halves embed settlement date D, so both are captured by one substring check. Explicit warning added in docstring NOT to re-introduce the D+1 condition. _fetch_fppmw_monthly_csv: Remove the spurious D+1 date condition from the inner-ZIP search (same reasoning — both inner ZIPs…3691b86 -
Fix missing second-half FPPMW ZIP for dates from 11 Jan 2026
ClaudeRoot cause: AEMO's second-half file (16:00–04:00 AEST) is named PUBLIC_NEXT_DAY_FPPMW_2_YYYYMMDDHHMMSS_<seq>.zip where the embedded YYYYMMDD is D+1 (the calendar date the half *ends*), not the settlement date D. The previous search `date_str in filename` only matched D's date and silently skipped the second half, yielding only 10 800 rows (the 04:00–16:00 first half) instead of the expected 21 600 for a full NEM market day. Fix 1 — _find_files_in: Extend the FPPMW exact-match search to also capture files whose name contains "_2_" AND the next-calendar-date string (D+1). This handles both naming conventions in one pass: • first half → "FPPMW" + date_str (D) in filename • second half → "FPPMW_2_" + next_date_str (D+1) in filename Fix 2 — _fetch_fppmw_monthly_csv: Add next_date_str parameter and apply the same dual-date logic to the inner-ZIP search inside monthly archive bundles (Jan 2026 bundles contain two inner ZIPs per market day from 11 Jan onward). Replace…5a2041e
-
Fix TypeError in compute_summary when std() returns None
Claudemw.std() returns None (not a float) when the series has fewer than 2 non-null values (std is undefined for n=1). The existing len(mw) > 0 guard only catches an empty series, not the n=1 case, so float(None) raised TypeError and the entire /api/data request returned 500. Replace the four inline guards with a _safe() helper that checks the scalar value itself for None before passing it to float(). https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxf074dd8 -
Catch all network errors; show HTTP status on frontend error
ClaudeTwo defensive fixes to expose the real failure cause: 1. aemo_fetcher.py: add except httpx.HTTPError fallback in both _list_directory and fetch_csv_for_date so that ConnectError, RemoteProtocolError, and other non-timeout/non-HTTP errors are converted to AEMOFetchError (JSON 503) rather than propagating as unhandled exceptions that could produce non-JSON responses. 2. app.js: replace the generic 'Unknown error' fallback with 'Server error <status> (no details available)' so the HTTP status code is always visible in the UI when the JSON body cannot be parsed (e.g. a 502/504 proxy timeout returning an HTML error page). https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxd707685 -
Fix docstring and remove wasteful D+1 fetch
ClaudeNow that fetch_csv_for_date downloads ALL files matching the settlement date (both the _YYYYMMDD_ first-half and _2_YYYYMMDDHHMMSS_ second-half ZIPs), the D+1 fetch in _fetch_day_pair was downloading two extra ZIP files that the boundary filter immediately threw away, making every request unnecessarily slow. - Remove _fetch_day_pair; replace with _fetch_day_csv (single call) - Update all three call sites in api.py accordingly - Remove now-unused timedelta import from api.py - Correct module docstring to document the actual two-file-per-day naming convention: FPPMW_YYYYMMDD_<seq> (first half) and FPPMW_2_YYYYMMDDHHMMSS_<seq+1> (second half), both sharing the same settlement date string https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx2420f34 -
Fetch ALL half-day ZIP files for a settlement date, not just the first
ClaudeThe root cause of the persistent data gap: AEMO publishes each 12-hour half of the NEM market day as a SEPARATE ZIP file, and both halves carry the same settlement date in their filename (e.g. two files both named PUBLIC_NEXT_DAY_FPPMW_20260115_*.zip). The old _find_file_in / _find_file pair always picked fppmw_exact[0], the lexicographically first match, which is consistently the first half (04:00–16:00 AEST). The second half was never downloaded. Fix: - Rename _find_file_in → _find_files_in: returns a LIST of every FPPMW file whose name contains date_str (sorted), not just the first one. - Rename _find_file → _find_files: propagates the list. - Rewrite fetch_csv_for_date: loops over every file in the list, downloads and extracts each, concatenates their raw CSV bytes. The existing _extract_csv_from_zip (opens all inner ZIPs) and the data_processor boundary filter [04:00 AEST D, 04:00 AEST D+1) together ensure the full market day is returned with no overlap or gaps, regardles…b7c0a6b -
Fix missing second 12-hour half of NEM market day (AEST, not UTC)
ClaudeRoot cause ---------- AEMO NEMWEB timestamps are in AEST (UTC+10, no DST), not UTC. The NEM market day runs 04:00–04:00 AEST. AEMO splits the FPPMW daily data into two 12-hour halves: • 04:00–16:00 AEST → outer ZIP named with date D • 16:00–04:00 AEST → outer ZIP named with date D+1 (next calendar day) Two bugs caused the gap: 1. _extract_csv_from_zip opened only inner_zips[0], so the second inner ZIP (second half, named with D+1) was never read. 2. filter_and_process had no time-boundary filter, so whatever was in the fetched files was returned unconstrained. Fix 1 — aemo_fetcher.py Rewrite _extract_csv_from_zip to open ALL inner ZIPs found in the outer ZIP and concatenate every CSV from each, instead of stopping at the first inner ZIP. fpp_bundle (flat outer ZIP) path unchanged. Correct all "UTC" timezone labels to "AEST" in comments/docstrings. Fix 2 — data_processor.py Add a strict [04:00 AEST target_date, 04:00 AEST target_date+1) boundary filter applied…73898ba -
Fix missing second 12-hour half of FPPMW daily ZIPs
ClaudeEach FPPMW daily inner ZIP contains two CSVs: one covering 04:00–16:00 UTC (named with the target date) and one covering 16:00–04:00 UTC (named with the *next* calendar date). The old date_str filter silently dropped the second CSV, producing data that ended at 16:00 and jumped to 04:00 the following day. Fix: when we are operating inside an inner ZIP (FPPMW daily format) all CSVs belong to the same market day, so take them all without filtering. The date_str filter is kept only for fpp_bundle outer ZIPs, where it is needed to isolate the correct day from a multi-day archive. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxadfdb79 -
Improve loading time estimate and add disclaimer
Claude- Loading message is now date-aware: dates from 11 Jan 2026 onward show a '2–3 minutes' estimate; older dates warn of '5–10 minutes or more' due to large NEMWEB archive files, replacing the misleading '10–20 s'. - Added a legal disclaimer paragraph to the footer covering data accuracy, liability, and directing users to verify with AEMO directly. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx12d7321 -
Unify hyperlink colour and add FPP source to footer
Claude- Extend the footer link style (var(--accent) sky-blue, no underline / underline on hover) to .panel a so the 'About This App' links match. - Add missing Frequency Performance Payments (FPP) scheme link to the footer alongside the existing FPPDAILY and NEM Generation Information sources. Also reword 'Data source:' to 'SCADA data:' for clarity. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx0a43438 -
Fix 12-hour gap in SCADA data by reading all CSVs from each inner ZIP
ClaudeEach FPPMW daily inner ZIP contains two CSV files covering the morning (04:00–16:00 UTC) and afternoon/overnight (16:00–04:00 UTC) halves of the NEM market day. The previous code took only csvs[0], discarding the second half and leaving a 12-hour gap in the output data. Both _extract_csv_from_zip and _fetch_fppmw_monthly_csv now read all matching CSV files from the inner ZIP in sorted order and concatenate their raw bytes. _parse_aemo_csv already tolerates multiple I-rows so no parser changes are needed. Updated comments to reflect this. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx7c86166 -
Fix chart truncation and CSV datetime format
Claude- Raise to_json_records cap from 5 000 to 25 000 rows so the chart shows a full 24-hour NEM day (21 600 readings at 4-second resolution) instead of only the first ~5.5 hours. - Remove explicit max_rows=5000 call in api.py; use new default. - Fix CSV download: format INTERVAL_DATETIME and MEASUREMENT_DATETIME with strftime before write_csv() so the output uses a space separator (2026-02-26 15:48:00) instead of the ISO 8601 T (2026-02-26T15:48:00). https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx3f12710 -
Remove time boundary filter causing data truncation to a few hours
ClaudeThe 04:00 UTC boundary filter applied naive UTC datetimes against NEM local-time timestamps stored in the CSV, resulting in only a few hours of data being returned instead of the full NEM market day. Replace the filter with a simple dedup + sort: raw CSV contents define the coverage window, and boundary rows shared between D and D+1 files are handled by deduplicating on MEASUREMENT_DATETIME. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx158d873 -
Only match FPPMW files; set DATA_START_DATE to 2025-02-28
ClaudeFPP files do not contain MEASUREMENT_DATETIME, MEASURED_MW or MW_QUALITY_FLAG and therefore cannot supply SCADA data. Any request that fell back to an FPP bundle (e.g. for Nov 2025 when both FPP and FPPMW monthly bundles have the same start date) would always fail with 'CSV missing expected columns'. Fix: remove FPP files from both exact-match and bundle-match logic in _find_file_in(). Only FPPMW files are ever selected now: - Exact match: FPPMW files containing the target date string - Bundle match: FPPMW files whose embedded date is an end-of-month date (monthly bundle convention, e.g. 20251031 covers November 2025) Also update DATA_START_DATE from 2024-12-09 to 2025-02-28, matching the first available FPPMW bundle (PUBLIC_NEXT_DAY_FPPMW_20250228.zip). The frontend date-picker min is sourced from /api/info so it updates automatically. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx10bbc1b -
Add FPPMW monthly archive support (Mar 2025–Jan 2026) + 24h day coverage
Claudeaemo_fetcher.py — complete rework: - Add _is_month_end(): FPPMW archive files whose embedded date is an end-of-month date are monthly bundles (e.g. 20250228 = March 2025 bundle); non-month-end FPPMW dates are individual daily archive copies. - _find_file_in() now returns (filename, url, kind) where kind is one of 'fppmw_monthly', 'fppmw_daily', or 'fpp_bundle'. - Bundle candidates now include FPPMW month-end files alongside FPP files, with FPPMW preferred over FPP for the same date (metered data > forecast). - _fetch_fppmw_monthly_csv(): uses remotezip (HTTP Range requests) to extract only the target day's inner ZIP from the multi-GB monthly bundle without downloading the whole file. - fetch_csv_for_date() dispatches to _fetch_fppmw_monthly_csv for 'fppmw_monthly' kind; downloads normally for 'fpp_bundle'/'fppmw_daily'. - New skip_future_check parameter for D+1 secondary fetches. data_processor.py: - _find_duid_col(): accepts both 'FPP_UNITID' (FPP format, ≤Jan 10 2026) …19b8b03 -
Disable all form controls while data is loading
ClaudeIntroduce setFormLocked(locked) which disables/re-enables selState, selBess, inpDate and btnLoad atomically. Called with true before the fetch and false in the finally block, so all controls are restored regardless of success or error. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx95bc2d3 -
Fix BESS columns, dropdown format, and date picker UX
Claudegen_info_fetcher.py: - Name column: 'Station Name' -> 'Unit Name' (row 4 column name) - MW column: 'Reg Cap' -> 'Unit Capacity (MW AC)' - Add MWh: 'Agg Nameplate Storage Capacity (MWh)' -> capacity_mwh field index.html: - Date input starts disabled; enabled only after a BESS unit is selected - Label 'Battery (BESS)' -> 'BESS Unit' app.js: - Dropdown options now show: 'DUID (Name) (MW / MWh)' - onBessChange() enables date picker and Load button when BESS is chosen - Changing state resets date picker to disabled again https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxd31601b -
Fix header row: use row 4 (index 3) not row 1 in Generator Information sheet
ClaudeThe AEMO NEM Generation Information XLSX has three metadata/title rows before the actual column headers, which appear on row 4. The previous code read rows[0] as headers and rows[1:] as data, causing rows 1-3 to be misinterpreted as data and producing phantom DUIDs like HORNSDALE_PWR3 that do not exist. Fix: read raw_headers from rows[3] and iterate data from rows[4:]. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx3f56f5a -
Source BESS list dynamically from AEMO generation information XLSX
ClaudeReplace static bess_list.json with live data from the AEMO NEM Generation Information spreadsheet: https://www.aemo.com.au/.../nem-generation-information-jan-2026.xlsx New service: app/services/gen_info_fetcher.py - Downloads the XLSX via httpx on first request - Parses the 'Generator Information' sheet using openpyxl - Filters rows: Technology Type = 'Battery Storage' Commitment Status = 'In Service' - Groups results by NEM region → state key (QLD1→QLD, NSW1→NSW, etc.) - Extracts DUID, Station Name, Reg Cap (MW), Region per unit - Caches the parsed result for 24 hours in memory - Falls back to stale cache, then to static bess_list.json, on any error Updated /api/bess endpoint to call fetch_bess_list() (now async). Added openpyxl==3.1.5 to requirements.txt. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxead45e6 -
Prefer FPPMW over FPP when both exact-match the target date
ClaudeThe Current directory contains both FPP and FPPMW files for dates from Jan 11 2026 onwards (e.g. PUBLIC_NEXT_DAY_FPP_20260305_XXX.zip and PUBLIC_NEXT_DAY_FPPMW_20260305_XXX.zip for the same date). The old code blindly returned exact[0] — whichever appeared first — which picked the FPP file. FPP files use the old format and carry a different suffix, so the wrong nested ZIP was opened and the wrong CSV was extracted. Fix: when multiple exact matches exist, prefer any FPPMW file. For dates before Jan 11 2026 only FPP files exist so fallback to exact[0] still works. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx540b503 -
Handle FPPMW nested-ZIP format introduced on 11 Jan 2026
ClaudeFrom 11 Jan 2026 AEMO changed FPPDAILY files from FPP to FPPMW naming. The new FPPMW ZIPs use a nested structure: outer ZIP → inner ZIP → CSV, instead of the old FPP structure: ZIP → CSV. The extraction code now: 1. Opens the outer ZIP and scans for CSV files (FPP path, as before). 2. If no CSVs are found, looks for an inner ZIP (FPPMW path), opens it, and extracts the CSV from there. Also updated the module docstring to document both formats clearly. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx7496930 -
Fix archive bundle logic: exclude FPPMW files; always try Current first
ClaudeTwo bugs fixed: 1. FPPMW files (PUBLIC_NEXT_DAY_FPPMW_YYYYMMDD.zip) are individual daily files, not multi-day bundles. They only contain data for their own date. The bundle-selection logic (largest date <= target) must not use them for a different date. Fix: filter to FPP-only (no 'FPPMW') before building the candidate list. 2. Current directory has 180+ files going back to at least Jan 10 2026 — far more history than the assumed 7-9 days. For recent dates (e.g. Feb 2026) the file is in Current, but the old code tried Archive first and wrongly returned an FPPMW file. Fix: always try Current before Archive. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxf0a0ff5 -
Fix Polars schema inference: scan all rows to detect float columns
ClaudeWith infer_schema_length=1000, columns like RAISE_PERFORMANCE were inferred as i64 because the first 1000 rows happened to have integer values. Later rows with floats (e.g. 0.06442) then caused a ComputeError. Setting infer_schema_length=None scans the entire file before deciding types. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx57e50b3 -
Rewrite archive ZIP selection: find largest start-date <= target date
ClaudeNEMWEB archive ZIPs are named PUBLIC_NEXT_DAY_FPP_YYYYMMDD.zip where the date is the FIRST day covered by that bundle (not end-of-month). A single archive ZIP spans multiple days (e.g. 20250228.zip covers Feb 28 - Mar 30). Previous approach (month-string matching) was wrong in two ways: - It matched FPPMW daily files as 'monthly archives' - It didn't account for bundles crossing month boundaries New approach: - Extract the 8-digit date from every filename in the archive listing - Select the ZIP whose date is the largest date that is still <= target date - This correctly navigates any bundle boundary Also: - Current dir files (with 16-digit suffix) still matched by exact date_str - _find_file now raises AEMOFetchError directly (cleaner call site) - Better error message lists which dates ARE available in the archive ZIP when the exact date CSV is missing - Extended _is_current window to 9 days for UTC/AEST lag tolerance - Added module docstring explaining both file naming convention…ff812da -
Fix monthly archive detection: exclude FPPMW daily files from monthly match
ClaudeThe previous logic matched PUBLIC_NEXT_DAY_FPPMW_20260201.zip as a 'monthly archive' for Feb 2026 because it contains '202602'. Monthly archives are named PUBLIC_NEXT_DAY_FPP_YYYYMMDD.zip (no 'MW') so the fix excludes any filename containing 'FPPMW' from the monthly match. Also always try Current URL as final fallback since NEMWEB may retain files longer than the assumed 7-day window. https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx0950d2d -
Fix FPPDAILY fetcher: add User-Agent, monthly archive support, and logging
Claude- Add browser User-Agent header to avoid NEMWEB blocking Python requests - Handle monthly archive ZIPs (PUBLIC_FPPDAILY_YYYYMM_xxx.zip) in addition to daily ZIPs — when archive match is monthly, find daily CSV inside the ZIP - Add INFO logging throughout fetch pipeline to diagnose future failures https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx3f2df04 -
Fix FPPDAILY fetcher: improve HTML parsing and add archive fallback logic
Claude- Use regex to extract hrefs (handles single/double quotes, full URLs) - Try archive monthly subdirectory (e.g. FPPDAILY/202602/) as fallback - Fall back from Current to Archive when no files match (not just on HTTP error) https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxd75e874 -
Initial implementation: NEM Battery SCADA Data Explorer
ClaudeFull-stack web app for exploring 4-second BESS SCADA data from AEMO NEMWEB: - FastAPI backend with async AEMO ZIP fetching, Polars-based CSV filtering, Parquet/CSV download endpoints, and SQLite analytics logging - Vanilla JS frontend with Plotly.js interactive time-series chart, state→BESS→date selection flow, data quality breakdown, and paginated table - Dockerfile configured for HuggingFace Spaces (port 7860) - Static BESS list (NSW/VIC/QLD/SA) with real DUIDs from AEMO NEM data - MW_QUALITY_FLAG explained to users (Good/Suspect/Bad) - IP-based analytics logging without user accounts https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjxfd0bd3b