# v0.1 implementation validation Validated on Apple Silicon on 2026-10-11 using the locked Python 3.12 environment. This is a software and short-training validation, not a completed research success claim. ## Data and training evidence Generator **0.1.2** produced the complete proposed dataset at `artifacts/poc/data`: 1,200 independently parameterized systems, seven days at 15-minute intervals, **806,400 transitions**, with **840 training / 180 validation / 180 test systems**. Source episodes include NumPy and Parquet files, simulator-only parameters, masks, seeds, policy IDs and segment IDs. The 35/30/25/10 policy mixture includes random hold intervals, step changes, ramps, feedback and boundary operation. The executed training comparison uses a smaller dataset at `artifacts/smoke/data-v2`: 90 systems, two days each, 63 training / 14 validation / 13 test systems. Validation is further divided into eight selection systems and six calibration systems. All four groups are disjoint. Training normalization and its fingerprint are saved with each checkpoint. MLP, GRU and transformer runs each use eight epochs, 4,096 sampled training windows, 512 selection windows, batch size 64 and seed 42. Transformer ensemble members use independent seeds 42, 43 and 44. Best checkpoints are selected by validation NLL. Transformer training and forward/backward compatibility were verified on host MPS in FP32. The default transformer has **1,570,434 parameters**; the 96-step variant has **2,750,082**. The full 150,000-window, up-to-50-epoch experiment has **not** been trained. Earlier smoke artifacts without the `-v2` suffix predate the final split/policy corrections and are superseded. ## Held-out-system comparison All main reports use the same 512 test anchors, seed 2026, identical system partitions and normalization. Anchors cover 184 battery, 165 tank and 163 thermal windows. Counterfactual evaluation uses 30 replayed snapshots with identical disturbance paths and noise streams across intervention branches. The comparison verifies fingerprints and matching anchor hashes. Errors below are MAE divided by each field's training standard deviation, averaged equally across field/horizon rows within each family. They are not averages of unlike physical units. Lower is better. | Model | Battery | Tank | Thermal | |---|---:|---:|---:| | Transformer | 0.24644 | 0.20519 | 0.08072 | | Three-member transformer ensemble | 0.24397 | 0.20475 | 0.08037 | | MLP | 0.26557 | 0.21290 | 0.09717 | | GRU | 0.26920 | 0.22744 | 0.10626 | | CatBoost | **0.17793** | **0.16986** | 0.08333 | | Fitted controlled affine dynamics | 0.19645 | 0.19399 | **0.05348** | | Persistence | 0.27418 | 0.20509 | 0.13506 | The transformer improves over the neural baselines in this short run, but the proposed 10% advantage over the strongest practical learned baseline is not demonstrated. Its tank forecast is essentially tied with persistence. These small, short runs do not establish what a fully trained model could achieve. Full physical-unit MAE/RMSE, Gaussian NLL, interval widths, intervention-effect errors, ranking accuracy and family-specific decision regret are retained separately by field and horizon in the raw reports. The simulator reference is oracle ground truth; hidden parameters are unavailable to the competing deployable models. Counterfactual action-effect MAE uses the same field-standardized averaging: | Model | Battery | Tank | Thermal | |---|---:|---:|---:| | Transformer | 0.33538 | 0.41009 | 0.06177 | | Three-member ensemble | 0.33930 | 0.40269 | 0.06547 | | CatBoost | 0.23052 | **0.26401** | 0.08128 | | Fitted controlled affine dynamics | **0.21354** | 0.37552 | **0.05729** | | Persistence | 0.44577 | 0.43092 | 0.13642 | This comparison also fails to demonstrate the required overall intervention advantage. There are only 12 battery, nine tank and nine thermal snapshots, so these figures are preliminary. ## Historical context ablation Three separately trained seed-42 transformers were evaluated on the same 512 held-out anchors eligible for 96 historical steps. Target values, planned controls and current states are identical; only the historical inputs differ. Each run has the same eight-epoch, 4,096-window optimizer budget. The longer-context model's eligible training-anchor population differs, and its larger temporal encoder increases parameter count. This is a preliminary comparison rather than an isolated causal estimate of history length. | Context | Battery standardized MAE | Tank | Thermal | |---|---:|---:|---:| | Six hours / 24 observations | **0.22588** | **0.24183** | **0.07896** | | 24 hours / 96 observations | 0.23028 | 0.24608 | 0.08463 | | Current state without history | 0.22862 | 0.25112 | 0.09654 | The longer history did not improve this short run. Historical context helps the thermal point estimate relative to the current-state-only model, but adaptation speed and slow-system identification remain unproven. These anchors differ from the main comparison, so its six-hour numbers must not be compared directly to the preceding table. Gaussian NLL and RMSE are recorded without calibrated-interval claims in `artifacts/smoke/report-context-v2/context.json`. ## Uncertainty limitations Nominal 90% intervals are calibrated on reserved validation systems, independently for each field/horizon and each checkpoint or ensemble. Coverage for the single transformer is 93.49% battery, 84.19% tank and 92.61% thermal; ensemble coverage is 93.82%, 86.18% and 92.53% respectively. **Thermal coverage applies to only 1,231 of 1,686 observed queries (73.01%).** Both reserved thermal calibration systems happen to be one-node systems, so there is no `mass_temp` calibration entry. Those predictions retain uncalibrated Gaussian dispersion and return null intervals. Reports explicitly mark calibration as partial. Battery and tank have entries for every observed test query. Correlated windows and the small number of calibration systems limit interpretation; no system-level coverage guarantee is claimed. Interval widths remain available in the reports, and coverage alone does not establish useful uncertainty. ## API, export and checks The typed FastAPI interface validates schemas, units, numeric types, control bounds, history alignment and horizon lengths. Single-model and ensemble requests have been executed against the trained v2 checkpoints and their matching calibration artifacts. An actual Uvicorn localhost server returned HTTP 200 for both `/health` and the example `/v1/predict` request; its response is saved at `artifacts/smoke/http-prediction-v2.json`. Unknown or unconfigured models do not return synthetic predictions. ONNX export of the trained single transformer passed graph validation and ONNX Runtime parity for dynamic batch and query counts, including one requested field/horizon. Exported metadata, normalization, calibration and checksums accompany the graph at `artifacts/smoke/onnx-transformer-v2`. History length, action length and controlled vocabulary remain fixed per exported graph. Portable ensemble packaging is not implemented; export members individually. Final checks: **72 pytest tests passed**, Ruff lint passed, and all 36 Python files passed formatting checks. Nine warnings concern the HTTP test client's deprecation and the explicitly selected legacy ONNX exporter/fixed-dimension tracing. Automated checks cover simulator replay and constraints, policy quotas and split counts, train-only normalization, missing values, window leakage, causal future-action prefixes, permutation/padding behavior, finite probabilistic loss, calibration provenance, matched evaluation anchors, strict API validation and actual ONNX Runtime execution. Latency artifacts record their exact timing scope; API profiling includes in-process HTTP validation, preparation, inference, decoding and serialization, excluding network sockets. Benchmark RSS is a process lifetime high-water mark, not isolated model peak memory. The single-model API was profiled after training finished, with five warmup requests and 50 timed requests: | Device | Request | p50 | p95 | |---|---|---:|---:| | CPU | One action / field / horizon | 2.69 ms | 3.04 ms | | MPS | One action / field / horizon | 4.95 ms | 5.22 ms | | CPU | Two actions / two fields / four horizons | 5.34 ms | 6.30 ms | | MPS | Two actions / two fields / four horizons | 5.31 ms | 7.20 ms | These in-process results meet the provisional latency gate for the single model on this host; they do not measure network serving latency. CPU is faster for the smallest request here. API requests use the short-history example and flag insufficient context. Hardware timings vary with request shape and concurrent host activity. ## Remaining research work Live CityLearn, BOPTEST and Minari validation has not been executed. Native collectors and contract fixtures are present; external field mappings, units, system grouping and normalization must be validated independently before combining data. Sensor dropout and bias experiments were executed for the transformer as part of Test C. New trajectories on known systems, shifted hidden-parameter regimes, unseen component configurations, held-out physical families and independently trained specialist comparisons remain unrun. Reports record these gaps explicitly. Auxiliary paired-action, conservation and multi-horizon objectives remain later experiments, as specified. ## Local artifacts Artifacts are ignored by source control and currently present in this workspace: - `artifacts/poc/data/manifest.json`: complete default source dataset. - `artifacts/smoke/comparison-v2.json`: verified baseline/model comparison and provenance. - `artifacts/smoke/report-{transformer,mlp,gru,ensemble}-v2/`: per-field/horizon reports, counterfactual results and calibration. - `artifacts/smoke/transformer-v2/seed-{42,43,44}/`: ensemble weights and reproducible run records. - `artifacts/smoke/onnx-transformer-v2/`: portable single-model bundle. - `artifacts/smoke/report-context-v2/context.json`: matched six-hour / 24-hour / no-history ablation. - `artifacts/smoke/api-latency-{cpu,mps,ensemble-cpu}-v2.json`: API timing scope and raw quantiles. The present evidence supports continuing controlled experiments. It does not yet justify the transformer complexity or demonstrate transferable low-data identification beyond the simpler models.