File size: 10,835 Bytes
1baf215 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | # Transition Foundation Model v0.1
A runnable research proof of concept for action-conditioned numerical physical systems. It predicts standardized state changes and Gaussian predictive dispersion for multiple fields, horizons and alternative actions in one non-autoregressive forward pass. The default schema-aware transformer has **1,570,434 parameters**.
This repository implements the synthetic experiment, baselines, calibration, API and portable export. A short validation run establishes that the software works; it does not establish the proposed research success gates or commercial advantage.
See [executed validation and results](validation.md) for the generated full dataset, short training comparison, uncertainty gaps and remaining experiments. The workspace contains trained smoke checkpoints; to try the API immediately:
```sh
TFM_CHECKPOINT=artifacts/smoke/transformer-v2/seed-42/checkpoint.pt \
TFM_CALIBRATION=artifacts/smoke/report-transformer-v2/calibration.json \
uv run --no-sync uvicorn tfm.api.server:app --host 127.0.0.1 --port 8000
```
## Setup
Python 3.11+; Python 3.12 recommended on Apple Silicon. Install the locked environment:
```sh
uv sync --locked --extra dev --extra baselines --extra export
uv run --no-sync pytest -q
uv run --no-sync ruff check tfm tests
```
PyTorch uses FP32 and chooses MPS when available. `--device mps` fails clearly if unavailable; CPU fallback from an operator failure requires `--allow-cpu-fallback`. GPU access may be unavailable inside a restricted process even when the host supports MPS. CPU operation works independently. See [PyTorch MPS documentation](https://docs.pytorch.org/docs/stable/notes/mps.html).
## Generate and train
```sh
uv run --no-sync python -m tfm.data.generate --output artifacts/poc/data --systems 1200 --days 7 --seed 42 --parquet
uv run --no-sync python -m tfm.training.train --data artifacts/poc/data --output artifacts/poc/transformer --model transformer --device mps
```
Defaults: 150,000 sampled training windows, history24, future8, batch256, AdamW3e-4, up to50 epochs, patience8. `configs/poc.json` records the proposed experiment and success gates; CLI flags are the executable configuration, and complete arguments are persisted in each run.
Generate complete source episodes, then derive windows. Each system has one fixed hidden parameter set, including randomized one/two-node thermal dynamics, tank storage/pump/outlet characteristics, and temperature-dependent batteries. A 35/30/25/10 mixture of held random controls, interventions, feedback and boundary operation provides diverse excitation. Exogenous paths are generated independently of controls.
The generator writes compressed NumPy episodes and optional Parquet, a versioned manifest, seeds, policy IDs, schema and simulator-only parameters. Each episode contains T+1 observations and T applied actions. History ends at current time; historical applied controls stop at t-1, and planned controls start at t. Future targets and hidden parameters are absent from neural inputs.
Systems are stratified 70/15/15 by family. Validation systems are further divided into model selection and calibration, preserving the untouched test split. Normalization uses observed training values only, records exact training IDs, and is fingerprinted by each checkpoint. Mean heads output normalized deltas; standard deviations use the same field scales.
## Baselines and experiments
Train MLP and GRU with the same data, windows and split seed:
```sh
uv run --no-sync python -m tfm.training.train --data artifacts/poc/data --output artifacts/poc/mlp --model mlp --device mps
uv run --no-sync python -m tfm.training.train --data artifacts/poc/data --output artifacts/poc/gru --model gru --device mps
uv run --no-sync python -m tfm.evaluation.benchmark --data artifacts/poc/data --checkpoint artifacts/poc/transformer/checkpoint.pt --output artifacts/poc/report --samples 4096 --calibration-samples 4096 --counterfactual-samples 100 --catboost --catboost-samples 8192 --stress-tests --device mps
```
Run each checkpoint with identical benchmark seed/sample settings to compare identical held-out windows. The report includes persistence, a per-window fitted affine controlled state-space model, and optional CatBoost. Physical-unit MAE/RMSE, standardized error/NLL, interval coverage and width are reported per family/field/horizon; values in unlike physical units are never pooled as a raw MAE.
Counterfactual evaluation replays the simulator to the held-out anchor, snapshots its complete state and RNG, and branches alternative plans against identical disturbances/noise. It reports intervention-effect error, pairwise action ranking and decision regret under explicit family-specific costs. Those costs are benchmark choices, not financial savings. Oracle trajectories establish simulation ground truth, not a deployable fair baseline.
`--history-steps 96` evaluates 24-hour context with a separately trained checkpoint; `--no-history` measures the identification benefit of history. `--ensemble-size 3` trains three independent initializations. Repeat `--checkpoint` during evaluation to combine member distributions by total variance and calibrate the ensemble independently. The generated `ensemble.json` lists its members.
`python -m tfm.evaluation.context --data DATA --h24-checkpoint PATH --h96-checkpoint PATH --no-history-checkpoint PATH --output REPORT` compares these context variants on identical anchors eligible for the longest history. `python -m tfm.evaluation.comparison --data DATA --config RUNS.json --output comparison.json` verifies the fingerprints, partitions and common test anchors of main benchmark reports before summarizing them.
Specialist comparison accepts `--specialist-checkpoint thermal=PATH` (similarly tank/battery) and remaps each specialist's training normalization onto the same test windows. Specialist datasets must preserve the original system partitions and exclude test systems from training/selection. Their checkpoints must actually be trained independently; no specialist superiority/transfer claim is inferred from an absent run.
Held-out-system evaluation is Test B. Sensor dropout/bias stresses cover part of Test C. New parameter regimes, new component configurations (D), unseen time periods on known systems (A), and leave-family-out training (E) require distinct experiment artifacts; `generalization.json` labels unmet prerequisites as unrun. Six-hour and 24-hour configurations are available; a longer context is not assumed to identify slow dynamics.
## API
```sh
TFM_CHECKPOINT=artifacts/poc/transformer/checkpoint.pt \
TFM_CALIBRATION=artifacts/poc/report/calibration.json \
uv run --no-sync uvicorn tfm.api.server:app --host 127.0.0.1 --port 8000
```
`GET /health`; typed OpenAPI docs at `/docs`; `POST /v1/predict`. `TFM_CHECKPOINT` accepts comma-separated ensemble checkpoints. `TFM_DEVICE` defaults to CPU. Without configured weights, prediction returns503 rather than generating illustrative outputs.
Schema fields carry `name`, `unit`, `role`, `type` (continuous), and optional `minimum`/`maximum`. v0.1 uses exact thermal/tank/battery schemas from `tfm.data.schema`; one-node thermal may omit `mass_temp`. Units and permissible control values are validated strictly; unknown names need explicit mapping. Numerical strings, booleans, invalid horizons and mismatched field sets are rejected.
`history` is at most the checkpoint's context length, includes current state at relative timestamp0, and is spaced15 minutes apart. Previous observations include their subsequent applied `actions` and `exogenous`; the final observation has neither. Missing state sensors use null. A requested output requires an observed current value because prediction decodes a delta.
`actions` maps scenario names to per-control arrays of8 values. `exogenous` maps known disturbance fields to arrays of8 values. `horizons_minutes` supports15/30/60/120, and `predict` selects observed state fields. All action scenarios form one inference batch. `examples/predict-thermal.json` is an executable request:
```sh
curl -s http://127.0.0.1:8000/v1/predict -H 'Content-Type: application/json' --data-binary @examples/predict-thermal.json
```
Response contains model version, per-action/horizon/field means, predictive standard deviations and calibrated90% intervals when a matching calibration entry is available. Uncalibrated Gaussian parameters are not labeled confidence intervals. Calibration identifies exact weight/normalization fingerprints and reserved systems. Missing entries produce null intervals. Intervals condition on supplied exogenous inputs; forecast-input uncertainty is not modeled. Physical clipping is disclosed per field.
`insufficient_context` identifies short histories. OOD is currently a six-standard-deviation current-state heuristic, not a learned reliability detector. Valid ranges do not identify instance-specific hidden capacity.
## ONNX
```sh
uv run --no-sync python -m tfm.export.onnx --checkpoint artifacts/poc/transformer/checkpoint.pt --data artifacts/poc/data --output artifacts/poc/onnx --calibration artifacts/poc/report/calibration.json
```
Export validates the graph and PyTorch/ONNX Runtime parity, including alternative-action batching and a single-field query. Batch/query dimensions are dynamic; field vocabulary and context/action lengths remain fixed per checkpoint. The package includes weights, schema, normalization, calibration when supplied, checksums and an interface manifest. A numerical graph alone does not reproduce preprocessing or physical decoding. Current export uses the legacy exporter explicitly for broad operator compatibility; see [PyTorch ONNX documentation](https://docs.pytorch.org/docs/stable/onnx.html).
API timing can be measured with `python -m tfm.api.profile --checkpoint PATH --request examples/predict-thermal.json --output latency.json --device cpu`. This includes FastAPI request validation, transfers, inference and decoding through an in-process HTTP client; it excludes network socket latency. Use `--device mps` for a synchronized host GPU measurement.
## External validation and limits
Native CityLearn/BOPTEST/Minari collectors keep external artifacts isolated until metadata and normalization have been independently validated. See [external validation](external-validation.md). The initial tests use native-contract fixtures; no live external-family transfer result is claimed.
The proposed 10% advantage over the strongest learned baseline, intervention advantage, calibrated85–95% coverage without overly wide intervals, sub20ms latency and absence of severe family regressions remain experiment gates. A small physical or learned specialist may win. Passing software tests does not justify added transformer complexity.
|