zhangrenchao commited on
Commit
1da5fc7
·
verified ·
1 Parent(s): 15aff58

Add English model card

Browse files
Files changed (1) hide show
  1. README.md +146 -0
README.md ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - OneScience
7
+ - Earth Science
8
+ - Data Assimilation
9
+ - Satellite Observations
10
+ - Multimodal Fusion
11
+ - Weather Forecasting
12
+ frameworks: PyTorch
13
+ ---
14
+
15
+ <p align="center"><strong><span style="font-size: 30px;">FuXi-DA</span></strong></p>
16
+
17
+ # Model Introduction
18
+
19
+ FuXi-DA fuses weather background fields and satellite observations into improved global atmospheric analyses for deep-learning data assimilation and forecast initialization. It learns background errors, observation biases, and analysis increments in a unified feature space to improve subsequent forecasts with multi-time geostationary satellite data.
20
+
21
+ Paper: FuXi-DA: a generalized deep learning data assimilation framework for assimilating satellite observations
22
+ https://doi.org/10.1038/s41612-025-01039-3
23
+
24
+ # Model Description
25
+
26
+ The method was proposed by teams from the Shanghai Artificial Intelligence Laboratory, Fudan University, and collaborating institutions. The paper trains on ERA5, FuXi background fields, and Fengyun-4B AGRI observations. FuXi-DA performs multi-scale fusion with background, observation, and condition encoders and uses frozen-FuXi forecast errors to supervise assimilation. The model supports 0.25-degree global background correction and geostationary multi-channel satellite assimilation.
27
+
28
+ # Use Cases
29
+
30
+ | Use Case | Description |
31
+ | :---: | :--- |
32
+ | Satellite assimilation | Fuse 70-channel backgrounds with multi-time AGRI brightness temperatures. |
33
+ | Multimodal fusion | Validate multi-scale interactions among background, observation, and condition branches. |
34
+ | Local validation | Execute sampled aligned tiles under full-variable and global-coordinate protocols. |
35
+ | ModelScope/OneCode execution | Validate data, training, inference, assimilation metrics, and visualization. |
36
+ | Multi-GPU training | Validate distributed training and checkpoint workflows through `torchrun`. |
37
+
38
+ # Usage Instructions
39
+
40
+ ## 1.OneCode
41
+
42
+ [Try intelligent, one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
43
+
44
+ ## 2. Download and Installation
45
+
46
+ ```bash
47
+ hf download OneScience-Group/FuXi-DA --local-dir ./FuXi-DA
48
+ cd FuXi-DA
49
+ ```
50
+
51
+ ### Environment Dependencies
52
+
53
+ **Hardware Requirements**
54
+
55
+ - A GPU or DCU is recommended.
56
+ - A CPU can be used for connectivity validation with the default small-sample configuration.
57
+ - DCU users must install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.
58
+
59
+ **DCU Environment**
60
+
61
+ ```bash
62
+ # Activate DTK and Conda first
63
+ conda create -n onescience311 python=3.11 -y
64
+ conda activate onescience311
65
+ pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
66
+ ```
67
+
68
+ **GPU Environment**
69
+
70
+ ```bash
71
+ # Activate Conda first
72
+ conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
73
+ conda activate onescience311
74
+ pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
75
+ ```
76
+
77
+ ### Training Data
78
+
79
+ This repository uses a small synthetic dataset for engineering validation, including 70-variable global backgrounds, eight AGRI times, 15 observation channels, six-hour relationships, and the real `[70,721,1440]` background and `[8,15,640,640]` observation protocols. Procedural full-coordinate fields and a few aligned original-coordinate tiles preserve variables, channels, and spatiotemporal specifications while reducing executed samples and coverage. These data validate the multi-branch U-Net, feature fusion, training, inference, and evaluation only and do not represent official ERA5 or AGRI distributions and scale.
80
+
81
+ ```bash
82
+ python scripts/fake_data.py
83
+ ```
84
+
85
+ ### Training
86
+
87
+ For single-GPU training, use:
88
+
89
+ ```bash
90
+ python scripts/train.py
91
+ ```
92
+
93
+ For multi-GPU training, use:
94
+
95
+ ```bash
96
+ torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
97
+ ```
98
+
99
+ Training combines latitude-weighted analysis L1 with multi-lead supervision from a frozen forecast proxy. The default reduces tile coverage, network width, iterations, and forecast steps while recording the paper's 6,000 iterations and ten forecast leads. Training artifacts are saved to:
100
+
101
+ ```text
102
+ result/checkpoints/fuxi_da.pt
103
+ result/training/metrics.json
104
+ ```
105
+
106
+ ### Trained Weights
107
+
108
+ The paper does not provide directly loadable official model weights, and no weights are bundled under `weight/`.
109
+
110
+ ### Inference
111
+
112
+ ```bash
113
+ python scripts/inference.py
114
+ ```
115
+
116
+ Inference loads the local checkpoint and takes a 70-channel background tile and corresponding multi-time AGRI observations. Background, observation, and condition branches generate an increment that is added to the background to form the analysis. Outputs retain targets, backgrounds, observations, analyses, coordinates, and coverage metadata and explicitly indicate incomplete global coverage. Inference results are saved to:
117
+
118
+ ```text
119
+ result/output/predictions.npz
120
+ ```
121
+
122
+ ### Evaluation and Visualization
123
+
124
+ ```bash
125
+ python scripts/result.py
126
+ ```
127
+
128
+ Evaluation compares latitude-weighted RMSE for the raw background, correction baseline, and FuXi-DA analysis and saves Z, T, U, V, R, and surface-variable groups. It also reports forecast-step errors and localization of analysis increments from a single-observation perturbation and generates a comparison of analysis schemes and variable-group errors. Synthetic tile results validate engineering only and do not represent complete global paper performance. Evaluation results are saved to:
129
+
130
+ ```text
131
+ result/evaluation/metrics.json
132
+ result/evaluation/comparison.png
133
+ ```
134
+
135
+ # Official OneScience Information
136
+
137
+ | Platform | OneScience Main Repository | Skills Repository |
138
+ | --- | --- | --- |
139
+ | Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
140
+ | GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
141
+
142
+ # Citation and License
143
+
144
+ This repository is an independent engineering reproduction of the public FuXi-DA specifications, with code licensed under the Apache License 2.0.
145
+
146
+ The original paper is licensed under CC BY-NC-ND 4.0; the paper, official model weights, ERA5 data, and Fengyun-4B AGRI data remain subject to the licenses and terms of their respective projects.