Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,169 @@
 ---
 license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+  - multimodal
+  - scientific
+  - protein
+  - rna
+  - dna
+  - molecule
+  - weather
+  - medical-imaging
+base_model:
+  - Qwen/Qwen3-VL-8B-Instruct
 ---
+
+<div align="center">
+
+[🤗 Model](https://huggingface.co/sais-org/Polaris_Pro>) &nbsp;•&nbsp; [💻 GitHub](https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro) &nbsp;•&nbsp; [📜 Technical Report (coming soon)](#) &nbsp;•&nbsp; [⚖️ License: Apache-2.0](./LICENSE)
+
+</div>
+
+# Polaris-Pro
+
+**Polaris-Pro is a unified scientific multimodal foundation model** that
+supports scientific **understanding and generation** across Earth science,
+proteins, RNA, DNA, and small molecules within a single **8B** model. Native
+scientific encoders/decoders wrap a shared **Qwen3-VL-8B-Instruct** backbone, so
+heterogeneous scientific data (sequences, molecular graphs, gridded physical
+fields, medical images) are reasoned about and generated in one representation
+space — natural language in and out, no per-task fine-tuning.
+
+> 📜 **Technical report coming soon.**
+
+## Key features
+
+- **Unified understanding *and* generation** across 7 modalities through one
+  natural-language interface.
+- **Seven modalities, one 8B backbone** (protein / RNA / DNA / molecule /
+  weather / medical-image / text) via a modality router.
+- **Native scientific encoders/decoders** (ESM-2, RNA/DNA ConvFormers, molecular
+  graph encoder, Swin-ViT weather tower, SAM-based image path) preserve domain
+  structure a generic tokenizer would destroy.
+
+## Capabilities
+
+| Modality      | Understanding | Generation |
+|:--------------|:-------------:|:----------:|
+| Protein       | ✅            | —          |
+| RNA           | ✅            | ✅         |
+| DNA           | ✅            | —          |
+| Molecule      | ✅            | ✅         |
+| Weather       | —             | ✅         |
+| Medical image | —             | ✅         |
+| Text          | ✅            | ✅         |
+
+<sub>**Understanding** = classification / regression / scientific QA. **Generation**: RNA sequence design · Molecule text → SMILES · Weather 10-day global ERA5 0.25° forecast · Medical-image text-prompted segmentation (needs SAM 3).</sub>
+
+## Benchmarks
+
+**Polaris-Pro** (**8B**) vs **Biology-Instructions** (Llama-3.1-**8B**, text-token,
+no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
+**Bold** = best; <u>underline</u> = second-best.
+
+### Biological sequence understanding
+
+| Task | Metric | Polaris-Pro (8B) | Biology-Instructions (8B) | Intern-S1-Pro (~1T) |
+|:-----|:------:|:----------------:|:-------------------------:|:-------------------:|
+| DNA · Epigenetic marks (EMP) | MCC | **71.99** | 3.64 | <u>14.02</u> |
+| DNA · Promoter det. 300bp (PD300) | MCC | **91.17** | 58.18 | <u>82.65</u> |
+| DNA · Core-promoter (CPD) | MCC | **66.35** | 44.54 | <u>54.60</u> |
+| DNA · Enhancer activity (EA) | PCC | 52.64 | <u>53.28</u> | **55.16** |
+| RNA · ncRNA function | Acc | **91.46** | <u>63.09</u> | 34.50 |
+| RNA · Modification | AUC | **96.03** | <u>59.06</u> | 57.77 |
+| RNA · APA isoform | R² | <u>79.87</u> | 59.01 | **82.95** |
+| RNA · CRISPR on-target | Spearman ρ | **28.76** | -0.02 | <u>15.69</u> |
+| Protein · Stability | Spearman ρ | **70.63** | 60.25 | <u>60.82</u> |
+| Protein · Fluorescence | Spearman ρ | <u>70.12</u> | 2.57 | **78.14** |
+| Protein · Enzyme Commission | Fmax | <u>68.65</u> | 19.79 | **72.70** |
+| Protein · Solubility | Acc | <u>67.26</u> | 63.02 | **67.60** |
+| Cross-modal · RPI (RNA–protein) | MCC | **76.49** | <u>74.26</u> | 58.51 |
+| Cross-modal · AAN (antibody–antigen) | MCC | <u>42.96</u> | 1.06 | **44.76** |
+| Cross-modal · EPI (enhancer–promoter) | MCC | <u>-0.03</u> | **3.37** | -1.30 |
+
+<sub>Aggregate over 20 biological-understanding benchmarks: Polaris-Pro matches or beats the ~1T Intern-S1-Pro on 10/20 and the same-scale 8B text-token baseline on 16/20.</sub>
+
+### Molecule understanding (SMolInstruct)
+
+| Task | Metric | Polaris-Pro (8B) | LlaSMol |
+|:-----|:------:|:----------------:|:-------:|
+| BBBP | Acc | **96.95** | 74.60 |
+| HIV | Acc | **97.00** | 96.70 |
+| SIDER | Acc | **71.00** | 70.70 |
+| ClinTox | Acc | 92.36 | **93.10** |
+| ESOL | RMSE ↓ | **0.550** | 1.150 |
+| Lipophilicity | RMSE ↓ | **0.628** | 1.010 |
+
+### Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)
+
+| Variable | Metric | Polaris-Pro (8B) | ECMWF HRES (NWP) |
+|:---------|:------:|:----------------:|:----------------:|
+| Z500 | RMSE ↓ | **≈740** | ≈810 |
+| T2M | RMSE ↓ (K) | **≈2.65** | ≈2.90 |
+| MSL | RMSE ↓ (Pa) | **≈680** | ≈745 |
+
+<sub>Polaris-Pro tracks or beats the operational physics-based HRES system, with the advantage growing at longer lead times.</sub>
+
+### Medical-image segmentation
+
+Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across
+nine imaging modalities, versus six modality-native segmentation specialists.
+
+| Modality | # Samples | Polaris-Pro | BiomedParse | MedSAM | SAM | SAM3 | DINO+MedSAM | DINO+SAM |
+|:---------|----------:|:-----------:|:-----------:|:------:|:---:|:----:|:-----------:|:--------:|
+| **All**    | 102,855 | **91.20** | <u>90.73</u> | 83.55 | 71.29 | 35.40 | 15.37 | 15.10 |
+| CT         |  45,306 | **93.36** | <u>92.25</u> | 83.87 | 74.10 | 28.93 |  9.59 | 10.34 |
+| MRI        |  30,990 | **85.29** | <u>85.25</u> | 75.90 | 68.34 | 53.64 | 13.28 | 12.39 |
+| OCT        |     283 | <u>85.31</u> | **86.63** | 56.26 | 55.99 |  8.69 |  6.68 |  6.98 |
+| X-ray      |  13,840 | <u>98.02</u> | **98.28** | 97.75 | 81.35 | 39.96 | 37.22 | 30.63 |
+| Dermoscopy |      65 | **98.08** | 97.11 | <u>97.35</u> | 88.23 | 51.47 | 81.28 | 78.29 |
+| Endoscopy  |     410 | **97.39** | 96.77 | <u>97.05</u> | 92.88 | 38.82 | 25.01 | 24.54 |
+| Fundus     |     800 | <u>91.33</u> | **91.50** | 88.06 | 57.16 | 18.58 |  3.19 |  2.73 |
+| Pathology  |     977 | **87.29** | <u>81.57</u> | 43.44 | 42.06 | 26.08 | 25.38 | 24.69 |
+| Ultrasound |  10,184 | <u>90.54</u> | **91.03** | 89.76 | 57.47 |  5.23 | 17.12 | 22.91 |
+
+<sub>Best overall Dice (All), and best on CT, MRI, pathology, dermoscopy, and endoscopy; on the four modalities where BiomedParse leads, the gap is ≤ 0.5 Dice.</sub>
+
+## Usage
+
+Runs via the accompanying code repository (custom multimodal architecture).
+
+```bash
+git clone <GITHUB_REPO> && cd polaris-pro
+pip install -r requirements.txt          # Python 3.10; transformers==5.0.0
+hf download <THIS_REPO> --local-dir ./model
+
+export PYTHONPATH=$PWD/code
+python code/inference.py --model_path model --greedy --max_new_tokens 64 \
+  --rna "GGATGCGATCATGTCTGCACTAACACACCGGATCCCATCAGAACTCCG" \
+  --system "You are a non-coding RNA family classifier. Output only the family name, no other text." \
+  --prompt $'<rna>\nWhich family does this non-coding RNA sequence belong to?'
+```
+
+All scientific encoder/decoder weights (ESM-2, the Suiren molecular graph
+encoder, the RNA/DNA ConvFormers, and the Swin-ViT weather tower) are embedded
+in `model.safetensors`. The only separate download is **SAM 3** for
+medical-image segmentation, under Meta's gated license.
+
+Each task has a specific `--system` prompt that fixes the output format; see
+`run_examples.sh` in the repository for per-task examples, weather, and segmentation.
+
+## License
+
+Apache-2.0. Built on Qwen3-VL (Apache-2.0); includes merged ESM-2 (MIT) and
+Polaris/Suiren-derived encoders. **Medical-image segmentation needs SAM 3,
+under Meta's separate gated license — not distributed here.** See
+`THIRD_PARTY_LICENSES.md` / `NOTICE` in the code repository.
+
+## Citation
+
+```bibtex
+@misc{polarispro2026,
+  title  = {Polaris-Pro: A Unified Scientific Multimodal Foundation Model},
+  author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
+  year   = {2026},
+  note   = {https://huggingface.co/sais-org/Polaris_Pro}
+}
+```

chat_template.jinja

@@ -0,0 +1,120 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,343 @@
+{
+  "architectures": [
+    "Qwen3VLForConditionalGeneration"
+  ],
+  "bio_token_ids": {
+    "_bio_seq_output_pad": {
+      "pad": 151687
+    },
+    "contact_map": {
+      "end": 151682,
+      "pad": 151683,
+      "start": 151681
+    },
+    "dna": {
+      "end": 151673,
+      "pad": 151674,
+      "start": 151672
+    },
+    "mol": {
+      "end": 151679,
+      "pad": 151680,
+      "start": 151678
+    },
+    "protein": {
+      "end": 151676,
+      "pad": 151677,
+      "start": 151675
+    },
+    "rna": {
+      "end": 151670,
+      "pad": 151671,
+      "start": 151669
+    },
+    "weather": {
+      "end": 151685,
+      "pad": 151686,
+      "start": 151684
+    }
+  },
+  "bos_token_id": null,
+  "dna_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "chunk_size_feed_forward": 0,
+    "conv_kernel_size": 7,
+    "dna_encoder_hidden_size": 512,
+    "dna_encoder_path": "dna",
+    "dna_max_seq_length": 2048,
+    "dna_projector_hidden_size": 2048,
+    "dna_vocab_size": 8,
+    "dropout": 0.1,
+    "dtype": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "qwen3_vl_dna",
+    "num_attention_heads": 8,
+    "num_encoder_layers": 8,
+    "num_latent_tokens": 64,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "problem_type": null,
+    "return_dict": true
+  },
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "epi_ohem_enable": false,
+  "epi_ohem_hard_fraction": 0.5,
+  "epi_ohem_weight": 2.0,
+  "image_token_id": 151655,
+  "med_seg_config": {
+    "cost_bbox": 5.0,
+    "cost_class": 2.0,
+    "cost_dice": 5.0,
+    "cost_giou": 2.0,
+    "cost_mask": 5.0,
+    "focal_alpha": 0.25,
+    "focal_gamma": 2.0,
+    "freeze_sam3": true,
+    "freeze_sam3_vision": false,
+    "loss_w_bbox_giou": 0.5,
+    "loss_w_bbox_l1": 0.5,
+    "loss_w_cls": 1.0,
+    "loss_w_mask_dice": 5.0,
+    "loss_w_mask_dice_high": 0.0,
+    "loss_w_mask_focal": 2.0,
+    "loss_w_mask_semantic": 0.5,
+    "loss_w_meta_ce": 0.2,
+    "mask_cost_res": 128,
+    "mask_h": 576,
+    "mask_loss_target_res": null,
+    "mask_w": 576,
+    "med_query_end_id": null,
+    "med_query_start_id": null,
+    "meta_num_classes": 15,
+    "proj_hidden_mult": 2,
+    "proj_mlp": true,
+    "sam3_model_path": null,
+    "sam3_text_dim": 256,
+    "system_prompt": "You are a medical imaging assistant specialized in image understanding for segmentation.\nYou will be provided with (1) a medical image and (2) a text query that specifies a target region to segment.\nYour goal is to produce a detailed, clinically plausible description that helps delineate the target region precisely.\nPlease describe:\n    1) The imaging modality/type if it is evident (e.g., CT, MRI, ultrasound, X-ray, endoscopy, dermoscopy, histopathology). If uncertain, explicitly say it is unknown.\n    2) The overall field of view and orientation (e.g., axial/sagittal/coronal for cross-sectional images, or a general view if not applicable).\n    3) The main anatomical structures/organs visible, and for each one provide approximate location using spatial terms. (left/right, superior/inferior, anterior/posterior, central/peripheral) and relative relationships between organs.\n    4) Most importantly, focus on the structure mentioned in the text query: describe its approximate position, boundaries, shape, size, and its appearance (intensity/color, texture, homogeneity, and contrast to surrounding tissues).\n    Be concise but detailed, use precise spatial language, and emphasize features helpful for pixel-accurate segmentation.",
+    "upsample_mask_loss": false,
+    "use_query_special_tokens": false
+  },
+  "med_seg_freeze_qwen": true,
+  "med_seg_visual_fp32": false,
+  "modality_configs": {},
+  "model_type": "qwen3_vl",
+  "mol_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "chunk_size_feed_forward": 0,
+    "dropout": 0.1,
+    "dtype": null,
+    "freeze_gnn": true,
+    "freeze_mol_gnn": true,
+    "gnn_drop_ratio": 0.0,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "qwen3_vl_mol",
+    "mol_decoder_hidden_size": 768,
+    "mol_decoder_max_seq_length": 512,
+    "mol_decoder_num_heads": 12,
+    "mol_decoder_num_layers": 6,
+    "mol_encoder_hidden_size": 256,
+    "mol_encoder_path": "mol",
+    "mol_projector_hidden_size": 1024,
+    "mol_vocab_size": 226,
+    "num_gnn_layers": 12,
+    "num_latent_tokens": 64,
+    "num_resampler_heads": 8,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "problem_type": null,
+    "return_dict": true
+  },
+  "pad_token_id": 151643,
+  "protein_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "chunk_size_feed_forward": 0,
+    "conv_kernel_size": 7,
+    "dropout": 0.1,
+    "dtype": null,
+    "esmc_model_name": "esm2_t30_150M_UR50D",
+    "freeze_backbone": true,
+    "freeze_esmc": true,
+    "freeze_protein_backbone": true,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "qwen3_vl_protein",
+    "num_attention_heads": 8,
+    "num_encoder_layers": 12,
+    "num_latent_tokens": 64,
+    "num_resampler_layers": 2,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "problem_type": null,
+    "protein_backbone_name": "esm2_t30_150M_UR50D",
+    "protein_convformer_vocab_size": 33,
+    "protein_encoder_hidden_size": 640,
+    "protein_encoder_path": "protein",
+    "protein_max_seq_length": 8192,
+    "protein_projector_hidden_size": 2560,
+    "protein_vocab_size": 31,
+    "return_dict": true,
+    "use_physicochemical_embed": true
+  },
+  "rna_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "chunk_size_feed_forward": 0,
+    "conv_kernel_size": 7,
+    "dropout": 0.1,
+    "dtype": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "qwen3_vl_rna",
+    "num_attention_heads": 8,
+    "num_encoder_layers": 8,
+    "num_latent_tokens": 64,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "problem_type": null,
+    "return_dict": true,
+    "rna_encoder_hidden_size": 512,
+    "rna_encoder_path": "rna",
+    "rna_max_seq_length": 4096,
+    "rna_projector_hidden_size": 2048,
+    "rna_vocab_size": 8
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "max_position_embeddings": 262144,
+    "model_type": "qwen3_vl_text",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 36,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_theta": 5000000,
+      "rope_type": "default"
+    },
+    "rope_theta": 5000000,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "vocab_size": 151936
+  },
+  "text_loss_weight": 1.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "deepstack_visual_indexes": [
+      8,
+      16,
+      24
+    ],
+    "depth": 27,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_vl",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 4096,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "weather_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attn_type": "flash",
+    "chunk_size_feed_forward": 0,
+    "const_chans": 6,
+    "decoder_depth": 12,
+    "dtype": null,
+    "embed_freq": 256,
+    "embed_mode": "add",
+    "embed_types": [
+      "step",
+      "hour",
+      "doy",
+      "lead_hour"
+    ],
+    "encoder_depth": 12,
+    "ffn_type": "geglu_ffn",
+    "frame_interval": "6h",
+    "hidden_size": 2048,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": [
+      721,
+      1440
+    ],
+    "in_chans": 70,
+    "in_frames": 1,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "lead_hour_scaling": true,
+    "lower_chans": 0,
+    "mask_type": "h",
+    "max_rollout_steps": 200,
+    "meteo_data_path": null,
+    "meteo_loss_coef": 1.0,
+    "meteo_loss_type": "l1_channel",
+    "meteo_loss_weight": 1.0,
+    "meteo_out_channels": 70,
+    "meteo_output_size": [
+      721,
+      1440
+    ],
+    "meteo_token_id": 151655,
+    "mlp_ratio": 4.0,
+    "model_type": "qwen3_vl_weather",
+    "n_kv_heads": null,
+    "norm_type": "adarms",
+    "num_heads": 32,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 6,
+    "pl_chans": null,
+    "problem_type": null,
+    "qwenvl_dim": 4096,
+    "return_dict": true,
+    "time_embed_dropout": 0.0,
+    "upper_chans": 70,
+    "window_size": 20
+  }
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.0,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "5.0.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bd9d8ccdde5b14d2daea4b6d43103093063a9bb26dc2b56518fe729a32c0bb0
+size 23422026570

processor_config.json

@@ -0,0 +1,65 @@
+{
+  "image_processor": {
+    "data_format": "channels_first",
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessorFast",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_pixels": 50176,
+    "merge_size": 2,
+    "min_pixels": 784,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 50176,
+      "shortest_edge": 784
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "data_format": "channels_first",
+    "default_to_square": true,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 8,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 802816,
+      "shortest_edge": 200704
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa662d183d411d13f6d7ef66e068bdfe8b4036ae5ba985cecf6ea4853c3891d0
+size 11426274

tokenizer_config.json

@@ -0,0 +1,37 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|rna_start|>",
+    "<|rna_end|>",
+    "<|rna_pad|>",
+    "<|dna_start|>",
+    "<|dna_end|>",
+    "<|dna_pad|>",
+    "<|protein_start|>",
+    "<|protein_end|>",
+    "<|protein_pad|>",
+    "<|mol_start|>",
+    "<|mol_end|>",
+    "<|mol_pad|>",
+    "<|contact_start|>",
+    "<|contact_end|>",
+    "<|contact_pad|>",
+    "<|weather_start|>",
+    "<|weather_end|>",
+    "<|weather_pad|>",
+    "<|bio_seq_pad|>"
+  ],
+  "is_local": true,
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}