| --- |
| language: |
| - en |
| base_model: |
| - Qwen/Qwen3-4B |
| - polymathic-ai/aion-base |
| datasets: |
| - Sand33p/AstroPRISM |
| tags: |
| - astronomy |
| - multimodal |
| - vision-language |
| - spectrum-language |
| - qwen3 |
| - aion |
| - prism |
| - adapter-alignment |
| - safetensors |
| - custom-code |
| --- |
| |
| # AstroPRISM: Joint AION–Qwen3-4B alignment checkpoint |
|
|
| This private research checkpoint is a single AstroPRISM model jointly aligned on two caption streams: |
|
|
| - calibrated four-band Legacy Survey images; |
| - observed-frame DESI spectra. |
|
|
| Both streams trained the same checkpoint with shared AION and Qwen components. Each example contained one non-text modality; simultaneous image-plus-spectrum prompting was not trained or evaluated in this run. |
|
|
| ## Architecture |
|
|
| - Language model: Qwen3-4B, frozen during alignment. |
| - Astronomy encoder: AION Base, frozen during alignment and shared by both routes. |
| - Image input: `float32[4,160,160]` in DES-G, DES-R, DES-I, DES-Z band order. |
| - Spectrum input: `flux`, `ivar`, and `wavelength` as `float32[7781]`, plus `mask` as `bool[7781]`. |
| - AION output: 576 image tokens or 273 spectrum tokens, width 768. |
| - Separate two-layer projectors map each modality from width 768 to Qwen width 2560. |
| - Trainable parameters: 17,059,840 (image projector 8,527,360; spectrum projector 8,532,480). |
|
|
| The checkpoint contains 1,974 tensors and 4,452,498,917 tensor elements, including the frozen Qwen and shared AION tensors plus both projectors. The checkpoint SHA-256 is recorded in `SHA256SUMS` and `provenance/model_manifest.json`. |
|
|
| ## Pinned bases |
|
|
| - Qwen3-4B: `Qwen/Qwen3-4B`, revision `1cfa9a7208912126459214e8b04321603b3df60c`. |
| - AION Base and codecs: `polymathic-ai/aion-base`, revision `40541618104bab0fa85c8af68daeb867a720bb8c`. |
|
|
| See `THIRD_PARTY_NOTICES.md` for upstream licensing information. No license for AstroPRISM's original contributions or training data is asserted by this private release. |
|
|
| ## Training data |
|
|
| Training used the private [`Sand33p/AstroPRISM`](https://huggingface.co/datasets/Sand33p/AstroPRISM) dataset. Each modality contains 646 candidate rows with a source-group-safe split: |
|
|
| | Split | Image-caption | Spectrum-caption | |
| |---|---:|---:| |
| | Train | 516 | 516 | |
| | Validation | 65 | 65 | |
| | Test | 65 | 65 | |
|
|
| The roster contains 537 source groups, so the 646 candidate rows are not guaranteed to represent 646 independent physical sources. |
|
|
| ## Alignment run |
|
|
| - Steps: 200 |
| - Batch size: 2 per rank |
| - Hardware: 12 Aurora XPU ranks |
| - Precision: BF16 mixed precision |
| - Optimizer schedule: cosine, learning rate `1e-3`, 10 warmup steps, no weight decay |
| - Seed: 42 |
| - Frozen components: Qwen3-4B and AION Base |
| - Optimized components: image and spectrum projectors |
|
|
| The sanitized run configuration is in `config/training_config.yaml`; `training/training_state.json` records final checkpoint state. Optimizer and per-rank RNG files are intentionally excluded because this release targets inference rather than exact optimizer-state resumption. |
|
|
| ## Evaluation |
|
|
| Token-weighted, teacher-forced causal-LM losses on the held-out test split: |
|
|
| | Stream | Test loss | |
| |---|---:| |
| | Combined | 1.4708027309 | |
| | Image-caption | 1.5210461083 | |
| | Spectrum-caption | 1.3005556689 | |
|
|
| These losses measure target-token prediction, not scientific correctness. Machine-readable values are in `metrics/loss_metrics.json`. |
|
|
| ## Held-out generations |
|
|
| Greedy generations for one held-out image and one held-out spectrum are in: |
|
|
| - `examples/README.md` — human-readable prompts and outputs; |
| - `examples/test_image43_spectrum48.json` — complete prompts, token IDs, references, provenance, timing, and stopping reasons. |
|
|
| The reference captions were not supplied during generation. Both uploaded examples reached EOS. |
|
|
| ## Loading and inference |
|
|
| This is a custom PRISM checkpoint and is **not** directly loadable with `AutoModelForCausalLM.from_pretrained`. Instantiate PRISM's `UnifiedTransformer`, construct the pinned Qwen and AION components, and then load `model.safetensors` using the PRISM checkpoint loader. Raw-input preprocessing and modality routing require the corresponding BaseMM_PRISM implementation and AION codecs. |
| |
| `inference/generate_astroprism_joint_examples.py` is the exact runner used for the uploaded qualitative examples. The uploaded configuration uses Hub IDs and sanitized paths; download the private dataset locally and map its manifests into the PRISM data configuration before running it. |
|
|
| Source provenance is based on branch `AstroPrism` at Git revision `993d7f3a8217e93202a604f198de502b2ca234e0`. The training implementation also contained uncommitted AstroPRISM worktree changes, so that revision alone is not a complete source snapshot. |
|
|
| ## Limitations |
|
|
| - This is a 646-candidate pilot trained on synthetic or weak-supervision captions. |
| - The held-out captions are not an independently expert-adjudicated benchmark. |
| - Qualitative outputs can invent morphology, emission lines, redshifts, or unsupported physical explanations. |
| - The uploaded image example calls the object lenticular while its reference describes an elliptical morphology. |
| - The uploaded spectrum example asserts `[O III]` while its reference emphasizes a possible 4000 Å break. |
| - Simultaneous image-plus-spectrum input was not evaluated. |
| - Do not use this model as a substitute for calibrated measurement or expert astronomical analysis. |
|
|
