Update hyperspectral image reader imports, add utils, and update README for Hugging Face

README.md

@@ -7,42 +7,56 @@ tags:
 - foundation-model
 - image-segmentation
 ---
+# HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone
 
-# HyperVision: Channel-Adaptive Ground-Based Hyperspectral Vision Foundation Models
-
-*This is the official repository for the paper "HyperVision: Channel-Adaptive Ground-Based Hyperspectral Vision Foundation Models".* 
+*This is the official repository for the paper "HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone".*
 https://arxiv.org/abs/2605.17286
 
+and **Hyperspectral Images Reader with multiple dataset support.**
+
 **Code:** [https://github.com/lronkitty/HyperVision](https://github.com/lronkitty/HyperVision)
 
+[![BMVC 2026](https://img.shields.io/badge/BMVC-2026-blue.svg)](https://bmvc2026.bmva.org/)
+[![Paper Status](<https://img.shields.io/badge/Status-Accepted%20to%20BMVC%202026-success.svg>)](https://bmvc2026.bmva.org/)
+[![arXiv](https://img.shields.io/badge/arXiv-2605.17286-b31b1b.svg)](https://arxiv.org/abs/2605.17286)
+[![GitHub](https://img.shields.io/badge/GitHub-HyperVision-green.svg)](https://github.com/lronkitty/HyperVision)
+
+> 📢 **Announcement**: Our paper **"HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone"** has been officially accepted by **BMVC 2026**! 🎉
+
+*Official PyTorch implementation for the BMVC 2026 paper:*
+**"HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone"**
+
 ---
 
-🚀 **News:**
-- **[2026-07-02]** Pre-trained checkpoints are now available! You can download them from [Hugging Face Checkpoints](https://huggingface.co/IronKitty/HyperVision/tree/main/checkpoints).
+## 🚀 News
+
+- **[31-Jul-2026]** 🎉 **HyperVision** has been officially accepted to **BMVC 2026**!
+- **[02-Jul-2026]** 📦 Official codebase, model checkpoints, and inference pipelines released.
 
 ---
 
 ## 📖 Abstract
 
-While hyperspectral imaging provides rich spatial-spectral information across hundreds of narrow wavelength bands for precise material identification, ground-based hyperspectral pre-trained backbones remain absent, constrained by varying spectral configurations across sensors, the scarcity and inconsistency of labels, and the limited scale and scene diversity of existing datasets. To address these challenges and enable universal perception, we propose HyperVision, the first ground-based hyperspectral pre-trained backbone. First, to handle varying spectral configurations, HyperVision adopts a channel-adaptive dynamic embedding mechanism to map heterogeneous inputs into a unified token space. Second, to address the scarcity and inconsistency of labels, we introduce a multi-source pseudo-labeling method that fuses semantic representations from both spatial structures generated by SAM2 and fine-grained spectral material information extracted by HyperFree. Third, to compensate for limited dataset scale and enrich scene diversity, a cross-modal knowledge distillation mechanism is utilized to transfer rich semantic representations from a pre-trained RGB vision model to our hyperspectral backbone. Pre-trained on a collection of 15k images from 26 diverse ground-based datasets, HyperVision demonstrates exceptional generalization. Requiring only efficient head-only adaptation without adjusting backbone parameters, it achieves state-of-the-art performance compared to task-specific methods across three downstream tasks under varying sensor configurations, yielding up to a 16.3% relative improvement in hyperspectral semantic segmentation $\text{Acc}_{\text{M}}$, a 2.1% relative gain in object tracking AUC, and a 35.5% reduction in salient object detection MAE. The source code and pre-trained model will be publicly available online.
+While hyperspectral imaging provides rich spatial-spectral information across hundreds of narrow wavelength bands for precise material identification, ground-based hyperspectral pre-trained backbones remain absent, constrained by varying spectral configurations across sensors, limited annotations and heterogeneous labeling schemes, and the limited scale and scene diversity of existing datasets. To address these challenges and enable universal perception, we propose HyperVision, the first ground-based hyperspectral pre-trained backbone. First, to handle varying spectral configurations, HyperVision adopts a channel-adaptive dynamic embedding mechanism to map heterogeneous inputs into a unified token space. Second, we develop an unsupervised representation learning framework. Specifically, to address limited annotations and heterogeneous labeling schemes, a multi-source pseudo-labeling method is introduced to fuse spatial structures from SAM2 and fine-grained spectral material information from HyperFree. Furthermore, to enrich scene diversity and compensate for limited dataset scale, a cross-modal knowledge distillation mechanism is utilized to transfer rich semantic representations from a pre-trained RGB vision model to our backbone. Pre-trained on a collection of 15k images from 26 diverse ground-based datasets, HyperVision demonstrates exceptional generalization. Requiring only efficient head-only adaptation without adjusting backbone parameters, it outperforms state-of-the-art task-specific methods across three downstream tasks under varying sensor configurations, yielding up to a 16.3% relative improvement in hyperspectral semantic segmentation $\mathrm{Acc}_{\mathrm{M}}$, a 2.1% relative gain in object tracking AUC, and a 35.5% reduction in salient object detection MAE.
+
+---
 
 ## 🛠️ Main Code (HyperVision)
 
-The core architecture and model initialization utilities are located under the [HyperVision](./HyperVision) directory. 
+The core architecture and model initialization utilities are located under the [HyperVision](./HyperVision) directory.
 
-To initialize the model, you can use the model builder functions from [HyperVision/build_HyperVision.py](./HyperVision/build_HyperVision.py). The pre-trained checkpoints can be downloaded from [Hugging Face Checkpoints](https://huggingface.co/IronKitty/HyperVision/tree/main/checkpoints).
+To initialize the model, you can use the model builder functions from [HyperVision/build_HyperVision.py](./HyperVision/build_HyperVision.py). We support different model configurations:
 
-We support different model configurations:
 - **HyperVision-B**: `build_HyperVision_b`
 - **HyperVision-L**: `build_HyperVision_l`
 - **HyperVision-H** (Default): `build_HyperVision_h`
 
 ### Python Example:
+
 ```python
 import torch
-from HyperVision import build_HyperVision_h, HyperVision_Predictor
-from hyperspectral_image_reader.read_dataset_image import load_hypervision_matrix
-from hyperspectral_image_reader.hyperspectral_pipelines import LoadHyperspectralImage
+from HyperVision import build_HyperVision_h, build_HyperVision_l, build_HyperVision_b
+from hyperspectral_image_reader import load_hypervision_matrix, LoadHyperspectralImage
 
 device = "cuda" if torch.cuda.is_available() else "cpu"
 image_size = 512
@@ -52,42 +66,46 @@ model = build_HyperVision_h(
     checkpoint="path/to/hypervision_distilled_h.pth",             # Optional checkpoint path (.pth weights)
     image_size=512,                                         # Input resolution
     vit_patch_size=16,                                      # Patch size of the backbone
-    encoder_global_attn_indexes=[15, 23, 31],               # Layers using global attention (can be adjusted for debugging/tuning)
-    merge_indexs=[8, 32],                                   # Layers doing patch merging (can be adjusted for debugging/tuning)
+    encoder_global_attn_indexes=[15, 23, 31],               # Layers using global attention
+    merge_indexs=[8, 32],                                   # Layers doing patch merging
     class_number=-1                                         # Number of classes for mask decoder
 )
 model = model.to(device)
 model.eval()
 
-# 2. Use HyperVision_Predictor to manage preprocessing and feature extraction
-predictor = HyperVision_Predictor(model)
-
-# 3. Load the preprocessed HSI image as a numpy array using load_hypervision_matrix
+# 2. Load and preprocess a hyperspectral image
 # Preprocessed shape: (H_ori, W_ori, C_hsi) with values in range [0, 255]
 img_matrix = load_hypervision_matrix("path/to/image.mat", dataset_name="harvard")
+h_ori, w_ori = img_matrix.shape[:2]
 
-# 4. Automatically retrieve the wavelengths list from the dataset pipeline config
+# Convert to torch tensor with shape (1, C_hsi, H_ori, W_ori)
+input_tensor = torch.as_tensor(img_matrix, device=device).permute(2, 0, 1).unsqueeze(0).float()
+
+# 3. Automatically retrieve the wavelengths list from the dataset pipeline config
 loader = LoadHyperspectralImage(dataset_type="harvard")
 wavelengths = loader.wavelengths
 
-# 5. Call predictor.set_image to preprocess the image and extract features (calculating embeddings)
-# GSD represents Ground Sampling Distance (defaults to 0.01)
+# 4. Extract features via the HyperVision Image Encoder
 with torch.no_grad():
-    predictor.set_image(img_matrix, test_mode=False, spectral_lengths=wavelengths, GSD=0.01)
-
-# 6. Extract multi-scale features from the predictor
-multi_stage_features = predictor.multi_scale_features
+    # GSD represents Ground Sampling Distance (defaults to [0.01])
+    multi_stage_features = model.image_encoder(
+        input_tensor, 
+        test_mode=True, 
+        input_wavelength=wavelengths, 
+        GSD=[0.01]
+    )
 ```
 
 ---
 
 ## 📊 Dataset Reading & Processing
 
-We provide [read_dataset_image.py](./hyperspectral_image_reader/read_dataset_image.py) to read and preprocess hyperspectral images from various datasets. The script utilizes the custom data pipeline class `LoadHyperspectralImage` from [configs/hypervision/hyperspectral_pipelines.py](./configs/hypervision/hyperspectral_pipelines.py) to normalize the images (scaled to `[0, 255]`) and reshape them into the format expected by HyperVision: `(H_ori, W_ori, C_hsi)`.
+We provide [read_dataset_image.py](./hyperspectral_image_reader/read_dataset_image.py) to read and preprocess hyperspectral images from various datasets. The script utilizes the custom data pipeline class `LoadHyperspectralImage` from [hyperspectral_image_reader/hyperspectral_pipelines.py](./hyperspectral_image_reader/hyperspectral_pipelines.py) to normalize the images (scaled to `[0, 255]`) and reshape them into the format expected by HyperVision: `(H_ori, W_ori, C_hsi)`.
 
 ### Python API Usage:
+
 ```python
-from hyperspectral_image_reader.read_dataset_image import load_hypervision_matrix
+from hyperspectral_image_reader import load_hypervision_matrix
 
 # Load and preprocess HSI matrix
 # dataset_name must be one of the supported dataset identifiers (e.g., 'harvard', 'arad_1k_31')
@@ -96,7 +114,9 @@ print("Processed shape (H, W, C):", img_matrix.shape)
 ```
 
 ### Command Line Interface:
+
 You can also run the script from the command line to load an HSI image and optionally save the processed matrix as a `.npy` file:
+
 ```bash
 python hyperspectral_image_reader/read_dataset_image.py --path /path/to/image.mat --dataset harvard --output output.npy
 ```
@@ -105,41 +125,41 @@ python hyperspectral_image_reader/read_dataset_image.py --path /path/to/image.ma
 
 ## 🗃️ Supported Datasets
 
-Below is a summary of the ground-based hyperspectral datasets supported by our pipeline, along with their respective keys to be used for the `--dataset` parameter:
-
-| Dataset Name | `--dataset` Key | # Bands | Wavelengths | # Images | Expected Extension | File Loader Detail / Required Accompanying Files |
-| :--- | :--- | :---: | :---: | :---: | :---: | :--- |
-| **50 Outdoor** | `fiftyoutdoor` | 33 | 400–720 nm | 50 | `.mat` | Custom Mat loader |
-| **Agricultural Plant** | `aphid` | 237 | 436–965 nm | 361 | `.npy` | NumPy array loader |
-| **ARAD1K16** | `arad_1k_16` | 16 | 400–1000 nm | 950 | `.mat` | HDF5 H5PY loader |
-| **ARAD1K31** | `arad_1k_31` | 31 | 400–700 nm | 949 | `.mat` | HDF5 H5PY loader |
-| **CAVE** | `cave` | 31 | 400–700 nm | 32 | `.mat` | Mat loader |
-| **DeepHS-NIR** | `deephsnir` | 252 | 950–1700 nm | 718 | `.bin` | ENVI binary (requires corresponding `.hdr` file in the same directory) |
-| **DeepHS-VIS** | `deephsvis` | 224 | 400–1000 nm | 3405 | `.bin` | ENVI binary (requires corresponding `.hdr` file in the same directory) |
-| **DeepHS-VISCOR** | `deephsviscor` | 249 | 400–1000 nm | 1566 | `.bin` | ENVI binary (requires corresponding `.hdr` file in the same directory) |
-| **Harvard** | `harvard` | 31 | 420–720 nm | 77 | `.mat` | Mat loader |
-| **HOT-2024-NIR** | `hotnir` | 25 | 665–960 nm | 477 | `.png` | PNG frame loader (requires corresponding false-color `.jpg` file) |
-| **HOT-2024-RedNIR** | `hotrednir` | 15 | 600–850 nm | 348 | `.png` | PNG frame loader (requires corresponding false-color `.jpg` file) |
-| **HOT-2024-VIS** | `hotvis` | 16 | 470–600 nm | 1070 | `.png` | PNG frame loader (requires corresponding false-color `.jpg` file) |
-| **HSI Drive v2.0** | `hsidrive20` | 25 | 600–975 nm | 752 | `.npy` | NumPy loader (requires corresponding pseudocolor `.png` file) |
-| **HSI Road** | `hsiroad` | 25 | 600–960 nm | 380 | `.tif` | TIF image loader |
-| **HSODBIT v2** | `hsodbitv2` | 200 | 400–1000 nm | 500 | `.mat` | HDF5 H5PY loader (requires corresponding color `.jpg` file) |
-| **HSSOD** | `hs_sod` | 81 | 380–720 nm | 60 | `.h5` | HDF5 H5PY loader (requires corresponding color `.jpg` file) |
-| **HyKo v2-NIR** | `hykov2nir` | 25 | 600–975 nm | 78 | `.mat` | Mat loader |
-| **HyKo v2-VIS** | `hykov2vis` | 16 | 470–630 nm | 163 | `.mat` | Mat loader |
-| **HyperBlood** | `hyperblood` | 128 | 377–1046 nm | 14 | `.mat` | Custom Mat loader |
-| **HyperDrive-VNIR** | `hyperdrivevnir` | 24 | 660–900 nm | 504 | `.npz` | NumPy archive (must contain `cube.npy` file inside) |
-| **HyperspectralCity v2** | `hyperspectralcityv2` | 128 | 450–950 nm | 1330 | `.hsd` | HSD raw data loader |
-| **ICVL** | `icvl` | 31 | 400–700 nm | 187 | `.h5` | HDF5 H5PY loader |
-| **LIB-HSI** | `libhsi` | 204 | 400–1000 nm | 393 | `.hdr` | ENVI header (requires corresponding raw binary data `.raw`/`.dat` file) |
-| **UM-EMM** | `umemm` | 33 | 400–720 nm | 3 | `.mat` | Mat loader |
-| **UM-LD 2015** | `umld2015` | 33 | 400–720 nm | 20 | `.mat` | Mat loader |
-| **UM-NS 2002** | `umns2002` | 31 | 410–710 nm | 8 | `.mat` | Mat loader |
-| **UM-NS 2004** | `umns2004` | 33 | 400–720 nm | 10 | `.mat` | Mat loader |
-| **UM-OS** | `umos` | 33 | 400–720 nm | 50 | `.mat` | Mat loader |
-| **UM-RI 2015** | `umri2015` | 33 | 400–720 nm | 33 | `.mat` | Mat loader |
-| **Virginia Tech Tree** | `virginia_tech_tree` | 420 | 400–1000 nm | 51 | `.hdr` | ENVI header (requires corresponding raw binary data file) |
-| **Apple Fire Blight** | `vnihdhiatlimafb` | 204 | 400–1000 nm | 420 | `.hdr` | ENVI header (requires corresponding raw binary data file) |
+Below is a summary of the 26 ground-based hyperspectral datasets (31 subset entries) supported by our pipeline, along with their respective keys for the `--dataset` parameter:
+
+| Dataset Name                   | `--dataset` Key       | # Bands | Wavelengths | # Images | Expected Extension | File Loader Detail / Required Accompanying Files                           |
+| :----------------------------- | :---------------------- | :-----: | :----------: | :------: | :----------------: | :------------------------------------------------------------------------- |
+| **50 Outdoor**           | `fiftyoutdoor`        |   33   | 400–720 nm |    50    |      `.mat`      | Custom Mat loader                                                          |
+| **Agricultural Plant**   | `aphid`               |   237   | 436–965 nm |   361   |      `.npy`      | NumPy array loader                                                         |
+| **ARAD1K16**             | `arad_1k_16`          |   16   | 400–1000 nm |   950   |      `.mat`      | HDF5 H5PY loader                                                           |
+| **ARAD1K31**             | `arad_1k_31`          |   31   | 400–700 nm |   949   |      `.mat`      | HDF5 H5PY loader                                                           |
+| **CAVE**                 | `cave`                |   31   | 400–700 nm |    32    |      `.mat`      | Mat loader                                                                 |
+| **DeepHS-NIR**           | `deephsnir`           |   252   | 950–1700 nm |   718   |      `.bin`      | ENVI binary (requires corresponding`.hdr` file in the same directory)    |
+| **DeepHS-VIS**           | `deephsvis`           |   224   | 400–1000 nm |   3405   |      `.bin`      | ENVI binary (requires corresponding`.hdr` file in the same directory)    |
+| **DeepHS-VISCOR**        | `deephsviscor`        |   249   | 400–1000 nm |   1566   |      `.bin`      | ENVI binary (requires corresponding`.hdr` file in the same directory)    |
+| **Harvard**              | `harvard`             |   31   | 420–720 nm |    77    |      `.mat`      | Mat loader                                                                 |
+| **HOT-2024-NIR**         | `hotnir`              |   25   | 665–960 nm |   477   |      `.png`      | PNG frame loader (requires corresponding false-color`.jpg` file)         |
+| **HOT-2024-RedNIR**      | `hotrednir`           |   15   | 600–850 nm |   348   |      `.png`      | PNG frame loader (requires corresponding false-color`.jpg` file)         |
+| **HOT-2024-VIS**         | `hotvis`              |   16   | 470–600 nm |   1070   |      `.png`      | PNG frame loader (requires corresponding false-color`.jpg` file)         |
+| **HSI Drive v2.0**       | `hsidrive20`          |   25   | 600–975 nm |   752   |      `.npy`      | NumPy loader (requires corresponding pseudocolor`.png` file)             |
+| **HSI Road**             | `hsiroad`             |   25   | 600–960 nm |   380   |      `.tif`      | TIF image loader                                                           |
+| **HSODBIT v2**           | `hsodbitv2`           |   200   | 400–1000 nm |   500   |      `.mat`      | HDF5 H5PY loader (requires corresponding color`.jpg` file)               |
+| **HSSOD**                | `hs_sod`              |   81   | 380–720 nm |    60    |      `.h5`      | HDF5 H5PY loader (requires corresponding color`.jpg` file)               |
+| **HyKo v2-NIR**          | `hykov2nir`           |   25   | 600–975 nm |    78    |      `.mat`      | Mat loader                                                                 |
+| **HyKo v2-VIS**          | `hykov2vis`           |   16   | 470–630 nm |   163   |      `.mat`      | Mat loader                                                                 |
+| **HyperBlood**           | `hyperblood`          |   128   | 377–1046 nm |    14    |      `.mat`      | Custom Mat loader                                                          |
+| **HyperDrive-VNIR**      | `hyperdrivevnir`      |   24   | 660–900 nm |   504   |      `.npz`      | NumPy archive (must contain`cube.npy` file inside)                       |
+| **HyperspectralCity v2** | `hyperspectralcityv2` |   128   | 450–950 nm |   1330   |      `.hsd`      | HSD raw data loader                                                        |
+| **ICVL**                 | `icvl`                |   31   | 400–700 nm |   187   |      `.h5`      | HDF5 H5PY loader                                                           |
+| **LIB-HSI**              | `libhsi`              |   204   | 400–1000 nm |   393   |      `.hdr`      | ENVI header (requires corresponding raw binary data`.raw`/`.dat` file) |
+| **UM-EMM**               | `umemm`               |   33   | 400–720 nm |    3    |      `.mat`      | Mat loader                                                                 |
+| **UM-LD 2015**           | `umld2015`            |   33   | 400–720 nm |    20    |      `.mat`      | Mat loader                                                                 |
+| **UM-NS 2002**           | `umns2002`            |   31   | 410–710 nm |    8    |      `.mat`      | Mat loader                                                                 |
+| **UM-NS 2004**           | `umns2004`            |   33   | 400–720 nm |    10    |      `.mat`      | Mat loader                                                                 |
+| **UM-OS**                | `umos`                |   33   | 400–720 nm |    50    |      `.mat`      | Mat loader                                                                 |
+| **UM-RI 2015**           | `umri2015`            |   33   | 400–720 nm |    33    |      `.mat`      | Mat loader                                                                 |
+| **Virginia Tech Tree**   | `virginia_tech_tree`  |   420   | 400–1000 nm |    51    |      `.hdr`      | ENVI header (requires corresponding raw binary data file)                  |
+| **Apple Fire Blight**    | `vnihdhiatlimafb`     |   204   | 400–1000 nm |   420   |      `.hdr`      | ENVI header (requires corresponding raw binary data file)                  |
 
 ---
 
@@ -147,15 +167,29 @@ Below is a summary of the ground-based hyperspectral datasets supported by our p
 
 If you find our work or this project helpful, please consider citing our paper:
 
+<!-- ```bibtex
+@inproceedings{fu2026hypervision,
+  title     = {{HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone}},
+  author    = {Fu, Guanyiman and Li, Jingtao and Cheng, Zihang and Li, Zhuanfeng and Chen, Diqi and Xu, Yan and Liu, Xiangyu and Xiong, Fengchao and Lu, Jianfeng and Chen, Chengrong and Zhou, Jun},
+  booktitle = {Proc. British Mach. Vis. Conf. (BMVC)},
+  year      = {2026}
+}
+``` -->
+
 ```bibtex
-@misc{fu2026hypervision,
-      title={HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone}, 
-      author={Guanyiman Fu and Jingtao Li and Zihang Cheng and Zhuanfeng Li and Diqi Chen and Yan Xu and Xiangyu Liu and Fengchao Xiong and Jianfeng Lu and Chengrong Chen and Jun Zhou},
-      year={2026},
-      eprint={2605.17286},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV},
-      url={https://arxiv.org/abs/2605.17286}, 
+@misc{fu2026hypervisionarxiv,
+  title         = {{HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone}}, 
+  author        = {Guanyiman Fu and Jingtao Li and Zihang Cheng and Zhuanfeng Li and Diqi Chen and Yan Xu and Xiangyu Liu and Fengchao Xiong and Jianfeng Lu and Chengrong Chen and Jun Zhou},
+  year          = {2026},
+  eprint        = {2605.17286},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CV},
+  url           = {https://arxiv.org/abs/2605.17286}
 }
 ```
-*(Note: Citation details will be updated upon official publication)*
+
+<!-- ---
+
+## 📄 License
+
+This project is licensed under the Apache 2.0 License. -->

hyperspectral_image_reader/__init__.py

@@ -0,0 +1,4 @@
+from .hyperspectral_pipelines import LoadHyperspectralImage
+from .read_dataset_image import load_hypervision_matrix
+
+__all__ = ['LoadHyperspectralImage', 'load_hypervision_matrix']

hyperspectral_image_reader/hyperspectral_pipelines.py

@@ -28,8 +28,16 @@ import rasterio
 import spectral.io.envi as envi
 import tifffile
 import random
-from configs.hypervision import ds_load
-from configs.hypervision.hsi3d import get_image_loader
+try:
+    from hyperspectral_image_reader.utils import ds_load
+    from hyperspectral_image_reader.utils.hsi3d import get_image_loader
+except ImportError:
+    try:
+        from .utils import ds_load
+        from .utils.hsi3d import get_image_loader
+    except (ImportError, ValueError):
+        from utils import ds_load
+        from utils.hsi3d import get_image_loader
 import cupy as cp
 from typing import Dict, Iterable, List, Optional, Sequence, Tuple, Union
 import logging

hyperspectral_image_reader/read_dataset_image.py

@@ -12,10 +12,13 @@ if parent_dir not in sys.path:
     sys.path.append(parent_dir)
 
 try:
-    from hyperspectral_pipelines import LoadHyperspectralImage
-except ImportError as e:
-    print(f"Error: Could not import LoadHyperspectralImage. Detail: {e}")
-    sys.exit(1)
+    from hyperspectral_image_reader.hyperspectral_pipelines import LoadHyperspectralImage
+except ImportError:
+    try:
+        from hyperspectral_pipelines import LoadHyperspectralImage
+    except ImportError as e:
+        print(f"Error: Could not import LoadHyperspectralImage. Detail: {e}")
+        sys.exit(1)
 
 DATASET_EXTENSIONS = {
     # .mat / .h5 formats

hyperspectral_image_reader/utils/__init__.py

@@ -0,0 +1,4 @@
+from . import ds_load
+from .hsi3d import get_image_loader
+
+__all__ = ['ds_load', 'get_image_loader']

hyperspectral_image_reader/utils/ds_load.py

@@ -0,0 +1,229 @@
+'''
+************************************************************************
+Copyright 2020 Institute of Theoretical and Applied Informatics, 
+Polish Academy of Sciences https://www.iitis.pl
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+************************************************************************
+
+HSI blood classification dataset by M. Romaszewski, P.Glomb, M. Cholewa, A. Sochan 
+Institute of Theoretical and Applied Informatics, Polish Academy of Sciences (ITAI PAS) https://www.iitis.pl
+Dataset DOI: 10.5281/zenodo.3984905
+
+HyperBlood API
+Basic loader for dataset files
+
+Warning:
+    * By default, data is cleared by removing noisy bands and broken line in the image. 
+    * Note that the 'F(2k)' image was captured with different camera. Its bands were interpolated 
+    to match remaining images. However, due to spectral range differences between cameras, it has 
+    less bands. After cleaning (default) all images have the same matching 113 bands. 
+
+NOISY_BANDS_INDICES = np.array([0,1,2,3,4,48,49,50,121,122,123,124,125,126,127])
+
+@author: mromaszewski@iitis.pl
+'''
+import warnings
+warnings.filterwarnings("ignore")
+import unittest
+import spectral.io.envi as envi
+import numpy as np
+import matplotlib.pyplot as plt
+# from geotiff import GeoTiff
+from osgeo import gdal
+gdal.PushErrorHandler('CPLQuietErrorHandler')
+# from osgeo import gdal_array
+
+IMAGES = ['A(1)','B(1)','C(1)','D(1)','E(1)','E(7)','E(21)','F(1)','F(1a)','F(1s)','F(2)','F(2k)','F(7)','F(21)'] 
+
+#change this to your DS location
+# PATH_DATA = '../'
+
+#------------------------ DATA LOADING ------------------------------------
+
+def get_data(name,remove_bands=True,clean=True):
+    """
+    Returns HSI data from a datacube
+    
+    Parameters:
+    ---------------------
+    name: name
+    remove_bands: if True, noisy bands are removed (leaving 113 bands)
+    clean: if True, remove damaged line
+    
+    Returns:
+    -----------------------
+    data, wavelenghts as numpy arrays (float32)
+    """
+    # filename = "{}data/{}".format(PATH_DATA,name)
+    hsimage = gdal.Open(name)
+    hsimage = hsimage.ReadAsArray()
+    wavs = np.asarray([376.8200 , 381.7583 , 386.7018 , 391.6505 , 396.6044 , 401.5636 , 406.5280 , 411.4977 , 416.4725 , 421.4525 , 426.4379 , 431.4284 , 436.4241 , 441.4251 , 446.4313 , 451.4427 , 456.4594 , 461.4813 , 466.5084 , 471.5408 , 476.5783 , 481.6211 , 486.6691 , 491.7224 , 496.7808 , 501.8445 , 506.9134 , 511.9876 , 517.0670 , 522.1515 , 527.2413 , 532.3364 , 537.4367 , 542.5422 , 547.6529 , 552.7689 , 557.8901 , 563.0164 , 568.1481 , 573.2849 , 578.4270 , 583.5743 , 588.7269 , 593.8846 , 599.0476 , 604.2158 , 609.3892 , 614.5679 , 619.7518 , 624.9409 , 630.1353 , 635.3348 , 640.5396 , 645.7496 , 650.9649 , 656.1853 , 661.4111 , 666.6420 , 671.8782 , 677.1195 , 682.3661 , 687.6179 , 692.8750 , 698.1372 , 703.4047 , 708.6775 , 713.9554 , 719.2386 , 724.5271 , 729.8207 , 735.1196 , 740.4236 , 745.7329 , 751.0475 , 756.3672 , 761.6923 , 767.0225 , 772.3580 , 777.6987 , 783.0445 , 788.3956 , 793.7520 , 799.1135 , 804.4803 , 809.8524 , 815.2296 , 820.6121 , 825.9998 , 831.3927 , 836.7909 , 842.1942 , 847.6028 , 853.0167 , 858.4358 , 863.8600 , 869.2896 , 874.7243 , 880.1642 , 885.6095 , 891.0598 , 896.5155 , 901.9764 , 907.4425 , 912.9138 , 918.3903 , 923.8721 , 929.3591 , 934.8514 , 940.3488 , 945.8514 , 951.3594 , 956.8725 , 962.3909 , 967.9144 , 973.4432 , 978.9773 , 984.5165 , 990.0610 , 995.6107 , 1001.1656 , 1006.7258 , 1012.2913 , 1017.8618 , 1023.4377 , 1029.0188 , 1034.6050 , 1040.1965 , 1045.7932]) 
+    data = np.asarray(hsimage[:,:,:],dtype=np.float32).transpose(1,2,0) # (520, 696, 128)
+    
+    #removal of damaged sensor line
+    fname = name.split('/')[-1].replace('.tif','')
+    if clean and fname!='F_2k':
+        data = np.delete(data,445,0)
+    
+    
+    if not remove_bands:
+        return data,wavs
+    return data[:,:,get_good_indices(fname)],wavs[get_good_indices(fname)] 
+
+def get_anno(name,remove_uncertain_blood=True,clean=True):
+    """
+    Returns annotation (GT) for data files as 2D int numpy array
+    Classes:
+    0 - background
+    1 - blood
+    2 - ketchup
+    3 - artificial blood
+    4 - beetroot juice
+    5 - poster paint
+    6 - tomato concentrate
+    7 - acrtylic paint
+    8 - uncertain blood
+    
+    Parameters:
+    ---------------------
+    name: name
+    clean: if True, remove damaged line
+    remove_uncertain_blood: if True, removes class 8 
+    
+    Returns:
+    -----------------------
+    annotation as numpy 2D array 
+    """
+    name = convert_name(name)
+    filename = "{}anno/{}".format(PATH_DATA,name)
+    anno = np.load(filename+'.npz')['gt']
+    #removal of damaged sensor line
+    if clean and name!='F_2k':
+        anno = np.delete(anno,445,0)
+    #remove uncertain blood + technical classes
+    if remove_uncertain_blood:
+        anno[anno>7]=0 
+    else:
+        anno[anno>8]=0
+           
+    return anno    
+
+
+#------------------------ UTILITY ------------------------------------
+
+
+def get_good_indices(name=None):
+    """
+    Returns indices of bands which are not noisy
+
+    Parameters:
+    ---------------------
+    name: name
+    Returns:
+    -----------------------
+    numpy array of good indices         
+    """
+    name = convert_name(name)
+    if name!='F_2k':
+        indices = np.arange(128)
+        indices = indices[5:-7]
+    else:
+        indices = np.arange(116)    
+    indices=np.delete(indices,[43,44,45])
+    return indices
+
+def convert_name(name):
+    """
+    Ensures that the name is in the filename format
+    Parameters:
+    ---------------------
+    name: name
+    
+    Returns:
+    -----------------------
+    cleaned name
+    """
+    name = name.replace('(','_')
+    name = name.replace(')','')
+    return name
+
+
+
+def get_rgb(data,wavelengths,gamma=0.7,vnir_bands=[600, 550, 450]):
+    """
+   Treturns an (over)simplified RGB visualization of HSI data
+    
+    Parameters:
+    ---------------------
+    data: data cube as nparray
+    annotation: wavelengths - band wavelenghts
+    gamma: gamma correction value
+    vnir_bands: bands used for RGB
+    
+    Returns:
+    -----------------------
+    rgb image as numpy array     
+    """
+    assert data.shape[2]==len(wavelengths)
+    max_data = np.max(data)
+    rgb_i = [np.argmin(np.abs(wavelengths - b)) for b in vnir_bands]
+    ret = data[:,:,rgb_i].copy()/max_data
+
+    if gamma!=1.0:
+        for i in range(3):
+            ret[:,:,i]=np.power(ret[:,:,i],gamma)
+    
+    return ret 
+
+class LoadTest(unittest.TestCase):
+    def test_load(self):
+        """
+        test image loading
+        """
+        for name in IMAGES:
+            data,wavelengths = get_data(name,remove_bands=True)
+            anno = get_anno(name)
+            self.assertEqual(data.shape[2],113)
+            self.assertEqual(data.shape[2],wavelengths.shape[0])
+            rgb = get_rgb(data,wavelengths)
+            plt.subplot(1,2,1)
+            plt.imshow(rgb,interpolation='nearest')
+            plt.subplot(1,2,2)
+            plt.imshow(anno,interpolation='nearest')
+            plt.show()
+            plt.close()
+            
+
+    def dis_test_indices(self):    
+        '''
+        Ensure F_2k is loaded correctly 
+        '''
+        _,wavs = get_data('F_2k',remove_bands=False)
+        assert 619.7518 in wavs 
+        _,wavs = get_data('F_2k',remove_bands=True)
+        assert 619.7518 not in wavs
+        _,wavs2 = get_data('F_1',remove_bands=True)
+        assert np.sum(wavs-wavs2)==0
+        
+        data,wavelengths = get_data('F_1',remove_bands=False)
+        self.assertEqual(data.shape[2],128)
+        self.assertEqual(data.shape[2],wavelengths.shape[0])
+        data,wavelengths = get_data('F_2k',remove_bands=False)
+        self.assertEqual(data.shape[2],116)
+        self.assertEqual(data.shape[2],wavelengths.shape[0])
+        anno = get_anno('F_1')
+        
+        
+if __name__ == '__main__':
+    unittest.main()

hyperspectral_image_reader/utils/hsi3d.py

@@ -0,0 +1,112 @@
+import os
+import os.path
+import torch
+import numpy as np
+import pandas
+from glob import glob
+import random
+from collections import OrderedDict
+import cv2
+import numpy as np
+
+def X2Cube(img, cellSize=4):
+
+    B = [cellSize, cellSize]
+    skip = [cellSize, cellSize]
+    # Parameters
+    M, N = img.shape
+    col_extent = N - B[1] + 1
+    row_extent = M - B[0] + 1
+
+    # Get Starting block indices
+    start_idx = np.arange(B[0])[:, None] * N + np.arange(B[1])
+
+    # Generate Depth indeces
+    didx = M * N * np.arange(1)
+    start_idx = (didx[:, None] + start_idx.ravel()).reshape((-1, B[0], B[1]))
+
+    # Get offsetted indices across the height and width of input array
+    offset_idx = np.arange(row_extent)[:, None] * N + np.arange(col_extent)
+
+    # Get all actual indices & index into input array for final output
+    out = np.take(img, start_idx.ravel()[:, None] + offset_idx[::skip[0], ::skip[1]].ravel())
+    out = np.transpose(out)
+    DataCube = out.reshape(M//cellSize, N//cellSize, cellSize*cellSize)
+
+    hsi_min = np.min(DataCube, axis=(0,1), keepdims=True)
+    hsi_max = np.max(DataCube, axis=(0,1), keepdims=True)
+    hsi_normed = (DataCube - hsi_min) / ((hsi_max - hsi_min) + 1e-6) * 255
+    hsi_normed = hsi_normed.astype(np.uint8)
+    return hsi_normed
+
+
+def X2Cube_fast(img, cellSize=4):
+    M, N = img.shape
+    if (M % cellSize) != 0 or (N % cellSize) != 0:
+        raise ValueError(f"Image shape {(M, N)} must be divisible by cellSize {cellSize}")
+
+    cubes = (
+        img.reshape(M // cellSize, cellSize, N // cellSize, cellSize)
+        .swapaxes(1, 2)
+        .reshape(M // cellSize, N // cellSize, cellSize * cellSize)
+    )
+
+    hsi_min = np.min(cubes, axis=(0, 1), keepdims=True)
+    hsi_max = np.max(cubes, axis=(0, 1), keepdims=True)
+    hsi_normed = (cubes - hsi_min) / ((hsi_max - hsi_min) + 1e-6) * 255
+    return hsi_normed.astype(np.uint8)
+
+
+def get_image_loader(img_file: str, cellSize=-1) -> np.array:
+    if cellSize > 0:
+        img = cv2.imread(img_file, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
+        img = X2Cube(img, cellSize)
+    elif cellSize == -1:
+        img = cv2.imread(img_file, cv2.IMREAD_COLOR)
+    else:
+        raise Exception
+    return img
+
+
+def _get_frames(self, seq_id, frame_ids):
+    frame_list = []
+    for frame_id in frame_ids:
+        frame_name = '%04d.png' % (frame_id+1)
+        image_path = os.path.join(self.RGBHDataRootDir, self.sequence_list[seq_id], 'img', frame_name)
+        if self.cur_modalName == 'NIR':
+            hsi_image = self.get_image_loader(image_path, cellSize=5)
+        elif self.cur_modalName == 'VIS':
+            hsi_image = self.get_image_loader(image_path, cellSize=4)
+        elif self.cur_modalName == 'RedNIR':
+            hsi_image = self.get_image_loader(image_path, cellSize=4)
+            hsi_image = hsi_image[:,:,:-1]
+
+        frame_name = '%04d.jpg' % (frame_id+1)
+
+        image_path = os.path.join(self.RGBHDataRootDir.replace('HSI-'+self.cur_modalName, 'HSI-'+self.cur_modalName+'-FalseColor'), self.sequence_list[seq_id], 'img', frame_name)
+        hsi_fc_image = self.get_image_loader(image_path, cellSize=-1)
+        frame_list.append(np.concatenate((hsi_fc_image, hsi_image), axis=2))
+
+    return frame_list 
+
+
+def get_frames(self, seq_id, frame_ids, anno=None):
+    frame_list = self._get_frames(seq_id, frame_ids)
+
+    if anno is None:
+        anno = self.get_sequence_info(seq_id)
+
+    anno_frames = {}
+    for key, value in anno.items():
+        if key == 'seq_belong_mask':
+            continue
+        anno_frames[key] = [value[f_id, ...].clone() for f_id in frame_ids]
+
+    object_meta = OrderedDict({'object_class_name': None,
+                                'motion_class': None,
+                                'major_class': None,
+                                'root_class': None,
+                                'motion_adverb': None})
+
+    return frame_list, anno_frames, object_meta
+