README.md

---
license: cc-by-nc-4.0
library_name: flexray
pipeline_tag: image-segmentation
datasets:
  - VictorButoi/flexray-data
tags:
  - pytorch
  - medical-image-segmentation
  - x-ray
  - radiograph
  - anatomy
  - flexray
---

# FleXray: `VictorButoi/flexray`

- Website and in-browser demo: [flexray.csail.mit.edu](https://flexray.csail.mit.edu/)
- Code: [github.com/VictorButoi/FleXray](https://github.com/VictorButoi/FleXray)
- Data: [`VictorButoi/flexray-data`](https://huggingface.co/datasets/VictorButoi/flexray-data)
- Tutorial: [Colab notebook](https://colab.research.google.com/drive/1jMBoOyV8PkRThHi3i6QIMjolmNoRE0cD)
- Paper: [FleXray: Universal Clinical X-ray Segmentation](https://arxiv.org/abs/2609.26756)

FleXray is a single 2D UNet that segments anatomy from standard radiographs
across body regions, projections, and acquisition settings. It predicts 60
anatomical structures (plus background) as independent sigmoid channels at
256 x 256 resolution.

This repository holds the **flagship model** and the four sibling models of
the FleXray ensemble, one bundle per `members/` subfolder (see
[Repository layout](#repository-layout)). `flexify` and
`FleXraySegmenter.from_pretrained` load the flagship by default.

## Quick start

```bash
python -m pip install flexray
flexify --input ./image.png --output-dir ./predictions
```

```python
from fxr.inference import FleXraySegmenter

segmenter = FleXraySegmenter.from_pretrained("VictorButoi/flexray")
prediction = segmenter.predict("./image.png", threshold=0.5)
prediction.masks          # uint8, BxCxHxW thresholded masks
prediction.probabilities  # float32, BxCxHxW sigmoid probabilities
prediction.logits         # float32, BxCxHxW raw scores
```

`flexify` writes `<name>_masks.npy`, `<name>_probabilities.npy`, and
`<name>_logits.npy` per image, each shaped `CxHxW`. The Python API keeps
the batch dimension (`B=1` for a single image). Channel order follows
`label_schema.json`.
Pass `--binary LABEL` (for example `--binary femurs`) to write one label. See
[docs/inference.md](https://github.com/VictorButoi/FleXray/blob/main/docs/inference.md)
for the full CLI and Python API.

## The FleXray ensemble

The flagship was trained with a 0.375 FluXray proportion in the training mix.
Four sibling models share its architecture, label schema, preprocessing, and
training recipe and differ only in that proportion:

| Subfolder | FluXray proportion | Role |
| --- | --- | --- |
| `members/flux000` | 0.0 | ensemble member |
| `members/flux025` | 0.25 | ensemble member |
| **`members/flux0375`** | **0.375** | **flagship (loaded by default)** |
| `members/flux050` | 0.5 | ensemble member |
| `members/flux075` | 0.75 | ensemble member |

`ensemble.json` at the repository root lists the flagship and the members.
Because the members share one output space, they are averaged in probability
space:

```bash
flexify --ensemble --tta-samples 16 --input ./image.png --output-dir ./predictions
flexify --subfolder members/flux000 --input ./image.png --output-dir ./predictions
```

```python
segmenter = FleXraySegmenter.from_pretrained("VictorButoi/flexray", ensemble=True)
prediction = segmenter.predict("./image.png", tta_samples=16)

member = FleXraySegmenter.from_pretrained(
    "VictorButoi/flexray", subfolder="members/flux000"
)
```

The website demo exposes the same choices as quality modes: **Low** runs the
flagship once, **Normal** runs the flagship with 8-pass TTA, **High** runs the
five-model ensemble once, and **X-High** runs the ensemble with 8-pass TTA.
The members are also listed in
[MODEL_ZOO.md](https://github.com/VictorButoi/FleXray/blob/main/MODEL_ZOO.md).

## Test-time augmentation

The reported results use 16 passes per model (`--tta-samples 16` or
`predict(..., tta_samples=16)`). The browser demo uses 8 passes per model in
Normal and X-High modes; its current settings are published in the
[demo manifest](https://flexray.csail.mit.edu/demo/demo_manifest.json).

`tta_samples=N` runs one un-augmented pass plus `N - 1` randomly augmented
passes and averages their sigmoid probabilities, then converts that mean
back to logits. The package and browser implement the released `tta_v3`
chain in this order:

| Transform | Probability | Parameters |
| --- | --- | --- |
| Horizontal flip | 0.5 | Exactly inverted on the prediction before averaging |
| Invert intensities | 0.5 | `1 - image` |
| CLAHE | 0.1 | Clip limit 1.0-2.0; 8 x 8 grid |
| Gamma | 0.25 | Gamma 0.9-1.1; gain 0.9-1.1; mutually exclusive with CLAHE |
| Contrast | 0.25 | Multiply intensities by 0.7-1.3 and clamp to [0, 1] |
| Sharpness | 0.5 | Factor 0.7-1.3 |
| Gaussian noise | 0.25 | Standard deviation 0.01 |

The flip is the only geometric transform; intensity transforms are not
inverted. The CLAHE/gamma branch leaves the image unchanged with probability
0.65. See the
[Python implementation](https://github.com/VictorButoi/FleXray/blob/main/src/fxr/inference/tta.py)
and [browser implementation](https://flexray.csail.mit.edu/demo/tta.js).

Use `predict(..., tta_samples=16, seed=42)` to reproduce the Python
augmentation draws without changing the global torch RNG. With no seed,
draws use the global torch RNG. The browser uses its own random-number
source, so matching augmentation settings do not imply identical random views.

With an ensemble, each view is drawn once and run through every member.
`M` models and `N` passes therefore require `M x N` forward passes: 80 for
the five-model ensemble at N=16, or 40 for the browser's X-High mode at N=8.
`tta_samples<=1` disables augmentation.

## Input contract

`preprocessing.json` is applied automatically by the public loaders:

- grayscale input (RGB is converted), any 8-bit or 16-bit PNG / JPEG / TIFF / BMP
- per-image percentile min-max normalization to `[0, 1]` (0.5th / 99.5th
  percentiles, `eps = 1e-8`)
- zero-pad to a square, then resize to 256 x 256
- outputs are `multilabel` sigmoid probabilities; masks use threshold 0.5

Outputs are at the 256 x 256 model resolution; the CLI and Python API do not
resample back to the original image size.

## Output labels

FleXray outputs 60 foreground masks (61 channels including `background`). The
broader dataset/evaluation protocol also recognizes aggregate `lumbar_spine` and
`thoracolumbar_spine` annotations; these are evaluated by combining the relevant
per-vertebra outputs and are not checkpoint channels. Channel order is stored in
each bundle's `label_schema.json`.

- **Skull / shoulder girdle:** skull, scapulae, clavicles
- **Upper limb:** humeri, radii, ulnae, carpals, metacarpals, phalanges
- **Lower limb:** femurs, patellae, tibiae, fibulae, tarsals, metatarsals, toes
- **Thorax:** rib_1 - rib_12, sternum
- **Spine:** vertebra_c1 - c7, t1 - t12, l1 - l5, sacrum
- **Pelvis:** hips
- **Soft tissue:** lungs, heart, liver, spleen, kidneys

Paired structures are merged (for example `femurs` covers both sides);
laterality is not predicted.

## Architecture

`fxr.models.UNet`, 2D, 1 input channel, 61 output channels; filters
`[64, 128, 256, 512, 512, 720, 1024]`, 3 convolutions per block, residual
blocks with instance norm, `align_corners=True` upsampling. The full
architecture is in each bundle's `config.yml`.

## Training data

The models were trained on three source types unified under the FleXray label
protocol. For training mixture proportions, please refer to the paper:

- **Real X-ray masks:** HandBones, FootBones, MURA forearm, and MURA humerus,
  with our own annotations.
- **Generated FluXray images:** digitally reconstructed radiographs from the
  MOOSE CTs, generatively edited toward real X-ray appearance, with exact
  overlapping masks for every protocol structure.
- **Online CT->DRR rendering:** MOOSE / ENHANCE-PET 1.6k, Shoulder-CT, HaN-Seg,
  PedsCT, RSNA cervical-spine fracture CTs, and ElbowCT, rendered to DRRs at
  random poses during training with per-label attenuation jitter.

Training used AdamW (lr 3e-4, cosine schedule), a Dice + binary cross-entropy
loss routed per dataset (partially labeled sources ignore unlabeled channels),
and separate augmentation presets for CT-derived and X-ray inputs. The exact
recipe is `fxr/configs/training/base.yml` in the code release.

Every dataset's license, redistribution status, and download pointer is
documented in the
[`VictorButoi/flexray-data`](https://huggingface.co/datasets/VictorButoi/flexray-data)
card. That repository ships the real X-ray sources whose licenses permit
redistribution as image/mask pairs with packaging manifests, the MURA masks,
and the FluXray database.

## Evaluation

FleXray was evaluated on eight real-radiograph datasets held out from training
(DarwinCVD19, DeepFluoro, ElbowLat, HipRay, LowerLimbs, RAM-W600, PedsTorso,
and VinDr-Rib), spanning lungs, ribs, peripheral bones, spine, and pelvis.
Against supported generalist baselines (FluoroSAM, TotalSegmentator2D, PAXray),
FleXray performs best or ties on all eight datasets, with significant
improvements on seven and no statistically detectable difference from PAXray
on VinDr-Rib. Per-dataset numbers and confidence intervals are in the paper;
the benchmark figure is on
the [project website](https://flexray.csail.mit.edu/#results).
Evaluation ignores ground-truth labels covering less than 0.1% of the image.

## Intended use and limitations

Research use only. FleXray is **not a medical device** and is not cleared for
clinical diagnosis, treatment planning, or patient-care decisions.

- Targets conventional radiographs; dental and mammographic images are out of
  scope.
- Predicts anatomy, not pathology.
- No laterality (left/right) and no uncertainty estimates.
- Performance on acquisition settings, views, or populations far from the
  training sources has not been validated.

## Repository layout

- `README.md`: this card.
- `ensemble.json`: the `flagship` subfolder and the `members` list with their
  FluXray proportions.
- `members/<name>/model.safetensors`: exported model weights.
- `members/<name>/config.yml`: architecture and protocol config consumed by
  `from_pretrained`.
- `members/<name>/label_schema.json`: ordered output labels.
- `members/<name>/preprocessing.json`: public preprocessing contract.
- `members/<name>/checksums.json`: SHA256 checksums of the bundle files.
- `members/<name>/onnx/flexray-<name>-256-fp16.onnx`: fp16 ONNX export
  (opset 18, sigmoid baked in) used by the in-browser demo; parity-checked
  against the PyTorch weights by `scripts.release.export_web_demo` in the
  release tooling.

## Licenses

- Code: MIT
- Weights: CC-BY-NC-4.0

## Citation

```bibtex
@misc{butoi2026flexray,
      title={FleXray: Universal Clinical X-ray Segmentation},
      author={Victor Ion Butoi and Vivek Gopalakrishnan and John V. Guttag and Adrian V. Dalca and Neel Dey},
      year={2026},
      eprint={2609.26756},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.26756},
}
```

Please also cite the source datasets listed in the
[`flexray-data`](https://huggingface.co/datasets/VictorButoi/flexray-data) card
for any dataset you use.