| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: image-classification |
| tags: |
| - medical-imaging |
| - x-ray |
| - fracture-detection |
| - contrastive |
| - clm |
| - research-prototype |
| datasets: |
| - FracAtlas |
| --- |
| |
| # Second Eye: CLM image state head for fracture detection (FracAtlas, C-RADIOv2-H) |
|
|
| > **Research prototype, not a medical device.** Not validated for clinical use. Do not use it to |
| > diagnose or treat anyone. |
|
|
| A small trainable image **state head** for [CLM (Contrastive Language Models)](https://github.com/Contrastive-LM/CLM). |
| Frozen `nvidia/C-RADIOv2-H` image features (the vision tower of NVIDIA Nemotron Nano 2 VL) go |
| through this head into CLM's 512-d action space. There they are scored by `scale · cos` against |
| CLM's frozen **action tower** (Qwen3-8B last-token pooling + the released |
| [CLM-v0.1-8B](https://huggingface.co/Contrastive-LM/CLM-v0.1-8B) action head) embedding the |
| answer options: |
|
|
| - Question: *"Is there a bone fracture in this X-ray?"* |
| - Options: *"Yes, there is a fracture."* / *"No, there is no fracture."* |
|
|
| Code, training and evaluation: [https://github.com/HarvinderBhullar/second-eye/tree/hackathon](https://github.com/HarvinderBhullar/second-eye/tree/hackathon) (see `docs/VISION.md`). |
|
|
| ## Files |
|
|
| | file | | |
| |---|---| |
| | `model.safetensors` | head weights (incl. train-set input standardisation) and `options`: the 2 frozen option vectors it was trained against | |
| | `config.json` | architecture (3840 → 1024 → 512, GELU), encoder, tile size, pooling, question and options | |
| | `calibration.json` | Platt scaling (`a`, `b`) and two thresholds on calibrated p(fracture), all fit on the validation split | |
|
|
| ## Usage |
|
|
| ```bash |
| git clone -b hackathon https://github.com/HarvinderBhullar/second-eye && cd second-eye |
| scripts/vision/step1_env.sh |
| python -m clm.vision.serve --model harvinderbhullar/second-eye --port 8701 |
| curl -s -X POST localhost:8701/predict -F image=@xray.jpg |
| ``` |
|
|
| Input: one `512×512` RGB tile per image (greyscale X-rays converted to |
| RGB, bicubic resize), RADIO summary vector, L2-normalised. Calibrated probabilities and the |
| decision (`p ≥ 0.0675`, the val operating point with sensitivity ≥ 0.90) are |
| valid only for the two trained options. Reworded options keep the ranking but not the calibration. |
|
|
| ## Evaluation |
|
|
| FracAtlas after integrity cleaning (truncated JPEGs, uniform bottom fills and byte duplicates |
| removed): 3,938 images, 70/15/15 split stratified on fracture, seed 42. Every choice was made on |
| validation. The test split (591 images, 104 fractured) was evaluated |
| once. AUROC CI: 1,000-sample stratified bootstrap. |
|
|
| | method | test AUROC [95% CI] | sens / spec @ val ≥0.90-sens threshold | ECE raw → Platt | |
| |---|---:|---:|---:| |
| | CLM head · radio_summary | 0.925 [0.891, 0.953] | 0.913 / 0.723 | 0.062 → 0.020 | |
| | Linear probe · radio_summary | 0.893 [0.850, 0.931] | 0.885 / 0.612 | 0.085 → 0.041 | |
| | Resolution only (leakage check) | 0.616 [0.577, 0.656] | 0.933 / 0.269 | 0.323 → 0.025 | |
| | (a) random option vectors · radio_summary | 0.913 [0.878, 0.945] | 0.875 / 0.733 | 0.075 → 0.027 | |
| |
| **Limitations.** |
| - With two fixed options, random option vectors do about as well as the CLM option embeddings |
| (see the last row). The head is effectively a classifier, and its value is the CLM interface, |
| not accuracy. |
| - The head is ahead of a linear probe in point estimate, but the CIs overlap. |
| - It is trained on one public dataset from a limited set of sources, with no patient IDs and no |
| external validation. |
| - Image size alone reaches about 0.62 AUROC on this data (a body-region confounder). |
| |
| ## Requirements and licences |
| |
| - These head weights: Apache-2.0. |
| - Using them requires [`nvidia/C-RADIOv2-H`](https://huggingface.co/nvidia/C-RADIOv2-H) (NVIDIA, |
| under its own licence) for the features, and Qwen3-8B plus the CLM action head only for custom |
| options. |
| - Training data: **FracAtlas**, Abedeen et al., *Scientific Data* (2023), |
| [doi:10.6084/m9.figshare.22363012](https://doi.org/10.6084/m9.figshare.22363012), CC BY 4.0. |
| - Built on [Contrastive-LM/CLM](https://github.com/Contrastive-LM/CLM) (Apache-2.0) at the Nebius × NVIDIA hackathon, |
| Berlin, October 2026. |
| |
