File size: 3,430 Bytes
c645013 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
library_name: transformers
license: llama2
base_model:
- lmsys/vicuna-7b-v1.5
- facebook/dinov2-large
tags:
- falcon-x
- vision-language
- image-segmentation
- custom_code
language:
- en
---
# FALCON
Functional Assembly and Language for Compositional Reasoning in X-ray.
[Paper](https://arxiv.org/abs/2606.25701) ·
[Project page](https://yonathan-kiflom.github.io/FALCON/page/) ·
[Code](https://github.com/yonathan-kiflom/FALCON)
This repository contains the completed Stage-3 model, including Vicuna-7B-v1.5,
DINOv2-L/14, the trained RF-DETR segmentation detector, multimodal adapters,
and unmerged LoRA weights.
## Usage
Install the pinned dependencies from the code repository:
```bash
pip install 'falcon-x[train] @ git+https://github.com/yonathan-kiflom/FALCON.git'
```
Use `JonathanJMK/FALCON` or a local downloaded model directory. Authenticate
with `huggingface-cli login` when accessing the private repository. Review the
custom code before trusting it; pin a Hub commit with `revision=` for reproducible use.
```python
from transformers import AutoModel
model = AutoModel.from_pretrained(
"JonathanJMK/FALCON", trust_remote_code=True,
).to("cuda").eval()
result, masks = model.predict(
image="image.png", prompt="Describe the image.", max_new_tokens=256,
)
print(result["answer"])
```
Use `predict_segmentation` for grounding prompts and `predict_panoptic` for
panoptic prompts. Tokenization, image preprocessing, and prompt formatting are
included. The model supports one device; automatic multi-device dispatch and
quantized loading are not supported. The saved per-component precision is
preserved; do not cast the entire model to half precision.
Evaluate using the same model:
```bash
python -m falcon evaluate run --model JonathanJMK/FALCON \
--dataset /path/to/falcon-x --split test --tasks all \
--run-dir runs/falcon-evaluation --device cuda
```
## Scope and limitations
Trained on falcon-x for X-ray descriptions, questions, component
presence/completeness, instance grounding, and segmentation. Generated answers
and masks can be wrong; this is a research model, not a certified screening
system. Available structured heads are declared in `config.json`; untrained
risk and physical-link heads are not presented as predictions. Counterfactual
completeness does not establish real-world danger or physical connectivity.
The package preserves the trained architecture and LoRA for further research.
It is not an optimizer-state checkpoint, and loading with Transformers alone
does not make the repository's stage-training CLI a general fine-tuning tool.
Task metrics and evaluation limitations are documented in the code repository.
## Licenses
Vicuna is derived from Llama 2 and retains the
[Llama 2 Community License](https://huggingface.co/meta-llama/Llama-2-7b/blob/main/LICENSE.txt)
and [Acceptable Use Policy](https://huggingface.co/meta-llama/Llama-2-7b/blob/main/USE_POLICY.md).
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta
Platforms, Inc. All Rights Reserved.
[DINOv2](https://github.com/facebookresearch/dinov2/blob/main/LICENSE) and
[RF-DETR segmentation 1.5.2](https://github.com/roboflow/rf-detr/blob/1.5.2/LICENSE)
use Apache-2.0. FALCON code is Apache-2.0 (`LICENSE-code`); this does not
relicense upstream model weights. Retain `LICENSE-Llama-2`, `LICENSE-code` and
`Notice` when redistributing the full package.
|