Model card: task collage, editing and generation prompts, LoRA adapters

.gitattributes

@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 assets/ovb_collage.jpg filter=lfs diff=lfs merge=lfs -text
+assets/ovb_collage.png filter=lfs diff=lfs merge=lfs -text
+assets/ovb_overview.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -24,12 +24,21 @@ tags:
 </p>
 
 <p align="center">
-<img src="assets/ovb_collage.jpg" width="100%" alt="Zero-shot OVB outputs on an in-the-wild photo: semantic, instance and referring segmentation, depth, surface normals, image editing and text-to-image generation, with the prompt below each output">
+<img src="assets/ovb_collage.png" width="100%" alt="Zero-shot OVB outputs on an in-the-wild photo: semantic, instance and referring segmentation, depth, surface normals, image editing and text-to-image generation, with the prompt below each output">
 </p>
 
 Outputs of the zero-shot 9B model on an in-the-wild photo, with the summarised prompt below each output.
 
-## Quickstart
+<p align="center">
+<img src="assets/ovb_overview.jpg" width="100%" alt="Overview of the Open Vision Banana recipe and analysis framework">
+</p>
+
+## 📦 Checkpoints
+
+- `final_models/{4B,9B}/{zeroshot,nonzeroshot}/`: the fine-tuned transformers for both base sizes. The zero-shot models are trained without the training splits of the evaluation benchmarks, and the non-zero-shot models include them. Each folder holds the intermediate `checkpoint-<step>` exports and the weight-averaged checkpoint used in the paper, each as a `transformer/` subfolder.
+- `ablation_models/`: one checkpoint per ablation setting of the paper, and the LoRA adapters described at the bottom of this page.
+
+## 🚀 Quickstart
 
 Install PyTorch with CUDA support first, then:
 
@@ -37,7 +46,7 @@ Install PyTorch with CUDA support first, then:
 pip install "diffusers>=0.37.0" transformers accelerate safetensors pillow
 ```
 
-The fine-tuned transformer is swapped into the [FLUX.2 [klein] base 9B](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B) pipeline. One model handles every task and the prompt selects it. This example runs semantic segmentation:
+The fine-tuned transformer is swapped into the [FLUX.2 [klein] base 9B](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B) pipeline. One model handles every task and the prompt selects it. This example runs <img src="assets/icons/semseg.png" width="26" alt=""> semantic segmentation:
 
 ```python
 import torch
@@ -79,11 +88,11 @@ output = pipe(
 output.resize(image.size, Image.NEAREST).save("output.png")
 ```
 
-### Quickstart prompts for the other tasks
+### 🎯 Quickstart prompts for the other tasks
 
 Replace `prompt` above with one of these.
 
-**Referring segmentation**
+<img src="assets/icons/refseg.png" width="26" alt=""> **Referring segmentation**
 
 ```python
 prompt = (
@@ -93,7 +102,7 @@ prompt = (
 )
 ```
 
-**Instance segmentation**
+<img src="assets/icons/instanceseg.png" width="26" alt=""> **Instance segmentation**
 
 ```python
 prompt = (
@@ -111,7 +120,7 @@ prompt = (
 )
 ```
 
-**Depth estimation**
+<img src="assets/icons/depth.png" width="26" alt=""> **Depth estimation**
 
 ```python
 prompt = (
@@ -124,7 +133,7 @@ prompt = (
 )
 ```
 
-**Surface normal estimation**
+<img src="assets/icons/normals.png" width="26" alt=""> **Surface normal estimation**
 
 ```python
 prompt = (
@@ -135,7 +144,7 @@ prompt = (
 )
 ```
 
-**Image editing**
+<img src="assets/icons/imgedit.png" width="26" alt=""> **Image editing**
 
 ```python
 prompt = (
@@ -143,7 +152,7 @@ prompt = (
 )
 ```
 
-**Text-to-image generation**
+<img src="assets/icons/generation.png" width="26" alt=""> **Text-to-image generation**
 
 Call the pipeline without `image` and give it a size:
 
@@ -168,12 +177,7 @@ These are quickstart prompts, one per task. The model was trained on many phrasi
 - **Image editing:** keep the `[TASK: IMAGE_EDITING]` tag and write the edit as a plain instruction.
 - **Generation:** a plain description without a tag and without an input image.
 
-## Model layout
-
-- `final_models/{4B,9B}/{zeroshot,nonzeroshot}/checkpoint-*/transformer`: fine-tuned transformers for both base sizes, loaded as in the quickstart with the matching base pipeline (`FLUX.2-klein-base-4B` or `-9B`).
-- `ablation_models/`: one checkpoint per ablation setting from the paper.
-
-## LoRA adapters
+## 🧩 LoRA adapters
 
 `ablation_models/f2kbase4b_lora_zeroshot_ablations/` contains the LoRA adapters of the paper's LoRA table, trained on FLUX.2 [klein] base 4B over the zero-shot data: ranks 64, 128 and 256 at 25K, 37.5K, 50K, 75K and 100K steps, and ranks 512 and 1024 at 37.5K steps. `manifest.md` lists every adapter. Each adapter loads into the unmodified 4B base pipeline: