Model card: task collage, editing and generation prompts, LoRA adapters

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/ovb_collage.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -11,8 +11,24 @@ tags:
 - referring-segmentation
 - depth-estimation
 - surface-normal-estimation
+- image-editing
+- text-to-image
 ---
 
+<h1 align="center">Open Vision Banana <img src="assets/ovb_logo.png" height="40" alt="">: A Fully Open Recipe for Generative Visual Perception</h1>
+
+<p align="center">
+<a href="https://openvisionbanana.github.io/"><img src="https://img.shields.io/badge/Project-Page-1f72b8.svg" alt="Project Page"></a>
+<a href="https://github.com/OpenVisionBanana/OpenVisionBanana"><img src="https://img.shields.io/badge/GitHub-Code-181717.svg?logo=github" alt="Code"></a>
+<a href="https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B"><img src="https://img.shields.io/badge/Base-FLUX.2%20%5Bklein%5D-8a2be2.svg" alt="Base model"></a>
+</p>
+
+<p align="center">
+<img src="assets/ovb_collage.jpg" width="100%" alt="Zero-shot OVB outputs on an in-the-wild photo: semantic, instance and referring segmentation, depth, surface normals, image editing and text-to-image generation, with the prompt below each output">
+</p>
+
+Outputs of the zero-shot 9B model on an in-the-wild photo, with the prompt below each output.
+
 ## Quickstart
 
 Install PyTorch with CUDA support first, then:
@@ -119,9 +135,60 @@ prompt = (
 )
 ```
 
+**Image editing**
+
+```python
+prompt = (
+    "[TASK: IMAGE_EDITING] Change the beach and ocean environment in the picture to a snowy mountain."
+)
+```
+
+**Text-to-image generation**
+
+Call the pipeline without `image` and give it a size:
+
+```python
+prompt = "an empty tram at a cobblestone stop on a rainy night"
+output = pipe(
+    prompt=prompt,
+    height=1024,
+    width=1024,
+    guidance_scale=3.5,
+    num_inference_steps=28,
+    generator=torch.Generator(device=device).manual_seed(0),
+).images[0]
+```
+
 These are quickstart prompts, one per task. The model was trained on many phrasings of each task, so you can adapt them:
 
 - **Semantic segmentation:** list any classes with an RGB colour each. Distinct, saturated colours work best. Pixels outside the listed classes come out black.
 - **Referring segmentation:** replace the description and the paint colour.
 - **Instance segmentation:** prompt one class at a time; replace every `car` in the prompt with any noun phrase. The left-to-right color ordering helps maintain greater separation between assigned colors. The model may not follow it exactly, which is fine: each instance only needs its own distinct colour.
 - **Depth and surface normals:** use the prompts as they are. Depth is relative inverse depth, with closer points brighter. Normals follow the encoding stated in the prompt.
+- **Image editing:** keep the `[TASK: IMAGE_EDITING]` tag and write the edit as a plain instruction.
+- **Generation:** a plain description without a tag and without an input image.
+
+## Model layout
+
+- `final_models/{4B,9B}/{zeroshot,nonzeroshot}/checkpoint-*/transformer`: fine-tuned transformers for both base sizes, loaded as in the quickstart with the matching base pipeline (`FLUX.2-klein-base-4B` or `-9B`).
+- `ablation_models/`: one checkpoint per ablation setting from the paper.
+
+## LoRA adapters
+
+`ablation_models/f2kbase4b_lora_zeroshot_ablations/` contains the LoRA adapters of the paper's LoRA table, trained on FLUX.2 [klein] base 4B over the zero-shot data: ranks 64, 128 and 256 at 25K, 37.5K, 50K, 75K and 100K steps, and ranks 512 and 1024 at 37.5K steps. `manifest.md` lists every adapter. Each adapter loads into the unmodified 4B base pipeline:
+
+```python
+import torch
+from diffusers import Flux2KleinPipeline
+
+pipe = Flux2KleinPipeline.from_pretrained(
+    "black-forest-labs/FLUX.2-klein-base-4B", torch_dtype=torch.bfloat16
+)
+pipe.load_lora_weights(
+    "FunAILab/Open-Vision-Banana",
+    subfolder="ablation_models/f2kbase4b_lora_zeroshot_ablations/r64/checkpoint-75000",
+    weight_name="pytorch_lora_weights.safetensors",
+)
+```
+
+Prompts and sampling settings are the same as above.