Open Vision Banana 4B Seg 1-Step LoRA

This adapter makes a prompted foreground mask in one denoising step with the Open Vision Banana 4B non-zero-shot transformer. It was trained for 1,000 optimizer updates on 200 ReasonSeg training images, with rank 64, alpha 64, and a 512²-pixel training area cap. Inference uses one conditional transformer pass, an Euler update from sigma 1 to 0, and VAE decoding. Use CFG 1.0.

Benchmarks

The 1,000-update LoRA uses one denoising step and CFG 1.0. Training used 200 images from ReasonSeg's train split.

Evaluation Model Images Inference area cap Steps ↓ gIoU ↑ cIoU ↑ Cached s/image ↓
ReasonSeg held-out validation OVB 4B non-zero-shot · published baseline 200 1024² pixels 28 74.60 — —
ReasonSeg held-out validation One-step LoRA 200 1024² pixels 1 76.89 74.79 2.46
Open Images test · seed 42 One-step LoRA 200 512² pixels 1 73.79 80.53 0.78
Open Images test · seed 42 One-step LoRA 200 1024² pixels 1 73.96 79.35 2.31

Bold values mark the better result within each benchmark where both rows have a value. The ReasonSeg runs use the full separate val split and OVB's bundled Gemma 4 31B query rewrites. The +2.29 gIoU difference uses the published 28-step result as the reference. The paper does not report matching cIoU or runtime.

Both Open Images runs use the same 200 test photos, direct object-name prompts, and starting noise seeds. The area caps preserve aspect ratio without upscaling. Open Images compares the two resolutions with the same adapter. The 28-step baseline in the table applies to ReasonSeg.

gIoU is mean foreground IoU per image. cIoU pools the foreground intersection and union across images. Cached time includes denoising, output VAE decoding, and image conversion. It excludes prompt encoding, input-image encoding, and setup.

OVB paper, Tables 3 and 5 · OVB evaluation protocol · Open Images annotations

Open Images test examples

The same test photo and annotation are shown next to the two one-step masks. IoU labels are per image.

Tart resolution comparison

Photo by Anne-Lise Heinrichs · CC BY 2.0.

Jeans resolution comparison

Photo by Donald · CC BY 2.0.

Guacamole resolution comparison

Photo by Rachel Kramer Bussel · CC BY 2.0.

The panels resize the source photos and add captions, annotations, and predicted masks.

Load with Diffusers

This adapter uses the OVB 4B non-zero-shot transformer together with the FLUX.2 [klein] Base 4B pipeline components. For ReasonSeg-style questions, supply an explicit referring phrase such as “the black billiard ball.” The benchmark used the bundled Gemma query rewrites for that conversion.

import numpy as np
import torch
from PIL import Image
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel

transformer = Flux2Transformer2DModel.from_pretrained(
    "FunAILab/Open-Vision-Banana",
    revision="be5da66250cc385ff9c75ebbe4114e13ec305848",
    subfolder="final_models/4B/nonzeroshot/checkpoint-avg90000-92500-97500-100000_uniform/transformer",
    torch_dtype=torch.bfloat16,
)
pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-base-4B",
    revision="a3b4f4849157f664bdbc776fd7453c2783562f4d",
    transformer=transformer,
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights(
    "ApacheOne/Open-Vision-Banana-4B-Seg-1Step-Lora",
    weight_name="ovb_4b_reasonseg_one_step.safetensors",
    adapter_name="seg_one_step",
)
# Preserve the trained adapter precision used for the benchmark.
for layer in pipe.transformer.modules():
    if hasattr(layer, "lora_A") and "seg_one_step" in layer.lora_A:
        layer.lora_A["seg_one_step"].to(dtype=torch.float32)
        layer.lora_B["seg_one_step"].to(dtype=torch.float32)
pipe.to("cuda")

image = Image.open("input.jpg").convert("RGB")
expression = "the black billiard ball"
prompt = (
    f'[TASK: REFERRING_SEGMENTATION] Segment the region that best matches this '
    f'description: "{expression}". Paint the matching region white [255, 255, 255] '
    'and leave everything else black [0, 0, 0]. Output only the segmentation mask image.'
)
raw = pipe(
    image=image, prompt=prompt, num_inference_steps=1, guidance_scale=1.0,
    sigmas=[1.0], generator=torch.Generator(device="cuda").manual_seed(0),
).images[0]
foreground = np.asarray(raw.convert("RGB"), dtype=np.float32).mean(axis=-1) > 127.5
mask = Image.fromarray(foreground.astype(np.uint8) * 255)
mask.resize(image.size, Image.Resampling.NEAREST).save("mask.png")

The example uses white for a simple preview. Benchmark inference used OVB's seeded named-color prompts and nearest-color decoding.

License and attribution

This is an independent modification of Open Vision Banana. See LICENSE.md for the FLUX Non-Commercial License and NOTICE.txt for the attribution notice. Rights to use the model and this derivative are granted directly by Black Forest Labs under that license. Cite the Open Vision Banana paper and ReasonSeg / LISA when using these results.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ApacheOne/Open-Vision-Banana-4B-Seg-1Step-Lora

Adapter
(1)
this model

Paper for ApacheOne/Open-Vision-Banana-4B-Seg-1Step-Lora