Why "red apple* would trigger safety filter?

#23
by wfen - opened

Does Ideogram make this model unusable intentionally?

are you using natural prompts with ideogram or json format prompts?

Here is what I observed:


  1. Safety filter fires randomly on completely safe prompts

Observation
The built-in safety filter is non-deterministic and false-positives
frequently. The same 100% benign prompt:

  "A red apple on a wooden table, soft natural light"

was submitted 7 times and returned the grey "Image blocked by safety
filter" card 6 times (~86% false-positive rate):

  quality=medium  -> blocked
  quality=high    -> blocked
  run 1           -> blocked
  run 2           -> blocked
  run 3           -> blocked
  run 4           -> PASSED (real image)
  run 5           -> blocked

For each run I spend nearly 1 mins, and this safety filter only shows up at the very end, ensure the user really has spent their inference cost.

If anyone else would be looking into this thread in the future:

Safety filter in the model's weights (gray "Image blocked by safety filter" screen) is kinda... broken. It triggers on regular plain prompts with nothing remotely suspicious in them.

If you'll keep on hitting the wall, you can just add some extra details to the prompt - this usually resolves the problem. Alternatively: use synonyms for some of the words in your prompt or randomize seed.

Alternatively you can use a skill I have made for prompting Ideogram 4.

BTW: the way this safety filter is built makes it a two-edged sword: one one hand it has a ton of false positives, but on the other: it's almost impossible to abliterate without crippling the model

Sign up or log in to comment