Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Sabith64 
posted an update 12 days ago
Post
101
We’re releasing Lunara Art Eval, an art evaluation dataset for studying aesthetic quality, emotional resonance and content integrity in image generation.

It contains 8,000 generated images across 1,000 shared prompts, covering diverse artistic styles. The eight models are Qwen-Image, AuraFlow, GPT-Image-1 Mini, Stable Diffusion 3.5 Large Turbo, HiDream-I1 Fast, FLUX.2-Klein-4B, Z-Image-Turbo and Moonworks Lunara.

Each image includes scores on the three dimensions, allowing comparisons across models using the same prompts and evaluator.

Dataset: moonworks/lunara-art-eval
Paper: https://arxiv.org/abs/2609.22272

The paper also introduces Lunara’s Diffusion Mixture Transformer architecture, with fewer than 10B active parameters, and CAT, an active-learning-inspired training algorithm that iteratively updates the training distribution.

We’re a new lab focused on model architectures and training algorithms, working closely with artists and creators. Our earlier public datasets are:

- [Lunara Aesthetic I]( moonworks/lunara-aesthetic): art images with human-refined prompts and region, style and topic labels.
- [Lunara Aesthetic II]( moonworks/lunara-aesthetic-image-variations): original images paired with contextual variations.

We’d appreciate feedback from people working on image generation and evaluation, particularly on where automated scores with your judgment.

The aesthetic headline sits inside the noise of your own judge.

I pulled all 8,000 rows of lunara-art-eval and paired them by prompt. Table 1 reproduces exactly, means and win rates.

Lunara vs GPT Image 1 Mini on Aesthetic Quality:

  • mean gap +0.016 (8.473 vs 8.457)
  • paired bootstrap over the 1,000 prompts, 95% CI [-0.038, +0.070]
  • head to head: 506 wins, 1 tie, 493 losses
  • Lunara comes out #1 in 73% of resamples

The other two axes are not close. GPT Image 1 Mini leads Emotional Resonance by 0.24 [0.18, 0.31] and Content Integrity by 0.67 [0.59, 0.74].

One thing about the judge itself. Appendix A says ties are allowed, do not force a winner. Yet 916 of 1,000 prompts give all eight images distinct aesthetic scores. Shuffle each model's scores across prompts (same marginals, independent ratings) and a prompt averages 7.00 distinct values. Observed: 7.91. Within a prompt, the side-by-side scores behave more like a ranking than independent ratings.

Small one: win rate counts a tie at the top as a win for every tied model, which is why the Content column sums to 103.4%.

Did you run the judge twice on any subset? The flip rate on Lunara vs GPT Image 1 Mini would tell you whether 506 vs 493 is signal.

This looks like a really interesting resource for understanding how image-generation models differ in artistic quality and emotional impact. I especially like the idea of combining automated scores with human judgment, since creativity can be quite subjective. It would be interesting to compare the results with tools from Adobe https://adobe.pissedconsumer.com/customer-service.html and see how different models perform in real-world creative workflows.