SVE-ASCII-Bench

Benchmark for evaluating ASCII art generation and understanding. From Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs.

Tasks and Files

The benchmark has three sub-tasks. Each eval script reads a specific JSON file (configurable via BENCHMARK_FILE at the top of the script).

File	Task	Script
generation_benchmark.json	Text → ASCII art (Generation)	eval_generation.py
understanding_benchmark.json	ASCII art → label (Understanding)	eval_understanding.py eval_understanding_selection.py

File	Records
generation_benchmark.json	200
understanding_benchmark.json	200

Schemas

Generation (generation_benchmark.json):

{
  "id": "orig_5",
  "type": "original",
  "instruction": "Create an ASCII art representation of a milk carton."
}

• type: "original" (in-distribution) or "variant" (OOD)
• instruction: English text prompt

Understanding (understanding_benchmark.json) — used Understanding:

{
  "title": "Seen",
  "name": "Cat",
  "output": "  /\\     /\\\n..."
}

• title: "Seen" (in-distribution) or "Unseen" (OOD)
• name: ground-truth label
• output: ASCII art text

Usage with SVE-ASCII

Copy or symlink the JSON files into the benchmark directory:

SVE-ASCII/benchmark/
  eval_generation.py
  eval_understanding.py
  eval_understanding_selection.py
  generation_benchmark.json      ← from this dataset
  understanding_benchmark.json   ← from this dataset

Then run the eval scripts as described in benchmark/README.md. Each script’s BENCHMARK_FILE (at the top of the file) points to the corresponding JSON by default (e.g. understanding_benchmark.json for eval_understanding.py and eval_understanding_selection.py).