SVE-ASCII-Bench

Benchmark for evaluating ASCII art generation and understanding. From Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs.

Tasks and Files

The benchmark has three sub-tasks. Each eval script reads a specific JSON file (configurable via BENCHMARK_FILE at the top of the script).

File	Task	Script
`generation_benchmark.json`	Text → ASCII art (Generation)	`eval_generation.py`
`understanding_benchmark.json`	ASCII art → label (Understanding)	`eval_understanding.py` `eval_understanding_selection.py`

File	Records
`generation_benchmark.json`	200
`understanding_benchmark.json`	200

Schemas

Generation (generation_benchmark.json):

{
  "id": "orig_5",
  "type": "original",
  "instruction": "Create an ASCII art representation of a milk carton."
}

type: "original" (in-distribution) or "variant" (OOD)
instruction: English text prompt

Understanding (understanding_benchmark.json) — used Understanding:

{
  "title": "Seen",
  "name": "Cat",
  "output": "  /\\     /\\\n..."
}

title: "Seen" (in-distribution) or "Unseen" (OOD)
name: ground-truth label
output: ASCII art text

Usage with SVE-ASCII

Copy or symlink the JSON files into the benchmark directory:

SVE-ASCII/benchmark/
  eval_generation.py
  eval_understanding.py
  eval_understanding_selection.py
  generation_benchmark.json      ← from this dataset
  understanding_benchmark.json   ← from this dataset

Then run the eval scripts as described in benchmark/README.md. Each script’s BENCHMARK_FILE (at the top of the file) points to the corresponding JSON by default (e.g. understanding_benchmark.json for eval_understanding.py and eval_understanding_selection.py).