| # SVE-ASCII-Bench | |
| Benchmark for evaluating ASCII art generation and understanding. From *Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs*. | |
| ## Tasks and Files | |
| The benchmark has three sub-tasks. Each eval script reads a specific JSON file (configurable via `BENCHMARK_FILE` at the top of the script). | |
| | File | Task | Script | | |
| |------|------|--------| | |
| | `generation_benchmark.json` | Text → ASCII art (Generation) | `eval_generation.py` | | |
| | `understanding_benchmark.json` | ASCII art → label (Understanding) | `eval_understanding.py` `eval_understanding_selection.py` | | |
| | File | Records | | |
| |------|---------| | |
| | `generation_benchmark.json` | 200 | | |
| | `understanding_benchmark.json` | 200 | | |
| ## Schemas | |
| **Generation** (`generation_benchmark.json`): | |
| ```json | |
| { | |
| "id": "orig_5", | |
| "type": "original", | |
| "instruction": "Create an ASCII art representation of a milk carton." | |
| } | |
| ``` | |
| - `type`: `"original"` (in-distribution) or `"variant"` (OOD) | |
| - `instruction`: English text prompt | |
| **Understanding** (`understanding_benchmark.json`) — used Understanding: | |
| ```json | |
| { | |
| "title": "Seen", | |
| "name": "Cat", | |
| "output": " /\\ /\\\n..." | |
| } | |
| ``` | |
| - `title`: `"Seen"` (in-distribution) or `"Unseen"` (OOD) | |
| - `name`: ground-truth label | |
| - `output`: ASCII art text | |
| ## Usage with SVE-ASCII | |
| Copy or symlink the JSON files into the benchmark directory: | |
| ``` | |
| SVE-ASCII/benchmark/ | |
| eval_generation.py | |
| eval_understanding.py | |
| eval_understanding_selection.py | |
| generation_benchmark.json ← from this dataset | |
| understanding_benchmark.json ← from this dataset | |
| ``` | |
| Then run the eval scripts as described in `benchmark/README.md`. Each script’s `BENCHMARK_FILE` (at the top of the file) points to the corresponding JSON by default (e.g. `understanding_benchmark.json` for `eval_understanding.py` and `eval_understanding_selection.py`). | |