File size: 1,935 Bytes
b036fd6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | # SVE-ASCII-Bench
Benchmark for evaluating ASCII art generation and understanding. From *Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs*.
## Tasks and Files
The benchmark has three sub-tasks. Each eval script reads a specific JSON file (configurable via `BENCHMARK_FILE` at the top of the script).
| File | Task | Script |
|------|------|--------|
| `generation_benchmark.json` | Text → ASCII art (Generation) | `eval_generation.py` |
| `understanding_benchmark.json` | ASCII art → label (Understanding) | `eval_understanding.py` `eval_understanding_selection.py` |
| File | Records |
|------|---------|
| `generation_benchmark.json` | 200 |
| `understanding_benchmark.json` | 200 |
## Schemas
**Generation** (`generation_benchmark.json`):
```json
{
"id": "orig_5",
"type": "original",
"instruction": "Create an ASCII art representation of a milk carton."
}
```
- `type`: `"original"` (in-distribution) or `"variant"` (OOD)
- `instruction`: English text prompt
**Understanding** (`understanding_benchmark.json`) — used Understanding:
```json
{
"title": "Seen",
"name": "Cat",
"output": " /\\ /\\\n..."
}
```
- `title`: `"Seen"` (in-distribution) or `"Unseen"` (OOD)
- `name`: ground-truth label
- `output`: ASCII art text
## Usage with SVE-ASCII
Copy or symlink the JSON files into the benchmark directory:
```
SVE-ASCII/benchmark/
eval_generation.py
eval_understanding.py
eval_understanding_selection.py
generation_benchmark.json ← from this dataset
understanding_benchmark.json ← from this dataset
```
Then run the eval scripts as described in `benchmark/README.md`. Each script’s `BENCHMARK_FILE` (at the top of the file) points to the corresponding JSON by default (e.g. `understanding_benchmark.json` for `eval_understanding.py` and `eval_understanding_selection.py`).
|