# SVE-ASCII-Bench

Benchmark for evaluating ASCII art generation and understanding. From *Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs*.

## Tasks and Files

The benchmark has three sub-tasks. Each eval script reads a specific JSON file (configurable via `BENCHMARK_FILE` at the top of the script).

| File | Task | Script |
|------|------|--------|
| `generation_benchmark.json` | Text → ASCII art (Generation) | `eval_generation.py` |
| `understanding_benchmark.json` | ASCII art → label (Understanding) | `eval_understanding.py` `eval_understanding_selection.py` |

| File | Records |
|------|---------|
| `generation_benchmark.json` | 200 |
| `understanding_benchmark.json` | 200 |

## Schemas

**Generation** (`generation_benchmark.json`):
```json
{
  "id": "orig_5",
  "type": "original",
  "instruction": "Create an ASCII art representation of a milk carton."
}
```
- `type`: `"original"` (in-distribution) or `"variant"` (OOD)
- `instruction`: English text prompt

**Understanding** (`understanding_benchmark.json`) — used Understanding:
```json
{
  "title": "Seen",
  "name": "Cat",
  "output": "  /\\     /\\\n..."
}
```
- `title`: `"Seen"` (in-distribution) or `"Unseen"` (OOD)
- `name`: ground-truth label
- `output`: ASCII art text

## Usage with SVE-ASCII

Copy or symlink the JSON files into the benchmark directory:

```
SVE-ASCII/benchmark/
  eval_generation.py
  eval_understanding.py
  eval_understanding_selection.py
  generation_benchmark.json      ← from this dataset
  understanding_benchmark.json   ← from this dataset
```

Then run the eval scripts as described in `benchmark/README.md`. Each script’s `BENCHMARK_FILE` (at the top of the file) points to the corresponding JSON by default (e.g. `understanding_benchmark.json` for `eval_understanding.py` and `eval_understanding_selection.py`).