File size: 3,310 Bytes
33d23e1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | # Reference Implementations for Dataset Evaluation
This directory contains reference implementations for evaluating models on H100 hardware. These implementations use standard diffusers pipelines and provide baseline metrics for comparison with TT-Metal implementations.
## Setup
1. `pip install -r requirements.txt`
2. Download COCO captions:
- `wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/coco2014/captions/captions_source.tsv`
- Rename file to `captions.tsv`
3. Download COCO validation statistics:
- `wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/tools/val2014.npz`
4. Login to HuggingFace: `huggingface-cli login`
- Enter your token
- Accept license agreements as required for each model
## Flux Evaluation
**File**: `flux.py`
**Steps**:
1. Load weights locally:
- `pip install accelerate`
- ```
huggingface-cli download black-forest-labs/FLUX.1-schnell \
--repo-type model \
--local-dir ./flux_schnell \
--include "*"
```
2. Run eval:
```
pytest flux.py::test_accuracy_model -v -s --start-from=0 --num-prompts=5
```
**Notes**:
- Default settings: 1024x1024, guidance_scale=1.0, 4 inference steps (schnell model)
- Results are saved to `test_reports/flux_test_results.json`
## Stable Diffusion 3.5 Evaluation
**File**: `sd35.py`
**Steps**:
1. Accept the license agreement for SD3.5 at: https://huggingface.co/stabilityai/stable-diffusion-3.5-large
2. Load weights locally:
- `pip install accelerate`
- ```
huggingface-cli download stabilityai/stable-diffusion-3.5-large \
--repo-type model \
--local-dir ./sd35_large \
--include "*"
```
3. Run eval:
```
pytest sd35.py::test_accuracy_sd35 -v -s --start-from=0 --num-prompts=5
```
**Notes**:
- SD3.5 uses three text encoders: CLIP-L, CLIP-G, and T5-XXL
- The T5 encoder provides enhanced text understanding and longer prompt support (up to 256 tokens)
- Default settings: 1024x1024, guidance_scale=3.5, 40 inference steps
- Requires ~40GB VRAM on H100 for full resolution inference
- Results are saved to `test_reports/sd35_test_results.json`
## Motif Evaluation
**File**: `motif.py`
**Note**: There isn't an existing diffusers pipeline for Motif yet, so this uses the official Motif inference script.
**Steps**:
1. Clone the Motif repository:
```
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Motif-Technologies/Motif-Image-6B-Preview
cd Motif-Image-6B-Preview
huggingface-cli download Motif-Technologies/Motif-Image-6B-Preview --local-dir ./ --include "checkpoints/*"
```
2. Download captions for eval:
- `wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/coco2014/captions/captions_source.tsv`
- `mv captions_source.tsv captions.tsv`
3. Accept the license agreement for Motif if required
4. Run eval:
```
python motif.py \
--images-dir outputs/seed_7777/scale_4.0 \
--prompts-file captions.tsv \
--start-from 0 \
--num-prompts 500
```
**Notes**:
- This script uses the official Motif inference.py script to generate images
- Results are saved to `dataset_eval_results.json`
|