code/models/tt_dit/tests/dataset_eval/reference/README.md

Reference Implementations for Dataset Evaluation

This directory contains reference implementations for evaluating models on H100 hardware. These implementations use standard diffusers pipelines and provide baseline metrics for comparison with TT-Metal implementations.

Setup

1. pip install -r requirements.txt
2. Download COCO captions:
   • wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/coco2014/captions/captions_source.tsv
   • Rename file to captions.tsv
3. Download COCO validation statistics:
   • wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/tools/val2014.npz
4. Login to HuggingFace: huggingface-cli login
   • Enter your token
   • Accept license agreements as required for each model

Flux Evaluation

File: flux.py

Steps:

1. Load weights locally:

   • pip install accelerate

   • huggingface-cli download black-forest-labs/FLUX.1-schnell \
       --repo-type model \
       --local-dir ./flux_schnell \
       --include "*"
     

2. Run eval:

   pytest flux.py::test_accuracy_model -v -s --start-from=0 --num-prompts=5
   

Notes:

• Default settings: 1024x1024, guidance_scale=1.0, 4 inference steps (schnell model)
• Results are saved to test_reports/flux_test_results.json

Stable Diffusion 3.5 Evaluation

File: sd35.py

Steps:

1. Accept the license agreement for SD3.5 at: https://huggingface.co/stabilityai/stable-diffusion-3.5-large

2. Load weights locally:

   • pip install accelerate

   • huggingface-cli download stabilityai/stable-diffusion-3.5-large \
       --repo-type model \
       --local-dir ./sd35_large \
       --include "*"
     

3. Run eval:

   pytest sd35.py::test_accuracy_sd35 -v -s --start-from=0 --num-prompts=5
   

Notes:

• SD3.5 uses three text encoders: CLIP-L, CLIP-G, and T5-XXL
• The T5 encoder provides enhanced text understanding and longer prompt support (up to 256 tokens)
• Default settings: 1024x1024, guidance_scale=3.5, 40 inference steps
• Requires ~40GB VRAM on H100 for full resolution inference
• Results are saved to test_reports/sd35_test_results.json

Motif Evaluation

File: motif.py

Note: There isn't an existing diffusers pipeline for Motif yet, so this uses the official Motif inference script.

Steps:

1. Clone the Motif repository:

   git lfs install
   GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Motif-Technologies/Motif-Image-6B-Preview
   cd Motif-Image-6B-Preview
   huggingface-cli download Motif-Technologies/Motif-Image-6B-Preview --local-dir ./ --include "checkpoints/*"
   

2. Download captions for eval:

   • wget https://github.com/mlcommons/inference/raw/4b1d1156c23965172ae56eacdd8372f8897eb771/text_to_image/coco2014/captions/captions_source.tsv
   • mv captions_source.tsv captions.tsv

3. Accept the license agreement for Motif if required

4. Run eval:

   python motif.py \
     --images-dir outputs/seed_7777/scale_4.0 \
     --prompts-file captions.tsv \
     --start-from 0 \
     --num-prompts 500
   

Notes:

• This script uses the official Motif inference.py script to generate images
• Results are saved to dataset_eval_results.json