YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

pixelart-alpha2rfm

A standalone repository for text-conditioned Rectified Flow Matching (RFM) with PixArt Alpha. The model starts from pretrained RAW epsilon-prediction weights and trains for 100,000 optimizer updates, from step 0 to step 100K.

This repository contains the model, dataset definitions, training, preparation, validation, and inference code. scripts/train_rfm.py is the single training entry point. All project code is local to this repository; datasets, caches, checkpoints, and pretrained VAE/T5 assets must be supplied separately.

Experiment configuration

The experiment is defined in configs/rfm.py.

Setting Value
Model PixArt XL/2, 512 ร— 512, QK RMSNorm
Initialization RAW epsilon weights from A_flux1_dev_mix50/checkpoint-010000
Output head Four channels; zero-initialized for new training
Flow path x(t) = (1 - t) * noise + t * data
Target and loss data - noise, unweighted MSE
Time sampling Continuous uniform sampling; model time embedding 999 * (1 - t)
Datasets FluxLocalizedNarratives: 50,000 images; FluxSynth: 76,708 images
Data mixture Exactly four samples from each dataset per GPU batch
Batch size Four GPUs ร— eight samples; global batch size 32; no gradient accumulation
Precision BF16; gradient checkpointing disabled
Optimizer AdamW, learning rate 1e-5, 500 warmup updates, zero weight decay
EMA decay 0.9999
Text dropout 0.1
Training duration 100,000 optimizer updates, from step 0 to step 100K
Validation RAW and EMA weights; dopri5, CFG 4, atol=1e-6, rtol=1e-3
Validation seeds 42, 43, 44, 45

The trained output head is restored unchanged when resuming or sampling. dopri5 uses adaptive steps: sampling_steps=50 does not impose 50 sampling steps. Each validation run records its sampler settings and number of function evaluations in sampler.json.

Repository layout

run.sh                                # Bash launcher for train, validate, and inference
configs/
  rfm.py                              # Experiment settings and resource paths
examples/
  validation_prompts.json             # Four fixed validation prompts and seeds
  inference_prompts.json              # Example custom prompts with case IDs
datasets/
  rfm_data.py                         # CachedDataset, cache loading, and balanced sampling
diffusion/
  rectified_flow.py                   # RFM loss and Heun/dopri5 sampling
  model/                              # PixArt XL/2 and T5 text encoding
  utils/rfm_checkpoint.py             # Checkpoints, EMA, RNG state, and text dropout
  utils/theta_import.py               # Pretrained weight conversion and provenance
scripts/
  train_rfm.py                        # Training, resume, validation, and optional background launch
  validate_rfm.py                     # RAW/EMA validation from a checkpoint
  inference_rfm.py                    # Custom text-to-image inference
  preparation/
    cache_synthetic_dataset.py        # VAE posterior and T5 embedding caches
    encode_image_latents.py           # Image preprocessing and latent caching
    prepare_theta_v.py                # Initialization weights and fixed text conditioning
    verify_rfm_smoke.py               # Continuous-training versus resume verification
tests/                               # Loss, sampling, data, entry point, and checkpoint checks

Dataset definitions and runtime data loading live in datasets/. Offline asset preparation tools live in scripts/preparation/.

Installation

The observed runtime used Linux, Python 3.12.13, and PyTorch 2.9.1+cu129. requirements.txt records the main dependency versions observed during extraction; it is not a complete lockfile of the original training environment.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.9.1 torchvision==0.24.1 --index-url https://download.pytorch.org/whl/cu129
python -m pip install -r requirements.txt

Use a CUDA driver and GPU compatible with the selected PyTorch/xformers build. The default training configuration uses four GPUs with BF16 support. An existing compatible PixArt environment can also be used.

Run the following commands from the repository root unless stated otherwise.

Bash launcher

bash run.sh <train|validate|inference> <config.py> [arguments...]

The configuration path is required. After preparing the resources described below, use:

# Train using config['world_size'] distributed workers.
CUDA_VISIBLE_DEVICES=0,1,2,3 bash run.sh train configs/rfm.py

# Validate both RAW and EMA weights.
CUDA_VISIBLE_DEVICES=0 bash run.sh validate configs/rfm.py \
  --checkpoint output/rfm_mix50/checkpoints/update-100000.pth \
  --output output/validation-100000

# Generate images from custom prompts using EMA weights.
CUDA_VISIBLE_DEVICES=0 bash run.sh inference configs/rfm.py \
  --checkpoint output/rfm_mix50/checkpoints/update-100000.pth \
  --prompts examples/inference_prompts.json \
  --weights ema --output output/inference-100000

The launcher changes to the repository root, so all relative paths are resolved from there. Set PYTHON_BIN to select an interpreter:

PYTHON_BIN=/path/to/env/bin/python bash run.sh train configs/rfm.py

CUDA_VISIBLE_DEVICES is passed through unchanged. Validation and inference use the first visible GPU. Training arguments such as --resume and --work-dir are forwarded to the trainer. bash run.sh train configs/rfm.py --help displays help without starting distributed workers.

Validation and inference read resource paths and sampling settings from the specified configuration. Its model architecture and flow conventions must match the checkpoint. Use --conditioning-cache and --vae-path to override resource paths. Both modes require a new or empty output directory.

Validation uses the checkpoint's fixed prompts and seeds and verifies the conditioning cache against its recorded SHA-256 hash. Inference reads prompts and seeds from --prompts; --cfg, --steps, --method, --atol, and --rtol override the sampling configuration. --steps affects Heun only.

Prepare external resources

The default configuration expects this layout. Resources may be copied into the repository or provided through symbolic links to existing storage.

assets/source_model/                  # Pretrained checkpoint with transformer/ and scheduler/
assets/pixart/                        # Model snapshot with vae/, tokenizer/, and text_encoder/
data/synth_images/flux1_dev_localized_narratives_synth512/
data/synth_images/flux1_dev_synth512/
output/theta_raw_to_v_setup/           # initial_raw_epsilon.pth and conditioning.pt
output/datasets_512/FluxLocalizedNarratives/
output/datasets_512/FluxSynth/

Each source dataset needs a metadata.jsonl file. Each completed training cache needs cache_settings.json, training_metadata.jsonl, latents/*.npz, and t5_embeddings/*.npz.

The trainer checks source metadata, cache counts, source paths, and SHA-256 hashes. Moving a source dataset can invalidate the paths recorded in an existing cache; regenerate the cache or preserve its original source paths.

Initialization weights and validation conditioning

python -m scripts.preparation.prepare_theta_v \
  --source assets/source_model \
  --snapshot assets/pixart \
  --prompts examples/validation_prompts.json \
  --output output/theta_raw_to_v_setup

The reference experiment used the PixArt snapshot 50f702106901db6d0f8b67eb88e814c56ded2692.

Dataset caches

cache_synthetic_dataset.py currently uses that snapshot at:

~/.cache/huggingface/hub/models--PixArt-alpha--PixArt-XL-2-512x512/snapshots/50f702106901db6d0f8b67eb88e814c56ded2692

Provide the model there or update the script's snapshot variable to point to assets/pixart before generating caches.

Generate and finalize the first dataset cache with one process:

python -m scripts.preparation.cache_synthetic_dataset \
  --source data/synth_images/flux1_dev_localized_narratives_synth512 \
  --output output/datasets_512/FluxLocalizedNarratives --shards 1 --shard 0

python -m scripts.preparation.cache_synthetic_dataset \
  --source data/synth_images/flux1_dev_localized_narratives_synth512 \
  --output output/datasets_512/FluxLocalizedNarratives --shards 1 --finalize

Repeat both commands for the second dataset, using data/synth_images/flux1_dev_synth512 as the source and output/datasets_512/FluxSynth as the output.

For four-GPU cache preparation, run four workers with --shards 4 and distinct --shard values from 0 through 3, assigning one GPU to each worker. Once every worker finishes, finalize with --shards 4 --finalize.

Training

After preparing all resources, start a new run:

HF_HUB_OFFLINE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 \
  bash run.sh train configs/rfm.py

The default output directory is output/rfm_mix50. Use a new --work-dir when starting another run or resuming an existing one. The launcher defaults OMP_NUM_THREADS to 4 and starts torch.distributed.run with the configured number of workers.

Optional background preparation and training

The training entry point also supports the original background controller:

bash run.sh train configs/rfm.py --launch

This mode requires a passing output/rfm_mix50_smoke/verification.json, the corresponding pilot caches, initialization weights, and model assets. It enforces four GPUs and a per-GPU batch size of eight. If the full caches already exist, use direct training as shown above; the smoke verification artifacts are not required for that path.

Controller state and logs are written to preparation/status.json and preparation/controller.log under the work directory. Training output is logged to console.log.

Both launch modes accept --work-dir, --max-train-steps, --validate-every-updates, and --save-every-updates. Background mode forwards these settings to the training process. --launch is for new training only and cannot be combined with --resume or invoked under torchrun.

Resume training

Use the same configuration and resource paths as the saved run, with a new output directory. For example, resume a run from step 5,000 toward a total of 100,000 optimizer updates:

HF_HUB_OFFLINE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 \
  bash run.sh train configs/rfm.py \
  --resume output/rfm_mix50/checkpoints/update-005000.pth \
  --work-dir output/rfm_mix50_resumed \
  --max-train-steps 100000

The trainer strictly compares the checkpoint's configuration and data hashes. Only the work directory, total update count, and save/validation/logging intervals may change. Preserve all other training settings and resource paths.

To resume a checkpoint from the original project, restore the configuration values recorded in that checkpoint and make its paths resolve correctly. The example paths assets/source_model and assets/pixart/vae cannot replace historical paths during resume without triggering Resume changed training semantics. No separate historical configuration is maintained in this repository.

Validation and reproduction

scripts/validate_rfm.py calls the same validation function used during training. It produces four RAW and four EMA images, along with latent tensors, prompts, and sampler metadata under <output>/validation/update-<step>/. The effective configuration is saved as <output>/validation_config.json.

For the original 100K checkpoint, the following command uses assets from the parent workspace and the local Hugging Face cache:

CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 \
  bash run.sh validate configs/rfm.py \
  --checkpoint ../output/rfm_from_A_flux_raw_mix50_100k/resume-030000-to-100000/checkpoints/update-100000.pth \
  --conditioning-cache ../output/theta_raw_to_v_setup/conditioning.pt \
  --vae-path "$HOME/.cache/huggingface/hub/models--PixArt-alpha--PixArt-XL-2-512x512/snapshots/50f702106901db6d0f8b67eb88e814c56ded2692/vae" \
  --output output/revalidate-100000

These resource locations are specific to the original workspace. Adjust them for another machine, and select an interpreter with the required dependencies using PYTHON_BIN if needed.

A local verification through run.sh reproduced all eight original 100K PNGs byte for byte and both latent tensors exactly. RAW and EMA sampling used 182 and 176 function evaluations, respectively. Identical results are not guaranteed across GPU models or dependency versions.

When calling python -m scripts.validate_rfm directly without --config, sampler settings and default resource paths come from the checkpoint. Resource paths can still be overridden with --conditioning-cache and --vae-path.

Custom inference

Supply a JSON list with a unique filename-safe case_id, a nonempty prompt, and a nonnegative integer seed for each image:

[
  {
    "case_id": "example-01",
    "prompt": "A small cabin beside a lake at sunrise",
    "seed": 42
  }
]

Use examples/inference_prompts.json as a starting point. Select weights with --weights raw or --weights ema; the default is raw.

Inference needs tokenizer/ and text_encoder/ alongside the configured vae/ directory. Prompts are lowercased and stripped before encoding and must fit the 120-token training limit. The output includes one PNG per case, prompt_embeddings.pt, and a validation.json report.

Through run.sh, sampling defaults come from the supplied configuration: configs/rfm.py uses dopri5 and CFG 4. Direct Python invocation without --config uses checkpoint resource paths and defaults to Heun with 40 steps and CFG 2.

Checks

python -m pytest -q tests
bash run.sh train configs/rfm.py --help
bash run.sh validate configs/rfm.py --help
bash run.sh inference configs/rfm.py --help

The tests run on CPU and cover RFM loss and sampling, dataset loading, sampling cursors, checkpoint restoration, and launcher behavior. Training and image generation require CUDA.

License

The code retains the original project's Apache-2.0 LICENSE and the author and license notices in copied files. External models and datasets remain subject to their respective licenses.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support