| --- |
| tags: |
| - model_hub_mixin |
| - pytorch_model_hub_mixin |
| - art |
| license: mit |
| pipeline_tag: unconditional-image-generation |
| metrics: |
| - name: FID |
| type: image |
| value: 80.4755 |
| dataset: https://www.kaggle.com/datasets/ayhantasyurt/pixel-art-2dgame-charecter-sprites-idle |
| split: test |
| --- |
| # Sprite-flow |
| Flow-based generative model for unguided generation of 128x128 RGBA pixel art characters. |
|
|
| ## Model Details |
| ### Model Description |
| - **Developed by:** [Mihailo Radović](https://www.linkedin.com/in/mihailo-radović-484070278/) |
| - **Model type:** Unconditional Image Generation |
| - **License:** MIT |
|
|
| ### Model Sources |
|
|
| <!-- Provide the basic links for the model. --> |
|
|
| - **Repository:** [GitHub Repo](https://github.com/mradovic38/sprite-flow) |
| - **Demo:** [Gradio App](https://huggingface.co/spaces/mradovic38/sprite-flow) |
|
|
| ## Uses |
|
|
| <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
| ### Direct Use |
| Predicts the vector field for generating 128x128 RGBA pixel art character images from Isotropic Gaussian Distribution by simulating an ODE with Linear Noise Scheduling. |
|
|
| ### Out-of-Scope Use |
| Could be used with Cosine or any other Noise scheduler. |
|
|
| ## How to Get Started with the Model |
| * Step 1 - **Clone the [GitHub Repo](https://github.com/mradovic38/sprite-flow)** |
|
|
| * Step 2 - **Initialize the model**: |
| ```py |
| from models.unet import PixelArtUNet |
| |
| model = PixelArtUNet( |
| channels = [128, 256, 512, 1024], |
| num_residual_layers = 2, |
| t_embed_dim = 128, |
| midcoder_dropout_p=0.2 |
| ).to(device) |
| ``` |
| |
| * Step 3: **Load Model weights**: |
| ```py |
| from huggingface_hub import hf_hub_download |
| from safetensors.torch import load_file |
| |
| repo_id = "mradovic38/sprite-flow" |
| filename = "model.safetensors" |
| file_path = hf_hub_download(repo_id=repo_id, filename=filename) |
| checkpoint = load_file(file_path) |
| model.load_state_dict(checkpoint) |
| model.to(device) |
| model.eval() |
| ``` |
|
|
| * Step 4: **Initialize the probability path**: |
| ```py |
| from sampling.conditional_probability_path import GaussianConditionalProbabilityPath |
| from sampling.noise_scheduling import LinearAlpha, LinearBeta |
| |
| path = GaussianConditionalProbabilityPath( |
| p_data=None, |
| p_simple_shape=[4, 128, 128], |
| alpha=LinearAlpha(), |
| beta=LinearBeta() |
| ).to(device) |
| path.eval() |
| ``` |
|
|
| * Step 5: **Simulate ODE**: |
|
|
| ```py |
| import torch |
| |
| from diff_eq.ode_sde import UnguidedVectorFieldODE |
| from diff_eq.simulator import EulerSimulator |
| |
| num_timesteps = 200 # example number of timesteps |
| num_samples = 3 # example number of samples |
| |
| ts = torch.linspace(0, 1, num_timesteps).view(1, -1, 1, 1, 1).expand(num_samples, -1, 1, 1, 1).to(device) |
| x0 = path.p_simple.sample(num_samples).to(device) # (num_samples, 4, 128, 128) |
| ode = UnguidedVectorFieldODE(model) |
| simulator = EulerSimulator(ode) |
| x1 = simulator.simulate(x0, ts) # (num_samples, 4, 128, 128) |
| ``` |
|
|
| * Step 6: **Turn torch tensor to PIL**: |
|
|
| ```py |
| from utils.helpers import tensor_to_rgba_image, normalize_to_unit |
| |
| x1 = normalize_to_unit(x1) # [-1, 1] -> [0, 1] |
| imgs = tensor_to_rgba_image(x1) |
| ``` |
