Use any model with diffusers without waiting for an official integration

Community Article
Published November 16, 2026

This is a guide on how to integrate a new model with Modular Diffusers so you can use it without waiting for an official integration. At the end you have two repos on the Hub, one with the converted weights and one with the custom blocks, and anyone can run the model with a few lines of code:

import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image


pipe = ModularPipeline.from_pretrained(
    "OzzyGT/idol_custom_blocks", trust_remote_code=True
)
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image(
    "https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/idol/nb-dragon-anime_input.png"
)
result = pipe(
    image=image,
    generator=torch.Generator("cuda").manual_seed(42),
    output=["gaussians", "camera_latents"],
)
gaussians = result["gaussians"]
camera_latents = result["camera_latents"]

Idol turns a single image into a 3D Gaussian splat. The pipeline returns the Gaussians as tensors, ready to render or save as a .ply file, plus its estimate of the camera that took the input picture.

You will need a coding agent that can read repos, run code and use skills. I'm using Claude Code with Claude Opus 5.5, which is the one that gave me the best results so far. Results can vary a lot between models. When I tried the same task with Astra, it didn't convert the model, and it didn't always follow the skill, making some decisions on its own that weren't the right ones.

Disclaimer: this is about using a model that isn't in diffusers yet for your own use, not about submitting a PR for an official integration. The model code runs with trust_remote_code=True, so only load repos you trust.

Why?

The main reason is to build an app or an API that uses many models from a single environment. Most models ship with their own code, and that code usually pins its own versions of torch, transformers, diffusers and sometimes Python. Put two of them in the same app and their dependencies conflict, which is the usual mess and why people give up on testing models or building their own apps.

Pixal3D is a good example. To run the original code you first need the TRELLIS.2 environment it builds on, which only runs on Linux. For that you will need the CUDA Toolkit to compile packages, a conda env with PyTorch 2.6.0, and a setup script that builds flash-attn, nvdiffrast, nvdiffrec, cumesh, o-voxel and flexgemm. After this you will need to install its own requirements, build natten 0.21.0 from source for your GPU architecture, and install utils3d from a wheel hosted on a personal GitHub release. The Modular Diffusers version needs PyTorch, diffusers, transformers, accelerate, safetensors, numpy and pillow, and runs in the same environment as all my other models.

It also helps in other ways:

  • All the diffusers optimizations work. The new model gets the same optimizations as any model in diffusers, and they work the same way for every model, so the application or API doesn't need special code per model. The most important ones are group offloading, torch.compile and attention backends. Check the optimization docs for the rest.
  • Easy to share. The integration lives on the Hub, so you or anyone else can load it with from_pretrained from anywhere.

Do you need to rent a GPU?

It depends on the model size and your hardware. Everything in this guide ran locally on my own PC. I'll assume your GPU has enough VRAM to run the model. If it doesn't, you can always use model offloading or group offloading.

The integration also needs a parity check against the original code, and ideally it runs in full precision, so it needs more memory than just running the model. For small models you can run the check on a consumer GPU or on CPU. For big models you will need a GPU with enough VRAM to hold each component while it runs, or you can use group offloading, which only keeps a few blocks on the GPU at a time. It's slower but doesn't change the results, so it works for the parity check too. The weights wait in RAM, so you still need enough RAM to hold the model in full precision.

If you don't have enough VRAM or RAM, you can run the parity check in the cloud. I suggest HF Jobs. You run a script on Hugging Face hardware with hf jobs uv run --flavor <hardware> parity.py and pay only for the seconds it runs. An A100 with 80 GB of VRAM and 142 GB of RAM costs $2.50 an hour at the time of writing, which covers most models.

For reference, this is the component parity check for Idol in full precision, locally and on HF Jobs:

Where Hardware Time Cost
Local RTX 5090 13 s -
Local CPU (32 threads) 49 s -
HF Jobs A10G (24 GB, $1/hour) 4 min 28 s about $0.07

The HF Jobs time includes starting the machine, installing the dependencies and reading the weights from the Hub. The check itself took 3.5 minutes. All three passed.

Prompt and integration

The prompt I use as a starting point is this one:

I want to integrate this new model with Modular Diffusers as custom blocks on the Hub: https://huggingface.co/speridlabs/idol. This is for my own use, not a PR or an integration into the diffusers core, so the model code ships in the Hub repos instead of in the diffusers source tree. Research how feasible it is and what we need to do for it, and use the skills in the repo for model integration and custom blocks. Before you start, update the main branch and the skills to their latest versions.

Replace the link with the model you want to integrate. If the model card links to the original code, the agent can find and clone it on its own, as it did for me, but adding the GitHub link to the prompt helps, especially with less capable models.

The agent will look for the skills with diffusers-cli, or use them directly if you already have them installed. It will research the model card and the GitHub repo if there is one. Depending on the model and the harness, it will probably also use its memory or history of previous integration work you have done.

If you prefer to install the skills yourself before starting, run this from the diffusers repo:

diffusers-cli skills list                    # see the available skills
diffusers-cli skills add model-integration
diffusers-cli skills add custom-blocks
diffusers-cli skills update                  # later, to get the latest versions

It detects your agent from the environment. Pass --claude, --codex or --cursor to choose one, and -g to install the skills for your user instead of only for the current project.

Reviewing the plan

This is the more involved part, where you need to make decisions about what the agent suggests. If the model is very similar to an architecture already in Modular Diffusers, its suggestions will mostly be right, but it's still worth double-checking them.

With Idol, the agent first read the model card and the original code, and came back with a report and a plan. For this model, the two image encoders it uses, DINOv3 and the FLUX.2 VAE, are already in transformers and diffusers. Only the denoiser and the two decoders needed porting. The original code also uses diffusers for the VAE, and its denoiser is adapted from the Wan transformer in diffusers, which made the port easier.

The plan also included porting BiRefNet for background removal and writing the .ply file inside the pipeline. I didn't want that, because preparing the input image and saving the result are things the user or the app can do separately, so I told it to leave the pre and post processing out. You'll have to make this kind of decision with most models, deciding what is part of the model and what is processing around it.

This makes a difference if you want to share it or want other people to use it. For example, if you return a specific kind of output, like a saved file, it can be cumbersome for apps that already handle that part and just want the raw tensors.

Another example is porting BiRefNet as a preprocessor. It's a good background remover today, but there are alternatives, and better ones will probably come out. If it's built into the pipeline, users can't swap it for the one they prefer.

For the Idol model integration the agent asked four questions before writing any code:

  1. Which Hub repo should it go to, and should it be private?

    My answer: OzzyGT/idol_diffusers.

  2. Does your account have access to the gated models? Idol uses DINOv3 and FLUX.2-dev, which are gated, so you need to accept their terms on the Hub first. It also asked before downloading about 3 GB of weights plus the encoders.

    My answer: Yes, the token in that environment has access.

  3. Can it install the original code in a separate environment for the parity tests?

    My answer: Yes.

  4. Where should the files live while it builds them?

    My answer: In a folder in my drive.

Keep in mind that this can vary a lot from model to model. It depends on the modality, whether some of its components already exist in transformers or diffusers, and how complex the architecture is. I wrote these only as a reference for the kind of questions you can get.

Building it

After the questions and answers, the agent should write everything in one go: a conversion script for the original weights, the models, and the blocks that make the pipeline. The models follow the diffusers conventions for attention, so the optimizations from the Why section work with them too.

Then it should run a quick end-to-end test. It can work the first time, or it can fail while the agent fixes bugs or quirks it missed. In my experience it usually takes several tries before it works.

With Idol it worked on the third try, after two small fixes. With 50 steps it produced 262,144 Gaussians in 32.5 s, with a 4.6 GB peak of VRAM. This only shows that the code runs. At this point we still don't know if it's correct, and that's what the parity check is for.

This whole part, from writing the conversion script to the working end-to-end test, took about 14 minutes for this model.

Parity check

The parity check compares the new code with the original code from the model authors' GitHub repository, using the same weights and inputs. For every model it should do something similar to this:

First each component on its own, in full precision on CPU. Both versions get the same input, and the check compares every value in their outputs and keeps the largest difference. The model-integration skill sets the limit for a pass at 0.001 (1e-3), small enough that any real bug in the port goes over it.

Then the full pipeline on GPU in half precision, which is how people will run it. Here the numbers never match exactly, because small differences in half precision grow with each step, so the real proof that the port is correct is still the per-component check in full precision. This check is to see that the whole pipeline works like the original, not to get exact numbers.

For the Idol model integration this is what the agent did (from the Claude Code log):

  • Each component in full precision on CPU. Both decoders matched the original exactly, with a difference of 0. The largest difference in the transformer was 0.00002 (2e-5), with output values up to about 5, well below the limit.
  • The full pipeline on GPU in half precision. To know how much difference was normal, it also ran the original code against itself with the two guidance passes batched into one, which changes the order of the GPU operations but not the math. The original differed from itself by 0.6% and the port differed from the original by 0.7%, so the port is at the same level. The decoders gave exactly the same result when both got the same latent.
  • A difference that wasn't a bug. At first the image tokens from DINOv3 were off by up to 0.24, which looked like a bug in the port. The values going into DINOv3 were the same on both sides, but the original code builds the image tensor with a different memory layout than the diffusers image processor. That made the GPU pick a different kernel for the first convolution, and the small difference grew through the 32 layers. With the same tensor on both sides, the tokens matched exactly. The agent found this on its own by checking the input DINOv3 received inside the pipeline.

The whole parity check took about 13 minutes.

Checking it yourself

For me, this is the most important check, more than the numbers. I have a good eye for details, and if the generations look bad to me, something is wrong, even if the parity check passes.

So always compare the outputs yourself, with the same input and seed on both sides. Look at images and 3D models, and listen to audio. The numbers can't tell you everything, and you are the one who is going to use the model. If you don't have an eye or an ear for this, I highly recommend asking someone you know who does.

For Idol, I asked the agent to run the original code and the diffusers version with the same input image and seed, and to render each result from four sides. This is the input:

Input image

Original:

Original Idol code, four views

Diffusers:

Diffusers version, four views

They look almost the same to me from every side, including the back of the wings and the spikes along the back, which the input image never shows. They differ in some small parts, like the wing bones and the position of the scales. That's the small half-precision drift from the parity check. For this seed, I actually like the diffusers one better.

Time and tokens

The whole integration, from the research to the four-side renders, took the agent about 55 minutes of work in one Claude Code session with Claude Opus 5.5. It wrote about 182K output tokens, and the conversation peaked at 375K tokens of context out of 1M, so it never had to compact.

With the API prices for Claude Opus 5.5 at the time of writing, the session would cost about $12.50.

Publishing on the Hub

When the integration works, you're ready to upload it. Here you can do whatever you want, but if you want other people to use it, you have to make some decisions.

First, you can publish everything in one repo, code and weights. It's simpler to maintain and upload, but it has two drawbacks:

  1. Your repo is tied to the license of the weights. If you choose a different license for your code, the repo ends up with more than one license, which makes it a lot harder for other people to use, especially the ones that have a legal department.
  2. Any change you make to the inference code or the README also marks the weights as updated, which is cumbersome for people who watch the repo for changes to the weights.

The other option is what I usually do. I keep the custom blocks and the modeling code in a separate repo, so I can use any license I want for the code. I recommend Apache-2.0, but it's your choice. This is harder to maintain if you ever need to change the modeling code and the weights together, but that's rarely the case. Most of the time you only change the inference code.

The weights repo only has the configs and the converted weights. Converted weights are still a copy of the original ones, so they keep the original license. Add a copy of it to the repo, and if it doesn't allow redistribution, don't publish the weights. Users can convert them themselves with the script from the blocks repo.

Each repo needs a model card. A simple one is enough:

  • The metadata block at the top (license, library_name: diffusers, pipeline_tag, base_model, tags). It sets the license and the task on the Hub, and links your repo to the original model.
  • The usage code.
  • Credit to the original authors, with links to their repo and paper.
  • The licenses of everything the pipeline loads.
  • A link to the matching repo. The blocks card links to the weights repo, and the weights card links back to the blocks repo.

For me this part often uses more tokens than the conversion, because I'm very picky with the cards. You don't need to follow my example, but I'm very happy with how they turn out and I recommend doing something similar, even if it costs you more tokens and time.

First I review the code. The agent often misses something, or adds something that isn't needed because it follows the original code too closely. For example, the first version of the usage code loaded each component in the precision the original Idol uses:

pipe.load_components(
    dtype={
        "transformer": torch.float16,
        "octree_decoder": torch.float16,
        "gaussian_decoder": torch.float16,
        "image_encoder": torch.bfloat16,
        "vae": torch.bfloat16,
    }
)

A single dtype for the whole pipeline also works:

pipe.load_components(dtype=torch.bfloat16)

With bf16 the result is a bit further from the original, but it looks the same to me and the code is much simpler, so I kept that one. I also removed num_inference_steps=50 and num_gaussians=262144 from the example, because those are the defaults, and in diffusers we don't write the defaults in the usage code.

After that, when I was reviewing the licenses, the agent noticed two things. The VAE came from a gated repo with a non-commercial license, and it was the same VAE as another one with an Apache-2.0 license. This is worth checking for any model that loads components from other repos, because the license and the gating of those repos apply to your integration too.

The official DINOv3 repo is gated with manual approval by Meta. Its license allows redistribution, so in the end I shipped both DINOv3 (in bf16, with a copy of its license as LICENSE_DINOv3) and the klein-4B VAE in the weights repo, and nothing the pipeline loads is gated. Read the license on Meta's page and not the copy in their Hub repo, because the page version is newer and also asks you to show "Built with DINOv3", so I added that line to both cards.

After that I cleaned up the text and the code, and rewrote the parts that sounded too much like an agent. I like my cards to read as written by a person, in simple language.

The last part was a video that shows the output. I asked the agent to run the model, render the result and record a turntable video of it. This is the final result:

The whole cards part, including the video, the license checks and testing the final repos from the Hub, used about 100K output tokens and would cost about $15.70 with the API prices for Claude Opus 5.5, more than the integration itself.

(Optional) Creating a Hugging Face Space

For some models, especially 3D ones, I like to create a Space, since it's practically the only way to test them quickly and see the results. I make my Spaces work both locally and on ZeroGPU, so I can upload them and users can also download them and run them on their own machine.

I use some of these Spaces together with local apps or scripts, so when the Space runs locally, I add a button to unload the models and free the memory.

This part is fast for me because I have Spaces I can use as templates for each kind of model: 3D, image, video, audio and so on. I just ask the agent to create a new Space using one of them as a guide. For Idol I used my Hi3D Space.

One change, related to what I said in Reviewing the plan, is the background remover. My other Spaces already use BEN2, so this one uses it too instead of BiRefNet, which the original model uses. The Space also does the framing of the original code (a square crop around the object, a bit looser for photos) and shows a preview of the image that goes into the model. The result is shown with Gradio's Model3D component, which renders Gaussian splats, with a button to download the .ply.

Creating the Space took the agent about 16 minutes, used about 34K output tokens and would cost about $4.80 with the API prices for Claude Opus 5.5.

Closing

I have done several models this way and I have never had a parity check that I couldn't fix. The diffusers skill doesn't say what to do when this happens, but if it happens to you, this is the order I'd follow:

  1. Don't let the agent raise the tolerance to make it pass. A failing check means something is different, and the job is to find what.
  2. Make sure the inputs are really identical: the same weights, input tensor, seed, noise and timesteps. The DINOv3 difference from the parity check is a good example. The values were the same, but the memory layout wasn't.
  3. Make the agent find the first place where the numbers split. Compare component by component, then the outputs inside the failing component. The first layer that differs is usually where the bug is.
  4. Check the known causes. The model-integration skill has a list of them in references/pitfalls.md, so point the agent to it.
  5. If a small difference remains, judge it. Compare it with how much the original differs from itself, and look at the outputs yourself. For your own use that can be enough.

If you have a model you want to use that isn't in diffusers, try the same prompt with it and share the repos so others can use it too. If you want to understand what the agent wrote, or write custom blocks yourself, the Modular Diffusers custom blocks docs are a good place to start.

Links:

Total cost

These is the total cost for this custom model integration. The Claude costs use the API prices for Claude Opus 5.5 at the time of writing.

Part Agent time Output tokens Cost
Integration, from the research to the renders 55 min 182K $12.50
Hub cards, including the video and testing the repos - 100K $15.70
Space 16 min 34K $4.80
HF Jobs parity check (A10G) 4 min 28 s - $0.07
Total 316K $33.07

Disclaimer (again)

This guide is for using models yourself or sharing them with other people. It isn't meant for opening PRs to add models to the diffusers repository. That needs a deep understanding of the codebase and more steps than the ones here, which I skipped to keep this guide simple.

Community

Sign up or log in to comment