ComfyUI-HF-Diffusers

HuggingFace Diffusers (and Transformers) for ComfyUI

A lot of custom nodes for ComfyUI are really just bindings to HuggingFace Diffusers and/or Transformers, but overly constrained to use them in very particular ways. This set of custom nodes is intended to be a generic set of bindings for HuggingFace Diffusers and Transformers, in theory allowing you to use any pipeline supported by HuggingFace.

It also supports bitsandbytes quantization, automatic device mapping, and the other advantages unique to HuggingFace.

Because HuggingFace pipelines output PIL images, you'll likely need a node to get PIL images into ComfyUI's format. That node isn't provided by this package; I recommend ComfyUI_Ib_CustomNodes.

Philosophy

I've tried to make everything work fairly generically, as unopinionated as I can manage. There are no nodes for specialized tasks. For example, while this node set is capable of being used for prompt enhancement, there is no “prompt enhancement” node, because that's simply built from a text-to-text pipeline.

Every node that loads a component has some default class that it loads, but you can replace the class simply by its name. For instance, the pipeline loader loads AutoPipelineForText2Image by default, but you can simply replace AutoPipelineForText2Image with, e.g., LongCatImagePipeline to load LongCat Image instead.

All of the nodes include a generic kwargs parameter, which simply takes text. Rather than trying to anticipate every possible parameter, I've included only the most obvious parameters, and for anything else, use kwargs. kwargs can be empty (for no extra args) or an object in JSON format. Of course, this is only sufficient for those arguments that take JSON-compatible values. kwargs overrides other parameters.

As in HuggingFace, Pipelines are self-loading and include all components, but you can override the VAE and text encoder components if you wish, so they're optional inputs. If you want to load without these components, use kwargs to set them to null (None).

Workflows

These are some example workflows, in simple and exploded forms (where by “exploded” I mean “each step done separately”).

GLM-Image (exploded) (Note: As of the writing of this README, requires the git version of both Diffusers and Transformers.)

• Prompt enhancer (image)
• LongCat Image (exploded, image, image exploded)
  • (Note: Requires my PR to use quantization.)
• SDXL (exploded, image, image exploded)
• AI trash converter (image)

Nodes

Only the “load pipeline” and “run pipeline” nodes are needed for many use cases. Plus “load LoRA” to load LoRAs. Other nodes are provided for low-level control.

HF Diffusers load pipeline

Loads a HuggingFace Diffusers pipeline. By default, loads AutoPipelineForText2Image. Obviously, if the pipeline is supported by AutoPipelineForText2Image, this is the right choice, but any other Diffusers class can be specified.

The device and dtype can be set, and the model can be set to automatically offload to the CPU (enable_model_cpu_offload). Quantization is in dtype.

The VAE and text encoder can optionally be overridden, and whether overridden or not, the VAE is exported.

HF Diffusers load LoRA

Loads a HuggingFace Diffusers LoRA onto a pipeline. Note that the original, un-LoRA'd pipeline cannot be used by simply connecting the original “load pipeline” node. The non-LoRA version is lost.

HF Diffusers run pipeline

Runs a HuggingFace Diffusers pipeline.

For image-to-image and inpainting pipelines, supports (optional) image and mask inputs.

The most common/necessary parameters (prompts and image size) are supported directly. Other arguments can be provided by kwargs. If num_inference_steps is set to 0, the default number of steps for the pipeline is used.

Supports outputting latents or PIL images, but not both at the same time (i.e., only one of the two outputs will be activated, based on which you chose).

HF Diffusers load AutoencoderKL (VAE)

Loads a VAE directly. Generally speaking, the pipeline loads its own VAE, but you can use this to use a custom VAE, or to, e.g., load the VAE on a different device than the rest of the pipeline.

HF Transformers load model

Load a HuggingFace Transformers (not Diffusers) model. Generally for the text encoder. The pipeline loads its own text encoder, but you can use this to use a custom encoder or load it on a different device, etc.

HF Diffusers encode prompt

Use a HuggingFace Diffusers pipeline to encode a prompt into prompt_embeds.

Unfortunately, the way that encode_prompt works varies from pipeline to pipeline. So, I simply provide four tensors as output, and you need to know how the pipeline works to know which you need and where.

Some examples:

• SDXL: (prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, pooled_negative_prompt_embeds)
• Flux: (prompt_embeds, pooled_prompt_embeds, _)
• Z Image: (prompt_embeds, negative_prompt_embeds)
• LongCat Image: (prompt_embeds, _) (but the pipeline doesn't support prompt_embeds)

HF Diffusers VAE decode/encode

Use these to decode/encode HuggingFace latents using HuggingFace VAEs. Automatic if pipelines are run in PIL mode.

HF Transformers load pipeline

Load a HuggingFace Transformers (not Diffusers) pipeline. Useful for using an LLM or VLM as part of a larger workflow, such as prompt enhancement or image description.

HF Transformers create conversation

Create or extend a conversation in the style used by HuggingFace Transformers. Can include text or images. To create a multi-turn conversation, create multiple “create conversation” nodes and chain them together.

HF Transformers unpack conversation

Unpack a part of a conversation in HuggingFace Transformers format. Mostly useful for unpacking the output of a Transformers pipeline.

HF Transformers run pipeline

Run a HuggingFace Transformers pipeline. Takes anything as input, but for most pipelines, you want a conversation as input. Outputs the raw result, and if a conversation was generated, also outputs the extracted conversation. In most cases, you will want to pass that conversation into an “unpack conversation” node.