DiffSynth: Stable Diffusion XL Loader

SDXL is the architecture with the most parts, so it gets the Loader with the most sockets. Two text encoders, two tokenizers, a UNet and a VAE - six things to point at files, one pipe out the other end. Set it up once and it's a template for everything else in the pack.

Inputs

Required:

• torch_dtype - bfloat16 (default), float16, float32. On the SD and SDXL pipelines specifically, this value is forced onto the models' onload and computation dtypes, so it overrides whatever dtype the attached VRAM Config asked for.
• device - cuda or cpu.
• model_configs - the merged list from Merge ModelConfigs. Required.

Optional:

• tokenizer_config - ModelConfig for the first CLIP tokenizer.
• tokenizer_2_config - ModelConfig for the second. Note this socket doesn't exist on the SD 1.5 Loader; it appears here because SDXL has two encoders. Both are worth wiring.
• vram_limit - from a VRAM Limit node.

Output: pipe → DiffSynth: Stable Diffusion XL Inference.

The file list

Split stabilityai/stable-diffusion-xl-base-1.0 like this, per the pack's own example graph:

| Into | origin_file_pattern | |---|---| | model_configs | text_encoder/model.safetensors | | model_configs | text_encoder_2/model.safetensors | | model_configs | unet/diffusion_pytorch_model.safetensors | | model_configs | vae/diffusion_pytorch_model.safetensors | | tokenizer_config | tokenizer/ | | tokenizer_2_config | tokenizer_2/ |

Merge the first four through a Merge ModelConfigs node (it takes up to eight, so no chaining required), then wire the two tokenizer configs straight into the Loader. That's six ModelConfig nodes on the canvas for one image model, which is a lot of wire spaghetti - and it's the clearest example of what this pack is buying you. Each of those nodes can carry a different VRAM plan or a different quantization scheme. Split them across two GPUs in your head: the text encoders could live on cpu while the UNet runs cuda.

Files land in models/<model_id>/ under ComfyUI's working directory, overridable with DIFFSYNTH_MODEL_BASE_PATH. Downloads happen at ModelConfig execution, which is when you want to be watching the terminal.

Install

Manager → search DiffSynth-ComfyUI → install → restart. Manual:

pip install git+https://github.com/modelscope/DiffSynth-Studio.git
cd ComfyUI/custom_nodes && git clone https://github.com/modelscope/DiffSynth-ComfyUI.git

One declared dependency, diffsynth>=2.1.9. Add the quant extras only if you intend to put a Quantization Config on a ModelConfig:

pip install "diffsynth[quant] @ git+https://github.com/modelscope/DiffSynth-Studio.git"

The honest verdict

If your goal is "run Juggernaut XL and generate a portrait," this is the wrong tool - native ComfyUI loads a fused checkpoint in one node, and every SDXL finetune on CivitAI is distributed that way. DiffSynth wants the component files of the base model. You'd only build this graph to compare architectures under identical sampling conditions, or to keep one consistent loading and memory setup across an SDXL → video → audio pipeline. Both defensible, neither the everyday case.

Common issues

• Text renders as gibberish or the image is oddly generic. A missing or mismatched tokenizer config. Both sockets matter and they aren't interchangeable.
• Re-downloading files you already have. The pattern doesn't match the on-disk layout; the resolution is a literal glob, so model.safetensors and model*.safetensors are different requests.
• Slow, RAM-hungry, then fine. Two text encoders plus a 2.6B UNet add up. Offload defaults are cautious by design; raise computation_device/computation_dtype in the VRAM Config if you have headroom.
• Warehouse full. Component-wise loading means duplicates of models you may already have in a fused form. Check before you download.