Guided Diffusion CLIP Loader

This is not the CLIP loader you already know. ComfyUI's built-in CLIP Loader feeds the text encoder so a KSampler can cross-attend. This one loads CLIP models the way Disco Diffusion needs them: as raw models that both encode your prompt and act as the vision tower that grades random crops of the image during sampling. Different format, different output socket (GUIDED_CLIP), and it only wires into this pack's sampler. If you grabbed it expecting conditioning, that's the mix-up.

How it works

Disco's sampler works by making 16 random cutouts of the painting at each step, embedding them, and pushing the image toward the prompt's embedding. Both halves of that need a real CLIP model - the text tower for the prompt, the vision tower for the cutouts - and this node loads both in one shot. It keeps them in memory with autograd enabled, because the whole point is to backprop a CLIP loss into the image. The author's source has a note to that effect, explaining that he deliberately loads with inference mode off because ComfyUI turns it on for speed by default.

The node's load path also handles the download-and-cache dance: the OpenAI models come from OpenAI's own CLIP repo, the OpenCLIP variants from Hugging Face pretrains like laion2b_e16 or laion400m_e31. First use downloads them; after that they're cached.

The inputs

Fourteen toggles, one per CLIP model, and you set them like checkboxes - True or False for each. The defaults are the classic Disco trio: ViT-B/32, ViT-B/16 and RN50 on. The rest (ViT-L/14, ViT-L/14@336px, RN50x4/x16/x64, RN101, and the OpenCLIP quickgelu variants) default to off.

Turn on more than one and they form a committee - Disco originally stacked several CLIPs and averaged their opinions, which produces a more stable, more "opinionated" steer at the cost of speed. A common upgrade is flipping on ViT-L/14 for stronger prompt-following; just know it's a heavier download and slower per step. For a first run, leave the defaults alone.

The output

GUIDED_CLIP - a list of the loaded models, straight into the Disco Diffusion Sampler's guided_clip input. One wire, that's it.

Installing it

It ships in the same pack as the sampler, so the install is identical:

cd ComfyUI/custom_nodes
git clone --recursive https://github.com/space-nuko/ComfyUI-Disco-Diffusion
cd ComfyUI-Disco-Diffusion
pip install -r requirements.txt   # into the venv your ComfyUI uses

Or via ComfyUI Manager, search "Disco Diffusion". The README also claims you can skip this node entirely and wire ComfyUI's own CLIP Loader and CLIP Vision Loader into the sampler instead - you'd copy one model (it suggests clip-vit-large-patch14) into both the clip and clip_vision folders, because ComfyUI can't load both roles from a single file. It's a nice escape hatch, but this node exists precisely so you don't have to juggle that.

Where people get burned

The common mistakes are misreading the toggles (they're booleans, not radio buttons - turn off what you don't want, or you'll load half a dozen models) and expecting the output to plug into a normal KSampler, which it won't. And the usual pack caveat applies: it's from 2023, so on a modern ComfyUI it may not import at all. When it works, though, it's the quietly important node in the pack - without a CLIP model to steer by, the sampler has nothing to guide toward.