VQGAN-CLIP Generator

This node runs the technique everybody was obsessing over before Stable Diffusion existed: VQGAN+CLIP, the CLIP-guided "dream" from the 2021 colab era. You know the look - painterly, glitchy, hallucinatory textures that melt toward whatever your prompt describes without ever quite landing on it. Modern diffusion can't reproduce it because it's a different mechanism, and if you got hooked on those early animations you've probably looked for it in ComfyUI and found nothing. This pack fixes that. It's a niche nostalgia piece, not a daily driver - but it's the real thing, and it's the only way to get that era's output inside a normal ComfyUI graph.

Fun historical footnote: VQGAN is Patrick Esser's work from the Taming Transformers paper, and the latent-diffusion paper by Esser, Rombach and Ommer - the same lab - is what Stable Diffusion literally implements. This node runs the granddaddy of the whole lineage; Disco Diffusion and deep-daze used the same "optimize an image to match a CLIP embedding" trick.

How it actually works

VQGAN is a GAN that squashes an image into a small grid of discrete tokens (a learned codebook), then decodes it back. The node starts from a random token grid z and runs gradient descent on it, not diffusion - there's no denoising anywhere. Each iteration it decodes z into an image, takes cutn random crops ("cutouts") of it, pushes those through CLIP, and computes how far the result is from the prompt's CLIP embedding. That loss backpropagates into z, the Adam optimizer nudges z a step, repeat. After iterations passes you get the final image. That's why it's slow and why it produces those morphing, painting-like results: it's literally the network dreaming toward your text.

One implementation detail: ComfyUI runs nodes under torch.inference_mode, which blocks autograd, so the author re-enables gradients inside the node. Handled for you - but it's why this ships as a bespoke node rather than a script wrapper.

The inputs that matter

You'll leave most knobs alone, but these three define the result:

• iterations (default 100) - how many optimization steps. 100 gives a rough sketch; 300–1000 gets the full look. This is the main quality/speed tradeoff, and it's the first thing you raise.
• learning_rate (default 0.1) - how hard each step pushes the image toward the prompt. Too high and it goes to mush, too low and it never gets there.
• cutn (default 32) - random crops scored by CLIP per step. Higher = more coherent but linearly more work.

Also on the required side: text_prompt (you can chain multiple prompts with | and weight them like surreal:0.5), width/height, seed, and two dropdowns - vqgan_model picks the generator and clip_model (ViT-B/32, ViT-B/16, ViT-L/14) picks the CLIP encoder. The optional init_image input seeds z from an existing image - that's how you get the classic "start from a photo, then zoom and feed the output back in" feedback loops. It outputs a single IMAGE, which wires straight into Save Image / Preview Image / Video Combine like any other generator.

Installing it

ComfyUI Manager can find it as "ComfyUI-VQGAN", or do it by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/n1ckfg/ComfyUI-VQGAN
# restart ComfyUI

The Python deps are light (the requirements.txt is just omegaconf and ftfy), because the repo ships its own copies of CLIP and taming-transformers inside the pack folder and stubs out pytorch-lightning so nothing heavy gets pulled in. The real download is the VQGAN checkpoint: the node throws FileNotFoundError until you run the model script inside the pack:

cd ComfyUI/custom_nodes/ComfyUI-VQGAN
bash download_models.sh        # Windows: download_models.bat

By default that pulls vqgan_imagenet_f16_16384 (about a gigabyte). The other options in the dropdown - wikiart_16384, faceshq, sflckr, the gumbel model - need you to flip the matching true flag in the script first.

Troubleshooting

The top failure is VRAM. The README's numbers hold up: roughly 10 GB for 512×512, 24 GB for 900×900. Hit CUDA out of memory? Drop width/height, then cutn. On the flip side, if your card is small, 380×380 at 8 GB still works and honestly suits the aesthetic.

Second gotcha: the first CLIP load downloads weights, so the very first run stalls while it fetches ViT-B/32 - that's normal, not a hang. No GPU at all? The node silently falls back to CPU, and "silently" is doing a lot of work there: you'll be waiting a very long time. If you're just testing, run 50–100 iterations before committing to a long render. And if you see CUSOLVER_STATUS_INTERNAL_ERROR, that's the README's classic warning - check that width/height are sane multiples of the model's latent scale (f16 models decode at 16×).

Set your expectations right and this node is a delight: a time machine to 2021, on purpose. Just don't feed it your whole render queue expecting SDXL speed.