PixArt ControlNet Conditioning

Standard ControlNet, the SD1.5/SDXL kind, works by training a copy of the UNet's encoder blocks and adding its output into the frozen UNet's skip connections. That's a UNet-shaped trick, and PixArt doesn't have a UNet - it's a diffusion transformer, all attention and MLP layers, no skip connections to hook into. So DiT models that support ControlNet at all reimplement the same "condition on spatial structure" idea against attention instead, trained from scratch for that specific architecture. That's why you won't find a normal "Apply ControlNet" node anywhere in a PixArt graph, and why a ControlNet file trained for SD1.5 will never load on PixArt no matter how you wire it - this node is PixArt's own version of that mechanism.

What it actually does

Rather than routing your control image through parallel duplicated encoder blocks, this node folds a latent representation of the control image directly into your text conditioning, and hands back a single conditioning object carrying both. You use it in place of your plain text conditioning going into the KSampler - not alongside a separate "apply" step.

• cond - your existing CONDITIONING, from PixArt's T5 text-encode node.
• latent - the control signal, as a LATENT. This is your control image (an edge map, a pose skeleton, whatever preprocessor output you're using) run through VAE Encode first - not the raw image tensor.
• Output: cond - conditioning with the control signal baked in. Feed this straight into your KSampler's positive (and usually negative) input where you'd normally put plain text conditioning.

Installing it

Part of the whole ComfyUI_ExtraModels pack (ComfyUI Manager: search "Extra Models for ComfyUI," or git clone https://github.com/city96/ComfyUI_ExtraModels into custom_nodes plus pip install -r requirements.txt, then restart). You'll also need the separate PixArt ControlNet model weights (PixArt-alpha/PixArt-ControlNet on Hugging Face) alongside a normal PixArt checkpoint and its T5 encoder.

Common issues

There's a real, documented bug worth knowing about before you spend an hour debugging your own setup: earlier releases of this pack reference an internal variable named lewei_scale inside PixArt/models/pixart_controlnet.py, which later versions renamed to pe_interpolation. If your installed version doesn't match whatever a workflow or guide expects, you'll hit an execution error naming one variable or the other. The fix is a five-minute manual edit of that file to match the name your version actually uses - annoying, but well understood; it's tracked in the pack's own GitHub issues.

Preprocessor choice matters more here than it does on SD1.5/SDXL ControlNet. Community reports consistently point to HED SoftEdge as the preprocessor that actually works well with PixArt's ControlNet - canny and most lineart preprocessors (the "AIO" bundled ones included) reportedly work badly or not at all. If your control image looks like it's being ignored entirely, swap preprocessors before assuming the node itself is broken.