Apply Z-Image Fun ControlNet

Z-Image won its place on consumer GPUs by being the 6B model that runs on a 3060 at about thirty seconds a picture. The Fun ControlNet gives that same small model real spatial control - pose, depth, canny - and this node is the ComfyUI-native way to apply it. The name "Fun" isn't whimsy, it's the Alibaba-PAI team that trained the controlnet (a different sub-team from the Tongyi-MAI group that trained Z-Image itself, which is a fact with consequences, see troubleshooting).

Where this fits: ControlNet is the layer that stops prompting from being a slot machine. The prompt decides what appears, the control map decides where it goes. Z-Image's union controlnet covers canny, depth, pose (OpenPose), HED, and MLSD, plus inpainting and a separate tile model. This node is the apply step - you feed it a control image and it wires the structure into the model for the sampler.

How it works

The controlnet weights arrive as a MODEL_PATCH, loaded by the built-in ModelPatchLoader node, and this apply node fuses them into your base model. It encodes your control image into the latent space with the VAE you provide, then patches the model's double blocks (and its noise refiner) so that at each denoising step the control map steers the prediction toward the structure you fed in. Strength slides how hard.

The mechanism reads like a hybrid: it's the same code path as Qwen Image's DiffSynth controlnet (this class literally subclasses it), with Z-Image-specific patch handling on top. The VAE matters because the control image has to be encoded into the same latent space the model denoises in.

Inputs and what matters

• model - your Z-Image model.
• model_patch - the MODEL_PATCH from ModelPatchLoader (the Fun controlnet file).
• vae - used to encode control images. Needs to be the model's VAE.
• strength - float, default 1.0, range -10 to 10. 1.0 is full control; 0.3-0.7 is loose guidance for creative reinterpretation.
• image (optional) - the control image. inpaint_image and mask (optional) switch it into inpainting mode when you provide them.

Output is a single MODEL for the sampler.

The controlnet file goes in ComfyUI/models/model_patches/ and is picked up by ModelPatchLoader. For the current Union controlnet that's the alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union family on HuggingFace - grab the 2.x version, not the initial release.

Common issues & troubleshooting

The step-cutoff trick is the whole game. Because the controlnet was trained on the Turbo model by a team that likely never touched the Base checkpoint, running it at full strength across all steps partially breaks the distilled model's quality. The community fix, well documented in the Z-Image threads: apply the controlnet at full strength for the first few steps, then remove it for the last few so the base model refines detail. In ComfyUI that means two Advanced KSampler passes - one conditioned with this node, one without.

Combining with LoRAs degrades quality for some people. There are consistent reports of output falling apart when controlnet and a LoRA are both active. If that happens, drop the control strength first, then try moving the LoRA before or after this patch in the chain.

Wrong VAE, wrong latent space. This node uses whatever VAE you wire in. Feed it an SDXL VAE by accident and the encoded control map will be garbage. Match the VAE to the model.