🎚️ DiffsynthControlNet Apply

This is ControlNet for the new generation of models - specifically the DiffSynth-style ControlNets that ship for Z-Image and Qwen-Image - and it works differently from the ControlNet you remember from SD 1.5 and SDXL. Instead of producing a conditioning that you blend into the sampler, it patches the model itself and hands you back a control-seeded latent. If you've been trying to bolt a classic Apply ControlNet node onto a Z-Image or Qwen graph and watching it refuse to connect, this is the node that actually fits those architectures.

The mechanism follows from its outputs: model and latent. You feed it your base model, a vae, and a control image, pick a model_patch, and it bakes that ControlNet into the model as a patch and prepares a latent from your control image. You then run that patched model and latent through your normal sampler. The model_patch dropdown is the important part - it lists the DiffSynth ControlNets it knows how to apply: the Alibaba-PAI Z-Image-Turbo-Fun-Controlnet family (Tile and Union variants, including 8-step distilled ones) and the qwen_image_{canny,depth,inpaint}_diffsynth_controlnet set for Qwen-Image. strength (default 1) sets how hard the control pulls, resolution sets the working size, and there's an optional mask input that's there for the inpaint controlnet in the list.

One thing that catches people: the preprocessor option here is basically just None or InvertImage. This node does not run Canny, depth, or pose extraction for you. You feed it an already-prepared control map - do your edge detection or depth estimation upstream with controlnet_aux, the preprocessor pack the README recommends as a companion - and this node applies it. InvertImage is the one convenience, for the cases where your map needs its values flipped (white-on-black vs black-on-white).

Worth knowing about these Z-Image Fun ControlNets specifically, because it explains a quirk you'll hit: they're built on the distilled Turbo model, and running a ControlNet on a distilled model costs quality. The community's standard workaround is a step cutoff - apply the control for the early steps and let the model finish clean - which is exactly why the dropdown carries both full and 8-step distilled variants. If your controlled output looks mushy or over-baked compared to an uncontrolled Z-Image render, that distillation tax is why, and dialing strength back or ending the control early is the lever.

Install is the pack's shared path. ComfyUI Manager, search SDVN_Comfy_node; or cd ComfyUI/custom_nodes && git clone https://github.com/StableDiffusionVN/SDVN_Comfy_node, then pip install -r custom_nodes/SDVN_Comfy_node/requirements.txt from your ComfyUI root and restart. The ControlNet weights in that model_patch list are separate downloads - the pack can fetch popular ones automatically, but on Windows/macOS you need aria2c installed for the auto-download to work, and controlnet_aux installed if you want to generate the control maps in the same graph.

The big troubleshooting rule is matching. A model_patch has to line up with the base model you fed in - a Z-Image Fun ControlNet on a Z-Image model, a qwen_image_* patch on Qwen-Image. Cross them and you get errors or noise, because these are architecture-specific weights, not the universal-ish ControlNets of the SDXL era. If the patch downloads but the run fails, check that your VAE matches the model family too. And if nothing happens at all - no visible control - confirm your control image is actually a usable map (a real Canny/depth output, not the raw photo) and that strength isn't sitting at zero.