ControlNext SVD Apply

This is the node that makes ControlNeXt-SVD feel like ComfyUI instead of a wrapped diffusers pipeline. Where the rest of the pack runs a self-contained diffusers wrapper, ControlNext SVD Apply does the classic ControlNet move: it takes a normal MODEL, injects the ControlNeXt adapter into it as a patch, and hands you back a patched model you can run through a stock KSampler and VAEDecode. The README calls the native implementation work in progress, and it is - but it's the path you reach for once you want actual graph control.

How it works

You feed it a MODEL loaded from the converted ControlNeXt-SVD UNet, plus your pose skeleton frames. The node keeps its own copy of the ControlNeXt adapter - a small controlnext-svd_v2-controlnet-fp16.safetensors that ships inside the repo, no download - and patches the model's input blocks with it. Three settings do the actual work:

• strength (default 1.0) - the overall control weight, same idea as a ControlNet's conditioning scale. This is the dial you actually tune.
• blocks (default "3") - a comma-separated list of UNet input blocks to patch, 0-3. The source's own comment says blocks 0-3 are testable and 3 after skip has been best so far, which matches the default. Don't feel the need to experiment; the default is the sweet spot.
• input_block_patch_after_skip (default true) - whether the injection lands after the skip connection. Leave it on.

The mechanism is worth understanding because it explains the "fiddly" reputation: the adapter's control latents get normalized to match the hidden state's mean/std, adaptive-pooled to the UNet's spatial size, and added as h + control * 0.2 * strength - the 0.2 is a fixed scaling baked into the patch. It also sigma-normalizes so the control strength stays meaningful across the denoising schedule, and it respects AnimateDiff's context windowing when present, reusing the ad_params sub-indexes so patches line up with AnimateDiff's own frame chunks. That's why the example native workflow pairs this node with ADE_StandardStaticContextOptions and ADE_UseEvolvedSampling.

Output is model (MODEL), wired straight into a KSampler.

The workflow around it

The native example (controlnext_svd_comfy_01.json) shows the shape: ImageOnlyCheckpointLoader for the SVD checkpoint, a UNETLoader pointing at controlnext-svd_v2-unet-fp16_converted.safetensors (from Kijai/ControlNeXt-SVD-V2-Comfy, into ComfyUI/models/diffusion_models/), SVD_img2vid_Conditioning for the added-time conditioning, then ControlNext GetPoses → this node → a KSampler at ~10 steps, cfg 2.5, euler/karras → VAEDecode → VHS_VideoCombine. Because it leans on AnimateDiff-Evolved's context nodes and VideoHelperSuite's video nodes, those packs are effectively required for the native workflow even though this node itself only needs ComfyUI core.

Installing it

The pack is the same - ComfyUI Manager (search "ComfyUI-ControlNeXt-SVD") or:

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-ControlNeXt-SVD
# restart ComfyUI

The only model you download by hand is the converted UNet (~a few GB) into models/diffusion_models/. The ControlNeXt adapter file is already in the repo, and unlike the wrapper path this node does not need the whole diffusers stack - which is precisely why people run it: fewer moving parts, native ComfyUI patching, AnimateDiff context.

Common issues

• Node complains the UNet is missing - you loaded the wrong weights. This node expects the ...-unet-fp16_converted.safetensors in models/diffusion_models/; the non-converted file in models/diffusers/controlnext/ is for the wrapper path only.
• Control has no effect - check blocks and input_block_patch_after_skip are at their defaults ("3", true), and that your pose frames actually detected a body in ControlNext GetPoses. With no skeleton, there's nothing to follow.
• Working with AnimateDiff context - if your clip is longer than the AnimateDiff context and the pose alignment drifts, the fix is to make sure the context options match the pose batch, not to crank strength - the patch reuses AnimateDiff's sub-indexes, so mismatched context settings break the mapping.