ControlNext Sampler

This is the workhorse of the pack. ControlNext Sampler is where the diffusers-wrapper path of Kijai's ControlNeXt-SVD actually generates the video: you feed it the pose frames from ControlNext GetPoses, a reference image, and the CONTROLNEXT_PIPE that (Down)Load ControlNeXt built, and out comes a LATENT you hand to ControlNext Decode. If you use this pack at all, you use this node.

For context: ControlNeXt-SVD is DVLab's (Peking University) pose-to-video model from mid-2024 - a ControlNet-style adapter for Stable Video Diffusion. The community filed it next to Mimic Motion and AnimateDiff when it dropped, and the honest verdict was that it gives you real pose control but the raw output looks soft, so most people run a detailer pass on top. Kijai's wrapper is essentially the only way anyone actually runs the thing, and this node is the core of that wrapper.

How it works

The node runs the whole diffusers StableVideoDiffusionPipelineControlNeXt - not a patched ComfyUI model. That's the trade-off that defines the wrapper path: it's self-contained and auto-downloading, but it's a black box compared to the native path that ControlNext SVD Apply gives you.

Mechanically, SVD here is the img2vid-xt-1.1 architecture. The reference image gets VAE-encoded and repeated across the frame sequence as the conditioning image; the pose skeleton frames get normalized to [-1, 1], preprocessed, and multiplied by controlnext_cond_scale to become the ControlNeXt condition. Guidance ramps from cfg_min to cfg_max linearly across the batch of frames, which is the same per-frame guidance ramp stock SVD uses. Sampling runs in chunks: context_size frames per batch with context_overlap frames of overlap, and overlapping regions are averaged rather than just cut - that's the "context windowing" that keeps long clips coherent and keeps the UNet's VRAM footprint reasonable.

The inputs that matter

A beginner genuinely sets about four of these:

• pose_images / ref_image - your DWPose skeleton frames and the still frame the video animates from. They have to be the same resolution or the node asserts.
• steps (default 25) - plain old diffusion steps. 25 is fine; the example workflows drop to 10 with the AYS scheduler.
• context_size (default 24) and context_overlap (default 6) - your VRAM levers. Fewer frames per batch = less memory; overlap smooths the seams between chunks. And note the assert: you need at least as many pose frames as context_size.
• controlnext_cond_scale (default 1.0) - how hard the pose condition grips the output. Lower it if the video stops following your skeleton; that's the "strength" dial of this path.

motion_bucket_id (127) is SVD's motion-intensity knob - higher is more motion - and cfg_min/cfg_max (both default 3) set the guidance ramp range. noise_aug_strength (0.02) adds conditioning noise to the ref image, the stock SVD trick that keeps videos from being static; you rarely touch it. keep_model_loaded (true) keeps the unet/VAE in VRAM between runs at the cost of having nothing left for other jobs - uncheck it when you're doing other stuff between generations. seed, fps, and steps are what they look like.

The only output is samples (LATENT) - no image yet. That goes into ControlNext Decode, not VAEDecode.

Installing it

Install the pack first - search "ComfyUI-ControlNeXt-SVD" in ComfyUI Manager, or clone it manually:

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-ControlNeXt-SVD
# restart ComfyUI

The wrapper path needs diffusers>=0.30.0, accelerate, transformers, opencv-python and huggingface_hub (in the pack's requirements.txt). Model downloads are automatic via DownloadAndLoadControlNeXt - the UNet plus ControlNeXt adapter land in ComfyUI/models/diffusers/controlnext/, and the SVD VAE/image encoder land in models/diffusers/stable-video-diffusion-img2vid-xt-1-1/. First run downloads several GB, so grab a coffee.

Common issues

• "The number of poses must be greater than the context size" - your pose batch is shorter than context_size. Reduce context_size or feed more frames.
• OOM - this is a video UNet at 1024x576; 24-frame batches want real VRAM. Drop context_size to 8-12 and context_overlap to 4, or switch the optional ControlNext Diffusers Scheduler to fewer steps.
• Soft, wobbly output - that's the model, not you. The community's standing fix is a detailer/animate pass on the decode, because ControlNeXt-SVD's pose control is the point, not its fidelity.