(Down)Load ControlNeXt

(Down)Load ControlNeXt is the front door of the pack's diffusers-wrapper path. One node, one output - controlnext_pipeline (CONTROLNEXT_PIPE) - but it does a lot of heavy lifting on the way: it downloads the ControlNeXt-SVD weights, downloads the Stable Video Diffusion components it needs, loads every piece into VRAM, and assembles the whole StableVideoDiffusionPipelineControlNeXt you'll hand to ControlNext Sampler and ControlNext Decode. Nothing else in the wrapper path runs until this node has done its thing, and "its thing" includes several gigabytes of first-run downloading.

The one input that matters

• precision (default fp16) - fp32, fp16, or bf16. fp16 is the right default for most cards. bf16 behaves better on Ada/Ampere-class cards that handle it natively; fp32 is the "doesn't fit, slow, but exact" escape hatch you reach for when fp16 artifacts are wrecking a clip. For a video UNet, fp16 is where you live.

The output is the single controlnext_pipeline wire. It carries the VAE, image encoder, UNet, scheduler, feature extractor, and the ControlNeXt adapter all bundled in one pipe object - which is why you can feed it into both the sampler and the decoder.

What it downloads (and where)

It pulls from two HuggingFace repos, and knowing where matters for debugging:

• Kijai/ControlNeXt-SVD-V2-Comfy → ComfyUI/models/diffusers/controlnext/ - the ControlNeXt-SVD UNet and the small ControlNeXt adapter. Note it deliberately ignores files matching *converted*; the converted UNet is for the native ControlNext SVD Apply path.
• vdo/stable-video-diffusion-img2vid-xt-1-1 → ComfyUI/models/diffusers/stable-video-diffusion-img2vid-xt-1-1/ - the SVD XT 1.1 VAE, image encoder, scheduler and feature extractor, excluding the UNet (ControlNeXt replaces it).

Then it loads: the ControlNeXt UNet via accelerate's empty-weights init so a big model doesn't blow past VRAM during load, the SVD temporal VAE, the CLIP vision encoder, and the ControlNeXt adapter. First run is the slow one - you'll watch a progress bar crawl through two snapshot downloads before anything generates. Second run skips straight to loading.

Installing it

The pack installs normally - ComfyUI Manager (search "ComfyUI-ControlNeXt-SVD") or:

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-ControlNeXt-SVD
# restart ComfyUI

The wrapper path's real dependency list is the pack's requirements.txt: diffusers>=0.30.0, accelerate, huggingface_hub, transformers, opencv-python. If ComfyUI Manager doesn't auto-install them (it usually does), pip install -r requirements.txt inside the pack folder. accelerate isn't strictly optional here - the loader uses its empty-weights path for the UNet.

Common issues

• It hangs on "Downloading model" - network, not you. The snapshot downloads can be slow; if it truly stalls, check the target folders exist and re-run, since huggingface_hub resumes.
• CUDA out of memory on load, before you even sample - the loader loads the UNet, VAE and image encoder all at once. On a small card, try bf16, and know that this path is inherently VRAM-hungrier than the native ControlNext SVD Apply route precisely because it holds the whole diffusers stack.
• Missing diffusers classes / import errors - an outdated diffusers. The pack needs >=0.30; a stale version breaks the pipeline import before the node even shows results.
• Wrong file in the wrong folder - if you manually moved model files, remember the loader wants the non-converted UNet in models/diffusers/controlnext/, and it will re-download if it doesn't find what it expects.