RGB SparseCtrl 🛂🅐🅒🅝

This is the preprocessor for the RGB flavor of SparseCtrl. The RGB SparseCtrl model conditions on actual images (a start frame, an end frame) rather than sketches, and to do that it needs those images encoded into the model's latent space and shaped to match your animation's frame batch. That encoding is what this node does. Feed it your image, a VAE, and a reference for the batch size, and it hands back a processed image ready for the SparseCtrl apply chain.

You reach for this specifically when you're using the RGB SparseCtrl model to animate from real frames - image-to-video, keyframe interpolation, that kind of thing. For the scribble SparseCtrl model you don't use this node.

How it works

SparseCtrl conditions only a few frames and lets the AnimateDiff motion module fill the rest. The RGB encoder expects its conditioning frames in a particular encoded form and sized to the full frame batch, with the un-conditioned frames left empty. This node runs your image through the VAE to produce that encoded representation and pads it out to the batch length implied by latent_size, so what comes out is something the RGB SparseCtrl model can actually read as sparse hints. In other words, it's the translation step between "here's a picture" and "here's a SparseCtrl-shaped conditioning tensor."

The inputs and outputs that matter

Three required inputs, all necessary:

• image (IMAGE) - the frame(s) you want to condition on.
• vae (VAE) - the VAE that encodes the image into latent space. Use the one that matches your checkpoint.
• latent_size (LATENT) - a latent that defines the batch/frame count and dimensions to shape the output to. Wire in the same empty latent your sampler uses, so the frame counts line up.

Output is proc_IMAGE - the processed image, which goes into the image input of your Apply Advanced ControlNet node (the one driving the RGB SparseCtrl model).

How to install it

ComfyUI Manager: search ComfyUI-Advanced-ControlNet, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet

then restart. No dependencies for the node itself. The workflow around it needs the RGB SparseCtrl model in ComfyUI/models/controlnet and ComfyUI-AnimateDiff-Evolved for the motion module.

Common issues & troubleshooting

Frame counts don't match / shape errors at sampling. The latent_size you pass has to describe the same batch your sampler runs. Feed in the identical empty latent so the frame count and dimensions agree - a mismatch here is the usual cause of a SparseCtrl blow-up.

Colors or details come out wrong. Make sure the vae matches your base checkpoint. A mismatched VAE encodes the image incorrectly and the conditioning drifts.

You're using the scribble model. Then you don't want this node - RGB preprocessing is only for the RGB SparseCtrl encoder. Sketch conditioning takes a different path.