SD-CN Animation Txt2Vid

This is the flagship node of ComfyUI-SD-CN-Animation, and the thing to understand before any of its siblings: it's a port of volotat's SD-CN-Animation, the A1111 script that went around in spring 2023 and briefly blew people's minds with optical-flow-guided video. The name is a lie only in the sense that it's not a rival to Wan or LTX - it won't give you coherent minutes-long takes or Hollywood camera moves. What it gives you is something those models can't: any SD1.5 or SDXL checkpoint, ControlNet, LoRA, and IPAdapter you already have, turned into a video generator. If your style lives in a specific checkpoint, this node is the cheapest way to animate it.

How it works

Each new frame is predicted, not just sampled. FloweR - a ~8MB U-Net that watches the previous 4 frames and outputs optical flow, an occlusion mask, and a next-frame estimate - guesses where the scene is going. Then two SD passes make it real:

1. Inpaint pass - the FloweR prediction is fed through img2img at processing_strength, with the occlusion mask as the noise mask. Occluded areas (things that were hidden and just appeared) get fully regenerated; coherent areas barely change.
2. Refine pass - a second img2img at the much lower fix_frame_strength to clean up seams.
3. Histogram match - the frame is color-matched to the first frame in LAB space so the whole clip doesn't drift hue over time.

The first frame is plain txt2img (or img2img if you feed init_image). Loop through this a few hundred times and you have a video, one frame at a time. That's why it's slow: two full sampling passes per frame, every frame. It's the price of frame-to-frame coherence that AnimateDiff doesn't give you the same way.

Inputs that matter

The obvious ones are standard KSampler fare: model, vae, positive/negative, seed, steps, cfg, sampler_name, scheduler, width/height (FloweR wants multiples of 128, so 512x512 is a happy default). You also need flower_model from a Load FloweR Model node - without it there's no flow prediction at all.

The ones beginners actually tune:

• num_frames - total frames. Start at 30 to test; 90+ for a real clip.
• processing_strength - how much SD re-invents each frame. Default 0.85 is aggressive. The README's smooth-morphing recipe is 0.55; lower = smoother, higher = more creative drift.
• fix_frame_strength - keep at 0.1–0.2, or 0 to skip the refine pass and roughly halve per-frame time.
• loop_frames - blend the tail back toward frame one for a seamless loop. 10–20 with a 12–14 fps video combine is the standard recipe.

Optional: init_image seeds the first frame, control_net + cn_strength applies per-frame ControlNet (the FloweR prediction becomes the hint each frame - great with depth or lineart), and motion_ctrl comes from the SD-CN Motion Control node.

Output is a single frames IMAGE batch. Wire it into VHS_VideoCombine and set the fps yourself - the node outputs raw frames.

Install

ComfyUI Manager (search "ComfyUI-SD-CN-Animation"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/pxl-pshr/ComfyUI-SD-CN-Animation

Then restart ComfyUI. The only heavy-ish deps are opencv-python, scikit-image, and scipy; the ~8MB FloweR weights auto-download from Hugging Face on first launch into ComfyUI/models/FloweR/.

Gotchas

Where people get burned: this is a 2023 approach, and the original was open about its faults - ghosting and occasional flickering. ComfyUI's port doesn't fix the algorithm, it just gives you knobs and a graph. If your output shimmers, that's the technique, not a broken install. And the biggest trap is expecting prompt scheduling to steer much at low processing_strength - the README calls this experimental for a reason, because below ~0.5 denoise the prompt barely has room to act. Budget your patience and your GPU: it's a slow, deliberate node, and that's fine, because the look you get out of it is the one you already curated in your checkpoint.