GetWarpedNoiseFromVideo

This node is the ComfyUI version of the "Go with the Flow" trick that blew up in early 2025, and it's the most fun thing in a tiny pack that Kijai calls a work in progress. The idea is almost insultingly simple: instead of feeding your video sampler random noise and hoping it invents coherent motion, you feed it noise that has been physically pushed around by the motion of a real video. The diffusion model starts with temporal structure already baked into the latent, and that makes image-to-video dramatically steadier and more controllable.

Kijai shipped this as a slightly modified port of RyannDaGreat's noise_warp.py (and his rp library) plus code from Eyeline Research's Go-with-the-Flow repo - the Netflix team behind the paper. GetWarpedNoiseFromVideo is the generic, bring-your-own-model version. Its siblings are hard-wired for CogVideoX, HunyuanVideo, and AnimateDiff; this one lets you pick the latent shape and frame count yourself, which makes it the one to reach for when the target model isn't one of the three dedicated wrappers.

How it works

For every consecutive pair of frames in your input video, the node runs a RAFT optical flow model (torchvision's raft_large, downloaded automatically on first run) to estimate where each pixel moved. It then takes a seeded random-noise field and warps it along that flow, frame by frame, using RyannDaGreat's scatter-add noise warper - a particle-swarm-style trick that keeps the noise properly gaussian as it's dragged around, so you don't end up with smeared blobs. The warped noise is downscaled to latent resolution, collapsed from video frame count down to target_latent_count latent frames, and finally handed to you as a LATENT.

Crucially, the reference video's pixels never reach the sampler. Only the motion does, in the form of noise that has been distorted to match it.

The inputs that actually matter

Of the seven required fields, a beginner really touches four:

• target_latent_count (default 13) - how many latent frames the warped noise is interpolated to. For a video VAE with 4x temporal compression, that's (num_frames - 1) / 4 + 1, which is 13 for 49 frames.
• noise_downtemp_interp - how you get from video-frame count to latent count. nearest (the default) samples the warped noise; blend averages neighbors; blend_norm averages then renormalizes; randn throws the warp structure away and returns fresh noise; disabled keeps every frame.
• degradation (0 to 1) - blends the warped noise toward fresh random noise. At 0 you get pure warp structure; at 1 the warp is gone entirely. This is the "how hard do I trust the reference motion" dial.
• seed - seeds the initial noise field, so the same video + seed reproduces the same noise.

Then the model-specific part: latent_shape must match your target model. The tooltip is the honest guide - CogVideoX wants BCTHW, HunyuanVideo wants BTCHW, SD/AnimateDiff-style models want BCHW. Wire the optional model and sigmas in and the noise is scaled to the sampler's sigma range, so it's drop-in usable as sampler noise. spatial_downscale_factor defaults to 8, matching the 8x spatial downsampling of every VAE you care about; leave it.

Outputs

Three of them: noise (LATENT, wire it into your sampler's latent input), visualization (IMAGE - the noise field's first three channels normalized and stretched back to your video's frame count, so you can see whether the noise actually follows your motion), and optical_flows (IMAGE - the RAFT motion field rendered as an HSV image where hue is direction and brightness is magnitude). That last one is your debug tool: if the flow looks like static, RAFT isn't seeing the motion, and neither will your model.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-VideoNoiseWarp

Or just search VideoNoiseWarp in ComfyUI Manager and hit install. The only declared Python dependency is einops, and there are no model files to fetch by hand - the RAFT model downloads itself the first time you run the node (you need internet for that one run).

Where people get burned

This pack is genuinely marked WORK IN PROGRESS and hasn't been touched since spring 2025, so treat it as an experiment that happens to work, not a maintained library. A few real gotchas: degradation at 1 silently disables the whole effect, so if your output looks like ordinary noise, check it. The flow visualization imports opencv-python at runtime even though the requirements file doesn't list it - normal ComfyUI installs have it, but a stripped environment will throw on the optical_flows output. And because RAFT runs at video resolution on every frame pair, long or high-res reference videos are slow and VRAM-hungry; that's expected, not a bug. For a specific model, the dedicated CogVideoX, HunyuanVideo, or AnimateDiff variants are a better starting point than this generic one.