RAFT Optical Flow 🌊

If you've spent any time with Wan video workflows, you've hit the point where "video ControlNet" means VACE. VACE is Alibaba's control framework for Wan - reference-to-video, extension, inpainting - and it's driven by control videos: pose, depth, or optical flow sequences that tell the model how things should move. FERaftFlow is the optical-flow option. It computes dense optical flow between consecutive frame pairs using RAFT-Large (from torchvision) and outputs a Middlebury-coded RGB flow video that plugs straight into a VACE V2V control input.

Translation: you feed it a video, it paints a colour-coded picture of the motion (each colour = a direction), and you feed that into VACE so a Wan-based pipeline preserves the original footage's movement instead of inventing its own. For the VFX crowd this pack targets - stabilizing, rotoscoping, restyling existing footage - that's the difference between "reimagined" and "the movement you actually shot."

How it works

The node is honest about being a thin wrapper: it loads torchvision.models.optical_flow.raft_large, runs it over your frames, and encodes the result as Middlebury-style RGB flow with per-clip global normalisation - a choice the source explicitly ties to VACE's training conventions, so magnitude and direction land the way the control model expects.

Two details separate this from a naive RAFT script, and both are about not blowing up your machine. First, RAFT builds an all-pairs correlation volume that is O(H²×W²). At native 2K (2048×1080) that's about 4.5 GB per batch item - enough to kill RAM before you ever see VRAM. max_flow_size caps the longer edge of the resolution RAFT actually runs at (default 768; set 2048 to disable downsampling), and the flow is bilinearly upsampled back to the original resolution afterwards with magnitudes rescaled. For a VACE control signal, 512–768 is plenty - the model doesn't need 2K of flow vectors to understand the motion. Second, chunk_size batches frame pairs per forward pass (default 4), so you trade speed against VRAM instead of hitting a wall.

The weights download automatically on first run to {ComfyUI}/models/raft/raft_large.pth (a ~20 MB file from the torchvision archive), so there's no manual model hunt.

Inputs and the output

• images (required) - source frames [N, H, W, 3] float32 in 0–1, e.g. straight out of VHS LoadVideo. Minimum 2 frames - flow needs pairs.
• iters - RAFT refinement iterations, default 20 (the paper's standard). Higher = more accurate, slower. You'll rarely need more than 20.
• chunk_size - frame pairs per forward call, default 4. Higher = faster but more VRAM; drop it if you OOM on long clips.
• max_flow_size - the longer-edge resolution cap described above. Default 768 is a good starting point.
• flow_frames (output) - the Middlebury-coded RGB flow batch, ready for a VACE V2V conditioning node or a video saver.

Installing it

FEnodes installs as usual - ComfyUI Manager (search "ComfyUI-FEnodes") or:

cd ComfyUI/custom_nodes
git clone https://github.com/FugitiveExpert01/ComfyUI-FEnodes.git

Restart ComfyUI. It uses torchvision, which any ComfyUI install already has; the only new download is the RAFT weights on first run. Note this node isn't in the README's node table yet - it shipped in the code and the docs lagged it. It's real, it loads, it's just a young pack.

Common issues

• OOM or system RAM death on big clips. That's the correlation volume. Lower max_flow_size toward 512–640 before touching anything else; if it still dies, drop chunk_size to 2 or 1.
• Single-frame input does nothing. The node needs at least two frames; flow is always between consecutive pairs, so frame 0 and the last frame of the output only have flow in one direction.
• Flow looks noisy or jittery. Bump iters toward the 20 default if you lowered it, and remember the flow is upsampled back to original resolution - visual softness there is normal and fine for a VACE control signal, which doesn't need razor-sharp vectors.