AP Apply RAFT Optical Flow (Latent)

The image variant of this node warps frames in pixel space. This one warps them in latent space instead, and that choice is the entire reason it exists.

Here's the situation it's built for. You're doing iterative video work: warp the previous result forward, sample again, repeat. If you keep everything in pixel space, every iteration needs a VAE decode of the previous result and a VAE encode of the new start - and each round trip through the VAE is time, VRAM, and a chance for the latent representation to drift. Do that a few dozen times across a loop and you've burned real minutes on nothing but encode/decode overhead. APApplyRAFTOpticalFlowLatent takes a LATENT in, warps the latent samples directly with the flow field, and hands you a warped LATENT out. No pixel detour.

That makes it the right tool for loop-based temporal pipelines where the loop body is latent - the pack's own APLoopOpen (Latent) / APLoopClose (Latent) are explicitly designed around "avoid repeated VAE encode/decode in iterative latent workflows."

The mechanism

The flow field was computed in pixel space at your frame resolution. Latent samples are smaller by the VAE's compression factor (8× for SD1.5/SDXL-class VAEs), so the node resizes the flow down to the latent's resolution and warps the samples with the same grid-sample math as the image version. It carries the latent's other metadata (like the original samples tensor's shape) through untouched, so what comes out is still a valid LATENT dict you can feed a sampler.

One caveat baked into the physics of this: flow estimated from pixels describes pixel motion, and after the VAE's compression the latent channels mix spatial information. A flow warp on a latent isn't pixel-identical to a pixel warp that's been encoded. In practice it's close enough for a start-of-iteration warp, which is exactly how the README positions it - warp in latent, let the sampler finish the job.

The inputs that matter

Mostly identical to the image apply node, with one naming swap: batch_mode uses repeat_latent instead of repeat_image. Otherwise you get the same flow_direction, strength, invert_flow, interpolation, padding_mode, flow_skip/frames_skip, and the optional current_frame_index for index-aware loop alignment.

The ones worth your attention as a beginner:

• latent - the LATENT to warp. Usually the previous frame's latent output from a sampler or VAE.
• flow_data - the AP_FLOW object from APGetRAFTOpticalFlow, computed on the corresponding pixel frames.
• strength - flow multiplier, same as the image version. At 1.0 the latent moves exactly as the flow says; lower for a partial nudge toward the next frame.
• flow_direction - ab to move from frame A toward frame B. Get it backwards and your temporal loop will run in reverse.

Outputs are warped_latent (the LATENT to hand to a sampler) and valid_mask (a MASK at latent resolution marking warp-valid regions, which is useful if you want to blend the warped latent with a fresh one regionally).

Installing it

Same as every node in this pack - ComfyUI Manager (search "AP_OpticalFlow") or:

cd ComfyUI/custom_nodes
git clone https://github.com/adampolczynski/ComfyUI_AP_OpticalFlow
python -m pip install -r custom_nodes/ComfyUI_AP_OpticalFlow/requirements.txt

Then restart. The only dependency is torchvision>=0.15; the RAFT weights download on first flow compute.

The honest tradeoff

Latent warping is the right call when you're iterating many times and the VAE round-trips would dominate your runtime - which is most recursive temporal pipelines. It's the wrong call when you need pixel-exact warping for compositing against the original image, because the composite needs to happen in pixel space anyway (that's APFlowComposite's job). And note the masked latent variant exists too: APApplyRAFTOpticalFlowLatentMasked if you need to restrict the warp to a region without leaving latent space. Pick based on whether your next step is a sampler or a composite, not on which looks fancier.