(deforum) Hybrid Motion

This is the node that turns a Deforum animation into a vid2vid pipeline. Instead of letting the sampler invent camera motion, you hand it a reference video and it borrows that video's motion: it computes the optical flow between frames of your generated image (or between your image and the reference), then warps your image along that flow so the scene moves with the reference. Think of it as "dance to this video's moves" - the backbone of the hybrid mode that the classic Deforum extension was famous for.

In the pack's workflow it sits between generation and the next frame: image is what your sampler just produced, hybrid_image is the reference frame you want it to follow, and the node returns a warped IMAGE that gets fed back into the loop.

Inputs

• image - your current generated frame.
• hybrid_image - the reference video frame. If you leave it unconnected, the node falls back to warping based on your own previous frames instead (self-motion, useful for camera-style motion on a static reference).
• deforum_frame_data - required. The node reads hybrid_flow_factor_schedule from here, resolved to the current frame, and uses that value as the warp strength.
• hybrid_method - the optical flow estimator: RAFT (most accurate, slowest, loads a model), DIS Medium, DIS Fine, or Farneback (CPU-cheap, rougher).

The output is a single IMAGE - the warped frame, ready to re-enter the loop as the next init.

How it actually works

Look at the source and the pattern is clear. On the first call it has no previous frame, so it stores one and passes your image through unchanged. From the second frame on it computes flow between the stored previous frame and the current reference, warps your image by that flow scaled by the flow factor, then keeps only the latest frame for the next comparison. It keeps a RAFT model around after the first load - which is why the first frame with RAFT selected can feel like it stalls: the model downloads and loads once, then stays resident.

Installation

It's part of Deforum Nodes:

cd ComfyUI/custom_nodes
git clone https://github.com/XmYx/deforum-comfy-nodes.git

Restart ComfyUI, or install "Deforum Nodes" by XmYx via ComfyUI Manager. This node is one of the heavy ones: RAFT and the DIS models come from the pack's deforum[comfy] backend (opencv-contrib, the deforum-studio fork, Python 3.10 per the README), and the first RAFT call downloads model weights. Don't panic at the pause.

Common issues

• Nothing happens on the first frame - expected. The node needs two frames to compute a flow; frame one is a pass-through by design.
• First RAFT call is slow - model download + load. Subsequent frames are the fast ones.
• Motion too strong or too weak - that's the hybrid_flow_factor_schedule from Hybrid Schedule, not this node. Drop the factor if the warp tears the image; a factor above 1 is asking for distortion.
• Warped frames get smeary - optical flow warping has no imagination; on fast reference motion it drags pixels. Lower the flow factor or use DIS Fine over Farneback for smoother results.

One opinionated note: hybrid motion is where Deforum's "it's not really motion, it's warping" reputation comes from. It's the right tool when you want a specific, controllable camera move on a hallucinated scene. If you want real temporally-coherent motion, the KB's line holds - a modern video model is the better end state.