cv2.warpFrame

cv2.warpFrame takes an RGB-D frame - colour image, depth map, mask - and reprojects all three into a new camera pose. Give it a 4×4 rigid transform and the intrinsics, and it hands back the frame as it would look from the other position. If you've ever wondered how engines do temporal reprojection or how a depth-driven parallax effect would look done "properly" instead of as a 2-D UV shift, this is the primitive: real geometry, real occlusion, no neural network involved.

Two honest caveats before you invest a weekend in it. First, this is a genuine OpenCV 5 function from the 3-D module, not one of the ancient workhorses - the pack is curated against opencv-contrib-python-headless 5.0.0.93, and the wrapper is only generated if the installed cv2 has the function. Second, in ComfyUI terms it's a research node. Nothing in this pack's example workflows drives a diffusion pipeline through it; it's geometry you attach to depth data you already have.

How it works

An RGB-D frame is a picture with a distance per pixel, which makes it almost a point cloud. Reprojection is then the standard three-step dance: unproject each pixel to a 3-D point using the depth and the camera matrix, apply the rigid transform Rt, project back into the destination camera, and resolve what overlaps what. The mask input is what you have that says which pixels are worth carrying. The result isn't a warp of the picture - it's a view, which is why disocclusions appear as holes: geometry has no opinion about what was behind the sofa.

The tooltip for Rt is the author's own, and it's the one that matters: "4x4 rigid transform from the source camera to the destination camera." Rotation plus translation, no scale, no shear - a camera move. A pure translation gives you parallax; a rotation gives you a pan.

Inputs and outputs

• depth - the depth map. NPARRAY, IMAGE or MASK all accepted; frame 0 of a batch.
• image - the colour frame.
• mask - the validity/support mask. It's required - pass the capture's mask, or make a constant one.
• Rt - the 4×4 transform, NPARRAY only (data, not a picture). Build it with CV Pose To Matrix, CV Compose Pose (Trajectory), or Parse Matrix if you're typing a matrix in.
• cameraMatrix - the 3×3 intrinsics, again NPARRAY only. CV Camera Matrix builds a K from fx/fy/cx/cy; CV Load Camera Params (JSON) reads one back from a calibration file.
• warpedMask - optional, only if you want to thread an existing warped mask in rather than have one produced.

The outputs are the auto-generated trio: nparray_0, nparray_1, nparray_2. Generated wrappers don't rename their returns, and this function returns the warped image, depth and mask together - hover the sockets or run it once and inspect the shapes (the pack's Inspect CV Data reports shape, dtype and statistics of any array) to confirm which is which before you wire them onward.

Where it fits

Depth in ComfyUI is mostly a conditioning signal: a depth estimator like Depth Anything or Marigold produces a monocular map, and you hand it to ControlNet. That map is relative, not metric, so it's good for look and bad for measurement. warpFrame wants the other thing - a depth map with real units and a known camera pose, which in practice means a captured RGB-D frame, a stereo reconstruction, or a render. The related wrappers in this pack are cv2.depthTo3d / cv2.depthTo3dSparse (depth + K to an XYZ map) and cv2.registerDepth (which uses the same Rt convention to register a depth frame into a colour camera).

Installing it

Ships in ComfyUI CV. Manager → search ComfyUI CV, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"
# restart ComfyUI

Python ≥3.12 and a recent, V3-API ComfyUI. If the node isn't in the menu at all, your cv2 build doesn't expose warpFrame - the wrapper set is probed against the installed library at import time, and missing functions are skipped rather than failing.

What goes wrong

• Holes everywhere. Disocclusion is physics, not a bug. Mask-fill or inpainting downstream if it bothers you, or keep the transform small.
• Everything projects to a point or off-canvas. Almost always a units mismatch: Rt's translation is in the same units as your depth, and if your depth is normalised 0–1 the geometry makes no sense. Depth is depth, not a heatmap.
• Depth and colour don't line up. warpFrame isn't a registration tool - the two inputs must already be aligned cameras. Run the pair through cv2.registerDepth first if they aren't.
• Missing node after install. Check your OpenCV wheel: it needs to be a 5.x contrib build, and only one of the four opencv-python* distributions gets to win the shared site-packages/cv2. tools/repair_opencv_contrib.py --check in the pack's repo diagnoses a clobbered install.