cv2.registerDepth

The problem it solves

On an RGB-D rig - a Kinect, a RealSense, any depth camera plus a colour camera - the two sensors are a few centimetres apart and have different lenses. So the depth map's pixel (x, y) and the colour image's pixel (x, y) are looking at different points in the world. Before you can do anything useful - texture a point cloud, put a virtual object behind a real desk, cut a person out with metric depth - you need the depth remapped onto the colour camera's image grid.

That's this node. It warps the depth map as viewed by the depth camera into the registered (colour) camera's frame, at the output plane size you ask for. Depth values pass through the warp unchanged in meaning; the pixels move, their depths don't.

Its core input is the pack's own tooltip for Rt: "4x4 rigid transform from the depth camera to the colour camera." That single sentence is the node's whole geometry - everything else is two sets of intrinsics and a size.

Inputs

• unregisteredCameraMatrix - 3×3 K of the depth camera, i.e. the camera the incoming depth map is expressed in.
• registeredCameraMatrix - 3×3 K of the camera you want the depth on (usually the colour camera).
• registeredDistCoeffs - the target camera's distortion coefficients. This overloaded name is the one place people trip: it's the destination camera's distortion, not the depth camera's - check the OpenCV signature if your result looks warped in a way that no amount of Rt-fiddling fixes.
• Rt - 4×4 extrinsics, depth camera → colour camera.
• unregisteredDepth - the depth map, single channel, in the depth camera's own frame.
• outputImagePlaneSize - a CV_TUPLE, the size of the target plane. In practice the colour image's width and height; author it with CV Tuple or type (w, h).
• depthDilation (optional, default false) - dilates the registered depth. This is the pinhole patch: where depth and colour disagree at silhouettes, the warp leaves one-pixel gaps, and growing the depth values covers them. Turn it on when your textured cloud has speckled holes along object edges.

Output: nparray - a depth map on the target camera's image plane, same size as outputImagePlaneSize.

Where it fits in this pack

This is one of the RGB-D helpers that sit in the top-level cv2 namespace in OpenCV 5 (the generator's core registry has it under a plain, undotted name), which is why it isn't a contrib-only node. On an older OpenCV it lived under cv2.rgbd; if the node is missing from your menu, that's likely why.

Downstream, the registered depth is the input to the pack's 3D lane: back-project to a coloured point cloud, filter it, export it (CV Write PLY), or view it in the pack's dedicated three.js viewer (CV Preview 3D (Calibrated Camera)), which takes a real K and depth-map occlusion. For a stereo-derived depth map on pixel grids that already match you don't need this at all - cv2.reprojectImageTo3D with a Q matrix is the shorter path, and it's what the pack's stereo examples use.

Be aware that no shipped example workflow wires this node. It's in the auto-generated wrapper set, not in the curated examples, so there's no canvas to copy from and the parameter tooltips are mostly OpenCV's docstrings passed through. Budget twenty minutes with the OpenCV reference before you budget two hours with your own confusion.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Or ComfyUI Manager → search "ComfyUI CV" (bmad4ever). The pack needs Python ≥ 3.12 and a ComfyUI recent enough for the V3 node API. Restart ComfyUI, reload the page.

Troubleshooting

The output is empty, black, or all the same value. Check the units first: depth in millimetres versus metres changes nothing about the warp (it's a rigid transform), but feeding a uint8-scaled depth map that lost its precision gives you a depth map that's quantised to 255 levels and useless for metric work. If your depth came out of a 16-bit PNG, get it in as an NPARRAY rather than as a ComfyUI IMAGE, and consider cv2.rescaleDepth to convert millimetres to metres and turn zeros into NaN before it goes any further.

Everything is offset in one direction. Wrong Rt, or Rt in the wrong direction - the tooltip's direction (depth camera → colour camera) is not symmetric, and a transposed rotation looks like a plausible-but-shifted result rather than a crash.

Gaps along object boundaries. That's exactly what depthDilation is for.

Wrong output size. outputImagePlaneSize is a size, (width, height) - and the tuple order catches everybody at least once.

A cv2 attribute error naming a function you can't find. If it's a contrib function from another node, verify your wheel:

python ComfyUI/custom_nodes/comfyui_cv/tools/repair_opencv_contrib.py --check