cv2.depthTo3dSparse

Where cv2.depthTo3d back-projects every pixel into an H×W×3 map, cv2.depthTo3dSparse back-projects only the points you name and returns their 3D coordinates: give it a depth map, camera intrinsics and a list of pixel positions, get a small list of XYZ points back. Same pinhole maths, no full-frame work.

That's exactly what you want when the interesting pixels are a handful of features rather than a dense surface - a set of corners you detected, a few landmarks, points you clicked on a preview.

Inputs

depth is the depth map and, like the dense version, accepts an NPARRAY link or an IMAGE/MASK directly. in_K is the 3×3 intrinsic matrix of the depth camera and accepts only an NPARRAY - the tooltip is blunt about it: a data array, not an image. in_points is the list of pixel positions, also NPARRAY-only, and this is where the shapes bite. cv2's point-list convention is an N×1×2 or 1×N×2 array of 2-channel points - the layout the pack's feature and corner nodes produce - not an N×2 float table. If nothing sensible comes back, check the shape first.

Note the prefix convention: the dense node takes K, this one takes in_K and in_points. cv2's own naming, inherited by the generated wrapper.

One NPARRAY output: the 3D positions of those points, in camera coordinates, in the depth unit's scale. Same unit trap as the dense version - normalised 0–1 AI depth gives you a cloud the size of a thimble.

What it's good for

Anything where you'd otherwise compute a full point cloud to read three numbers out of it. Triangulation and PnP-style workflows: if you have 2D points in one view with reliable depth, this turns them into 3D correspondences, which is the input CV Solve PnP Pose and its iterative siblings want. Augmented-reality-style anchoring on a specific feature. Distance-between-two-pixels measurements, where you back-project both endpoints and take a norm. Sampling a depth map at landmark positions that came out of a face or pose detector.

It's also the cheap way to sanity-check a depth pipeline before you commit to a dense reconstruction: pick a few ground-control pixels where you know roughly how far away the object is, run them through, see whether the numbers are plausible. Every geometry bug I've met - swapped focal lengths, transposed intrinsics, wrong units - shows up immediately in three points and stays invisible in a 1920×1080 cloud for another hour.

If you do want the dense surface, the pack's CV Depth to 3D Points node is the practical one to reach for: it gives you an N×3 cloud with colours and drops invalid pixels, which is what CV Write PLY consumes.

Install

Manager → Install Custom Nodes → ComfyUI CV, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Restart ComfyUI. Python ≥ 3.12 plus a recent V3-API ComfyUI; the contrib headless OpenCV wheel is the only dependency and there are no models to download for the low-level nodes. A stray non-contrib opencv-python install overwrites the shared site-packages/cv2 and strips the contrib nodes - python tools/repair_opencv_contrib.py --check, then --apply. The pack is GPL-3.0, forked from geroldmeisinger/opencv-comfyui, mostly AI-written per the author's own disclosure, with a firm "not production-ready, support not promised" note.

Common issues

• An assertion about channels or shape on in_points. cv2 wants 2-channel point data (N×1×2 / 1×N×2). Reshape with the pack's array nodes, or feed points straight from CV Detect Corners / CV Detect Features.
• Zero or NaN results for every point. Those pixels have no depth (holes, sky, out-of-range). Sample from a region with valid values; CV Depth to 3D Points logs how many pixels it dropped for the same reason.
• Points are in reversed or mirrored positions. Your point list is (x, y) but arrived as (row, col), or the image was transposed on the way in. Print the first few coordinates.
• Numbers are tiny. Relative 0–1 depth rather than metric. Scale it before the node.