cv2.omnidir.projectPoints

Projection is the forward half of camera geometry: take known 3D points, a camera pose, and a lens model, and predict where those points land in the image. cv2.omnidir.projectPoints does that for the unified omnidirectional model (cv2.omnidir, the CMei formulation) - fisheye and catadioptric cameras, anything past 180° that a pinhole model can't describe.

The reason you'd run this deliberately is verification. Calibrate a fisheye, project a checkerboard's ideal grid back through the calibrated intrinsics, and overlay the result on the frame you calibrated from. If the dots land on the corners, your calibration is real. If they drift, they drift in a direction that tells you which parameter is wrong. It's also the mechanism behind AR-style overlays and measurement annotations in a wide-angle frame: project your 3D points, draw, and you're done.

Inputs

objectPoints, rvec, tvec, K and D are arrays; xi is a scalar FLOAT. That asymmetry is worth noting because the sibling cv2.omnidir.initUndistortRectifyMap takes xi as an array - same physical quantity, two different C++ signatures, and the generated wrappers reproduce the signatures rather than normalizing them. Read the socket type, not your memory.

• objectPoints - N×3 float32, the 3D points in the world/board frame. The pack's CV Grid Points node exists precisely to author a planar calibration grid; Parse Matrix takes it from text.
• rvec / tvec - the camera pose: rotation as a Rodrigues vector, translation as a 3-vector. These come from a calibration run or a solved pose.
• K / D - intrinsics and the omnidir distortion vector (four values in this model). xi is the mirror parameter: 0 is a pinhole, 1 a strongly curved mirror or fisheye.

Shape and dtype matter more than usual here: rvec wants 3×1 (or 1×3), tvec a 3-vector, K 3×3, and mismatches produce a raw cv2 assertion rather than a friendly message. CV Reshape Array is the usual fix, and the pack ships it for exactly this.

Outputs

Two NPARRAYs, unnamed - nparray_0 and nparray_1. In the underlying call the projected image points are the first output and the Jacobian is the second. The Jacobian is the derivative of image coordinates with respect to the parameters, which is what a solver consumes when it's optimizing a calibration; unless you're building one, you can ignore the second socket and let it dangle.

The projected points are N×2 or N×1×2 pixel coordinates. To look at them: convert with the pack's CV Points To Contour bridge and draw with the annotation/drawing nodes, or push them into a plot. Note the perspective-vs-omnidir difference as a sanity check - if you project the same points through the pinhole model and through this one, the omnidir result spreads them correctly toward the frame edge where a fisheye stretches, which is precisely the distortion a pinhole model can't reproduce.

Install

ComfyUI Manager → search ComfyUI CV → install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Python ≥ 3.12, ComfyUI on the V3 node API. cv2.omnidir is contrib-only, so a non-contrib opencv-python(-headless) installed over the contrib wheel silently removes this node along with the rest of the contrib set. tools/repair_opencv_contrib.py --check / --apply handles that. The blank tooltips on this node's parameters are the generator's doing - the contrib stubs carry no documentation - so OpenCV's own cv2.omnidir reference is where you go for the model's math.

Where it fits

If you're looking for 3D from ordinary photos, this is not the node - the ecosystem's default answer is a learned monocular model, one image, no rig, no calibration. omnidir.projectPoints is for the case where you do have the hardware: a fisheye or mirror rig, a calibration you want to verify, and a geometric answer you intend to trust. The pack's CV Omnidir Calibrate (Chessboard) node and the 54_omnidir_calibration example are the front half of that pipeline.