cv2.fisheye.projectPoints

cv2.fisheye.projectPoints is the node that answers "if the camera really is here, where do these 3D points land in the picture?" That makes it the sanity check for every fisheye pose you get from a solver, and the thing that draws overlays - a projected chessboard, a model's axes, a wireframe of a tracked object - back onto a wide-angle frame.

It's one of ~470 auto-generated raw cv2.* wrappers in ComfyUI CV (bmad4ever), which wraps OpenCV 5.0 as ComfyUI nodes. Worth knowing about the pack: it's a GPL-3.0 fork of geroldmeisinger's opencv-comfyui, rewritten on ComfyUI's V3 node API, and the author says plainly in the README that it was built with heavy LLM assistance and shouldn't go into production un-audited. The plus side of that origin is breadth; the minus is that these raw wrappers hand you OpenCV's signatures and expect you to know what they mean. There is essentially no community chatter about this pack on Reddit - this is one you read the source for, not the comment section.

Why a separate fisheye function exists

A wide lens doesn't obey the pinhole model. cv2.fisheye implements the equidistant (Kannala–Brandt) model with four coefficients k1..k4 instead of the pinhole camera's five, and the projection is a theta polynomial rather than a lens-formula divide. That's why a pose solved with the pinhole solver on a fisheye image looks nearly right in the centre of the frame and drifts at the edges - the classic "my overlay lines up in the middle and slides off at the corners" symptom.

How it works

You feed object points in world/model space plus the pose (rvec rotation vector, tvec translation), the intrinsics K and the four distortion coefficients D, and OpenCV applies the fisheye projection. The optional alpha composes a new camera matrix on the way in, so the projected points come out in the straightened frame rather than the raw one - blank leaves it at OpenCV's default of 0, which keeps the original K.

Two sockets come out and neither is a picture: nparray_0 is the projected image points (and depth-coverage info for the ones behind the camera), nparray_1 is the Jacobian. The Jacobian is a derivative matrix used by hand-rolled bundle adjustment; if you don't know why you'd want it, you don't want it.

The sockets that matter

• objectPoints - an Nx3 cloud, but remember this is a fisheye entry point: OpenCV's fisheye functions want the (1, N, 3) convention that the pack's own CV Fisheye Calibrate node also works around. Grab one from CV Grid Points for a chessboard, or reshape a hand-built matrix with CV Reshape Array.
• rvec / tvec - straight from cv2.fisheye.solvePnP or a tracking node's pose stack. Here's the format rabbit hole: both are NPARRAY sockets, so they pass a real ndarray through untouched, and a (3,1) float vector is what cv2 expects.
• K / D - K is the 3×3 from calibration; D must be four values. Feed a five-element pinhole vector in and you get a plausible-looking wrong projection, not an error. Get them from CV Fisheye Calibrate (Chessboard) or CV Load Camera Params (JSON).
• alpha is an advanced optional string - type a Python literal or leave it blank.

You'll normally preview the result by pulling the frame back through cv2.projectPoints' cousin draw node, or wire the points into CV Draw Points. In general, every socket in this pack marked NPARRAY is a raw OpenCV ndarray, so data nodes and drawing nodes are separate by design - the data nodes never draw, which is deliberate.

Install

ComfyUI Manager, search ComfyUI CV, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install -r comfyui_cv/requirements.txt

That installs opencv-contrib-python-headless~=5.0.0.93. Needs Python ≥3.12 and a current ComfyUI (V3 API). Restart after installing.

Common issues

The whole fisheye category is missing from your node search. This is almost always a contrib-wheel problem, not a pack problem: all four OpenCV pip distributions share one site-packages/cv2, so installing plain opencv-python over the contrib wheel empties the contrib submodules silently. The pack ships a diagnostic:

python ComfyUI/custom_nodes/comfyui_cv/tools/repair_opencv_contrib.py --check

"cv2.fisheye.projectPoints failed" with the input shapes listed after it. The wrapper catches cv2.error and re-raises with the arguments that were actually passed, which is the fastest way to spot a (N,1,3) where the fisheye code wanted (1,N,3).

Everything projects to the centre. Your rvec/tvec are the wrong shape or all zeros - a (3,) row instead of a (3,1) column is the usual culprit, and the pack has CV Reshape Array for exactly this ("required before cv2 arithmetic on (3,) rows").