cv2.fisheye.estimateNewCameraMatrixForUndistortRectify

This is the fisheye equivalent of getOptimalNewCameraMatrix, and it answers a question that has nothing to do with the maths and everything to do with taste: once you've straightened a wide-angle image, how much of it do you want to keep?

Straightening bends the corners outward, so the result either has black wedges where the sensor never saw anything, or you crop in and lose the extreme field of view that was the reason you bought the lens. This node estimates a new 3×3 camera matrix that describes a chosen compromise, and you feed that matrix into the mapping step that actually warps the picture.

Inputs

K is the original 3×3 camera matrix, D the four fisheye coefficients (k1..k4 - equidistant model, not the pinhole five-coefficient vector), both from CV Fisheye Calibrate (Chessboard) or a saved calibration.

image_size is a two-component Size value - (width, height) of the image you're undistorting - authored with CV Tuple or typed in place. It's required, and it doesn't default to anything sensible: (0,0) will get you an error rather than a guess.

R is the 3×3 rectification rotation. Identity for a plain undistort; for a stereo fisheye rig it's the rectification rotation from cv2_fisheye_stereoRectify. Type it with Parse Matrix, or get it from the rectification step.

Then the three optional ones, which are the reason you're here. balance is the compromise knob: 0 keeps the longest focal length cv2 considers valid - maximal crop, no black corners, the framing you'd get from a normal lens - and 1 keeps the shortest, which preserves the widest view at the cost of empty wedges. You can type any literal in between (0.5), and blank means OpenCV's default applies. new_size is the output size, if you want a different canvas from image_size; leave it at (0,0) to mean "same". fov_scale is an extra divisor on the estimated focal length: above 1 zooms out (more scene, smaller), below 1 zooms in. It's a blunt instrument on top of balance; most people never touch it.

Output is a single nparray: the new 3×3 camera matrix.

The part that will annoy you

On the shipped example, balance 0.0 gives an fx around 166 and balance 1.0 gives around 50 - that's not a subtle difference in framing, that's two different lenses. And which end leaves black corners is lens-dependent, not a rule you can apply blind. On that example, balance 0.0 leaves about 3.7% of the frame black and balance 1.0 leaves about 0.3%: the opposite of the intuition most people arrive with. So look at both rather than assuming, and pick the one that actually looks right for your footage. Preview CV Array will show you.

Second annoyance, and it's a known OpenCV wart: when the calibration left k3/k4 unconstrained, the estimator can return an all-NaN matrix while the calibration's RMS error still looks healthy. If your new K comes out as NaNs, that's what happened - the fix is to recalibrate with CALIB_FIX_K3 | CALIB_FIX_K4 set, which is exactly what the pack's 53_fisheye_calibration.json example demonstrates. The curated CV Fisheye Undistort node detects this case and falls back to the original K with a warning; through the raw wrapper, you have to notice it yourself with Inspect CV Data.

Where the matrix goes next

Into cv2.fisheye.initUndistortRectifyMap as the P argument (with the original K and D, and R = identity for a plain undistort), then the two resulting maps go into cv2_remap. That's the whole mechanism, and CV Fisheye Undistort wraps it in one node if you'd rather not assemble it: it also returns the K the output image is expressed in, which is what any later projection or point-mapping needs.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Manager: "ComfyUI CV" → install → restart. Python ≥ 3.12, V3-API ComfyUI. cv2.fisheye is contrib-only; a non-contrib OpenCV wheel installed by another pack empties the contrib submodules and this node vanishes with them. tools/repair_opencv_contrib.py --check then --apply.

Common issues

cv2 error about sizes. image_size is required and must be real numbers - (0,0) is not "auto". It also has to match the K you're using, since focal length is in pixels at that resolution.

NaNs in the output. Unconstrained k3/k4 in the calibration, as above. Recalibrate with those coefficients fixed.

The undistorted result is correct but weirdly framed. You're seeing exactly what balance asked for. Try the other end before deciding your calibration is broken.