cv2.omnidir.stereoRectify

Stereo rectification is what makes a stereo pair usable: you rotate both cameras' views so they share a common image plane with the baseline purely horizontal, and after that corresponding points differ only in their x coordinate - which is what every disparity matcher on earth assumes. cv2.omnidir.stereoRectify provides that alignment for the unified omnidirectional model (fisheye and catadioptric rigs, cv2.omnidir in opencv-contrib).

The caveat first, because it's the thing that will confuse you: as generated, this node takes just two inputs, R and T, and returns two arrays (nparray_0, nparray_1). That's the small overload of the module's API - the relative rotation and translation between the two cameras in, the rectification rotations out. The calibration-heavy overload, the one that also wants both cameras' K, D, xi and the projection flags, is not exposed here. So don't wire your calibration bundle in expecting the full pipeline: it goes past this node, not through it.

What to do with the two outputs

They're the two per-camera rectification rotations (call them R1 and R2). Their natural use is one per camera, as the R input of cv2.omnidir.initUndistortRectifyMap alongside each camera's own K, D and xi - that pair of map sets turns your two fisheye frames into a rectified stereo pair, which is what a disparity matcher wants. The projected output size and the virtual camera matrix (P) are your choices at that step, which is where you decide how much of the field of view survives rectification.

Worth knowing for context: the pack's higher-level CV Omnidir Stereo Reconstruct node does its rectification internally, so if you're using that node you may never call this one at all. This is the piece you reach for when you want to build the rectified pair yourself - to inspect it, to feed it to your own matcher, or to keep the two frames in pixel space where you can look at them.

R and T come from calibration. On the omnidir path that means the calibration nodes in this pack - the CMei chessboard calibration for each camera, then a stereo calibration for the relative pose - or the JSON/CSV sidecars the pack can save and load (CV Save Camera Params (JSON) / CV Load Camera Params (JSON) and the stereo-from-CSV bridge). Both R and T are array inputs expecting real matrices: a 3×3 rotation and a 3-element translation. The sockets will technically accept an IMAGE link as a storable array, but a uint8 picture is not a rotation matrix, and the result will be nonsense rather than an error.

Install

ComfyUI Manager → search ComfyUI CV → install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Python ≥ 3.12, ComfyUI on the V3 node API. cv2.omnidir ships in contrib - if this node is missing from your menu, check CV Build Information and be suspicious of any non-contrib opencv-python-headless that got installed over the contrib wheel (tools/repair_opencv_contrib.py --check / --apply).

Perspective check

Calibrated stereo is the minority position in ComfyUI's 3D landscape, and it's worth being honest about that: the mainstream answer to "get depth from images" is a learned monocular model that needs no rig and no calibration, and the KB's depth docs treat that as the default. Classical rectification like this is for people who actually have a fisheye or catadioptric rig and want geometry they can trust rather than a plausible-looking prediction. The pack's README adds its own warning on top: the bundled stereo workflows were tuned against a specific dataset (StereoGeo-CARLA) and aren't production-grade. Individual nodes can still be exactly what you need; just don't treat the demo pipelines as finished tools.