CV Camera Pose To 3D View

Core ComfyUI's 3D preview nodes take a LOAD3D_CAMERA dict for their camera - a camera_info input on Preview 3D (Advanced), Preview Splat and Preview Point Cloud. This node builds one from an OpenCV camera model, so a camera you measured can drive the Three.js viewer. That's the difference between orbiting a render for fun and checking whether your reconstruction, your point cloud or your virtual object sits where the real camera saw it.

The chain is: intrinsics from a calibration node (CV Calibrate Camera (Chessboard), ChArUco, circle grid, or CV Camera Matrix if you're approximating), plus extrinsics - rvec/tvec from CV Solve PnP (Pose). The viewer camera is then placed at the physical camera's position relative to the board or object, looking down its optical axis.

How it works

Two coordinate conventions get reconciled. OpenCV is right-handed with Y down and the camera looking along +Z; Three.js is right-handed with Y up and the camera looking down -Z. The conversion negates Y and Z. The orbit target is the projection of orbit_anchor onto the optical axis, so OrbitControls' lookAt reproduces the pose's viewing direction exactly - orbit around the thing you're looking at, not the world origin. Roll is not preserved; OrbitControls keeps +Y up, and that's a viewer limitation, not something a parameter fixes.

Inputs

• camera_matrix - the 3×3 K. Required; it sets the reported vertical FOV as fov_y = 2*atan(image_height / (2*fy)).
• rvec / tvec (optional) - from a pose solve. A 3×3 rotation matrix is accepted in place of rvec. Leave them unwired and you get a frontal default view.
• orbit_anchor (optional) - a 3D point in the same object/world coordinates as your solve-PnP object points, e.g. the board centre as [[4, 2.5, 0]]. Default is the object-space origin.
• dist_coeffs (optional) - rides along in the dict as an extra key for viewers that can render distortion (CV Preview 3D (Calibrated Camera) in this pack). The core preview nodes ignore it.
• image_width / image_height - the calibration image size, for aspect and FOV. 0 (the default) estimates them as 2*cx and 2*cy.
• scale - object units (chessboard squares, mm) to scene units, applied to camera position and orbit target. This is how you match camera distance to the size of the model you're previewing.

Outputs

camera_info is the LOAD3D_CAMERA dict - position/target/zoom/cameraType plus quaternion/fov/aspect/near/far - and it's the one you wire. fov_y_deg and distance are there so you can read what the dict ended up containing.

One thing to know before you spend time on FOV: the current frontend applies only position, target and zoom from a camera_info input. The viewer keeps its own FOV. So a perfectly computed fov/aspect/quaternion is recorded and ignored by core's preview - the pose (where the camera is and what it looks at) is what you actually see. Don't debug a "wrong" FOV in the viewer.

Failure is soft: with no rvec/tvec - or with the zero vectors that a pose solve emits on found=false - you get a front view of orbit_anchor at 10× scale distance. So a failed solve doesn't error, it just quietly gives you the default view. If your camera mysteriously snaps to a front-on shot, check the pose node's found first.

Install

Manager → ComfyUI CV, or:

cd ComfyUI/custom_nodes && git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Python ≥ 3.12 and a recent ComfyUI on the V3 node API. The 3D preview nodes themselves are core, so there's nothing else to install - though the pack's own example workflows bundle three.js in web/lib/ for its own viewers, which is why the repo is bigger than the node count suggests.

Where people get burned

Scale mismatches are the number one surprise: your PnP translation is in marker-length or square-size units, and the GLB you're previewing is in metres. Set scale so the camera distance lands in the same world as the model, or the camera will sit inside the mesh or a kilometre away.

Passing a matrix that came from a different image size. Same trap as CV Camera Matrix: cx/cy are pixel coordinates of a specific frame, and if image_width/image_height disagree with them you get a skewed default view that looks like a bug in the viewer.