cv2.solvePnPRefineLM

solvePnPRefineLM is not a solver - it's a refiner, and the distinction matters more than any parameter on the node. You give it a pose you already have (rvec, tvec) and it runs Levenberg-Marquardt to shave the reprojection error down from there. It's a local optimizer: start it near the answer and it converges beautifully; start it somewhere wrong and it stays somewhere wrong, happily.

Where it earns its place in a ComfyUI graph is after a RANSAC solve. solvePnPRansac gives you a robust pose built from a minimal subset of points; the refiner then uses all your correspondences to tighten it. The pack uses exactly this in its CV Rapid Pose Refine subgraph - track a 3D mesh through a frame, solve, refine, feed the refined pose into the next frame as a warm start.

Inputs

Six required, and unlike cv2.solvePnP, everything is required including the distortion coefficients:

• objectPoints - the 3D points, Nx3. At least a handful; a pose needs enough constraints to be constrained at all.
• imagePoints - the matching 2D points, Nx2, in the same order.
• cameraMatrix - the 3×3 intrinsics K.
• distCoeffs - required here. For an ideal pinhole, wire in a zero vector: CV Scalar with (0, 0, 0, 0, 0) in float32 is the quick way, and OpenCV treats an empty/NULL vector as zero distortion anyway. Real values come from the calibration nodes.
• rvec / tvec - the pose to refine. Straight off solvePnP or solvePnPRansac.

Then the optional convergence settings, which the pack renders as a named group rather than a raw TermCriteria literal - a nice touch, because TermCriteria is one of those parameters everybody types wrong:

• criteria_type (default max count or epsilon, whichever first) - stop after the iteration cap, stop when the change drops below epsilon, or whichever happens first.
• criteria_max_count (default 30) - the iteration cap; ignored in epsilon-only mode.
• criteria_epsilon (default 0.001) - the accuracy target; ignored in max-count-only mode.

Outputs

Two sockets, same names as the inputs they replace: rvec and tvec - the refined pose. Note there's no boolean: this node doesn't report success. Whether the refinement helped is something you check by reprojecting (cv2.projectPoints) or by watching the distance between your points and their projections.

Install

Manager → search ComfyUI CV, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Restart ComfyUI afterwards. Python ≥ 3.12 and a recent V3-API ComfyUI. No models - pure geometry.

Where people get burned

It can't fix a bad start. If your rvec/tvec came out of a failed RANSAC (retval false), refining them just produces a confident-looking pose in the wrong place. Check the upstream flag first; that's why the raw wrapper's retval exists.

distCoeffs is required here. The optional-on-solvePnP, required-on-refine asymmetry catches people out - the node won't run with the socket unconnected, and a missing distortion vector is not the same as no distortion.

Outliers get a vote. LM fits all the points you hand it. Feed it the RANSAC inliers only (index the arrays with CV Take By Index) or one surviving bad match can pull the refined pose off target even though the robust pose was fine.

Local minima on repetitive geometry. Symmetric or near-planar point sets have ambiguities, and a refiner will happily settle into the wrong branch. If the refined pose flips orientation between frames, that's this - and it's the reason the pack's tracking subgraph warm-starts from the previous frame rather than cold-solving each one.

LM versus VVS. cv2.solvePnPRefineLM is the default choice; cv2.solvePnPRefineVVS is the virtual-visual-servoing variant with a tunable gain. If LM stalls or oscillates, VVS with a smaller gain is the thing to try. Try LM first - it has fewer knobs to get wrong.

Contrib-wheel collisions. All four OpenCV distributions share one site-packages/cv2, and installing a non-contrib wheel over the contrib build leaves contrib nodes missing from the menu with no error. tools/repair_opencv_contrib.py --check diagnoses it.