cv2.rapid.convertCorrespondencies

RAPID is the classic "track a known 3D model in an image" algorithm: project the model's silhouette, shoot search lines perpendicular to it, find the strongest gradient along each line, and use the resulting 3D–2D pairs to refine the pose. This node is the second-to-last step of that loop. It takes the search-line bundle plus the locations where the search actually found something, and produces the image points - the actual 2D observations - plus a mask saying which control points produced a usable match.

Why you'd touch it directly

If you're using the pack's curated CV Rapid Track or the CV Rapid Pose Refine subgraph, you never call this node by hand; it's inside. You'd wire it explicitly when you're assembling the loop yourself - for instance to inspect the correspondences before a solve, to substitute your own matching step, or to feed a different PnP solver.

The value is in the mask output. Roughly, per control point: did the search line find an edge, or did it fall through to background clutter? That's the difference between a refinement that converges and one that quietly drifts. Feed the mask into your solve so outliers don't drag the pose, and look at it when tracking fails on the frames you'd expect to be easy.

Inputs and outputs

• cols - the correspondence/colour bundle that the search stage produced (rapid.findCorrespondencies and its relatives output this shape; so does rapid.extractLineBundle's bundle when paired with a matcher). NPARRAY in practice.
• srcLocations - the pixel locations of the samples along each search line, i.e. the locations output of rapid.extractLineBundle.
• Optional mask - an input mask to restrict which correspondences are considered. Leave it unconnected for the full set.

Outputs:

• pts2d - the image-space point each matched search line landed on. The pack's own return documentation describes this as aligned with extractControlPoints' ctl3d, ready for cv2.solvePnP - so you pair it with the 3D control points from rapid.extractControlPoints and hand both to your solve.
• mask - which control points produced a valid correspondence. Use it to filter before solving.

Everything here is matrices, not pictures. All three inputs share the pack's polymorphic socket type so they can take an IMAGE, but if you've wired an image into this you've made an error somewhere upstream.

Where it sits in the loop

The raw building blocks in the rapid module that matter, in order:

1. rapid.extractControlPoints - sample control points along the projected silhouette: 2D pixel positions (ctl2d) and the matching object-space points (ctl3d).
2. rapid.extractLineBundle - sample image intensities along each control point's normal: the intensity bundle and the pixel locations of those samples.
3. (matching) - rapid.findCorrespondencies in the raw set, or your own.
4. rapid.convertCorrespondencies - this node: bundle + locations → pts2d + validity mask.
5. A solve - pair ctl3d with pts2d and run PnP. The pack's CV Rapid Track (Sequence) does this with optional RANSAC and warm-starts each frame; the raw module exposes rapid.rapid as a whole iteration if you prefer one node.

The pack's CV Rapid Pose Refine subgraph packages steps 1–5 as a single node with found, rvec, tvec, correspondences, search_lines, bundle, cols, scores and inliers outputs, and a note that it refines a pose rather than finding one - the incoming rotation and translation must already be close.

Install

ComfyUI CV by bmad4ever. ComfyUI Manager, search comfyui_cv, or:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Restart. Python ≥ 3.12 and a current V3-API ComfyUI. No models.

Common issues

The rapid nodes are missing from the menu. cv2.rapid is an opencv-contrib module: it only exists in a contrib build. If a non-contrib opencv-python wheel has been installed over the contrib one, the module silently disappears and the pack skips those entries. python tools/repair_opencv_contrib.py --check inside the pack folder, then --apply.

Empty pts2d. Nothing matched. In practice the search lines are too short (len is the half-length in pixels - too short and they never reach the edge) or the pose is too far off for the silhouette to sit near real edges, so the search locks onto the wrong gradient.

Pose jitters even with clean correspondences. Filter with the mask output before the solve, and check scores upstream - that's what it's there for.

Everything looks like garbage after one bad frame. The loop is warm-started. One diverged frame poisons the next; have your graph detect it (a found flag) and reset from a fresh global estimate rather than letting it coast.