Point Tracker

If the compositor is the lip-sync pipeline's payoff, Point Tracker is the part that decides whether it looks good or looks like a badly-dubbed puppet. You give it a frame batch and one starting coordinate on the first frame; it follows that point through the whole video using pyramidal Lucas-Kanade optical flow and hands back a per-frame path. Wire that path into the Tracked compositor and your mouth rides the character's head instead of floating on top of it.

How the tracking actually works

Classic Lucas-Kanade assumes small motion, so this node stacks it in a pyramid - coarse levels first, refining down - which is what lets it handle a head that moves several pixels between frames. On top of that base it layers real recovery machinery:

• Sub-pixel accuracy via Scharr gradients, so the point doesn't snap to integer pixels.
• Multi-stage fallback. When confidence drops, it tries template matching with an adaptive template (refreshed roughly every 5% of the clip), then the original first-frame template, then a full-frame search. A lost point isn't a dead end; it's a re-acquisition.
• Periodic drift validation against the original template, so a slowly-accumulating offset gets caught and corrected rather than compounding.

That last one is the difference between this and a bare calcOpticalFlowPyrLK wrapper. Optical flow drifts; this node explicitly fights it.

The inputs that matter

• start_x / start_y - usually wired from Point Preview's click output, or typed.
• window_size (default 31, up to 1025) - the search window. The tooltip is blunt: use 201+ for full-frame tracking of fast-moving objects. Bigger window = more robust to fast motion, slower.
• pyramid_levels (default 4, up to 8) - more levels handle larger frame-to-frame motion; 6–8 for very fast motion.
• search_radius_percent (default 0) - when it's >0, the template-match recovery searches a percentage of the whole frame instead of just the window. 100 means "search the entire frame" on recovery. Zero means the recovery only searches within window_size.
• smoothing (default 0.3) - temporal smoothing of the path. Raise it if the points jitter, but too high and the mouth lags the face.

The outputs

points is the POINT_SEQUENCE the Tracked compositor's points mode wants. tracking_masks is a per-frame mask centered on each point - feed it to the compositor's masks mode, or run it through Points To Masks for a soft Gaussian version. preview shows the point drawn on the frames, so you can see where it went without rendering anything else.

Installing it

Part of TrentNodes; install the pack once:

cd ComfyUI/custom_nodes
git clone https://github.com/TrentHunter82/TrentNodes.git
cd TrentNodes
pip install -r requirements.txt

ComfyUI Manager (search "Trent Nodes") also works, though the author's day-one repo rename left two registry entries and Manager occasionally flags the pack as "unsafe" - the clone is the dependable path. The pack needs OpenCV (in requirements) and nothing else for this node; no model downloads.

Where it goes wrong

• Fast motion, lost track - raise window_size or pyramid_levels before you blame the node.
• Slow drift - that's the smoothing vs. recovery trade. A drifting point usually means the original click was on a low-contrast patch; re-pick at a feature edge.
• Frames vs. video mismatch - if your batch is at a different resolution than the point was picked on, coordinates land wrong. Track on the actual frames you'll composite onto.