TracksToTrajectories

TracksToTrajectories is the node that turns "this pixel moved this way in 2D" into "this point moved this way in 3D space." Feed it 2D point tracks from EstimateTracks (which runs CoTracker3), a depth sequence, and camera poses, and it unprojects everything into world-space 3D trajectories - [T,M,3] - that a dynamic Gaussian splat can actually follow. In the pack's video_to_4d_world workflow, this is the step that gives your moving objects their third dimension.

The math is the standard unprojection dance, and it's worth understanding one gotcha up front. The node expects radial (ray) depth, not Z-depth. Each track's pixel coordinate is sampled into the depth map (nearest-neighbor, to avoid mixing foreground/background at edges), converted to a unit ray direction using the input_projection (PINHOLE/FISHEYE/EQUIRECTANGULAR) and input_horizontal_fov, then multiplied by that radial depth to get a camera-frame 3D point. The per-frame camera pose then carries it into world space. Feed it Z-depth from a naive estimator and everything bows outward at wide FOVs - that's the single most common reason the resulting trajectories look wrong.

The inputs that matter:

• tracks - [T,N,2] pixel coordinates (x,y) straight out of EstimateTracks.
• visibility - [T,N], 0/1 per track per frame, also from EstimateTracks.
• depth_seq - [T,H,W] radial depth. VideoPoseEstimator already outputs ray-corrected depth, so its depths output works directly. Any other source, run it through ZDepthToRayDepthNode first.
• input_horizontal_fov - must match the depth's actual FOV.
• min_visible_frac - default 0.5. Tracks visible in fewer than half the frames get dropped. This is your noise filter; raise it for cleaner, sparser trajectories.

The optional trajectory input is world-to-camera poses [T,4,4]; leave it out and the node assumes a static camera (identity poses), which is the right call for locked-off footage. Outputs are trajectories3d [T,M,3] and track_valid [T,M] - a per-track-per-frame validity mask you can use downstream to ignore the filled-in garbage. Invalid points get a linear fill plus a short temporal median filter, so the output is smooth-ish even where tracks blink out.

One thing to know: if no tracks survive min_visible_frac, the node raises an error rather than returning an empty tensor. If you hit that, the visibility input is probably wrong (or all zeros) - lower the threshold, don't fight it.

Installation is the pack's normal story - ComfyUI Manager, search camera-comfyUI, Install, restart:

git clone https://github.com/Alexankharin/camera-comfyUI.git custom_nodes/camera-comfyUI
cd custom_nodes/camera-comfyUI && python install.py

The only heavy dependency this node itself needs is CoTracker3, which EstimateTracks downloads via torch.hub on first use - expect a one-time download. The quality of what you get out is bounded by the depth map you feed in; shaky per-frame depth means shaky trajectories, and no amount of median filtering fully rescues that. Get the depth right and this node is quietly the backbone of every dynamic 4D scene this pack can make.