EstimateTracks

The 4D splat pipeline's whole premise is "things move." EstimateTracks is the node that watches your video and figures out which pixels move and where they go - it runs CoTracker3 across the frame sequence and returns per-pixel trajectories: for every tracked point, a path of 2D coordinates through time, plus a visibility flag saying when that point is actually in frame.

Without this, you'd have a static 3D scene from video. With it, the dynamic bits - the person walking, the car passing - become 3D tracks that TracksToTrajectories lifts into world space, and eventually BuildSplats4D turns into animated splats. It's the motion-tracking backbone of the entire video_to_4d_world workflow.

How it works

Feed it frames - an IMAGE batch shaped [T,H,W,3] float 0..1, i.e. the video as a stack of frames - and it tracks a grid of points. The grid_size input (default 20) sets a grid_size × grid_size grid of starting points, so the default is 400 tracks across the frame. More tracks = denser, more faithful motion; fewer = faster and cheaper. For a 4D scene you'll typically keep the default or nudge it up for content-rich shots, then drop it way down while previewing.

The tracking itself is CoTracker3, an offline model fetched on first use via torch.hub - expect a one-time download and make sure ComfyUI has network access on the first run. The source is doing real engineering here: it pre-resizes the video to CoTracker's internal ~384×512 shape before uploading to the GPU, so you don't blow up VRAM shipping a full-resolution clip to the card. Good news for people whose 4D experiments live on one GPU.

Two outputs come back:

• tracks - [T,N,2], pixel coordinates for every track across every frame.
• visibility - [T,N], 0/1 flags for whether each track is currently visible (occluded tracks get flagged rather than hallucinated).

Wire both straight into TracksToTrajectories, which unprojects them into 3D using the depth sequence and camera poses. The track_valid info that node produces later becomes the per-track quality signal BuildSplats4D uses to avoid binding splats to dead tracks.

The practical reality

This is a heavy node. CoTracker3 is a real neural network doing dense tracking, and it's part of a chain (VGGT pose+depth, tracking, lifting, splat fusion) that is genuinely ambitious and genuinely early. Plan for it: short clips, moderate resolution, one GPU's worth of patience. The README calls the full video-to-4D run a pipeline, not a one-click, and EstimateTracks is a solid chunk of that runtime.

Install

Shared camera-comfyUI install: Manager → search "camera-comfyUI" → Install, or clone https://github.com/Alexankharin/camera-comfyUI into custom_nodes/ and run python install.py. CoTracker3 isn't a pip dependency - it downloads via torch.hub on first use, so the first run needs network and a little time. device (auto/cpu/cuda) is available to pin execution.

Troubleshooting

• Node hangs on first run: it's downloading CoTracker3. Let it finish; check the console for progress.
• All tracks flagged invisible: frames aren't in the expected [T,H,W,3] layout, or the video has huge scene cuts where no point survives. Shorten the clip / drop grid_size.
• 4D motion is wobbly: noisy tracks from motion blur or fast movement. Cleaner input video beats any parameter here; lower grid_size can also avoid tracking noise in textureless regions.
• VRAM is the wall: keep frames short and let the node's internal pre-resize do its job - don't pre-upscale your video before feeding it.