BuildSplats4D

Regular 3D Gaussian splatting is a still photo you can orbit. BuildSplats4D is the node that upgrades that still to a short film: it takes your frozen splat cloud and attaches it to a set of moving "skeleton" points, so the whole scene can deform, walk, and bend over time. It's the payoff node of camera-comfyUI's 4D pipeline - the reason you'd bother tracking pixels and lifting them into 3D in the first place.

This is the part of the pack that's genuinely ambitious and genuinely early. The README positions it as the canonical "video → 4D world" stage, and the demo is a monocular video turned into a navigable scene with moving content. Expect it to be fiddly, expect VRAM pressure, and expect to feel like a grad student while you get the shapes right.

What it actually does

You feed it two things: a set of canonical splats (your static scene's dynamic objects, frozen at one reference timestep) and 3D control-point trajectories - essentially 3D landmarks that move through the scene across time. BuildSplats4D then lets every splat follow the motion of its nearest control points.

Each splat's position is a weighted blend of the control points around it. The knn input (default 4) sets how many control points each splat listens to; more points means smoother, softer motion transfer, fewer means more rigid local warping. rbf_gamma controls how those neighbors are weighted: leave it at 0 and you get plain inverse-distance weights, push it above 0 and it switches to RBF falloff (exp(-gamma·d²)), which makes influence decay faster with distance - handy when you want cleaner local deformations and less smearing.

The other inputs matter less at first. reference_index (default 0) tells the node which timestep the canonical splats were captured at; times defaults to a simple linspace over 0..1 if you don't pass your own; and track_valid is the per-track validity mask from TracksToTrajectories, which you should wire in because dead tracks produce garbage motion. The output is a single splats4d (GSPLAT4D) object that RenderSplats4DVideo or RenderSplats4DFrame knows how to evaluate at any time value.

Where it sits in the workflow

It's late in the chain. Upstream you have pose + depth (VideoPoseEstimator / VGGT), a motion mask to separate moving from static pixels, CoTracker3 tracks via EstimateTracks, 3D trajectories via TracksToTrajectories, and SplitSplatsByMask to carve the dynamic splats out of the background. BuildSplats4D is where those parts finally marry. Wire the static background in through the optional static input so it stays put while the dynamic content moves.

The one big gotcha is depth type. Depth estimators in this pack (including VideoPoseEstimator) output Z-depth, but the point-cloud and splat lifting nodes expect ray depth - skip the ZDepthToRayDepthNode conversion and your geometry bows outward at wide FOVs, which corrupts every 3D track downstream. It's the pack's single most common silent failure mode.

Install and requirements

This is a camera-comfyUI node, so install is the shared one: ComfyUI Manager → search "camera-comfyUI" → Install, or git clone https://github.com/Alexankharin/camera-comfyUI into custom_nodes/ followed by python install.py. That script wires up the heavy friends: gsplat (JIT-compiles CUDA kernels), CoTracker3 (fetched via torch.hub on first use - expect a one-time download), and for pose estimation upstream, VGGT with its ~5 GB facebook/VGGT-1B weights.

If the node shows up gray/disabled in your graph, one of those optional deps failed to load - the pack degrades gracefully and only kills the 4D nodes, so check the console and re-run python install.py.

Troubleshooting

• Geometry melts into mush: knn too high with sparse, noisy tracks. Drop it to 2–3 and feed track_valid from TracksToTrajectories.
• VRAM: you're holding CoTracker tracks, splats, and a diffusion scene all at once. Use the device input to pin cuda explicitly, keep the video short, and lower grid_size in EstimateTracks upstream (default 20×20 = 400 tracks already).
• Static content wobbling: you forgot the static input, so the background got bound to dynamic tracks too.

Save your finished scene with SaveSplats4D (.npz) and reload it later with LoadSplats4D - building it is the expensive part, so don't rebuild every session.