VideoToFusedSplats

VideoToFusedSplats is where a video stops being frames and becomes a place. It runs SHARP - Apple's single-image-to-gaussian-splat model - on keyframes of your clip, transforms each keyframe's splat cloud from its own camera frame into a shared world frame, and fuses them into one GSPLAT you can render from any camera. This is the "static world" half of the pack's video_to_4d_world workflow: the frozen scene you can walk around in, distinct from the dynamic objects handled separately.

The mechanism, in order: pick keyframes (keyframe_stride, default 8 - SHARP runs on every Nth frame), lift each one to splats, then optionally do two cleanups before fusing. If you feed depths (metric depth, [T,H,W]), each keyframe's splats get scale-aligned to it via a robust median disparity ratio - this is what makes the cloud metric and prevents per-keyframe drift. If you feed a static_mask ([T,H,W], 1 = static/keep), any splat whose source pixel is below 0.5 gets dropped - that's how moving people and cars get scrubbed out of the world. Then each cloud is rotated into world space by the inverse of its camera pose, and all of it is fused with a voxel-style reduction controlled by stitch_voxel_size and stitch_mode.

The inputs that matter:

• frames - [T,H,W,3].
• trajectory - world-to-camera poses [T,4,4] (a single [4,4] broadcasts). Feed it VideoPoseEstimator's trajectory output directly.
• horizontal_fov - default 60; use the FOV from VideoPoseEstimator, not the default, or the fusion won't line up.
• checkpoint - the SHARP .pt. It can download the default model for you; pick a local checkpoint from your input folder if you have one.
• keyframe_stride - 8 is a sane default. Lower = more keyframes = slower but denser; raise it if a pass takes forever.
• stitch_voxel_size / stitch_mode - fusion controls. smart is the default and usually the right call; average, discard, and keep are there when smart is smearing.

Optional depths, static_mask, and device round it out. One output: splats (GSPLAT), which wires straight into RenderSplat, SavePlySplat, SplatPolish, or SplitSplatsByMask.

Gotchas, and these are the ones that actually bite:

• Skip the double ray-depth correction. Feed it VideoPoseEstimator's already-ray-corrected depth, or you'll bow the geometry at wide FOVs.
• Dynamic pixels will ghost without a static_mask. A moving person fused into the world becomes a smear across all keyframes. Run MotionMaskFromDepth and pass its output in.
• Wrong FOV = splats in the wrong place. This node trusts your horizontal_fov; a mismatched one fuses clouds that don't overlap, and the result looks like a broken hologram.
• SHARP and gsplat are required. The node errors if the SHARP submodule isn't available. install.py (run automatically by ComfyUI Manager) initializes the submodules/ml-sharpt checkout and pip-installs gsplat, whose CUDA kernels JIT-compile on first use - expect a long first run.

Install the pack via ComfyUI Manager (search camera-comfyUI, Install, restart), or:

git clone https://github.com/Alexankharin/camera-comfyUI.git custom_nodes/camera-comfyUI
cd custom_nodes/camera-comfyUI && python install.py

Expect a GPU with real VRAM to be happy (8 GB works at reduced settings; more is better), and expect this node to be the slow one in the graph. But when it works - and it does work - you get a world-frame splat scene from a phone video, and that's the whole reason this pack exists.