(Down)Load sam2_realtime Model

This is the model loader for ComfyUI-SAM2-Realtime, a pack that tracks objects across video frames with Meta's SAM 2. And the headline is genuinely good news: you never hunt down a checkpoint. The first time you hit Run, this node downloads the SAM 2 weights it needs into ComfyUI/models/sam2/ and instantiates the model from a config that ships inside the repo. The download is the entire setup story, and it's a rare custom node that can say that.

Why would you reach for SAM 2 at all? Meta's "Segment Anything" sequel is the version that added video - click once on an object and it follows that object frame to frame. Most ComfyUI SAM2 packs, including the well-known image ones and the tools bundled into fat packs like ComfyUI-RMBG, only give you single-image masking. This pack is one of the few that runs the camera predictor path - the frame-to-frame tracker SAM 2 was actually built for - so it's what you want when your mask has to survive motion, not just one still.

How it works

The node takes your pick of checkpoint, downloads it if it isn't already in models/sam2/ (from Meta's public dl.fbaipublicfiles.com bucket), then builds the model from the vendored Hydra configs in sam2_configs/. The repo forks Meta's SAM 2 realtime code wholesale - Apache 2.0, LICENSE included - with a SAM2TensorPredictor that operates directly on ComfyUI IMAGE tensors instead of decoded video files. What comes out is a SAM2MODEL handle that only the pack's other node, Sam2RealtimeSegmentation, knows how to consume.

The inputs that matter

• model - sam2_hiera_tiny.pt or sam2_hiera_small.pt. For video tracking, tiny is the one you actually want: fast, low VRAM, and SAM 2's tiny backbone holds up surprisingly well on tracked masks. Small buys you a bit of edge fidelity at real cost on long batches.
• device - cuda, cpu, or mps. Here's the truth the docs would tell you if there were any: the code ignores your choice and forces cuda whenever a CUDA GPU is available, silently falling back to CPU otherwise. Set it to whatever you like; it's mostly decorative.
• precision - fp16 (default), bf16, or fp32. Don't overthink it: the segmentation node always runs fp16 autocast anyway, so this mostly acts as a guard that refuses fp16/bf16 on CPU.
• segmentor - a single choice, realtime. It's stored in the model dict and never actually branches on anything. Vestigial, but harmless.

Output

One output: sam2_model (SAM2MODEL). It wires straight into Sam2RealtimeSegmentation. Nothing else in ComfyUI accepts it - this type is private to the pack.

Installing it

ComfyUI Manager: search "ComfyUI-SAM2-Realtime" and install. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/pschroedl/ComfyUI-SAM2-Realtime
cd ComfyUI-SAM2-Realtime
pip install -r requirements.txt

A couple of caveats the missing README doesn't warn you about. First, requirements.txt ends in a self-referential git+https://github.com/pschroedl/ComfyUI-SAM2-Realtime.git@main install, which runs setup.py - and setup.py compiles a CUDA extension (connected_components.cu) with torch's build tools. That means you need nvcc/a CUDA toolkit on the machine, not just a CUDA-capable GPU. Second, the build hardcodes TORCH_CUDA_ARCH_LIST to 8.0 8.6+PTX 8.7 9.0 9.0a - no sm_120, so RTX 50-series/Blackwell users and anyone on older pre-Ampere cards can end up with a "no kernel image" error or a build that compiles for the wrong arch. If you hit that, install the CUDA toolkit and set TORCH_CUDA_ARCH_LIST yourself (e.g. 12.0 on Blackwell) before installing. There's no torch in requirements.txt either - it relies on ComfyUI's own install, which is the normal pattern.

This is a niche pack by Peter Schroedl, the same author as ComfyUI-FasterLivePortrait, aimed squarely at the realtime/streaming crowd. It works, but the rough edges are real: expect a compile step, and don't expect the device picker to be honest with you.