Meta SAM-Audio

ComfyUI SAM-Audio

ComfyUI nodes for Meta SAM-Audio, which separates a described sound from an audio mixture.

Supports text prompts, positive and negative time spans, and visual masks. Each separator outputs the isolated target and the remaining audio.

Requirements

• Python 3.11 or newer
• Git
• A CUDA GPU is strongly recommended
• An internet connection for the first model download

Installation

Clone the repository into ComfyUI/custom_nodes:

cd ComfyUI/custom_nodes
git clone https://github.com/ethanfel/ComfyUI-SAM-Audio.git

Run the installer with the same Python environment used by ComfyUI:

python ComfyUI-SAM-Audio/install.py

Then restart ComfyUI. If installed through ComfyUI Manager, the installer should run automatically.

The node uses ComfyUI's decoded media tensors and does not install or load TorchCodec.

The installer does not install or replace PyTorch, TorchCodec, or xFormers. It installs the listed inference dependencies and three pinned Meta source packages into the Python environment running ComfyUI. Existing packages are kept when they already satisfy the requirements; an older transformers may be upgraded to the required version.

Model download

Select a model in SAM-Audio Model Loader and queue the workflow. The loader first tries Meta's official repository. If Hugging Face denies gated access, it automatically downloads the matching pinned and checksum-verified checkpoint from the public mirror collection.

You can authenticate in the ComfyUI environment if you prefer the official repository:

hf auth login

Models download on first use to the active ComfyUI models directory under models/sam_audio/. For example, when ComfyUI is started with --models-directory /path/to/models, SAM-Audio checkpoints are stored in /path/to/models/sam_audio/. Both official and mirrored checkpoints remain subject to Meta's SAM License.

The required T5 asset is pinned in the Hugging Face cache and reused locally after its first download. The separate 5.8 GiB PE span predictor is loaded only when predict_spans is enabled. reranking_candidates above 1 similarly downloads Meta's optional 5.7 GiB Judge checkpoint once and requires a text description.

Nodes

• SAM-Audio Model Loader — loads an official or local checkpoint
• SAM-Audio Text Separate — separates audio using a text description
• SAM-Audio Span Prompt — creates positive or negative time spans
• SAM-Audio Span Separate — separates audio using text and time spans
• SAM-Audio Visual Separate — separates audio using image frames and a mask
• SAM-Audio Video Separate — accepts a native ComfyUI VIDEO, uses its embedded audio, and separates the object selected by a mask

All nodes are in audio/SAM-Audio.

Basic use

1. Load a model with SAM-Audio Model Loader.
2. Load audio with ComfyUI's audio loader.
3. Connect both to a separator and enter a short prompt such as man speaking or dog barking.
4. Preview or save the target and residual outputs.

Long audio is processed in overlapping chunks by default. Set chunk_duration to 0 for a single full-clip pass.

precision=auto uses BF16 on supported GPUs and FP32 otherwise. initial_device=cpu keeps the normal ComfyUI offload path; choose gpu to materialize checkpoint weights directly on the GPU and avoid the CPU-first checkpoint stage.

Example workflows are included in the examples directory.

Notes

• Output is mono at 48 kHz.
• Adjacent chunks use a normalized crossfade; the defaults are 10-second chunks with 1-second overlap.
• Progress and cancellation are checked during every ODE inference step.
• White pixels in a visual mask identify the target object.
• Native VIDEO inputs use the video's real frame rate. Frames and embedded audio are trimmed to their common timestamped duration. Use ComfyUI's Create Video node to attach separate audio when needed.
• Span boundaries are quantized to the model's 40 ms audio-feature grid; zero-width and contradictory positive/negative anchors are rejected.
• Local model folders must contain config.json and checkpoint.pt.
• All configured ComfyUI sam_audio model roots are searched, including roots from extra_model_paths.yaml.
• The model loader offers PyTorch SDPA (default) or Comfy Kitchen INT8 attention. Comfy Kitchen requires a supported current ComfyUI build and GPU and may slightly change results because its attention is quantized.
• ComfyUI's Unload Models action moves SAM-Audio out of VRAM but keeps the cached loader output in system RAM. Use Unload Models and Execution Cache when you also want ComfyUI to discard that cached CPU model.
• Because Meta's source packages are installed without their training-only dependency trees, pip check may report their optional packages as missing. This is expected for this inference-only integration.

License

This ComfyUI integration is released under the MIT License. SAM-Audio and its checkpoints use Meta's separate SAM License.