Yogurt InstructSAM Model Loader

This is the boring-but-everything-goes-wrong-here half of the Yogurt InstructSAM pack. It loads a vision-language segmentation model - InstructSAM, a fusion of Qwen3-VL and SAM3 - and hands the loaded handle to the Yogurt InstructSAM Image Segmentation node. One output, model, type YOGURT_INSTRUCTSAM_MODEL. That's all it does. But the model storage and download behavior are unusual enough that this loader is where almost every "it won't run" starts.

Why the model layout is weird

Most ComfyUI packs dump weights in the Hugging Face cache and let them be. This one doesn't: everything must live in your current ComfyUI's models/instructsam/ folder, as two directories:

ComfyUI/models/instructsam/
├── CircleRadon--InstructSAM-2B/   # the ~5.2 GB checkpoint + tokenizer
└── facebook--sam3/                # metadata only - NO sam3 weights

The main weights are from CircleRadon's InstructSAM-2B repo. The facebook--sam3 folder is just config, processor, and tokenizer files - the actual segmentation weights are baked into the InstructSAM checkpoint, so don't go hunting for a model.safetensors to drop in there. The loader scans for a config.json plus a checkpoint file per directory, and it validates the resolved path stays inside ComfyUI's registered model folders - so symlinks pointing at an external D:\Models tree will be rejected. Copy whole directories, not just the weights file.

The download toggle

download_model (default false) is the switch that makes this pack deliberately offline-first. Set it to true on a Loader node and execute once: it pulls a fixed manifest of files straight into models/instructsam/ - no Hugging Face cache, no background update checks - using an official pinned revision. There's a catch hiding in the manifest: facebook/sam3 is a gated repo. Before the downloader can fetch its metadata files you have to log into Hugging Face, accept the terms on the facebook/sam3 page, and make a token visible to the ComfyUI process (set HF_TOKEN in the environment ComfyUI actually launches with). Getting 401 or 403 means one of those three steps is missing. After it's downloaded, flip download_model back to false and you're fully local and air-gapped.

The knobs that matter

• dtype - auto (default) picks bfloat16 when your GPU supports it, else float32. That's the right call; explicit bfloat16 or float32 is for when you have a reason.
• cache_model (default true) - keeps the loaded model handle cached between runs, so repeated executions are fast, at the cost of holding CPU memory. Set it false and the handle is released after each segmentation, then auto-reloads on the next queue run - slower but kinder on a tight box. There's deliberately no separate "unload" node.
• force_reload (default false) - only meaningful with caching on. Flip it after you swap model files, and it tears down and rebuilds the cached handle.

Install and the transformers pin

Same shared install as the whole pack: ComfyUI Manager (search "Yogurt InstructSAM") or git clone https://github.com/yogurt7771/ComfyUI-YogurtNodes-InstructSAM into custom_nodes, then install one requirement - transformers>=4.57.3,<6 - into the same Python running ComfyUI (Windows Portable: python_embeded\python.exe -m pip install -r requirements.txt), and fully restart. That version floor exists because InstructSAM needs Qwen3-VL support, which only landed in 4.57.x.

The version pin is the part worth respecting. Under transformers 4.57.x the pack uses its own vendored SAM3 compatibility implementation; under 5.x it uses transformers' native SAM3. Either way it avoids the Triton dependency that trips up the ComfyUI-RMBG SAM3 node on Windows - a genuinely nice call for the platform where SAM3 setups usually die. Just don't let another plugin force-transformers upgrade your whole install to 5.x for this pack's sake; 4.57.x is the compatibility-safest lane, and force_reload plus a full restart covers the "I replaced the model files" cases.