CLIP Vision Loader (Unified Memory)

Most ComfyUI users learn the CLIP loader for text and the UNET loader for the big diffusion model, and then run into their first image-to-video or IPAdapter workflow and stare at the graph wondering where the vision model loads from. That's this node's job. CLIP Vision Loader (Unified Memory) pulls a CLIP vision model from clip_vision/ and hands you a CLIP_VISION output - the image-encoder half of the family that WAN-style image-to-video, IPAdapter, and reference-based workflows all depend on. It's the DGX Nodes pack's take on ComfyUI's stock CLIPVisionLoader, aimed at DGX Spark and GB10 owners.

Why it exists

In the DGX Nodes pack, every loader shares one design goal: load weights directly into CUDA so they stay resident on a unified-memory machine. The stock ComfyUI loader doesn't handle the shared CPU/GPU pool gracefully - it can stage through CPU, then re-read or re-copy at inference. The DGX path reads the safetensors straight into CUDA, builds the vision model from already-resident tensors, and corrects ComfyUI's memory tracking so the model isn't needlessly evicted the moment something else loads.

Honest scaling check: a CLIP vision model is small - a few hundred MB, nothing like a Flux UNET. So this node isn't about shaving minutes off a load. It's about keeping your whole graph on one loader family so a DGX workflow stays consistent, and it does add the practical benefit of the model staying warm in unified memory between runs rather than being dropped and re-staged.

The inputs

Simple node, short list:

• clip_name - the file from clip_vision/ (e.g. a CLIP-ViT-H or ViT-bigG for IPAdapter).
• dgx_mode - ON (default) uses the DGX direct-to-CUDA path; OFF uses stock loading.
• device - which CUDA device, cuda:0 by default.
• storage_backend - auto, instanttensor, or fastsafetensors, same semantics as the rest of the pack.

One implementation detail the source handles for you: OpenCLIP-style files carry a visual. key prefix and sometimes need key conversion, and the node detects and normalizes that so the model actually constructs.

Wiring it up

The CLIP_VISION output feeds nodes that consume an image encoder: CLIPVisionEncode in IPAdapter setups, or the vision-conditioning path in image-to-video workflows. If you're already running a DGX checkpoint or UNET loader, this is the matching piece for the vision side.

Installing it

Same pack, same install - once. ComfyUI Manager: search DGX Nodes. Manual:

cd ComfyUI/custom_nodes
git clone https://github.com/broken-gage/ComfyUI-DGX-Nodes
pip install -r requirements.txt
pip install instanttensor fastsafetensors   # optional backends for the DGX path

Restart ComfyUI; nodes appear under DGX Nodes. Requires ComfyUI 0.24.0+.

Gotchas

• Safetensors only on the direct path. Other formats fall back to stock loading automatically - the node just does the standard thing, no error.
• It's a no-op off DGX hardware. Windows, x86 boxes, anything without the DGX backends installed: this node transparently falls back to stock ComfyUI loading. Keep it in shared workflows without guilt, but it only earns its keep on a Spark/GB10.
• Remember the pack is experimental, GPLv3, and authored with heavy AI assistance per its own README. For a small vision loader the downside of a bug is modest, but don't blind-trust it for anything safety-critical.