ClipProj Device Loader

ComfyUI's stock CLIPLoader gives you exactly two choices for where a text encoder lands: default and cpu. That's fine until you have a multi-GPU machine and a card that's sitting there idle. This node exists to fill that gap - it loads any text encoder onto the GPU you name, and nothing else. No projection, no magic.

It's the quiet workhorse of the ComfyUI-ClipProj pack. If you're on a single card, you probably want the stock Load CLIP plus ClipProj Apply instead of this (the pack's README says so in so many words). But on a five-GPU box, being able to say "the encoder goes on cuda:3 and stays there" is genuinely useful.

What it loads

• clip_name - a dropdown of everything in ComfyUI/models/text_encoders/. A Qwen3-VL-4B or -8B is the target here, but it'll load any encoder ComfyUI knows.
• type - defaults to auto, which reads the checkpoint header and picks the matching architecture. This is a real safety feature, not decoration: it checks for a vision tower, so a text-only Qwen3 (same hidden width as the VL, would silently ignore your prompt) gets refused up front.
• device - which GPU receives the encoder (cuda:0, cuda:1, …; xpu on Intel; cpu as fallback).
• mode - resident, streaming, or dynamic.

Output is a single CLIP, which you can wire straight into ClipProj Apply to add a projection, or use directly if you want the raw encoder.

The mode tradeoff is the whole game

The three modes differ on two independent axes: where the encoder folds when it isn't in use, and whether it loads in one block or gets paged layer by layer.

• resident (default) - loaded in one go and pinned to the chosen card. Fastest to encode, but it never leaves. If that card is also running the DiT, the diffusion model keeps 4–9 GB less headroom at every single sampling step.
• streaming - loaded in one go but folds back to RAM when done. Same encoding speed, and the VRAM is returned for sampling. Costs one full transfer each time the encoder is used again.
• dynamic - ComfyUI pages the weights layer by layer. Lowest peak usage, slowest to encode.

The pinning in resident isn't paranoia. It exists because if the encoder were left to ComfyUI's memory manager, an unload would free nothing while removing the model from ComfyUI's accounting - the manager would oversell the VRAM and you'd OOM on a card that looks empty. The cost is that pinned things never leave on their own, which is what the pack's ClipProj Free VRAM node is for.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/nicolab28/ComfyUI-ClipProj

Restart ComfyUI (Manager: search "ComfyUI-ClipProj"). No requirements.txt, nothing to pip install - it only imports torch and ComfyUI's own modules. Requires ComfyUI 0.31.0+; tested on Windows 11 with NVIDIA (3090/4070/3060), and the code is deliberately platform-neutral.

Rough edges worth knowing

• int8 encoders (like qwen3vl_4b_int8_convrot) historically only worked in resident mode - the paged path would hand the dequantizer bf16 where it expected int8 and fail with a message naming nothing. Since 0.1.13 they work in all three modes, with a warning if you leave resident. If you hit a dequantize_int8_embedding error, that's the thing to check first.
• Single card, default settings, is a trap. With resident on your only GPU, the encoder takes 4–9 GB away from the DiT at every step and the model starts paging its own weights. On an 8 GB card someone measured the stock 32B encoder winning against a pinned projected 4B, because ComfyUI unloads the 32B before sampling while the pinned 4B stayed put. Use streaming, or use the stock loader plus ClipProj Apply and let ComfyUI handle the offloading.

If you do run it on a single card, streaming is the mode that keeps the VRAM win - same fast block load, and the encoder actually leaves when it's done.