LLM CLIP Loader (Qwen)

This is the node that loads the "LLM" half of the LLM CLIP trick: replace SDXL's ancient CLIP text encoder with a 4-bit Qwen3-4B and a small adapter, so your prompts get read like sentences instead of a bag of tags. The name isn't a lie. There's no API call, no key, and nothing remotely hosted - everything runs on your own GPU. It's also an honest proof of concept, so set expectations before you start.

Why this exists

The whole LLM-encoder thing isn't a gimmick - by 2026 it's the mainstream: Z-Image, Klein and Anima all encode with Qwen3 variants. But those are new models built that way. This pack's author (molbal) went the weird direction and trained a small adapter so an LLM can condition classic SDXL checkpoints. The motivation, from his own announcement post: CLIP is weak at spatial language ("left of", "behind"), negations ("no moustache"), and chokes at 77 tokens. An LLM reads the whole prompt as one coherent instruction.

The mechanism is where it gets clever. Qwen3-4B outputs a hidden state that's a sequence of 2560-dim vectors - not at all the shape SDXL expects. So he trained a Perceiver-style resampler (~280M params, 4 layers of cross-attention) that condenses Qwen's output into exactly 77 vectors at 2048 dims plus a 1280-dim pooled embedding: the exact shape OpenCLIP-G feeds SDXL's attention. A small LoRA on Qwen steers the hidden states toward CLIP-like values. The trained weights were fitted to 10,000 image-caption pairs from the Spright dataset, cached from the real CLIP outputs. That's the whole trick - an LLM encoder glued onto SDXL's existing geometry.

The inputs that matter

This loader is the "load the pieces" node. Three fields, two of which you'll probably leave alone:

• base_repo_id - Hugging Face ID of the base LLM. Defaults to unsloth/Qwen3-4B-unsloth-bnb-4bit, a 4-bit bitsandbytes quant of Qwen3-4B (Apache 2.0). You can point it at any HF causal-LM repo if you're experimenting.
• resampler_name - a dropdown of .pth files in ComfyUI/models/text_encoders. This is the trained resampler you downloaded. If the dropdown is empty, you skipped step zero of the install.
• lora_repo_id - optional, defaults to molbal/qwen-clip-resampler-adapter. The LoRA that nudges Qwen's hidden states. It's pre-tuned; leave it.

Installing it

cd ComfyUI/custom_nodes
git clone https://github.com/molbal/ComfyUI-LLM-CLIP
# restart ComfyUI
pip install transformers accelerate peft bitsandbytes

Or just search ComfyUI-LLM-CLIP in ComfyUI Manager and let it handle it. Then the bit everyone forgets:

1. Grab resampler.pth from https://huggingface.co/molbal/qwen-clip-resampler-adapter and drop it in ComfyUI/models/text_encoders. Not anywhere else - the loader's dropdown reads that folder specifically.
2. First run downloads the 4-bit Qwen3-4B (~2.5 GB) from Hugging Face. Needs internet, takes a few minutes, and the same download cost repeats on every fresh install.

Where people get burned

• It's a POC, not a production swap. The resampler and LoRA were trained at 4-bit on just 10k images by one person on a rented 5090. Expect a quality drop versus native SDXL CLIP, not an upgrade. The author says it himself: "nothing special, but enough to demonstrate the idea works." Read it as an experiment worth playing with, not a daily driver.
• Slow cold start. Loading a 4B LLM plus LoRA plus resampler takes tens of seconds, and it happens every time ComfyUI starts - CLIP loaded in the blink of an eye. On an 8 GB laptop it works, but you feel the difference.
• Windows + bitsandbytes. The 4-bit load needs bitsandbytes, which on Windows still occasionally fights with your torch version. If the loader dies at import or "CUDA error", that's usually the culprit.
• VRAM math. The 4-bit Qwen (~2.5-3 GB) sits in VRAM alongside your SDXL checkpoint and VAE while it encodes. It's freed afterward, but if you're right at the edge, this nudges you over.

The output is a single LLM_CLIP_BUNDLE, and it feeds the other node in this pack, LLM CLIP Text Encode, which turns your words into conditioning. One loader, two text-encode nodes (positive and negative), and your existing KSampler graph stays untouched.