ClipProj Apply

MiniMax H3 is a great video model and a terrible VRAM citizen. A huge chunk of its footprint - 15.7 GB in NVFP4 - is the text encoder, a Qwen3-VL-32B chopped to 50 layers, whose only job is turning your prompt into a [seq, 5120] conditioning tensor. It runs once, and pins the card for the whole session.

ClipProjApply is the honest fix: load a small Qwen3-VL-4B with ComfyUI's own Load CLIP, run it through this node with a learned projection matrix, and get out an object that behaves like the official 32B. The author's own recommendation for a single GPU is a three-node chain:

Load CLIP (type: krea2)  →  ClipProj Apply  →  the H3 node's clip input

That's the whole trick: no rewiring, no new vocabulary. The node returns a CLIP that drops into the H3 pipeline where the stock one sat, and ComfyUI manages the small encoder like any other - offloading it to RAM once the prompt is encoded, so the diffusion model gets the whole card back for sampling. The 4B plus projection lands around 4.5 GB instead of 15.7.

How it actually works

This only works because the 4B and the 32B share a tokenizer (151,936 tokens), so a prompt produces the same tokens at the same positions in both. That makes a position-by-position map between their hidden states learnable:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

The node reads one hidden layer of the small encoder (the matrix file records which "tap"), standardizes it, multiplies by a learned matrix W, and un-standardizes back into the 5120-dim space the DiT expects. Calibration was ridge regression, not training - no gradients, no epochs - so the whole thing is a few tens of MB. Two wrinkles: token 0 is an "attention sink" whose raw value would wreck short prompts, so the node substitutes its measured value; and the -mlp files carry a residual network on top of the matrix that starts out reproducing it exactly.

The two inputs that matter

• clip - a small encoder, already loaded. Pick the 4B (krea2 type) or 8B (boogu) on the stock Load CLIP.
• projection - the dropdown lists every .safetensors in ComfyUI/models/clip_projections/, plus three <control:...> entries: zero, identity, and random.

The controls are not projections - they're deliberate baselines to prove W is doing the work. Zero ignores your prompt entirely; identity copies the raw dims with no learning. They never error and run on any encoder, so run them first before trusting any new matrix.

The node outputs one thing: clip, the projected CLIP, which wires into the H3 node's clip input.

Which matrix file

Grab it from NicoLab28/ClipProj-MiniMax-H3. Start with mmh3-4b-ClipProj-v3-mlp.safetensors (4B) or mmh3-8b-ClipProj-v3-mlp.safetensors (8B). The matrix must match the encoder's size - 4B is 2560 dims, 8B is 4096 - and the node refuses a mismatch with an error that names the file you should have picked. Prefer a -celeb variant if you prompt named people; the v3.1 files need node 0.1.13+.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/nicolab28/ComfyUI-ClipProj

Restart ComfyUI (Manager users: search "ComfyUI-ClipProj"). There's no requirements.txt and nothing to pip install - the nodes import only torch and ComfyUI's own modules, which any working install already has. It needs ComfyUI 0.31.0 or newer. On first launch the pack creates ComfyUI/models/clip_projections/; that's where the matrices go. The encoder lives in models/text_encoders/ - a qwen3vl_4b_* file from the Comfy-Org/Krea-2 repo works, and a matrix calibrated on bf16 works on fp8 and int8 variants of the same size.

Where people get burned

• KeyError: 'W' - you're on a node version older than 0.1.13 loading a v3 -mlp file. Those files carry no linear matrix, which is why the loader prints | residual only when it does load one: expected, not a truncated download. Update the pack.
• "This projection does not go with this encoder" - size mismatch; pick the mmh3-4b-… or mmh3-8b-… file for the encoder you loaded.
• Output ignores your prompt - you almost certainly loaded a text-only Qwen3. It has the same hidden width as the VL, loads without complaint, and produces conditioning that never reads your prompt. The checkpoint must be a Qwen3-VL.

One honest caveat: this is a proof of concept, tested on Windows/NVIDIA, and you do lose some of the 32B's knowledge - simple and structured prompts hold up, while exotic named identities and multi-language speech are where the small encoder shows its size. For most workflows the trade is overwhelmingly worth it.