Apply Jina IP-Adapter

This is the node that actually does the IP-Adapter thing. The two loaders before it are setup; here's where a reference image stops being a picture and becomes part of the model's cross-attention. Same decoupled trick Tencent published in 2023 - image tokens ride their own attention path while your text prompt stays untouched, which is why you can still steer the result with words - just rebuilt around Jina's vision tower and trained specifically for Mugen, an anime SDXL finetune. Use it for style transfer, character look, mood-copying; it's the IP-Adapter you've always used, with a better image encoder behind it.

How it works

One breath: it runs your image through the Jina vision model, pushes the hidden states through the 16-token projector, then writes ip_to_k / ip_to_v projection weights into eleven SDXL cross-attention blocks (input blocks 4/5/7/8, the middle block, and the six output blocks). At every sampling step, each cross-attention pass computes the image tokens' contribution and adds it to the text attention output, scaled by weight. Two details worth knowing from the code: it clones your model rather than mutating it, and it zeroes the image embedding on the unconditional pass so your negative prompt doesn't accidentally pick up the reference. That last one is a genuinely clean touch that a lot of adapters get wrong.

Inputs

• model - the SDXL checkpoint you're generating with. Mugen is the natural partner; the trainer confirms it works on Mugen with the original CLIP-G/CLIP-L text encoders, and since the adapter only touches UNet cross-attention, other SDXL anime finetunes are worth a shot.
• jina_ip_adapter - output of the model loader.
• jina_vision_model - output of the vision loader.
• image - your reference, from a Load Image node. One image; for several references use the Multi variant.
• weight - the one dial that matters. Default is 1.0, and the model author explicitly says that's too hot for most cases. This is a style adapter above all: 0.6–0.9 for transfer, and the usual IP-Adapter rule applies - above roughly 0.8 the reference overrides your prompt, below 0.4 it's a whisper. Negative weights push the output away from the reference, which is occasionally exactly what you want.

Output: MODEL - the patched clone, straight into a KSampler. That's it; nothing else needed.

Install and models

Shared with the whole pack:

cd ComfyUI/custom_nodes
git clone https://github.com/lRemixl/ComfyUI_JinaCLIP_SDXL_IP_Adapter

then restart, or ComfyUI Manager → search "ComfyUI_JinaCLIP_SDXL_IP_Adapter". No requirements.txt ships, so if the pack fails to import, pip install transformers. Two model downloads: jina-clip-v2 (~3.5GB) into ComfyUI/models/LLM/jina-clip-v2/, and Mugen-Jina-IP-Adapter.safetensors (790MB) into ComfyUI/models/ip_adapter/. The adapter's HuggingFace repo also carries WorkflowMugenJinaIPAdapter.json - pull it and drag it into ComfyUI for a working reference graph instead of rebuilding from scratch.

Troubleshooting

• "Model loading failed" on the vision loader - it can't find jina-clip-v2 in the LLM folder, or transformers is missing.
• Image has no visible effect - check weight isn't near zero, and make sure the jina_vision_model and jina_ip_adapter outputs come from their loaders, not fresh copies.
• Weak results on realistic photos - expected. This adapter was trained on anime data for Mugen; the KB's standing verdict on SDXL IP-adapters applies double here: they lean stylized. It's an anime style tool, not a photorealism tool.
• OOM on an 8GB card - the 368M-parameter adapter plus a 1.7GB vision tower add up fast. Run the vision loader on cpu (or cuda:1 if you have a second card) via its device input, and set its dtype to fp16.

And the licensing line that keeps coming up with this stack: the adapter is Apache 2.0, but the jina-clip-v2 vision model underneath is CC BY-NC 4.0. Non-commercial. Worth checking before you point this at paid work.