Load Jina CLIP Vision

The name does two jobs: it loads the vision tower of jina-clip-v2, the multilingual multimodal embedding model Jina AI shipped at the end of 2024, and then it quietly throws away the text half of that same model to keep your VRAM intact. This is the front door of the whole ComfyUI_JinaCLIP_SDXL_IP_Adapter pack - every Jina IP-Adapter workflow starts by wiring this node's jina_vision_model output into one of the Apply nodes. It's fully local: AutoModel.from_pretrained off your own disk, no Jina cloud API, no key.

Why you'd reach for it: the pack re-implements the classic IP-Adapter trick - reference image goes in, gets turned into extra cross-attention tokens, style or character conditioning comes out without training a LoRA - but with a modern image encoder instead of the OpenCLIP ViT-bigG that Tencent used for SDXL back in 2023. Jina's vision tower is EVA02-based and noticeably better at actually reading a reference image. For anime style transfer on Mugen, the SDXL finetune this adapter was trained for, that better encoder is the whole point.

How it works

Loading a model with trust_remote_code gets you the full jina-clip-v2, which is overkill for image conditioning. So the node explicitly deletes the text modules and empties the CUDA cache, keeping only the vision tower alive. What comes out of the output socket isn't the raw model either - it's a small wrapper object that exposes encode_image_hidden_states, which resizes your reference to the processor's crop size, normalizes it, and returns the per-image hidden states the Apply node feeds to the projector.

The inputs that matter

• model_name - a dropdown of whatever model folders live directly inside ComfyUI/models/LLM/ (the code also accepts a lowercase llm folder; on Linux the case matters). The folder's name is what appears. If the dropdown is empty, the model isn't where the node is looking.
• device - auto means cuda:0 if you have a GPU, else CPU. If you've got a second card, cuda:1 offloads the vision tower there and keeps your generation VRAM clean.
• force_reload - false by default; flip it when you've swapped files on disk but ComfyUI keeps serving the cached model.
• dtype - auto maps to bf16. On an RTX 20-series or older, pick fp16 instead; bf16 wants Ampere or newer. fp32 exists for debugging.

Outputs: jina_vision_model (type JINA_MODEL) into the Apply node's matching socket, plus an info STRING the author wrote as a diagnostic - it prints the resolved model path and device. Wire info to a text display or ignore it.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/lRemixl/ComfyUI_JinaCLIP_SDXL_IP_Adapter

then restart ComfyUI - or ComfyUI Manager → search "ComfyUI_JinaCLIP_SDXL_IP_Adapter" and let it handle the clone. There's no requirements.txt in the repo; the pack needs transformers for the AutoModel loading, and ComfyUI's default venv often doesn't have it:

cd ComfyUI && pip install transformers

Then the heavy part: download the whole jinaai/jina-clip-v2 repo (~3.5GB, the safetensors alone is 1.7GB) and drop it at ComfyUI/models/LLM/jina-clip-v2/ as a direct subfolder. It uses custom code, so keep the folder intact - config.json and model.safetensors both have to be there.

Troubleshooting

• Empty model_name dropdown - jina-clip-v2 isn't in the right spot. It must be its own subfolder under models/LLM/ (or models/llm/).
• trust_remote_code / missing-key errors on load - your transformers is too old. pip install -U transformers.
• Node fails to import at startup - transformers is missing entirely; install it.
• Loads fine but generation dies on an older GPU - the auto→bf16 default; switch dtype to fp16.

One honesty note: this is a small, fresh hobbyist pack with essentially no community footprint yet - the adapter model it pairs with has had almost no downloads. That's not a knock; the model card ships a working workflow, and the code is clean. It just means you're early, and "early" comes with rough edges like the missing requirements.txt. And one licensing gotcha worth knowing before you build anything on it: the adapter weights are Apache 2.0, but the jina-clip-v2 vision model this node loads is CC BY-NC 4.0 - non-commercial. If money's involved, read that license first.