🖥️Easy VLM local Loader

Not every vision-LLM workflow needs an API key. comfyui_LLM_party's README explicitly lists three local vision-language models it's tested support for - Meta's Llama-3.2-Vision, Alibaba's Qwen2.5-VL, and DeepSeek's Janus-Pro - and this node is the loader for all three, running entirely on your own hardware via the transformers library.

This is one of the places the pack goes head-to-head with lighter alternatives like comfyui-ollama, which people also reach for specifically to keep local models in the loop. The tradeoff here is more setup (a real transformers-based load, more VRAM, more disk) for direct access to models Ollama may not have packaged yet, and finer control over dtype and device placement.

How it works

Point it at a model and tell it which of the three supported architectures that model actually is - this matters because loading code differs by family, so a mismatch between the type you pick and the model you're actually pointing at will break the load. Once loaded, it hands back the model and its paired tokenizer/processor for a downstream VLM chat or run node to use.

The inputs and outputs that matter

• model_name_or_path - which model to load, from a dropdown ComfyUI populates by scanning your local models. If it's empty, you haven't downloaded a VLM into the expected location yet - the README's general guidance for local-model nodes in this pack applies here too: fill in a local path, or a Hugging Face repo id (like meta-llama/Llama-3.2-11B-Vision-Instruct or Qwen/Qwen2.5-VL-3B-Instruct) if you want it fetched for you.
• type - llama-v (Llama-3.2-Vision), qwen-vl (Qwen2.5-VL), or deepseek-janus-pro (Janus-Pro). This has to match the actual model family at model_name_or_path.
• device (auto / cuda / cpu / mps) and dtype (auto / float32 / float16 / bfloat16 / int8 / int4) - where it runs and at what precision. Lower precision trades quality for VRAM headroom, same tradeoff as the pack's other local model loaders.
• is_locked (default true) - keeps the model pinned in memory across runs rather than reloading it every time. Turn it off if you need to swap models mid-session without restarting ComfyUI.

Outputs: model and tokenizer(processor) - both CUSTOM types, wired into whichever downstream node in the pack runs the actual VLM inference.

How to install it

Search comfyui_LLM_party in ComfyUI Manager and install, restart, then download or point at your chosen VLM checkpoint before your first run. Manual install:

cd ComfyUI/custom_nodes
git clone https://github.com/heshengtao/comfyui_LLM_party.git

Run pip install -r requirements.txt from the pack's own folder with ComfyUI's Python, then restart - and note the README's own caveat: if a VLM or GGUF model errors, update llama-cpp-python to the latest release from its GitHub, since older builds are a common source of load failures for these models.

Common issues & troubleshooting

Load fails with a shape or architecture mismatch error. Check type against what you actually downloaded - these three model families aren't interchangeable, and picking the wrong one is the most common way this node breaks.

Out of memory on load. Multi-billion-parameter VLMs are large; step dtype down (bfloat16 or float16 before int8/int4) rather than assuming your GPU can't run any VLM at all.

First run is very slow. If model_name_or_path is a Hugging Face repo id rather than a local path, that's a multi-gigabyte download happening on first use - expect it, and make sure you have the disk space.