🖥️VLM local Loader

This is the full-precision (or transformers-quantized) sibling of the pack's VLM-GGUF Loader - same idea, different backend. If GGUF is llama.cpp's compact single-file format, this node instead loads a vision-language model straight through the HuggingFace transformers library, in whatever distributed format the model was published in. Which one you want depends on your model: some VLMs only ship as transformers checkpoints with no GGUF conversion available, and this node is what handles those.

The pack currently supports three VLM families through this loader, and you pick which with the type field: llama-v (Meta's Llama-3.2-Vision), qwen-vl (Qwen2.5-VL), and deepseek-janus-pro (DeepSeek's Janus-Pro). This isn't a free-for-all "load any vision model" node - it's built around these specific architectures, so if your model isn't one of the three, this loader isn't the right tool. model_name_or_path accepts either a local directory (the README's own example is a Windows path like E:\model\Llama-3.2-1B-Instruct) or a HuggingFace repo ID directly, which will trigger a download on first use if it's not already cached locally. device (auto/cuda/cpu/mps) and dtype (auto/float32/float16/bfloat16/int8/int4) are the usual transformers loading knobs - auto on both is a reasonable default, but if you're tight on VRAM, dropping to int8 or int4 trades some quality for a much smaller memory footprint, same tradeoff you'd make with any quantized transformers load. is_locked keeps the model resident in memory across runs instead of reloading it every execution - leave it on unless you're actively swapping between VLMs, in which case pair it with the pack's Clear Model node to release memory before loading the next one.

Two outputs come out of this node: model and tokenizer(processor) - vision models need both a tokenizer for text and a processor for images, and this pack bundles them into one output slot rather than splitting them, which is a small but convenient design choice. Both wire into the pack's Local LLM node, with model_type set to match whichever type you picked here (VLM(llama-v), VLM(qwen-vl), or VLM(deepseek-janus-pro)).

Installing it means installing the pack: search "comfyui_LLM_party" in ComfyUI Manager, or git clone https://github.com/heshengtao/comfyui_LLM_party into custom_nodes and restart, then pip install -r requirements.txt from inside the pack folder using ComfyUI's own Python (portable builds need the explicit python_embeded\python.exe -m pip install -r requirements.txt).

The README's own troubleshooting advice for this exact node is worth repeating: Qwen2.5-VL support specifically requires an up-to-date transformers install (pip install -U transformers), and if your VLM reports errors, that's the first thing to check before assuming the model files themselves are broken. Beyond that, expect the usual VRAM math that comes with any full-precision vision model load - these are noticeably heavier than a comparably-sized text-only GGUF quant, so if you were planning to run a large VLM the way you'd run a small GGUF LLM, budget more headroom than you think you need, or drop dtype down before you drop the model size.