Multimodal — LLM (local first: GGUF + optional transformers)

This is the node that makes the whole pack make sense. It runs a real LLM or vision-language model as a node inside your ComfyUI graph - same Python process, no separate server, no OpenAI-style API, no key. You hand it text and up to three images, it hands you a text string, and whatever you do with that string is your business. The classic use: image → VLM describes what it sees → LLM writes a prompt → that prompt feeds CLIP Encode → KSampler. A closed loop where the machine reads its own output and decides what to generate next, with no copy/paste between windows.

The name "Multimodal - LLM" undersells it. The thing to internalize first is that there are two entirely different engines under the same hood.

GGUF first, transformers as the fallback

The default path is GGUF + mmproj through llama-cpp-python - the same format and library that powers llama.cpp. You point it at two files sitting in ComfyUI/models/llm/<your_model>/: the main weights and the vision projector (mmproj-*.gguf). Both have to come from the same release family; mixing a Qwen main with a Gemma mmproj fails loudly or silently degrades. Supported families are Qwen3-VL, Qwen2.5-VL, Gemma 3, and LLaVA 1.5/1.6, selected via gguf_vlm_handler. This is the part where people get burned: pick the wrong handler and you get "can't see the image" or empty/odd output. If your weights are Qwen3-VL, gguf_vlm_handler must be qwen3-vl, and your llama-cpp-python build has to actually export Qwen3VLChatHandler (see the wheel picker article in this pack for that).

The optional second engine is transformers (Hugging Face). Fill combined_model with an HF id like Qwen/Qwen3-4B or a local snapshot folder and the node loads it through the transformers pipeline instead. load_source (auto/local/huggingface) only applies to that field - it tells the node whether combined_model is a path on disk or something to fetch. Leave combined_model empty for pure GGUF workflows, which covers text-only too.

The inputs that matter

use_vision is the big switch. On, the model sees the linked image/image2/image3 sockets; off, linked images are ignored - ComfyUI can't hide a socket, but the model never sees the pixels. For an image-only run, leave the text fields empty. Then the sampling stack, which you'll recognize from any LLM UI: temperature (0.7), top_p (0.95), top_k (20), repetition_penalty, and max_new_tokens (default 5000 - drop it if the node feels sluggish). generation_seed_mode is either fixed or increment_each_run; the seed controls sampling randomness, not a chat session. There's no built-in chat memory - that's what system_prompt + extra_context are for, and this pack's helper nodes (System prompter, Context handler) exist to feed those two fields.

keep_models_loaded is worth a warning: it only caches this extension's HF pipelines and llama-cpp instances. It does not stop ComfyUI from unloading SD checkpoints. If you're fighting VRAM, offload_image_mode asks ComfyUI to free image-model memory before loading the LLM, and log_load_details prints which branch actually ran (GGUF+VLM, GGUF+text, HF pipeline, causal LM) to the console - invaluable when the output makes no sense.

Install and the one dependency trap

Install it like any custom node: ComfyUI Manager, search "Multimodal" and install theonetwoone/Multimodal-LLM-ComfyUI-node, or:

cd ComfyUI/custom_nodes
git clone https://github.com/theonetwoone/Multimodal-LLM-ComfyUI-node
pip install -r Multimodal-LLM-ComfyUI-node/requirements.txt

The requirements pull in transformers, accelerate, sentencepiece, and llama-cpp-python>=0.3.0 into the same Python environment ComfyUI uses. The trap: GGUF vision needs a CUDA-capable llama-cpp-python wheel that matches your Python version, platform, and CUDA tag, and the default pip one often isn't it. The pack ships a wheel-picker node and a CLI (python -m llm_comfy_multimodal.install_llama_wheel) to find a matching build from GitHub Releases - use them. Then restart ComfyUI.

Where people get stuck: "cannot import name Qwen3VLChatHandler" means your llama-cpp-python build predates Qwen3-VL support; "slow or OOM" usually means gguf_n_ctx too high, streaming on, or a second ComfyUI instance squatting on the GPU. If the dropdown fields look scrambled on an old workflow, delete and re-add the node.