H3 Qwen3-VL Model (Local ComfyUI)

What it is

The local option for the compact director. Instead of shipping your reference images to OpenRouter and paying per prompt, this node loads a Qwen3-VL text encoder from your own models/text_encoders folder through ComfyUI's native CLIP loader, and hands the director an llm_model that writes its prompts on your own GPU, offline, for free, and without a hosted filter deciding what you're allowed to describe.

That last part is why people bother. Qwen3-VL is Alibaba's open-weights vision-language family (Apache 2.0, a ViT feeding a Qwen3 backbone), and it has quietly become infrastructure in this scene rather than a model you generate with - the default local captioner, and increasingly the text encoder new image models are built on. Its one rough edge is the same one that explains this node's preferred filename: the Instruct builds carry assistant-style refusals, so people run abliterated versions for anything explicit.

How it works

Three dropdowns, one output:

• clip_name - populated from ComfyUI's text_encoders list. If qwen3-vl-4b-heretic_int8.safetensors is present it becomes the default; otherwise you get the first file in the folder, which is rarely what you want.
• clip_type - ltxv (default) or minimax. This is which ComfyUI loader recipe gets used for the file, not a MiniMax-specific mode.
• load_device - default or cpu.

Output: llm_model, into the director's llm input.

The loading is lazy. Nothing is read from disk when you queue the graph - the connection only pulls the model up when the director actually asks for text, and it reuses ComfyUI's single-entry CLIP cache and memory manager rather than loading its own copy. That matters because you are now budgeting VRAM for two models at once, and because it means load_device: cpu is a real escape hatch when the encoder and the diffusion model start fighting over your card.

The request plumbing is the part worth understanding, because it explains the failures. The node converts the director's chat messages into Qwen's own prompt format, decodes each embedded image and inserts the vision placeholders where they belong - keeping image order and source labels exactly as sent, so a video frame never gets silently remapped to a picture. Then it generates through the CLIP object with fixed sampling settings (top-k 40, top-p 0.9, min-p 0.05, repetition penalty 1.05, sampling only when temperature is above zero), decodes, and checks the answer is valid JSON. Fenced code blocks are tolerated; anything else raises: "Local Qwen returned invalid/truncated JSON. Increase max_tokens or reduce the scene count; no paid fallback was called." That's a deliberate design choice. A local model that rambles fails loudly instead of shipping mangled JSON into your conditioning.

Install and setup

cd ComfyUI/custom_nodes
git clone https://github.com/elgalardi/ComfyUI-VisionPromptAssistant
# or install "Vision Prompt Assistant" through ComfyUI Manager, then restart

No pip dependencies. ComfyUI 0.30 or newer is required, and that floor is not decorative - the pack is written against the newer node API, and so is the text-generation path this node uses.

Then put the model where ComfyUI can see it: ComfyUI/models/text_encoders/. The node does not download anything. If the filename isn't in that folder, the dropdown won't offer it and execution stops with a file-not-found error from ComfyUI's own path lookup.

The caveat

As shipped in 2.0, the local path's connection class reaches back into the pack module for a helper (LocalVisionPromptGenerator._load_clip) that no longer exists in the 2.0 tree. The node itself loads fine, the filename validation passes, and then generation can die with an AttributeError naming that class. If you hit exactly that, it isn't your model file - switch to the OpenRouter, Ollama or API provider while it's sorted out, and check the repo's issues. Everything above about how the local path should behave comes from reading the shipped code; treat the local route as the experimental one in this pack.

Common issues

• AttributeError: ... LocalVisionPromptGenerator - the caveat above. Not a config problem.
• Truncated or invalid JSON, usually with several scenes - raise the director's max_tokens, lower continuous_scene_count, or shorten the request. Small local models degrade fast as output length grows.
• Refusals on explicit material - that's the Instruct training, not your prompt. An abliterated build is the fix, which is what the -heretic- filename is hinting at.
• Out of memory mid-run - set load_device to cpu and accept the speed hit, or free VRAM elsewhere. The encoder and the video model share the card.
• The prompt comes out worse than the API's - expected. Be honest with yourself about the tradeoff: you bought offline, uncensored and free-per-call, and paid for it in writing quality.