VLM Prompt HF (PyTorch)

"VLM Prompt HF (PyTorch)" does the same job as the GGUF nodes - look at an image or video, write a prompt - but swaps the backend from llama-cpp-python to Hugging Face transformers with PyTorch. Same inputs, same presets, same single prompt output. The practical difference is which models you can run and what it costs to load them.

Why would you pick the HF flavor over the GGUF one? The honest answer is model availability and headaches. The GGUF path needs a multimodal llama-cpp-python build with the right chat handlers, and that wheel situation has sent more than one person down a rabbit hole. The HF path leans on transformers, accelerate, and huggingface_hub, which you very likely already have in a ComfyUI environment, and it lets you run up-to-date PyTorch checkpoints that may not have a clean GGUF conversion yet. The tradeoff is weight: a raw fp16 model is a lot fatter than a Q4 GGUF, which is where the quantization options come in.

The inputs that matter

• model_name - the HF catalog in models_hf.json is small on purpose: Qwen3.5-2B-abliterated (default) and Qwen3.5-4B-Claude-HighIQ-THINKING. The JSON even carries rough VRAM notes per model.
• quantization - None (FP16/BF16) (default), 8-bit, or 4-bit, via Hugging Face's BitsAndBytes. The 2B model notes 4.5 GB at fp16, 2.5 GB at 8-bit, 1.5 GB at 4-bit; the 4B is 9.5/5.0/2.5. If bitsandbytes isn't installed for your platform, the node logs a fallback and just runs fp16 - it doesn't crash.
• preset_prompt / system_prompt / custom_prompt - the same 27 presets from system_prompts.json (Wan I2V Prompt is the default); custom_prompt overrides the preset when filled.
• max_tokens, keep_model_loaded, seed - same semantics as the GGUF node.
• image / video (optional) - wire in either to get actual vision; text-only runs work fine.

Output: prompt (STRING), wired into a Text Encode or your video model's prompt input.

Installing it

Same pack install either way:

cd ComfyUI/custom_nodes
git clone https://github.com/mexxmillion/ComfyUI-VLMPrompt.git
pip install -r ComfyUI-VLMPrompt/requirements.txt

requirements.txt covers the HF backend (transformers, accelerate, huggingface_hub, numpy, pillow). If you want 4/8-bit quantization, add a working bitsandbytes for your platform. Model snapshots download to ComfyUI/models/LLM/HF on first use.

One structural nicety: the pack's __init__.py wraps the HF import in a try/except, so if transformers is genuinely missing, ComfyUI still loads the GGUF, Muse, and VRAM nodes and just skips the HF ones. No crash, just missing nodes in your search.

Common issues

• Huge first download - fp16 snapshots are big. The 4-bit path is worth it on small cards.
• "HF backend unavailable" in the console - transformers isn't in your environment. Install requirements and restart.
• Quantization silently ignored - that's the BitsAndBytes fallback to fp16. Confirm your bitsandbytes import actually works before debugging anything else.