Image-Text to Text

The other "caption" nodes just tell you what's in the picture. This one lets you talk back. Image-Text to Text is the pack's visual-QA node: you hand it an image plus a prompt - "what color is the hat?" or "turn this sketch into a detailed scene description" - and it answers with text. It's the node you reach for when a plain caption isn't enough, and it's the closest thing in ComfyUI-Transformers to a full vision-language model inside your graph.

It's a thin wrapper around Hugging Face's image-text-to-text pipeline, same as every node in this pack. The node grabs the first frame of your IMAGE tensor, converts it to PIL, loads whatever model you name from the Hugging Face Hub (with trust_remote_code=True), and runs your prompt as a chat message alongside the image. The default model is Salesforce/blip2-opt-2.7b, a 2.7B-parameter vision-language model that's genuinely good at instructed captioning and simple questions.

The inputs that matter

• image - a standard ComfyUI IMAGE. Only the first frame is used, so batch a bunch of images if you like, but know you're only captioning frame zero.
• prompt - the instruction or question, in plain text. This is the whole point of the node; a blank prompt behaves a lot like the plain Image to Text captioning node.
• model_name - a string, no dropdown. Type the exact Hugging Face model ID.
• max_new_tokens - cap on how long the answer can be, default 50, up to 512.

The single output, generated_text, is a STRING - wire it into a text display, a file saver, or onward into a prompt that gets fed to a sampler.

How to install it

The README is gloriously minimal - it's basically one command and a depth-estimation screenshot. Install the whole pack via ComfyUI Manager (search "ComfyUI-Transformers") or clone it:

cd ComfyUI/custom_nodes
git clone https://github.com/kadirnar/ComfyUI-Transformers

Then restart ComfyUI. The pack's requirements.txt pulls in transformers>=4.42, torch, numpy, Pillow, accelerate, sentencepiece and friends. There's no model file bundled - the first run downloads BLIP-2 from the Hub, which is roughly 4GB and takes a while. Nothing will appear to happen while it downloads; be patient.

Where people get burned

The big one: BLIP-2 opt-2.7b is not small, and it runs eagerly on GPU. On a mid-range card you'll wait a few seconds per answer. If it's too slow, swap model_name for something lighter - smaller BLIP-2 variants or a modern small VLM - at the cost of answer quality. Also remember every node in this pack shares one quirk: the model is loaded fresh into a per-session cache on first execution, so the first run after a restart is always the slow one.

And a word of caution worth repeating for this pack: trust_remote_code=True means whatever model repo you point at can run arbitrary code. Stick to well-known model IDs from the defaults and reputable publishers - this is the same reason the ComfyUI ecosystem keeps reminding people to be careful what custom nodes and remote code they load.