Visual Question Answering

This is the pack's "self-check" node, and it's genuinely fun. VisualQuestionAnsweringPipeline takes an image, takes a question, and returns the model's answer - wire it after a sampler and ask your own generation "is there a cat in this picture?" or "what color is the car?" before you commit to saving it. It turns ComfyUI from a fire-and-forget generator into something that can evaluate its own output. That loop - generate, ask, re-roll on a bad answer - is where this node earns its keep.

It's from kadirnar/ComfyUI-Transformers, the pack that wraps Hugging Face's transformers.pipeline() as one node per task. The default model is dandelin/vilt-b32-finetuned-vqa - ViLT, a vision-language transformer (BERT-sized, ~400 MB) fine-tuned on VQA v2.

How it works

Under the hood it converts the first frame of your IMAGE tensor to PIL, calls pipeline("visual-question-answering", model=model_name), and returns the result as formatted JSON. For ViLT on VQA v2, the answer is a single short word or phrase - "cat," "blue," "yes" - drawn from a fixed answer vocabulary. It's an old-school approach compared to modern generative VLMs (a 2021 model, distilled-style answers), which has a predictable consequence: it's good at one-word factual answers and bad at anything that needs a sentence or reasoning.

Two behaviors worth knowing:

• Only the first frame is looked at. Feed it a batch and frames 2+ are ignored, same as the pack's other image nodes.
• The output is a JSON string of candidate answers with scores - e.g. a list like [{"answer": "cat", "score": 0.9}, ...] - not a single clean string. You'll be reading the JSON to get the top answer.

Inputs and outputs

• image - an IMAGE tensor, straight off a VAE decode.
• question - a single-line STRING. Phrase it simply: "Is there a cat?" beats "Could you describe any feline subjects present."
• model_name - default ViLT-VQA; free text, so you can drop in any visual-QA model on the Hub.
• Output: answers_json - STRING.

The question field is where you'll spend your effort. VQA models are trained on short, literal questions, so "how many birds are flying?" works and "what is the mood of this scene?" gets you a shrug.

Installing it

Pack-standard install - ComfyUI Manager (search "ComfyUI-Transformers") or:

cd ComfyUI/custom_nodes
git clone https://github.com/kadirnar/ComfyUI-Transformers
cd ComfyUI-Transformers
pip install -r requirements.txt

then restart. The README's install line (cd custom/nodes) is a typo. First run downloads ViLT from the Hub to ~/.cache/huggingface, and subsequent runs re-load it each time (the pack's pipeline cache is unused by these nodes).

Common issues

• Answers come back as JSON, not plain text. Wire a Show Text node in and read the top entry.
• Nonsense answers to complex questions. That's the VQA-vocabulary limit, not a config problem. Ask simpler questions.
• Slow on CPU. ViLT isn't huge, but image+text inference plus a per-run model load adds up. Not a node you want firing every frame.

The honest verdict: as a quality gate it's fun but shallow - a modern VLM node will answer better and in full sentences. Where this one wins is that it's small, local, no API key, and built to take a ComfyUI IMAGE directly. For a first-pass "did it actually draw what I asked" check, that trade is often the right one.