Kosmos-2

Most captioning VLMs describe what's in an image. Kosmos-2 goes one step further: it grounds its descriptions to the actual regions of the image. When it says "a dog sitting on a bench," it can reference where the dog is, not just that the dog exists. That grounding makes it genuinely useful for visual question answering and for workflows where you care about where things are, not just what's there.

What it is

This is the pack's Kosmos-2 node, running Microsoft's kosmos-2-patch14-224 - "Grounding Multimodal Large Language Models to the World," if you want the paper. It's a small enough model (roughly 1.6B parameters) that it runs on a modest GPU or even CPU in a pinch, which puts it in the "fast enough to iterate with" camp rather than the "wait five minutes per query" camp.

The inputs

Only two, which is refreshing:

• image - your input image.
• text_input - a STRING prompt, your question or instruction. "Describe the image," "What's on the table?", "Where is the cat?" - whatever you want the model to do with the picture.

The output is a STRING containing the model's grounded response. You can read it directly, send it to a text display, or push it into the pack's prompt-generation chain as raw material.

How it works

On first use the node automatically downloads the model files into models/LLavacheckpoints/files_for_kosmos2 - no manual model hunting, no GGUF wrangling, which makes it a nice contrast to the pack's llama.cpp-based nodes. It runs through the transformers stack and stays cached for subsequent runs, so the first call is slow and everything after is fast. You don't pick a checkpoint from a dropdown; the model is baked into the node.

Install

Pack-level install, one command:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes

Restart ComfyUI (or install via Manager by searching "VLM Nodes"). Python 3.9+ is required. Because the model auto-downloads, the only real "setup" is patience on the first run and making sure you have a few GB free on disk.

Gotchas

The download is the main failure surface. If the first run fails, check your connection and that models/LLavacheckpoints/files_for_kosmos2 isn't half-written - deleting the partial folder and re-running fixes most cases. Also worth knowing: Kosmos-2's grounding output is great for captions and VQA, but it's not a prompt-polisher. Pair it with CreativeArtPromptGenerator or LLavaPromptGenerator if your goal is a diffusion prompt, not a description.