VLM Spatial Prompt Builder

Vision-language models are shockingly good at describing an image and quietly bad at answering with coordinates you can use. Ask "where's the cat?" and some models will happily hand you boxes in a format they invented five minutes ago, with units you can't trust. The VLMSpatialPromptBuilder exists to close that gap from the prompt side: it writes the instruction your VLM needs to return structured, bounded, parseable coordinates - and its sibling VLMStructuredSpatialParser reads the answer back out.

You reach for this node any time you want actual geometry out of a VLM rather than prose. The classic loop: hand the builder a request, feed the resulting prompt plus your image to a VLM, take the raw JSON response, and run it through the parser to get real VLM_DETECTIONS and VLM_POINTS types that the rest of the pack can consume.

How it works

The builder is pure string construction with a strict contract. You give it your plain-language request (the default is "Detect and segment every visible object, and return useful key points."), and it wraps that request in the constrained instruction - most importantly declaring the coordinate mode the model must reply in. Choose one of three:

• pixel - absolute pixel coordinates for your image,
• normalized_0_1 - fractions of width/height,
• normalized_0_1000 - normalized but scaled to 0–1000, which some models handle more stably.

Then tell it the canvas: width, height, frame_count (for video, where the model must say which frame each box belongs to), and fps if you want timestamps back too. A single prompt (STRING) comes out, ready to wire into the text input of your VLM node.

The trick that makes this whole scheme work: because the parser is strict, the prompt has to be too. The builder and parser are built to the same contract, so as long as you keep the coordinate mode and dimensions in sync between the two, whatever the model returns can be validated and normalized into typed geometry. Get the mode wrong and the parser will reject or mis-scale your boxes - the prompt half and the parse half are a matched pair, not independent widgets.

The inputs that matter

Only a few are worth your attention:

• request - say what you want detected or pointed at. Be explicit; "detect every person and return their key points" beats "find stuff."
• coordinate_mode - the units contract. Use pixel for single images where you know the exact dimensions; normalized_0_1000 is the safer choice for video or mixed resolutions.
• width / height - must match the actual image or video-frame dimensions you're feeding the model. They don't have to be the original source pixels if you downscaled for analysis - just what the model actually saw.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). Install via ComfyUI Manager (search "VLM Nodes") or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run pip with ComfyUI's Python and don't let the repo install its own torch. This node does no inference and downloads no models - it's instant on any backend.

Gotchas

The number-one way people burn themselves: mismatched coordinate_mode between builder and parser, or dimensions that don't match the image actually sent to the model. If your parser suddenly produces boxes that are all in one corner, that's the first thing to check. And while a good prompt makes structured output likely, small models still sometimes answer with prose. Feed the raw response into VLMStructuredSpatialParser and let it fail loudly and clearly rather than trying to fix broken JSON by hand.