VLM Structured Spatial Parser

The moment a vision-language model hands you back coordinates in JSON, you have a parsing problem. Where do the boxes start? Is that [x, y, w, h] or [x1, y1, x2, y2]? Which field is the label, and are those coordinates pixels, fractions, or something a model invented? VLMStructuredSpatialParser is the node that turns that raw, semi-trusted JSON from any VLM into the typed VLM_DETECTIONS and VLM_POINTS formats this pack actually uses downstream.

It's the receiving end of VLMSpatialPromptBuilder's contract. You build the strict prompt, the model answers with structured JSON, and this node validates, normalizes, and converts it into something you can wire into a crop, a mask, a tracker, or a bounding box without writing a single line of JSON-cleaning Python.

How it works

It's a strict parser, not a forgiving one - that's the point. You feed it the model's raw response JSON and tell it the geometry: coordinate_mode (one of pixel, normalized_0_1, normalized_0_1000), the width and height of the image the model actually saw, frame_count for video, and fps if you want timestamps. The source field is just a provenance tag stamped onto the records so you can trace where a detection came from later.

Under the hood it validates the JSON against the expected spatial schema, checks that coordinates are in range and finite, and converts everything into a canonical normalized form (hence the normalized_json output - the same detections re-serialized in the pack's stable format, ready for VLMDetectionsToJSON or VLMDetectionsFromJSON if you want to persist or hand-edit them).

The parser also refuses to be lenient in useful ways. Out-of-range values, non-finite numbers, or malformed JSON fail the node with an actionable error instead of quietly producing a garbage box you'll discover three nodes downstream. That's a feature: fail early, fail loud, fix the prompt or the model, don't ship broken geometry.

Inputs and outputs

The inputs you'll actually touch:

• response - the model's raw JSON output. Paste it or wire it from the VLM's string output.
• coordinate_mode - must match what the model was told to return. Mismatches here are the classic source of "all my boxes are in the corner."
• width / height - the analysis resolution, not necessarily your original source resolution.

Outputs:

• detections (VLM_DETECTIONS) - labeled boxes, ready for VLMCropDetections, VLMDetectionsToMasks, VLMTrackDetections, or Florence region tasks.
• points (VLM_POINTS) - key points from the same response, in the pack's pixel-coordinate format.
• normalized_json (STRING) - the canonical JSON, for persistence or inspection.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). ComfyUI Manager → search "VLM Nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run with ComfyUI's Python; don't let the repo touch torch. Pure parsing, no model, no downloads, works on any backend.

Gotchas

Two things trip people up. First: coordinate mode and dimensions must match what the model actually saw and was told - change one side and the geometry silently warps. Second: this parser is strict on purpose, so a model that ignores your structured-output request and replies with prose will just error. That's the design; go fix the prompt (or switch to a model that obeys structured output) rather than wishing the parser were sloppier.