Generate Text with Molmo

Generate Text with Molmo is the pack's answer to a different kind of question. Pixtral and Llama Vision describe; Molmo points and counts. Ask it "how many apples?" and you get a number. Ask it "where is the red one?" and it answers with <points> tags holding percentage coordinates - which this pack's own Parse Points and Plot Points nodes are built to turn into dots on the image. That's a genuinely different capability than plain captioning, and it's why Molmo has a devoted corner of the ComfyUI community for targeted questions like "what color is her hair?" rather than full descriptions.

How it works

The node builds Molmo's chat format (<|im_start|>user...), passes images and text through Molmo's custom processor (with the model's own code loaded - trust_remote_code is on and non-negotiable here), then calls generate_from_batch. Images are inserted at the start of the prompt; there's no inline placement like Pixtral. Note the README's findings on its limits: Molmo is surprisingly bad at bounding boxes, and pointing at two objects in one prompt tends to merge the labels - split it into two simple prompts and it cooperates.

Inputs that matter

• molmo_model - a VISION_MODEL from Load Molmo Model.
• system_prompt - optional (default empty).
• prompt - default "Describe this image. " - but this is the node where you should rewrite the prompt to ask a specific question. That's its strength.
• images (optional) - an IMAGE list; no batch text generation, one task at a time.
• max_new_tokens - default 256.
• temperature / top_p / top_k - defaults 0.3 / 0.9 / 40. For pointing tasks, low temperature is your friend - you want deterministic coordinates.
• stop_strings - default <|endoftext|>.
• seed, include_prompt_in_output, unload_after_generate - the standard set. No repetition_penalty; the author says it doesn't work for this model.

Output is a single STRING, which for pointing questions is XML-ish <points ...> markup you'd feed into Parse Points.

Install

One-time pack setup - Manager search ComfyUI-PixtralLlamaVision, or clone the repo. Requirements: transformers >= 4.45.0, accelerate, bitsandbytes, torchvision. Molmo is a 7B model you source yourself into ComfyUI/models/LLM (the loader needs the architecture name MolmoForCausalLM in its config).

Troubleshooting

• One prompt pointing at two things gives one merged label - known limitation; split into two prompts.
• Bounding box requests fall apart - expected. Molmo points; it doesn't box. Use the Llama/Pixtral path for boxes.
• Coordinates come back as percentages - that's normal; Parse Points divides by 100 to normalize, so they're wired for exactly this output.
• Custom-code load errors - update transformers; Molmo needs a current version and its own repo code.

For building "click on the thing" style automation or asking the image a focused question, this is the node. Just don't ask it for a bounding box.