Shrug VLM

This is the node the pack exists for. ShrugVLM is a chat call to a vision-language model running on your own machine - usually a Qwen3-class model on Apple silicon via MLX. You feed it a prompt, optionally some images, and it returns text: a caption, an edit instruction, a structured description, a judgment call. No cloud, no API key, no cost per token, and the images never leave your LAN.

Where does it actually earn its keep? Captioning your own generations so you can re-prompt them. That's the loop people rave about with VLM-in-the-graph: generate, describe what you got, feed the description back in with a style tweak. It's also great for turning frames into structured scene descriptions before a video pipeline, or as a decision node - "which of these looks right?" - before routing.

How it works

The node builds an OpenAI-style chat message list: an optional system prompt, your user prompt, and any images as JPEG content. Then it POSTs to the server's /v1/chat/completions (images base64-encoded as data: URLs) or /v1/chat/completions/multipart (raw image bytes as file uploads) depending on the transport widget. The heavy lifting - tokenizing, vision tower, generation - happens server-side; the ComfyUI side just encodes tensors to JPEG and decodes the reply.

The inputs that matter

• user_prompt and system_prompt - the actual prompt. system_prompt is where you'd wire a ShrugTemplate output.
• images (optional) - an IMAGE tensor batch. All images go into one request.
• max_tokens (default 1024) - how long an answer you'll tolerate.
• temperature (default 0.7) - crank down for captioning consistency, up for creative re-prompting.
• enable_thinking (default off) - turns on Qwen3-style reasoning blocks.
• transport - base64 (default) or multipart. The tooltip says it plainly: multipart is ~57 ms/image faster but requires server support. Unless you know your server build handles it, leave base64.
• image_resize_max (default 0) - if you set it above 0, the longest edge is resized to that before upload. This is the one you'll actually touch for speed; huge images get slow and token-hungry.
• image_quality (default 85) - JPEG quality for the upload. 60-70 is fine for captions.

Two outputs: response (the visible answer) and thinking (the <think> reasoning block, if the model emitted one). Wire response to a ShowText node and you're done.

How to install

Standard pack install - ComfyUI Manager, search Shrug-Prompter, install, restart. Or clone it:

cd ComfyUI/custom_nodes
git clone https://github.com/fblissjr/shrug-prompter

Dependencies are already in any modern ComfyUI (httpx, orjson, pillow, torch, numpy); nothing heavy to install, nothing to download into ComfyUI. What you do need is the heylookitsanllm server running with a vision model loaded, plus a ShrugConnection node feeding this one. Recent ComfyUI is also required - the pack uses the V3 extension API.

Common issues

• Blank or erroring response. The server isn't up, the model id on your connection doesn't exist, or the model has no vision support. Check the server logs first.
• "Images ignored." Usually a multipart transport on a server build that doesn't support it. Drop back to base64.
• Slow uploads. Your images are being base64'd at full resolution. Set image_resize_max to 1024 (or 512 for captions) and lower image_quality - you lose almost nothing for description tasks.
• Empty thinking output. enable_thinking is off, or the model doesn't emit thinking blocks. It's a Qwen3 feature; don't expect it from every checkpoint.