✨📷 Groq VLM API

✨📷 Groq VLM API takes an actual IMAGE tensor - not a file path, the tensor - and sends it to a hosted vision-language model with your question attached. You get text back. That's it, and that's plenty: describe this image, what style is this, what's the dominant colour, is this person's left hand doing something weird.

The two jobs people actually use it for are captioning (building .txt sidecars for a LoRA dataset - same-basename pairs, one caption per image) and prompt recovery (generate a caption from an image you liked and feed it back into txt2img to get variations). Because the input is a tensor, the image can come from anywhere in the graph: a Load Image, a previous generation, a decoded latent, a whole batch.

Setup is the same as the other Groq nodes - a .env in the pack root:

cd ComfyUI/custom_nodes/ComfyUI-mnemic-nodes
cp .env.example .env
# edit .env -> GROQ_API_KEY=gsk_...

Key from console.groq.com/keys. No key, no node: it raises rather than failing silently.

Inputs

image is required. preset gives you the canned jobs - the README's examples are short caption, medium caption, long caption, primary colour, and a fully custom prompt. Custom prompt is the general-purpose one: system_message for standing instructions, user_input for the specific question, and if you mention USER in the system message it refers to that field. Then temperature (default 0.85 - high for captioning; 0.2-0.4 is more consistent), max_tokens, top_p, seed, max_retries, stop, and json_mode (needs the word JSON in your prompt, incompatible with stop).

Outputs are api_response, success, status_code - check success in any loop, or an error string becomes your caption.

Size limits, and why your run 400s

This bites everybody once. Groq's own restrictions, quoted in the pack README:

• A request carrying an image URL input is capped at 20MB.
• A request carrying a base64-encoded image is capped at 4MB, and over that you get a 413.

The node sends the image base64-encoded, so the 4MB number is the one that matters. A 4096×4096 PNG from a two-pass upscale blows straight through it and you get a 413 that looks like a ComfyUI error but isn't. Downscale before the VLM node - you're asking for a description, not a pixel-level review - or feed it the first-pass image instead of the upscaled one.

What it's good at, and what it isn't

For a dataset captioner this is the cheapest option you have: no download, no VRAM, free tier, good English. It's genuinely competitive with running a local VLM for a first pass, and vastly faster than hand-captioning 500 images.

But keep expectations calibrated, because every vision-language model has the same failure mode: multi-subject attribution. With two people in frame it mixes up who's wearing what and who's doing what. It's the one weakness that has held across two years and every model in this category - JoyCaption's own release notes say so about itself. So: auto-caption a large set, then hand-audit a small one. And don't trust a caption to reproduce the source image faithfully; a good-sounding caption often isn't specific enough to generate anything like the original back.

It's also not uncensored. Groq's endpoints enforce their providers' policies, and there's no local weights to patch around it. For NSFW dataset captioning, a local captioner is the way.

Install

ComfyUI Manager → ComfyUI-mnemic-nodes, or:

cd ComfyUI/custom_nodes
git clone https://github.com/MNeMoNiCuZ/ComfyUI-mnemic-nodes

Restart. groq and python-dotenv arrive with the pack requirements - no model weights involved at all, since inference happens on Groq's hardware. The pack is at v3.0.0 and written against ComfyUI's newer V3 node API, with automatic migration for workflows built on the older versions, so a stale workflow should still load rather than throwing missing-node errors.