AcademiaSD LLM Vision 👁️

The pack's local captioning pipeline has two halves: a loader that brings a vision-language model into memory, and this node - the one that actually looks at the image and writes the description. It's the workhorse of a fully offline dataset captioning workflow, and it's built to slot straight into the "captions for LoRA training" assembly line.

How it works

You connect a MODEL from the pack's AcademiaSD VLModel (Down)Loader (the ACADEMIA_MODEL type they share), hand it an image, and give it an instruction - the default is a solid generic "describe this image in detail, focus on lighting, composition, and subjects" - plus a max_tokens budget (default 512). It builds a chat-style prompt, runs the VLM under torch.no_grad() with a low temperature, and returns the generated text on the caption (STRING) output.

The interesting bit is the post-processing. The node tries to parse the model's output as JSON, and if it finds a compositional_deconstruction structure with bounding boxes, it rescales those boxes from the model's normalized 0–1000 coordinate space into real pixel coordinates using the optional width and height inputs (defaulting to 1000×1000 if you don't provide them), converting to standard x1,y1,x2,y2 format. So it's not just a captioner - it's a captioner that can hand you usable detection output for dataset work. If the JSON parsing fails for any reason, it returns the raw text unchanged rather than breaking.

The inputs that matter

• model - from the VLModel loader. This is a matched pair; don't try to feed it a checkpoint MODEL.
• instruction - what you want said about the image. This is where you steer the captioning style.
• max_tokens - caption length ceiling, 64–2048.
• image, external_prompt, width, height - optional, for context injection and bbox rescaling.

Install

Pack install as usual: ComfyUI Manager → "AcademiaSD", or git clone https://github.com/AcademiaSD/comfyui_AcademiaSD, restart. This node itself adds no dependencies, but the loader it depends on needs bitsandbytes + accelerate for its 4-bit mode (see that node's article - nothing in the pack auto-installs them).

Where it sits in your workflow

The intended chain: VLModel loader → LLM Vision → the pack's Save Dataset Caption (.txt) node, with Batch Image Loader feeding the image and path, and a counter driving the batch index. Compared to the cloud option (AcademiaSD_GeminiVision), this trades caption quality for privacy, cost, and reliability - no API key, no spend, works offline. A small local Qwen VLM won't match Gemini's scene understanding, so for the best captions use cloud; for a private, unattended, get-it-done dataset run, this is the dependable choice. One tip from the training playbook to keep in mind: what you leave out of the caption is as important as what's in it - that's what keeps the trainer from baking fixed attributes into your trigger word.