Auto Caption

This is the node that runs Joy Caption alpha - the uncensored vision-language model that became the community's default captioner for Flux and SDXL LoRA training. If you're here, you almost certainly have a folder of images you want natural-language captions for. Auto Caption does it, but it's the heavyweight way to do it: you're spinning up an 8B Llama model to write alt text. It works, it's been the community favorite for a reason, and it has quirks you should know about before you caption 200 images with it.

How it works

The captioning brain is three pieces: a SigLIP vision encoder (google/siglip-so400m-patch14-384), a Llama-3.1-8B text model, and a small "image adapter" LoRA that maps vision features into the LLM's embedding space. The loader node - "Joy Model load" - assembles all of that. At run time, Auto Caption encodes your image through SigLIP, pushes the features through the adapter (two linear layers with a GELU between them), concatenates the result with the embeddings of your text prompt, and runs the LLM's generate. Out comes one decoded caption string.

The setup is the heavy part, not the node itself. The wpkklhc6 adapter lives in models/loras-LLM/ and the README is emphatic that it's manual download only - the auto-downloader can't fetch it. The SigLIP and Llama weights it will pull from Hugging Face on first run, but manual is recommended there too; the 8B model is several gigabytes and you don't want it silently redownloading mid-workflow.

The inputs that matter

• JoyModel - from the "Joy Model load" node. That loader lets you pick between the 4-bit Llama (unsloth/Meta-Llama-3.1-8B-bnb-4bit) and full-precision meta-llama/Meta-Llama-3.1-8B. On a normal card, pick 4-bit. Full is a memory exercise.
• image - an IMAGE tensor. One image in, one caption out; there's no batch output here, which is the alpha version's biggest limitation.
• prompt - the instruction. Default is "A descriptive caption for this image"; this is your main knob for steering output.
• max_new_tokens (default 1024) and temperature (default 0.6) - caption length budget and randomness. Lower temperature for consistency, raise it if every caption reads identically.
• cache - the trap. Default false means the entire pipeline is unloaded from VRAM after every single caption. Caption 50 images and you reload an 8B model 50 times. Set it true to keep the model resident - faster batch runs, but it stays in VRAM next to your diffusion model.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/Cyber-BlackCat/ComfyUI_Auto_Caption

or use ComfyUI Manager (search "ComfyUI_Auto_Caption") and restart. Then install dependencies - on Windows, double-click install_req.bat in the pack folder (it pip-installs into ComfyUI's embedded Python). Manually that's:

pip install -r requirements.txt

The heavy hitters are transformers>=4.44.2, bitsandbytes (plus bitsandbytes-windows>=0.37.5 on Windows), accelerate, sentencepiece, opencv-python, and huggingface_hub==0.24.3.

Troubleshooting

• Missing image_adapter.pt - the classic. The adapter folder models/loras-LLM/wpkkklhc6 has to exist and contain the file; it will not download itself. Grab it from the fancyfeast Joy Caption pre-alpha HF space and drop it in.
• Transformer errors on load - the README warns the transformers version can't be too low; the pinned requirement is ≥4.44.2. If you run an older ComfyUI python env, update it.
• Windows import failures - you need the bitsandbytes-windows wheel, not the Linux one.
• OOM - you're loading an 8B LLM. With 4-bit on an 8 GB card it's tight but workable; free VRAM first (or use cache and expect slow reloads). On 16 GB+ this is a non-issue.

If your use case is batch captioning a full dataset, honestly look at Auto Caption 2 from this same pack first - it's the improved version with list output and real control. Come back to this one when you want the simplest single-caption pipeline.