Auto Caption 2

Auto Caption 2 is the node from this pack you'll actually end up using. It's the Joy Caption alpha two pipeline with the rough edges filed off: instead of one image in / one caption out, it takes an image batch and returns a list of captions - one per image - and it hands you actual control over the style of caption you get. That's the difference between "write a descriptive caption" and "write a Booru-style tag list for training," which for LoRA work is the whole ballgame.

Captioning quality is the single biggest lever in training, and this is the current community-favorite approach for LLM-encoder models like Flux: natural-language captions from an uncensored VLM, in bulk. This node is that, in ComfyUI form.

How it works

Same recipe as the alpha version - SigLIP vision encoder, an 8B Llama text model, and a small image adapter LoRA - but built around Joy Caption alpha 2's prompt system. The loader is "Joy caption 2 model loader" (class Joy_Model2_load), which offers the Instruct Llama or the uncensored Llama-3.1-8B-Lexi-Uncensored-V2, an nf4/bf16 dtype choice, and a device picker.

At run time, Auto Caption 2 assembles an instruction from your caption_type, caption_length, optional extra options, and user_prompt, then streams the image through the model to generate text. The image input is a batch, and the output is a list of strings - feed it from the pack's "Load Many Images" node (which, unlike the old Load Image dir, sorts numerically so 10.jpg doesn't load before 2.jpg) and you can caption a whole folder in one pass.

The inputs that matter

• caption_type - nine styles: Descriptive, Descriptive (Informal), Training Prompt, MidJourney, Booru tag list, Booru-like tag list, Art Critic, Product Listing, Social Media Post. For training you'll live in Training Prompt or Booru tag list; the rest are fun toys.
• caption_length - any, very short through very long, or a numeric word count (20–260). This gets turned into "within N words" instructions. Big win for keeping your captions consistent.
• user_prompt - free-form extra instruction, appended to whatever the type/length system builds.
• ExtraOptionsSet (optional input) - wire the Extra Options Set node's string in here to add those flavor toggles.
• top_p / temperature / max_new_tokens - sampling knobs; defaults of 0.8 / 0.6 / 1024 are sane starting points.
• device - picks which GPU to run on (or cpu, if you're masochistic or CUDA is invisible).

Install and models

cd ComfyUI/custom_nodes
git clone https://github.com/Cyber-BlackCat/ComfyUI_Auto_Caption

or via ComfyUI Manager (search "ComfyUI_Auto_Caption"), then restart. Install deps - on Windows double-click install_req.bat, otherwise pip install -r requirements.txt (needs transformers>=4.44.2, bitsandbytes, and on Windows bitsandbytes-windows>=0.37.5).

Model placement is where people get burned. The loader auto-downloads the SigLIP encoder and the LLM, but the adapter is manual download only - README is explicit about it. You need:

models/clip_vision/siglip-so400m-patch14-384/
models/LLM/Meta-Llama-3.1-8B-Instruct/          # or Llama-3.1-8B-Lexi-Uncensored-V2
models/loras-LLM/cgrkzexw-599808/image_adapter.pt   # manual!

Chinese readers get faster mirrors on ModelScope (the README links them). The adapter file itself lives in the John6666 joy-caption-alpha-two HF repo.

Troubleshooting

• Missing image_adapter.pt is the #1 failure - no cgrkzexw-599808 folder, no captions. Download it by hand and put it exactly there.
• VRAM - you're loading an 8B LLM. The code clears VRAM when free memory drops below ~10 GB for nf4 or ~20 GB for bf16. On 16 GB+, use nf4 and you're fine; bf16 is for the 24 GB crowd. cache off unloads the model after each run (slow but frees memory); cache on keeps it resident (fast, but it'll be eating VRAM alongside your diffusion model).
• The device dropdown only shows cpu - that means ComfyUI can't see your GPU; check the driver/CUDA setup before blaming the node.
• Old transformers - the README warns the version can't be too low; keep it at or above 4.44.2.

One thing to note: this pack is genuinely "alpha" software with a bilingual README and rough edges - but the Joy Caption alpha two model it wraps is well-proven. If the caption node errors, it's almost always the model paths, not the model.