Joy caption 2 model loader

It's a loader with a job to do

Joy Caption is the community's favorite captioner for training data - the KB tracks 133 mentions in the first half of 2026 alone - and it's the standard answer for FLUX-class (LLM-encoder) models that want natural-language captions instead of a comma-farm of Danbooru tags. This node is the front door to the Joy Caption 2 half of Cyber-BlackCat's ComfyUI_Auto_Caption pack. It captions nothing itself: it loads the whole pipeline and hands it to the Auto Caption 2 node through a single custom-type output named JoyModel2.

Internalize this before you hit queue: it is not a lightweight tagger. It's a Llama 3.1 8B as the text brain, a SigLIP vision model to actually look at the image, and a PEFT LoRA that teaches the LLM to write captions. The first run downloads gigabytes and looks exactly like ComfyUI has frozen. It hasn't. Probably.

What it actually does

Three required inputs, and only two are worth your attention.

• llm_model - pick between unsloth/Meta-Llama-3.1-8B-Instruct and Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2. The second is the uncensored variant; reach for it if your training set isn't work-safe, because plain Instruct will politely censor whatever it's describing.
• dtype - nf4 or bf16. This decides whether you run at all. nf4 is 4-bit quantized through bitsandbytes (double-quant, bf16 compute, roughly half the memory). bf16 loads the model at full precision and wants something like 16GB+ of free VRAM. The node even auto-cleans your VRAM when free memory dips below 10GB on nf4 or 20GB on bf16 - that threshold is your hint about the real requirements.
• device - generated from your actual GPU count, so a normal single-GPU box just shows cuda:0, and cpu only appears if you have no CUDA device. Not worth stressing over.

At load time it checks models/clip_vision/siglip-so400m-patch14-384 and models/LLM/<model-name>, downloading either from Hugging Face if missing, then merges the Joy Caption 2 LoRA into the LLM and loads the image adapter. It caches the result, so re-runs are cheap - but switching llm_model wipes the cache and reloads the whole ~8GB thing.

Installing it (the README's three steps, decoded)

1. Install the pack: ComfyUI Manager (search "ComfyUI_Auto_Caption") or

   cd ComfyUI/custom_nodes
   git clone https://github.com/Cyber-BlackCat/ComfyUI_Auto_Caption
   

   then restart.
2. Install dependencies - the pack needs transformers>=4.44.2, bitsandbytes, sentencepiece, accelerate. On Windows there's a clickable install_req.bat; Windows also needs bitsandbytes-windows (the classic pain point for anything 4-bit).
3. Manually drop the Joy Caption 2 LoRA into models/loras-LLM/cgrkzexw-599808 from John6666/joy-caption-alpha-two-cli-mod. This one the pack will not download for you. The LLM and SigLIP it fetches itself, though the README recommends manual download too. And heads-up: the README's link for the Lexi model is a copy-paste typo - it points at the Instruct repo. The code uses Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2.

Where people get burned

• The first run "hang." It's downloading SigLIP plus an 8B LLM. Watch the console; the model loader prints progress.
• NoneType errors. In the wild, the #1 cause of JoyCaption-in-ComfyUI failures is the LoRA files sitting in the wrong folder - one r/comfyui poster downloaded 20GB of models, misplaced the LoRA, and got object of type 'NoneType' has no len(). If you see that, the path table above is your checklist.
• OOM. Switch dtype to nf4, or free VRAM before loading.
• It's an obscure pack from an individual author. This pack has essentially no community footprint, and a node that merges a downloaded PEFT LoRA and runs a local 8B model is a lot of trust to hand out. The usual rule applies: skim the source before running it. (ComfyUI's ecosystem history has a whole cautionary tale about an LLM-vision node with malicious code - enough said.)

Once it's loaded, wire the JoyModel2 output into Auto Caption 2's JoyModel2 input and you're captioning for your next LoRA.