Download and Load JoyCaption Model

This node doesn't caption anything. It's the unglamorous prerequisite that hands a ready-to-run vision-language model to the actual JoyCaption and JoyCaption (Custom) nodes - and it's where almost every first-run problem happens, because on first use it quietly downloads roughly 16GB. That download is not the node being broken. It's just an 8-billion-parameter model moving from HuggingFace to your drive.

Why bother? Because JoyCaption is the community's default auto-captioner for training datasets. The KB's lora-training essay calls it the uncensored VLM built specifically for training captions, and it's been the favorite for years - for natural-language models like Flux and its descendants you want sentence captions, not tag soup, and JoyCaption writes sentences. What's easy to miss is that this pack is first-party: it's by fpgaminer, the person who actually built JoyCaption. This isn't a random wrapper, it's the official one.

How it works

The model input defaults to fancyfeast/llama-joycaption-beta-one-hf-llava - JoyCaption Beta One, a LLaVA-style model (Llama-3.1-8B + a SigLIP vision tower). On first run the node snapshots the whole repo into ComfyUI/models/LLavacheckpoints/ - note the odd capitalization, that's the standard ComfyUI LLaVA folder, not a typo and not where you'd expect it - then loads it with transformers. It is not a GGUF or llama.cpp thing, so don't go looking for a .gguf file to drop somewhere.

The inputs that matter

• model - a HuggingFace repo ID or a local path. The default is fine. Because it's a plain string, you can point it at a newer LLaVA-format JoyCaption checkpoint or a folder you already downloaded; only the default is guaranteed tested.
• memory_mode - Default (full bf16, needs ~16GB VRAM), Balanced (8-bit), or Maximum Savings (4-bit). Quantization trades quality for VRAM, and it genuinely degrades captions - which matters, because a bad caption poisons the LoRA you're training. If you have the VRAM, use Default. That's the take.
• keep_loaded - keep the model resident between runs. This matters most in the quantized modes: without it, an 8-bit/4-bit model is unloaded after every run and rebuilt from scratch next time, which makes batch captioning painfully slow.

Wiring it up

The single output, joycaption_model (type JOYCAPMODEL), plugs only into the other two nodes in this pack. There's nothing else to do with it. Loader → JoyCaption → save the caption text.

Install and the one real trap

Install via ComfyUI Manager (search "JoyCaption Nodes") or:

cd ComfyUI/custom_nodes
git clone https://github.com/fpgaminer/joycaption_comfyui

then restart ComfyUI. Dependencies auto-install from the pack's own manifest (transformers, torch, huggingface-hub, accelerate, bitsandbytes).

The trap: the quantized modes need a recent bitsandbytes, and the stock version often isn't one. The classic error is Using bitsandbytes 8-bit quantization requires the latest version of bitsandbytes. The fix is exactly what it says - pip install -U bitsandbytes - and this bites hardest on Mac and Windows where bitsandbytes support is flaky to begin with. Default mode avoids bitsandbytes entirely, another reason to just run Default when you can.

Two more things to expect: the first run looks frozen while it pulls 16GB (it isn't), and the download lands in that LLavacheckpoints subfolder, which confuses everyone once. That's the whole node, honestly. Get it loaded once and it stops being the thing you think about.