JoyCaption Download And Load

This is the zero-setup way to get JoyCaption running. Drop the node in, pick a model from the dropdown, and it downloads the weights into models/LLavacheckpoints on first run, then hands you a ready-to-use model. No hunting for a HuggingFace mirror, no manual folder wrangling - the "And Load" is the whole point.

The model dropdown offers the three choices the pack ships with, defaulting to fancyfeast/llama-joycaption-beta-one-hf-llava - the current release, the Llama 3.1 8B + SigLIP2 LLaVA model that's become the community's default natural-language captioner. The other two are the older alpha-two build and its vqa-test-1 variant, which you almost certainly don't want unless you're deliberately testing the old lineage.

How it works

Mechanically it's simple: if the folder doesn't already exist in LLavacheckpoints, it runs snapshot_download from HuggingFace to pull the whole repo, then constructs a JoyCaptionPredictor around it. That's also the one real constraint: the dropdown only lists models published as full HuggingFace repos in the pack's config, so you can't point this node at an arbitrary path. If you've got a model file already or you want something not on the list, use the plain JoyCaption Loader instead - that one reads whatever directories are sitting in LLavacheckpoints.

The loader inputs worth knowing:

• memory_mode - Default (full precision), Balanced (8-bit), or Maximum Savings (4-bit, nf4 via bitsandbytes). The quantized modes save a chunk of VRAM but degrade caption quality; the README says so outright.
• precision_default - fp16 (default), bf16, or fp32. On modern cards bf16 is a fine drop-in if you want a touch more headroom.
• device - pick which device runs inference. The dropdown lists cpu and your CUDA cards; cpu is slow but genuinely works as a fallback when the diffusion model needs the VRAM.
• lora (optional) - plug in a JOYCAPTIONLORA from the JoyCaption Lora node to load a captioner fine-tune on top.

It outputs one thing: joycaption_model (type JOYCAPTIONMODEL), which wires into the joycaption_model input on the main JoyCaption or JoyCaption (Custom) node.

Install

Same pack as the rest: ComfyUI Manager → search wo_joycaption_comfyui, or

cd ComfyUI/custom_nodes
git clone https://github.com/without-ordinary/wo_joycaption_comfyui

then restart. Requirements include transformers, torch, torchvision, accelerate, bitsandbytes, and huggingface-hub - bitsandbytes is what the 8-bit/4-bit modes run on, so if those modes throw an import error, that dependency is the first suspect.

Common issues

The first run is a big download - an 8B model is several gigabytes - and it happens inside your first queue, so the node can look "stuck" for a while. Let it finish; the console shows progress. If it dies mid-download or you're offline, grab the model another way and put it in models/LLavacheckpoints yourself, then switch to the non-downloading loader.

One folder thing that trips people: LLavacheckpoints is a convention several different LLM/VLM node packs use, but not all of them - some want their own folder. If another captioner you have insists on a different path, that's fragmentation in the ecosystem, not you doing it wrong.