joy model loader

Every workflow in this pack starts here, and it's the node that decides how much VRAM you need. "joy model loader" doesn't caption anything itself - it assembles the three pieces of the Joy Caption model into a JoyPipeline object and hands it to the pack's two captioning nodes. But it's also where the 6GB downloads happen, where the one manual setup step lives, and where you make the only real choice this pack asks you to make: which Llama backbone to run.

Why it exists

Joy Caption is a vision-language model: a SigLIP vision encoder turns your image into features, a small adapter squeezes those into Llama 3.1 8B's embedding space, and Llama writes the caption. It's the community's favorite natural-language captioner for LoRA training on Flux-class models - the KB tracks it as the top pick for training captions, still heavily used years after it shipped. That power costs VRAM, and this loader is where the cost is set in stone.

How it works

The loader pulls three components on first run. SigLIP (google/siglip-so400m-patch14-384) auto-downloads to models/clip/. Your chosen Llama auto-downloads to models/LLM/ - note the capital folder, which surprises people who went looking in the usual spots. The third piece, models/Joy_caption/image_adapter.pt, is not auto-downloaded - you must place it manually or the loader raises a FileNotFoundError. That's the single most common setup failure in the whole pack.

The one input that matters

Just one: model, a dropdown with two choices.

• unsloth/Meta-Llama-3.1-8B-bnb-4bit - the one to pick. ~5GB download, runs comfortably in ~11GB VRAM thanks to bitsandbytes NF4 quantization, and the loader caps GPU memory at 10GiB, which this build fits neatly.
• meta-llama/Meta-Llama-3.1-8B - the full fp16 model. Requires approval on Hugging Face, wants a much bigger GPU, and because the loader still applies that 10GiB memory cap, a big chunk spills onto the CPU and the whole thing crawls. There's a case for it if you have a 24GB+ card and want max caption quality, but honestly the 4-bit build gets you 95% of the way there at a fifth of the footprint.

The output is a single JoyPipeline - wire it into "joy image caption" or "joy diretory caption".

Install

cd ComfyUI/custom_nodes
git clone https://github.com/YMC-GitHub/ymc_node_joy

Restart and let deps auto-install, or use ComfyUI Manager (search "ymc_node_joy"). The registry install also works: comfy node registry-install ymc_node_joy.

What actually gets you

The first run isn't hung - it's downloading gigabytes, and SigLIP plus the 4-bit Llama is roughly 6GB over whatever connection you have. The 4-bit path needs bitsandbytes in your environment; ComfyUI Desktop ships it on Windows, but on a hand-rolled Linux setup you may need pip install bitsandbytes first. And once everything's loaded, the loader's IS_CHANGED is deliberately NaN, so it re-runs every time - that's what makes the caption nodes' cache flag matter: with cache off, every caption reloads the pipeline, which is exactly what the directory node is built to avoid.