Joy Model load

"Joy Model load" is the model loader half of the Joy Caption alpha pipeline bundled in ComfyUI_Auto_Caption. If you're captioning a dataset for a Flux or SDXL LoRA - the KB still calls JoyCaption the community favorite for that job - this node is what hauls the heavy machinery into VRAM so the pack's "Auto Caption" node can write natural-language captions instead of tag soup. It's not a captioner itself. It's the shotgun you load before firing.

What it actually loads

The node assembles the full Joy Caption alpha stack in one go, straight out of the original fancyfeast joy-caption-alpha-two space:

• SigLIP CLIP vision (google/siglip-so400m-patch14-384) - reads your image. This one is hardcoded, no choice.
• Meta-Llama-3.1-8B - the LLM that writes the caption. This is the choice you get.
• The image adapter - a small two-layer MLP that maps SigLIP's vision features into Llama's embedding space. It lives in models/loras-LLM/wpkklhc6/image_adapter.pt, and the README is blunt that this one is manual download only.

Under the hood it's simple and a little rough: loadCheckPoint() downloads whatever's missing with snapshot_download, then the caption node concatenates BOS token + image embeddings + your prompt text and lets Llama generate. The loader caches the pipeline on the node, so the 8B model loads once and stays hot - switching the model dropdown triggers a full reload, which is the "go make tea" moment.

The one input that matters

There's exactly one required input, an enum:

• unsloth/Meta-Llama-3.1-8B-bnb-4bit - the quantized option. ~5-6 GB of weights. This is the sensible default on 8-12 GB cards.
• meta-llama/Meta-Llama-3.1-8B - full fp16, ~16 GB of weights alone. Only if you've got headroom.

Two gotchas hide in that dropdown. First, the full model is gated on Hugging Face - you must accept Meta's license and run huggingface-cli login or the download 401s. The unsloth bnb-4bit mirror is public, which is half the reason it's the better default. Second, the bnb-4bit choice needs bitsandbytes (or bitsandbytes-windows on Windows) or the load dies mid-way.

The single output is JoyModel - a custom type that only wires into this pack's Auto Caption node's JoyModel input. Don't expect it to plug into anything else.

Installing

ComfyUI Manager, search "ComfyUI_Auto_Caption". Or:

cd ComfyUI/custom_nodes
git clone https://github.com/Cyber-BlackCat/ComfyUI_Auto_Caption

Then restart ComfyUI. On Windows, run install_req.bat; otherwise pip install -r requirements.txt (needs transformers>=4.44.2, bitsandbytes, huggingface_hub, accelerate, sentencepiece, opencv-python). Installing newer transformers into ComfyUI's env is the classic trap - it can break nodes pinned to an older version, so keep a backup venv handy.

Models auto-download into ComfyUI/models/ (clip_vision/, LLM/, loras-LLM/) on first run. The LLM download is the multi-gigabyte part; grab the wpkklhc6 LoRA manually from the fancyfeast pre-alpha space and drop it at models/loras-LLM/wpkklhc6 or the node fails on a missing image_adapter.pt. If Hugging Face is unreachable, the README lists ModelScope mirrors.

Troubleshooting

• OOM on load: you picked full 8B with 12 GB of VRAM. Switch to the bnb-4bit option.
• Download fails on the meta-llama model: gated repo, no token. Login and accept the license.
• bitsandbytes errors: the Windows wheel is a separate package; install bitsandbytes-windows.
• Missing image_adapter.pt: that LoRA is never auto-fetched - the README and the source agree it's manual only.

The name overpromises slightly - it only loads the alpha stack; Joy Caption 2 has its own loader in this pack (Joy_Model2_load). For the beginner captioning a Flux dataset, this node plus Auto Caption is the whole recipe: load, caption, train.