Joy_caption_load

Joy_caption_load is the boring half of a two-node team, and it's better that way. It doesn't caption anything - it builds the JoyCaption pre-alpha pipeline (SigLIP vision encoder, a Llama 3.1 8B text model, and the image adapter that connects them) and hands it to Joy_caption as a JoyPipeline. You only need to care about it when it's slow to load or eating your VRAM, which is most of the time, honestly.

What it actually does

Pick a model and the node downloads everything it needs into ComfyUI/models/:

• clip/siglip-so400m-patch14-384 - Google's SigLIP vision encoder.
• LLM/<chosen model> - the language model that writes the caption.
• Joy_caption/image_adapter.pt - the MLP that projects image features into the LLM's embedding space. This one you download by hand from the fancyfeast pre-alpha space (wpkklhc6); the other two auto-download on first run.

The model dropdown has two choices. Use unsloth/Meta-Llama-3.1-8B-bnb-4bit. It's the default, it's 4-bit quantized so it fits in around 8–10 GB of VRAM, and - unlike meta-llama/Meta-Llama-3.1-8B - it isn't gated behind Hugging Face's Llama license approval form. If you picked the meta option and get a 401 from the Hub, that's why. Accept the license and use an access token, or just switch to the unsloth build.

What's on the node

One input, one output:

• Input: model - the two Llama 3.1 8B choices above.
• Output: JoyPipeline - wire this into Joy_caption's joy_pipeline input.

Nothing else. There's no VRAM toggle here; you control memory on the run node with its cache flag (keep it False unless you want the pipeline to stay resident).

Installing

Same pack install as everything else - ComfyUI Manager (search "Comfyui_CXH_joy_caption") or clone and pip:

cd ComfyUI/custom_nodes
git clone https://github.com/StartHua/Comfyui_CXH_joy_caption
cd Comfyui_CXH_joy_caption
python -m pip install -r requirements.txt

Then restart. The requirements.txt pins transformers>=4.44.2, and the README warns the same thing in Chinese: keep transformers recent or the model load breaks.

Where people get burned

• First run downloads ~10 GB (SigLIP + 4-bit Llama). It looks hung while snapshot_download runs; it isn't. Give it time.
• Missing image_adapter.pt is the number-one error - the node doesn't fetch it automatically, and without it Joy_caption fails at inference. Get it from https://huggingface.co/spaces/fancyfeast/joy-caption-pre-alpha/tree/main/wpkklhc6 and put it at models/Joy_caption/image_adapter.pt.
• Switching model mid-session triggers a full teardown and reload, because the loader keeps one pipeline alive and rebuilds when the choice changes. Fine, just slow.
• The loader only works on NVIDIA - the code calls .to("cuda") directly, so AMD/Apple users are out of luck without editing the source.