Simple LLM Caption Loader

The node everything else plugs into

Every other node in this pack - SimpleLLMCaption, its Advanced and Batch variants - needs a pipeline input, and exactly one node produces it: SimpleLLMCaptionLoader. You place it once, pick an LLM, and it loads the whole Joy Caption stack: the vision model, the adapters, the language model. Its LLM_PIPELINE output then feeds every captioning node in your graph.

What makes this loader unusual is that you never place a single model file. On first use it downloads everything itself. That's the pack's whole pitch, and it's why the setup section is basically "clone, install requirements, restart."

Inputs - there are two

• llm_model - a dropdown with two AUTO-DOWNLOAD choices: Llama-3.1-8B-Lexi-Uncensored-V2-nf4 (the recommended default) and Meta-Llama-3.1-8B-Instruct-bnb-4bit. Both are Llama-family, because Joy Caption is a Llama pipeline - the dropdown deliberately filters to llama/lexi/meta-llama names. The "Lexi" one is the uncensored variant, which is how the pack advertises NSFW captioning without the model refusing to cooperate.
• use_4bit - a toggle, on by default. Here's the honest bit: in the current source, that toggle doesn't actually quantize anything. Both bundled repos are already 4-bit quantized (NF4 and bnb-4bit respectively), so there's nothing left for it to do. Don't go hunting for a memory difference - the quantization is baked into which model you pick.

Output: a single LLM_PIPELINE, the shared custom type used by the other three nodes.

What actually happens on first run

The loader downloads three things into ComfyUI/models: the chosen LLM into models/LLM, SigLIP (siglip-so400m-patch14-384) into models/clip, and the Joy Caption adapter kit into models/Joy_caption - that last one being image_adapter.pt (86MB), clip_model.pt (1.7GB), and a text_model folder holding the 671MB LoRA adapter plus its tokenizer.

Then the interesting part: the loader loads the tokenizer from text_model, rewrites text_model/adapter_config.json to point the adapter's base model at your local LLM, and loads the LLM with the LoRA already attached. That's how captions come out clean, with no "Here is a caption:" post-processing anywhere in the chain.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/Pixelailabs/Joy_Caption_Two_PixelaiLabs.git
cd Joy_Caption_Two_PixelaiLabs
pip install -r requirements.txt

Restart ComfyUI and the loader appears under image/captioning. ComfyUI Manager (search "Joy Caption Two") does the same without the terminal. The requirements list is modest - transformers, torch, accelerate, bitsandbytes, sentencepiece, protobuf, huggingface_hub - but sentencepiece and protobuf matter for the Llama tokenizer, and bitsandbytes is required for the 4-bit models.

Gotchas worth knowing

• Budget 6-8GB of disk and a patient first run. The loader prints "First-time setup: Downloading..." and if a download fails it prints the HuggingFace repo so you can fetch it manually and drop it in models/LLM.
• The pipeline is cached by the loader node, so it survives across runs. But models move to GPU on demand and back to CPU after captioning - that's the VRAM management the README brags about, and it's what lets an 8GB card run this at all.
• Custom model folders: this pack writes to models/Joy_caption and models/LLM, neither of which is a standard ComfyUI folder. If you've routed models to an external drive via extra_model_paths, these won't follow - people with portable or NAS-symlinked setups hit exactly this, and it's a known pain point in the community.
• Only one LLM loads at a time, and switching models in the dropdown clears and reloads. Give it a minute when you swap.

The author, by the way, is a YouTube tutorial creator (aiconomist / PixelaiLabs), so this pack reads like it was shaped by "make it work without touching a terminal" - which is mostly a good thing. The auto-download is the feature, but it's also the thing that makes the first run feel like it's hung. It isn't.