LayerUtility: Load JoyCaption2 Model(Advance)

Before Beta One, there was "JoyCaption-alpha-two" - this is that model, sometimes called JoyCaption2 in this pack to distinguish it from the newer Beta One line. Under the hood it's actually a small stack of models working together: a Llama-based LLM (either an uncensored Lexi fine-tune or a stock Meta Instruct model), a SigLIP vision encoder, and a set of adapter weights (cgrkzexw-599808) that tie them together into the John6666 joy-caption-alpha-two-cli-mod implementation this node is based on. This node loads that whole stack; the actual captioning happens in the paired LayerUtility: JoyCaption2 node, and there's also a split-inference variant elsewhere in the pack for sharing the loaded model across multiple captioning calls.

llm_model picks the underlying language model - either Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2 (the uncensored fine-tune, and the one that matches JoyCaption's whole design intent) or unsloth/Meta-Llama-3.1-8B-Instruct (the stock instruct model, if you specifically want more conservative output). device is cuda only for this generation - no CPU option here, so you need a GPU to run it at all. dtype picks precision: nf4 (4-bit quantized, the lighter option) or bf16 (full precision, better quality, more VRAM). vlm_lora is a bit of a trap if you don't read it closely - despite the name suggesting a vision LoRA, it actually controls whether the text model gets loaded (text_model versus none); the underlying architecture uses a LoRA-adapted text decoder, and this toggle is what turns that half of the pipeline on. Output is joy2_model, feeding into the JoyCaption2 captioning node.

Install the pack via ComfyUI Manager (search "ComfyUI Layer Style Advance") or git clone https://github.com/chflame163/ComfyUI_LayerStyle_Advance into custom_nodes/, restart, run the requirements installer. This is genuinely the heaviest model download in the pack's captioning lineup, because it's three separate downloads that all need to land in the right places: the LLM (either Orenguteng's Lexi-Uncensored or unsloth's Instruct build) goes in ComfyUI/models/LLM; the SigLIP vision encoder (google/siglip-so400m-patch14-384) goes in ComfyUI/models/clip; and the cgrkzexw-599808 adapter folder goes in ComfyUI/models/Joy_caption. All three are available from HuggingFace directly or bundled via the author's Baidu links if you'd rather pull from one source.

Because this predates Beta One's cleaner packaging, it's more prone to a specific class of error: UnboundLocalError: local variable 'clip_processor' referenced before assignment or the same for text_model. If you hit that with all three model pieces confirmed present in the right folders, it's not a missing file - it's your transformers version, and it needs to be at least 4.43.2. If updating that then surfaces a protobuf complaint about descriptors needing regeneration (common on transformers 4.45.0+), downgrade protobuf to 3.20.3 or set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python. Given how many moving pieces this loader has, it's worth using this over the newer Beta One loader only if you specifically need JoyCaption2's particular caption style - for most new setups, Beta One is the simpler, more actively-referenced choice.