JoyCaption(NYJY)

If you've trained a LoRA, you know the misery: thousands of images, each needing a caption, and the quality of your captions directly caps the quality of your model. This node puts the joy-caption captioner - the one the training community adopted as the natural-language gold standard in 2024 - inside ComfyUI, running locally. Feed it an image and it writes a full descriptive sentence, the kind of caption that teaches a model composition and style rather than just "1girl". It's the prose half of this pack's captioning pair; JoyTag is the tag-list half, and for LoRA training you'll often run both.

How it works

This is the pre-alpha joy-caption stack, genuinely local: a SigLIP vision encoder (google/siglip-so400m-patch14-384) reads the image, a small learned ImageAdapter bridges its features into a Meta-Llama-3.1-8B language model (the 4-bit bnb build), and the Llama generates the caption token by token. Both models plus the adapter weights download automatically on first run. The author tested it on a 4090 at roughly 9–10 GB VRAM, so this is a heavy node - you're loading an 8B LLM, and you should only reach for it when you actually need that prose quality.

The inputs that matter

• image - the IMAGE to caption. First frame of a batch gets analyzed.
• prompt - the instruction given to the Llama. Default is A descriptive caption for this image:, and honestly that's right for most purposes - the joy-caption recipe is built around it. Change it only if you want a specific style of caption.
• model - fixed at Meta-Llama-3.1-8B-bnb-4bit; there's only the one choice.
• max_new_tokens - caption length cap, default 300 (10–1000).
• top_k (default 10) and temperature (default 0.5) - sampling knobs; the defaults are sensible, leave them alone unless you know what you're doing.
• clear_cache - the VRAM escape hatch. Off by default, meaning the models stay loaded between runs (fast). Turn it on if you're VRAM-starved: it unloads everything after each caption, trading a long reload on every image for fitting in less memory.
• newbie - the other escape hatch, and a good one: switch this on and the node stops using local models entirely and calls the online joy-caption space instead. All the other parameters except image become irrelevant. It's literally a "try this before you commit to the download" mode.

Output: captions, a STRING.

Installing

Standard pack install: ComfyUI Manager → Install via Git URL → https://github.com/aidenli/ComfyUI_NYJY, restart. Then the heavy part: first run auto-downloads SigLIP (into models/clip_vision/siglip-so400m-patch14-384) and the Llama (into models/llm/Meta-Llama-3.1-8B-bnb-4bit). The Llama is several gigabytes and the 4-bit build needs bitsandbytes installed - both come via the pack's requirements.txt, but on a fresh install that first download is a wait. The author's docs also note you must download the whole model directory, not just the safetensors.

Common issues & honest caveats

First run is slow and heavy. Several GB of downloads plus ~10 GB VRAM. If that's a wall, newbie mode gets you the same captioner without any of it. The pack's newbie is a shared online space too, so same reliability caveats as the online variants - quotas and busy servers.

JoyCaption (the pre-alpha line) is old enough now that newer captioners exist and are often better for specific jobs. But for the classic job - rich, descriptive prose for a training set - it's still a workhorse, and having it run locally means no quotas, no uploads, and no per-image fees. If you're captioning hundreds of images for a LoRA, that local cost is exactly the trade you want.