JLC CaptionForge Joy Caption

JoyCaption is the LoRA crowd's default captioner for a reason: it's the uncensored vision-language model built for training captions, and it's been the community favorite for years. This node is that model wrapped for ComfyUI, and inside CaptionForge it plays the role of the first "witness" - the Pass A caption engine whose account gets weighed against the others later. It's also a perfectly good standalone JoyCaption if you never touch the rest of the pack.

Here's the thing you should know first: CaptionForge is a brand-new, still-labeled-v0.1.x project (2026, MIT, by J. L. Córdova), so googling "CaptionForge" will mostly turn up an unrelated folder-tracking tool with the same name. Make sure you're on Damkohler/CaptionForge before you clone.

How it works

Under the hood it's a Python/Hugging Face engine running the JoyCaption/LLaVA-family model. Models are expected in ComfyUI/models/LLM/JLC_JoyCaption/, and if the folder's empty the node will download the weights from Hugging Face on first use - that's a multi-GB pull, not a typo on your part. The download_probe_only toggle, sitting at the very bottom of the widget list by design, fetches just the lightweight metadata and returns a status message instead of captioning. Use it to check your wiring before committing to a full download.

The node has two personalities. With nothing connected it's a standalone captioner. Plug a pipeline_plan in and it switches into Pass A evidence mode: the Pipeline Planner takes over image routing, per-run seeds, sampling schedules, and shared output paths, and your caption gets appended to the run's JSONL evidence file. Most of the sampling widgets stop mattering in that mode - the planner overrides them.

Inputs and outputs that matter

The few you'll actually touch:

• model - defaults to llama-joycaption-beta-one-hf-llava. There's a model registry, so the dropdown may grow as models get added.
• memory_mode - Balanced (8-bit) is the default and the author's recommendation for 16 GB VRAM. It's bitsandbytes load-time quantization. If you can't load the 8-bit version, Default is your fallback.
• caption_type / caption_length - the structured template path. Joy has thirteen caption styles (Descriptive, MidJourney, Danbooru tag list, Art Critic…) and a length target from "very short" to explicit token counts.
• custom_prompt_mode - overrides the template path when enabled; custom_prompt wins over prompt_preset if non-empty.
• forbidden_phrases / replace_pairs - a cheap cleanup filter. One phrase per line to drop captions containing it, or old=>new lines for replacements.

Outputs: caption (the text you actually want), resolved_prompt (what was actually sent to the model - handy for debugging), plus passthroughs image_out, pipeline_plan_out, and template_options_out so you can chain nodes cleanly. Wire caption into a text display or into the capstone's JSONL feed.

Install

One install serves the whole pack. ComfyUI Manager, search "CaptionForge", or:

git clone https://github.com/Damkohler/CaptionForge.git ComfyUI/custom_nodes/CaptionForge

Restart ComfyUI, then make sure the Python deps are in your ComfyUI environment:

cd ComfyUI/custom_nodes/CaptionForge
pip install -e .

The repo ships no weights - Joy downloads itself into ComfyUI/models/LLM/JLC_JoyCaption/. Optional 8-bit support needs pip install bitsandbytes.

Common issues

First run downloads a big model, so give it time or use download_probe_only to test the graph first. On 16 GB systems keep memory_mode on Balanced (8-bit). And if you're running the full pipeline, know that Joy and Qwen are process-local Python models while the distillation stages live in Ollama - CaptionForge clears the Python models before handing work to Ollama, so don't be alarmed if the next image feels like a fresh load. That eviction dance is the pack trying to fit two model ecosystems in one GPU.