JLC CaptionForge Orchestrator

CaptionForge's core bet is that one captioner is easy to fool, so its main node is the part that doesn't look at the image first. JLC CaptionForge Node - the capstone - takes the raw "witness" captions your Joy/Qwen/Ollama nodes produced in Pass A, distills them with a text-only LLM, checks the result against the actual image with a vision model, and then writes your final captions. It's the orchestration hub of the pack, and it's the slowest part of it.

The design intent is worth understanding before you run it: the semantic work mostly ends at the validated long caption. The short and taggy outputs are deliberately lightweight formatting steps derived from that validated paragraph, not fresh reinterpretations of the image. The formatter prompt even says, in effect, "do not rewrite the validated paragraph." So if you want better captions, tune the distiller and validator, not the formatter.

How it works

Three Ollama-backed stages, in order:

1. Fat Draft (Pass B) - a text-only LLM (default mistral-small:24b) merges all the witness captions into one deliberately over-complete draft. It's told not to validate against the image, not to summarize aggressively, and to preserve concrete details. That's why it's called fat.
2. Validator (Pass C) - an image-aware VLM (default gemma4:26b) looks at the actual image and returns a corrected paragraph keeping only image-supported details. This is where hallucinations get cut.
3. Formatter (Pass D) - a text-only LLM derives the comma-separated taggy caption from the validated paragraph.

Then it exports: _long.txt, _short.txt, and _taggy.txt sidecars written beside each source image, plus a JSONL audit trail (_B_DISTILL.jsonl, _C_VLM_VALIDATED.jsonl, _D_FINAL_EXPORT.jsonl, run configs) under your output root. If you ever need to defend a caption to a client or debug why a training set learned something weird, that trail is the selling point.

Inputs that matter

• Input - captions JSONL - the Pass A raw captions your caption nodes appended to. This is the node's primary fuel.
• Input - image path - the folder/file root the validator uses to resolve source images. It has to be findable, or Pass C can't do its job.
• Input - include caption families - joy,qwen,ollama by default; use all or * for everything.
• LoRA - trigger word and LoRA - user caption anchor - your trigger token and any identity/style anchor passed through to distiller and validator.
• Fat Draft / Validator / Formatter - model - dropdowns loaded from config/captionforge_ollama_models.json; pick Custom to type any installed Ollama tag.
• Final - TXT export format - natural, taggy, or both_separate.
• Ollama - URL - defaults to http://127.0.0.1:11434. Nothing happens until that server is up.

Outputs: natural_captions, taggy_captions, final_jsonl_records, output_paths_json, and status. The JSONL/paths outputs are the audit trail in string form if you want to inspect or re-export.

Install

Same as every node in this pack - clone and restart:

git clone https://github.com/Damkohler/CaptionForge.git ComfyUI/custom_nodes/CaptionForge

Then the real requirement: a working local Ollama with the stage models pulled.

ollama pull mistral-small:24b
ollama pull gemma4:26b

The pack ships no weights and pulls nothing for this node by itself - it's pure Ollama calls.

Common issues

If you see connection errors, Ollama isn't running or the URL is wrong - check ollama serve. It's slow by design: three model passes per image, and a 24B Ollama model can churn for a while on a 16 GB card (the author's own dev box is an RTX 4090 Laptop). The request timeout default is a generous 1800 seconds because large models may still be pulling on first call. And remember: this node clears the resident Joy/Qwen Python models before it calls Ollama, so the GPU handoff can feel like a reload. The fat draft is supposed to contain conflicting details - the validator is the one that trims them, so don't panic when Pass B looks overstuffed.