Image to Prompt (TJ)

Image to Prompt (TJ) runs a local vision LLM over an image and hands you back a prompt - plain caption, SD/Booru tags, pose analysis, a quality check, or whatever custom instruction you give it. The name is a bit of a lie in the best way: there's no API, no key, no cloud call. It loads a GGUF vision model into your own GPU (or CPU), so it works offline and costs nothing per image. If you've ever wanted to caption a dataset for LoRA training, or reverse-engineer a prompt from a reference image, this is the node for it.

How it works

It supports two backends, selectable in model_backend:

• GGUF / llama.cpp (default) - needs two files in ComfyUI/models/text_encoders/: the model GGUF plus a vision projector mmproj-*.gguf. The default it looks for is a Qwen3.5 4B build, but any compatible GGUF works (the README's recommended lineup is Qwen2.5-VL 7B/3B, MiniCPM-V, LLaVA). Vision models need both files - the mmproj is what lets the model see images at all.
• ComfyUI TextGenerate - the alternative backend, using a gemma4_e4b_it_fp8_scaled.safetensors text encoder; clip_loader_type picks the right loader type (the "Auto" option infers from filename).

Then you choose what you want from the image via vision_task: Caption (plain description), Caption + Format (applies model_format below), SD/Booru Tags, Pose & Anatomy Focus, Content Quality Check, or Custom Instruction (write your own in custom_instruction). model_format shapes the output for a target model family - KREA2 prompt enhance, Flux natural-language prose, SDXL tags+weights, Pony/Illustrious booru tags, video-motion prose for LTX/Wan, and more. There's even an aesthetic dropdown to inject a style direction into the caption.

Sampling knobs (n_gpu_layers, n_ctx, max_tokens, temperature, seed) work like any local LLM. lock_in is the caching switch: lock it and the node stops re-analyzing on every run, holding the cached prompt - essential when you're using it as a fixed caption in a training loop instead of live.

Outputs: image_prompt (the STRING you feed into a prompt/text node) and thought_process (the model's reasoning - genuinely useful for debugging why it wrote what it wrote).

Inputs and outputs that matter

image (required), model_backend, gguf_model + mmproj_file (for GGUF), vision_task, model_format, lock_in. Output: image_prompt is the one that matters; thought_process is the debug channel.

Install

ComfyUI Manager → Install Custom Nodes → search TJ_NODE, or:

cd ComfyUI/custom_nodes
git clone https://github.com/designloves2/ComfyUI-TJ_NODE

Restart ComfyUI. This is the heavy one. The GGUF backend needs llama-cpp-python, which the pack's requirements install - but the plain pip build is CPU-only, which is slow and won't run the newest vision models well. For GPU acceleration you need a source build (CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir, or the pack's LLAMA_GPU_SETUP.md / build_llama_gemma4.bat on Windows). Then drop a vision GGUF + its mmproj into models/text_encoders/. Plan for a ~5–6GB download for the recommended Qwen2.5-VL 7B Q4_K_M pair.

The honest verdict

This is the node to reach for when you're building a dataset - the captioning step in LoRA training is exactly this job, and doing it locally means your 10,000-image set doesn't cost API credits. The wrinkle is setup: llama-cpp-python on CPU is a bad time, so budget the GPU build. And remember vision models need the mmproj file; load the base GGUF alone and the dropdown shows NO_VISION_HANDLERS_AVAILABLE for a reason. Get past that and it's a genuinely strong captioner with the format presets doing the heavy lifting of matching your target model's prompt style.