Nodes/ComfyUI-FeiFei/Image Captioner (API / Local HF)
ComfyUI Node

Image Captioner (API / Local HF)

An image in, a prompt out — but you bring the vision model

By aifeifei798·Created 2 months ago·Updated 18 days ago· 2
Image Captioner (API / Local HF)
    • chinese
    • english
    • thinking
    ◄sourceAPI (config.json)►
    ◄image▾►
    ◄instructionLook at this image carefully and output ONLY one JSON object, nothing else: {"chinese": "describe the image content, subject, action, scene, lighting and style in detail in Chinese", "english": "English image-generation prompt, comma-separated tags and quality words, directly usable in Stable Diffusion or Qwen-Image, e.g. '1girl, ... , masterpiece, best quality'"}►
    ◄temperature0.70►
    ◄max_tokens1024►
    ◄max_side1024►
    ◄thinking_modeOurs (8-step)►
    ◄timeout180►

    The display name tells you the important part: (API: edit config.json). This node does not contain a model. It sends your image to an OpenAI-compatible endpoint you configure, and the endpoint is the whole story - specifically, it has to be a vision model, because a text-only LLM behind that URL will just get confused.

    What it's for is the two jobs VLM captioning has always had: generating captions for a LoRA training set, and seeding a prompt from an image you already have for img2img or i2v. If you're captioning a big set, this is the automation; if you're seeding one prompt, it's a faster start than staring at the image and typing.

    What you set

    The image input is a dropdown of files in ComfyUI/input/, with an upload button - it is not an IMAGE socket. This is the first thing that trips people: you can't wire your freshly decoded latent into it. Save or upload the image to input/ first, then pick it. The node resolves the path and reads the file from disk.

    instruction is the contract. The default asks for exactly one JSON object with a chinese key and an english key, and it's worth reading rather than skimming, because the node parses those keys back out of the reply: chinese becomes one output, english the other. If the model replies in prose instead of JSON, the raw text lands in the first output and the second is empty - a signal you need to tighten the instruction or drop the temperature.

    max_side (default 1024) caps the longest side before upload. The image is re-encoded as JPEG, scaled to fit, and base64'd into a data URL, so this is your token/cost/VRAM dial. temperature (0.7) and max_tokens (1024) do what they look like. thinking_mode has three options - Ours (8-step) (default), Model native, Both - and it's a real behavioural switch: Model native ignores your instruction box entirely and uses a built-in minimal instruction so the model's own reasoning runs, with enable_thinking set true. Ours (8-step) uses your box with thinking off.

    Outputs are three strings: chinese, english, thinking. Wire english into a CLIP Text Encode and you've turned a reference image into a prompt.

    Setting the endpoint

    Everything lives in config.json in the pack folder - there is no api_base widget and, notably, no environment variable is read anywhere in this pack, not OPENAI_API_KEY, not proxy vars. There's a per_node block, and the Captioner is the one that usually needs it, since your vision model is often on a different host than your text LLM:

    {
      "api_base": "http://127.0.0.1:8080",
      "model": "",
      "per_node": {
        "FeiFeiImageCaptioner": { "api_base": "http://127.0.0.1:11434", "model": "qwen2.5vl" }
      }
    }
    

    Leave api_key empty for local llama.cpp. The file is re-read on every run, so no restart. Requests go through the openai SDK when it's installed and fall back to urllib when it isn't - a missing package degrades, it doesn't kill the node.

    One honest note about this category. The most damaging security incident in this ecosystem's history was ComfyUI_LLMVISION, a captioner that shipped credential-stealing malware and ended in a federal prosecution; a vision node is tempting to weaponise precisely because it's expected to touch the network and load weights. That's a reason to read a pack before installing, and a reason the no-env-vars design here is the right call: the only credential that can travel is what you wrote in config.json.

    Install

    ComfyUI Manager → ComfyUI-FeiFei, or:

    cd ComfyUI/custom_nodes
    git clone https://github.com/aifeifei798/ComfyUI-FeiFei.git
    pip install -r ComfyUI-FeiFei/requirements.txt   # openai>=1.40; optional, urllib fallback exists
    

    No local model files - the model is on the other end of the URL.

    Common issues

    • "API Error: api_base is empty in config.json". Exactly what it says; the top-level value is empty and there's no per_node override.
    • The text LLM answers with a description of nothing useful. You pointed a text-only model at an image payload. Use Qwen-VL, MiniCPM-V, or whatever vision model your server is serving, and set model accordingly on vLLM/Ollama-style servers.
    • 400 on enable_thinking. Some servers reject the field; the pack retries once without it rather than failing.
    • Long captions truncated. Raise max_tokens. Bigger max_side means a better description and a bigger request - the two costs move together.
    • Two-subject images get their attributes crossed. Not this node's fault; it's the known failure mode of every VLM captioner. Audit your captions before training on them, especially at small dataset sizes.
    CategoryFeiFei

    Inputs (8)

    NameTypeDefaultDescription
    sourceCOMBOAPI (config.json)API (config.json) posts the image to the endpoint in config.json; Local HF model runs the `model_dir` folder listed in config.json with transformers.
    imageCOMBO1 options: example.png
    instructionSTRINGLook at this image carefully and output ONLY one JSON object, nothing else: {"chinese": "describe the image content, subject, action, scene, lighting and style in detail in Chinese", "english": "English image-generation prompt, comma-separated tags and quality words, directly usable in Stable Diffusion or Qwen-Image, e.g. '1girl, ... , masterpiece, best quality'"}—
    temperatureFLOAT0.700.1–1.5—
    max_tokensINT102464–8192—
    max_sideINT1024256–4096—
    thinking_modeCOMBOOurs (8-step)3 options: Ours (8-step), Model native, Both
    timeoutoptINT18010–600API mode only: request timeout in seconds.

    Outputs (3)

    NameTypeDescription
    chineseSTRING—
    englishSTRING—
    thinkingSTRING—