ComfyUI-ImageToPrompt

A custom ComfyUI node that sends images to OpenAI-compatible vision APIs (like GPT-4 Vision, LLaVA, etc.) and returns the LLM's text response. Perfect for image analysis, captioning, and visual question answering workflows.

Features

• 📷 Upload images via ComfyUI's image input nodes
• 🤖 Send to any OpenAI-compatible API (GPT-4 Vision, local LLaVA, llama.cpp, etc.)
• 💬 Optional system and user prompts for controlling the analysis
• ⚙️ Configurable parameters: temperature, max tokens, model selection
• 📝 Text output that can be used in downstream nodes or displayed in the UI
• 🔍 Debug logging to monitor API calls and responses

Installation

Prerequisites

• ComfyUI installed and working

Install via ComfyUI Manager (Recommended)

1. Open ComfyUI Manager
2. Search for "ComfyUI-ImageToPrompt"
3. Click Install
4. Restart ComfyUI

Manual Installation

1. Navigate to your ComfyUI custom nodes directory:

   cd ComfyUI/custom_nodes/
   

2. Clone this repository:

   git clone https://github.com/yourusername/ComfyUI-ImageToPrompt.git
   

3. Install dependencies:

   cd ComfyUI-ImageToPrompt
   pip install -r requirements.txt
   

4. Restart ComfyUI

Usage

1. Add the node: In ComfyUI, add the "Image to LLM Prompt" node (found under the "LLM" category)

2. Connect an image: Connect an image output (from LoadImage, etc.) to the node's image input

3. Configure the API:

   • base_url: Your OpenAI-compatible API endpoint (e.g., http://localhost:8080/v1)
   • api_key: Your API key (use sk-test for local servers that don't require authentication)
   • model: Model name (e.g., gpt-4o, llava, etc.)

4. Add prompts (optional):

   • system_prompt: Instructions for the AI (e.g., "You are an expert art critic")
   • user_prompt: Specific question about the image (e.g., "Describe this image in detail")

5. Adjust parameters (optional):

   • temperature: Controls randomness (0.0 = deterministic, 2.0 = very random)
   • max_tokens: Maximum response length

6. Execute the workflow: The LLM's response will appear in the node's output and can be passed to other nodes

Example Workflow

LoadImage → ImageToLLMPrompt → (use text output in other nodes)

The node outputs a STRING that can be connected to:

• Text processing nodes
• Prompt nodes for image generation
• File writing nodes
• Any node that accepts STRING input

API Compatibility

This node works with any OpenAI-compatible vision API endpoint, including:

• OpenAI GPT-4 Vision (https://api.openai.com/v1)
• Local LLaVA servers (via llama.cpp, ollama, etc.)
• Azure OpenAI with vision models
• Other compatible backends

Configuration Examples

OpenAI GPT-4 Vision

base_url: https://api.openai.com/v1
api_key: sk-your-actual-api-key
model: gpt-4o

Local LLaVA (llama.cpp server)

base_url: http://localhost:8080/v1
api_key: sk-test
model: llava

Troubleshooting

• "LLM API call failed" errors: Check that your base_url is correct and the server is running
• Empty responses: Verify your model supports vision/image inputs
• Timeout errors: Increase the timeout in the code or use a faster model
• Node not appearing: Restart ComfyUI and check the console for import errors

Dependencies

• openai - OpenAI Python client library
• pillow - Python Imaging Library for image processing
• numpy - Array processing (typically already installed with ComfyUI)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.