Extensions/ComfyUI Gemini Expanded API
ComfyUI Extension

ComfyUI Gemini Expanded API

Node for usage of Gemini API

By silveroxides·Created about a year ago·Updated 4 days ago· 5
silveroxides/ComfyUI_Gemini_Expanded_API
Nodes5
On cloudLocal install
Categoryutils/api_keys, API/Gemini
Stars5
Updated4 days ago

Nodes (5)

Get API Key from JSON

Keep your Gemini key out of your workflow file

utils/api_keys
Configure Gemini API Key

The node that hands your key to Gemini — without calling it

API/Gemini
Configure Gemini Response Schema

Gemini Answers in Prose Unless You Hand It a Schema First

API/Gemini
Expanded Gemini Text/Image

Text, vision, and image generation in one ComfyUI node

API/Gemini
Configure Gemini Video Input

Configure Gemini Video Input

API/Gemini
Readme

ComfyUI Gemini Expanded API Node

This is a Google Gemini API integration node for ComfyUI, supporting text generation and image generation functions. With this node, you can directly use Google's Gemini 2.0 series models in your ComfyUI workflow. Special Note: Regarding the error [ERROR]API call error: 'NoneType' object has no attribute 'parts', it means that the image or prompt you uploaded violates the "Generative AI Prohibited Use Policy". Please test with general scene or product images first to ensure compliance.

Updates

2024.3.19: Updated to support multi-image processing.

Features

  • Supports Gemini 2.x and 3.x series models
  • Supports text-to-text generation
  • Supports image-to-text generation (image understanding)
  • Supports text-to-image generation (Note: this functionality is implemented in separate .py files, not directly within this node.)
  • Built-in proxy support for convenient use by users in China
  • Automatic dependency checking and installation
  • Comprehensive error handling and logging

Installation Method

  1. Ensure you have ComfyUI installed.
  2. Clone or download this repository into ComfyUI's custom_nodes directory: 
    cd ComfyUI/custom_nodes
git clone https://github.com/silveroxides/ComfyUI_Gemini_Expanded_API.git
  3. Install dependencies: 
    cd ComfyUI_Gemini_Expanded_API
pip install -r requirements.txt
  4. Restart ComfyUI.

How to Use

1. Configure Gemini API or Enterprise/Vertex AI

Option A(Gemini API)

First, you need to obtain a Google Gemini API key:

  1. Visit Google AI Studio or Google Cloud Console
  2. Create an API key (In Google Cloud Console it is recommended to set key restriction to the Generative Language API)
  3. In ComfyUI, locate and use the Configure Gemini API Key node to enter your API key.

Option B (Gemini Enterprise Agent Platform, formerly Vertex AI)

  1. Start by visiting Google Cloud Console Vertex AI API and enable it.
  2. Visit the Vertex AI Model Garden and select a Google Gemini model and then "view code" button.
  3. Read the instructions for the chosen model at "Try (Python)" for environment authentication or "Try while using express mode (Python)" for API-key authentication.
  4. For environment authentication, set GOOGLE_CLOUD_PROJECT=your-project-id, GOOGLE_CLOUD_LOCATION=global, and GOOGLE_GENAI_USE_ENTERPRISE=true before launching ComfyUI. Existing installations may continue using GOOGLE_GENAI_USE_VERTEXAI=true; the node supports both names, with GOOGLE_GENAI_USE_ENTERPRISE taking precedence if both are set.
  5. In ComfyUI, locate and use the Configure Gemini API Key node and set use_vertexai_env to true.
  6. For Enterprise express mode, enter its API key and enable vertexai_express; project and location are not required for this mode.

2. Text Generation

Use the Expanded Gemini Text/Image node for text generation:

  • Connect the API key configuration node to the config input.
  • Enter your prompt text in prompt.
  • Adjust generation parameters (temperature, top_p, top_k, etc.).
  • If needed, connect images through image_inputs. For video, pass a native ComfyUI video through Configure Gemini Video Input, then connect its output to video.
  • Configured video is normalized to MP4/H.264 and sent with embedded audio at the selected sampling FPS. pad_at_start optionally prepends black video and silence. duration_aware_padding derives that padding from the video's fractional-second duration and rounds it down to native video frames for comparison testing. The normalized video must be smaller than 100 MB.
  • Enable use_cache on the configuration node to reuse unchanged prompt, system instruction, image, and video context through Gemini after the local result cache misses.

3. Proxy Settings

If you are in China or other regions requiring a proxy:

  1. Set use_proxy to True.
  2. Set proxy_host (default is 127.0.0.1).
  3. Set proxy_port (default is 7890).

Parameter Description

API Key Configuration Node

  • api_key: Google Gemini API key
  • use_cache: Enable Gemini API context caching. Disabled by default.
  • cache_ttl_minutes: Number of minutes Gemini retains cached context.
  • cache_seed: Generation seed used while use_cache is enabled.

Video Input Configuration Node

  • video: Native ComfyUI video input.
  • fps: Gemini sampling rate from 1 through 24 FPS.
  • pad_at_start: Prepend matching-resolution black video and silence for half one sampling interval before encoding.
  • duration_aware_padding: With start padding enabled, use the video's fractional-second duration rounded down to native video frames instead of the standard padding duration.

Response Schema Builder

Add Configure Gemini Response Schema under API/Gemini. Choose the Type of the response. Object starts empty, with an Add field control.

  1. Use Add field to choose a type, such as Text or Integer. A new bordered entry appears inside its parent.
  2. Enter its Name, set Required, and add a Description if needed. Hover over a control for a plain-language explanation.
  3. Choose Object for a group of named fields or Array for a list of items. Children appear inside the parent's border with a small indentation.
  4. Choose Disabled to leave a field out while retaining its settings. Use Remove to delete the field and all its nested definitions completely.
  5. Queue the node to produce the GEMINI_RESPONSE_SCHEMA dictionary and formatted schema_json string. The JSON is also returned as a text preview.

There is no fixed field count or container-depth cutoff in the builder. Only actual authored entries are stored. Generated property order follows field creation order. This does not remove runtime or provider schema-complexity limits.

  • Text: defaults to Any text. Choose From list under Allowed values to enter one allowed answer per line.
  • Integer / Number: Minimum and Maximum each offer No limit / Set limit.
  • Boolean: asks the AI for a yes-or-no answer. It does not set that answer in advance.
  • Names are trimmed and must be nonempty and unique among enabled siblings. Descriptions retain their text. Invalid active enums or numeric bounds identify the affected field.
  • An empty object emits empty properties; not every provider/model necessarily accepts that schema.

Type changes retain inactive settings, including object children and array items. Save and reopen the workflow to restore the complete configuration. The JSON authoring state is stored internally; you do not edit it directly.

The editor is one ComfyUI DOM widget with scoped styles. Entries use compact borders and padding. Each section has a Collapse / Expand button: it hides settings while leaving the header accessible and does not omit any information from the output. Collapsed sections are saved with the workflow. Descriptions use a single line; allowed-value lists remain multiline. Resize the node to adjust the editor area; long content scrolls inside it. Existing version-1 builder state remains compatible.

After updating, restart ComfyUI, refresh the page, and recreate nodes from the previous fixed-slot prototype. Old prototype inputs are not silently migrated. Check nested field creation, disabling/re-enabling, type switching, queue output, and save/reopen behavior in your live frontend.

This builder makes no API requests. Connect its schema output to Response format on Expanded Gemini Text/Image. That optional input appears after video and immediately before the growing image inputs. Leave it disconnected for a normal answer. The schema output can also connect directly to Core's Preview as Text; schema_json is the same schema already formatted as text.

With a response format connected, the request uses JSON formatting and returns the final JSON in both text and structured_output. Input images and video can still be analyzed. Active image generation cannot be combined with a response format. Invalid JSON is reported as a processing error rather than repaired. Changing the format causes a new generation, while unchanged cached input context can be reused.

For development tests, run npm ci once to install the test-only DOM library, then use tests/run_tests.py --group response_schema with the configured ComfyUI Python. The browser editor itself has no npm runtime dependency. DOM tests check structure and events; they do not replace checking actual canvas positioning, zoom, and layout in ComfyUI.

Text Generation Node

Required Parameters

  • config: API key configuration
  • prompt: Prompt text
  • model: Select model (gemini-2.0-flash, gemini-2.0-flash-exp, gemini-2.0-pro)
  • temperature: Generation temperature (0.0-1.0), controls the randomness and creativity of the output.
  • top_p: Nucleus sampling parameter (0.0-1.0)
  • top_k: Number of candidate tokens to consider (1-100). Higher values mean more diversity.
  • max_output_tokens: Maximum output tokens (1-8192)
  • include_images: Whether to include images in the response (True/False)

Optional Parameters

  • video: Optional GEMINI_VIDEO_CONFIG input. Its video is sent before images and prompt with embedded audio.
  • response_schema (Response format): Optional GEMINI_RESPONSE_SCHEMA input from the builder's schema output; placed immediately before Autogrow.
  • image_inputs: Optional ordered image autogrow inputs for image understanding.
  • use_proxy: Whether to use a proxy (True/False)
  • proxy_host: Proxy host address
  • proxy_port: Proxy port

Output

Text generation node output:

  1. text: Final answer text, excluding returned thoughts.
  2. image: Generated image when image generation is enabled; otherwise the existing empty image output.
  3. final_actual_seed: The final generation seed.
  4. structured_output: Final JSON text when a connected response format succeeds; otherwise empty.
  5. thoughts: Returned thought text from the accepted response; otherwise empty. include_thoughts controls requesting it.

The first three output positions remain unchanged. Errors and timeout fallbacks remain in text; structured_output and thoughts are empty on failure. Failed requests are not stored as successful cached results. JSON parsing checks syntax, not factual correctness or full schema compliance.

Precautions/Notes

  • According to Google's "Generative AI Prohibited Use Policy", Gemini API has the following restrictions:

  • Must not be used to generate content that violates laws and regulations.

  • Must not be used to generate harmful, fraudulent, pornographic, or violent content.

  • Must not be used to generate content that infringes on others' privacy or intellectual property rights.

  • Image generation may have additional restrictions, and certain types of images might not be generatable.

  • Using this node requires a stable network connection or effective proxy settings.

  • API requests may be affected by Google server load.

  • Large requests may require longer processing times.

  • The image generation feature requires a model that explicitly supports it (e.g., gemini-2.0-flash-exp).

Troubleshooting

  • If you encounter network connection problems, please check your proxy settings.
  • If API requests fail, please check if your API key is valid.
  • If dependency installation fails, please manually install the required dependency packages.

Acknowledgements

Thanks to Google for providing the Gemini API service. Thanks to tatookan for creating the original custom node repository for me to expand upon.