Modal Endpoint Chat

This is the node that makes the "ComfyUI on Modal" fantasy real without you touching a cloud GPU. Modal Endpoint Chat is a plain chat-call node: you give it a Modal Direct endpoint URL, type a prompt, and it talks to whatever OpenAI-compatible model that endpoint serves, returning the text into your graph. If you've already deployed a model on Modal (scale-to-zero, ~$30/month of free credits, cold boots in seconds with memory snapshots), this is how that model becomes a node in your workflow.

The mental model is simple and worth nailing down, because the pack around it is not. Everything else in ttulttul/ComfyUI-Modal (branded "Modal Sync") is about remote execution - marking nodes Run on Modal and shipping chunks of your graph to a cloud GPU. This node is the opposite: it makes no remote island, exposes no Run on Modal toggle of its own, and needs no GPU worker at all. It's a local node that makes one HTTP call, exactly like ComfyUI's built-in OpenAI ChatGPT node - the layout is deliberately modeled on it. The name is slightly misleading in the other direction: "Endpoint Chat" means you host the endpoint, not that this node provisions one.

How it works: it posts to the /v1/chat/completions route on your modal.direct URL. Leave model blank and it asks /v1/models and uses the first advertised model - handy, though I'd set it explicitly once your endpoint is stable. Images and files ride along in OpenAI's multimodal format, so a vision-capable serving recipe just works. Two details matter more than they look. First, Modal endpoints scale to zero, so the first request after idle gets an empty HTTP 503 while a replica cold-starts; the node treats that as a cold-start signal and retries with backoff until timeout_seconds (default 600) expires. Second, credentials: it wants proxy tokens - MODAL_KEY/MODAL_SECRET with wk-/ws- prefixes, not your API creds (ak-/as-). Resolution order is environment variables, then the OS keyring, then auto-creation via modal workspace proxy-tokens create (the values go to Keychain, never into your workflow JSON). It also refuses anything that isn't an HTTPS modal.direct origin and rejects redirects.

The inputs you'll actually set: endpoint_url, prompt, and model. max_tokens (default 4096) and temperature (default 0.7) work like any LLM call. timeout_seconds is the cold-start allowance - raise it if you keep hitting 503 on a warm-down endpoint. The optional environment only matters if your endpoint lives in a non-main Modal environment. Everything funnels into one output: response, a STRING you can wire into a prompt-prefix node, a text display, or another LLM call.

Install via ComfyUI Manager - search for Modal Sync - or:

cd ComfyUI/custom_nodes
git clone https://github.com/ttulttul/ComfyUI-Modal

Restart ComfyUI. This is a V3 node pack (requires ComfyUI ≥ 0.11.1), so update ComfyUI first if Manager complains. Setup has three prerequisites: a Modal account, a model endpoint you've already deployed there, and an authenticated Modal CLI on the ComfyUI machine (<venv>/bin/python -m modal setup). The whole pack is alpha, and this node is flagged experimental - expect rough edges.

Where people get burned: cold starts (watch the retries, bump timeout_seconds), pasting ak- API keys instead of proxy tokens, and running headless Linux with no keyring backend - there you must set MODAL_KEY and MODAL_SECRET in the environment, or auto-creation has nowhere to store the result. It's the rare LLM node where the model isn't the hard part; the Modal plumbing is. But once your endpoint is up, this is the cheapest, cleanest hosted-LLM call you can drop into a workflow - free monthly credits, per-second billing, no VRAM budget on your card at all.