HF Transformers create conversation
How to actually talk to an LLM inside ComfyUI
- previous
- image
- HFT_CONVERSATION
HuggingFace Transformers doesn't take a raw string and generate - it takes a conversation: a list of messages with roles, the same shape the OpenAI-style chat templates expect. This node is how you build that from inside ComfyUI. It's the input side of the pack's LLM story, and it's the first thing you'll wire up in any prompt-enhancement or image-description workflow built on these nodes.
Three inputs, one output:
- role - defaults to
user; typesystemorassistantfor the others. This is how you slip an instruction preamble in front of your actual request, which is exactly what the pack's "prompt enhancer" example does: asystemmessage teaching the model how to expand prompts, then ausermessage with your raw one-liner. - text - a multiline string, the message content.
- image (optional) - a
PIL_IMAGE. Feed one in and the message becomes a multi-part message with an image, for the vision-language models. Yes, you can chat about an image inside a ComfyUI graph. That's the VLM version of "describe this for me."
There's also an optional previous input: chain the output of one HFTCreateConversation into the previous of the next to build a multi-turn history. First turn, leave it unplugged; the node starts fresh.
The output is a single HFT_CONVERSATION, which feeds directly into HFTRunPipeline's input. That's the whole loop: create a conversation, run the pipeline, get a (possibly extended) conversation back.
Why it exists
This pack's design rule is "generic, not specialized." There's no dedicated "prompt enhancer" node, because prompt enhancement is just a text-to-text pipeline plus a system message. This node is the "plus a system message" - the unopinionated primitive the author expects you to build on. It also means the same node serves totally different purposes: rewriting prompts, describing an image for a img2img loop, classifying, whatever.
Install
cd ComfyUI/custom_nodes
git clone https://github.com/Yahweasel/ComfyUI-HF-Diffusers
or search ComfyUI-HF-Diffusers in ComfyUI Manager, then restart. The pack pins diffusers~=0.36.0 in requirements.txt; for the Transformers side you're relying on the transformers ComfyUI already carries, and the author's GLM example notes some new models want the git versions of both.
Gotchas
- A conversation isn't text. If you wire
HFT_CONVERSATIONinto a text input, nothing connects. To get the assistant's reply as a string, you needHFTUnpackConversationon the other end. - Image in, image type matters. The
imageinput isPIL_IMAGE, so it needs a PIL bridge (Chaoses-Ib'sComfyUI_Ib_CustomNodes) if your image is native ComfyUI. - First message isn't necessarily the user's. If you're building a system prompt, create that message first and chain the user message onto it. Get the order backwards and the model reads your instructions as user content - usually still works, but sloppy and sometimes embarrassing.
Inputs (4)
| Name | Type | Default | Description |
|---|---|---|---|
| role | STRING | user | — |
| text | STRING | — | |
| previousopt | HFT_CONVERSATION | — | |
| imageopt | PIL_IMAGE | — |
Outputs (1)
| Name | Type | Description |
|---|---|---|
| HFT_CONVERSATION | HFT_CONVERSATION | — |