Nodes/ComfyUI-HF-Diffusers/HF Transformers create conversation
ComfyUI Node

HF Transformers create conversation

How to actually talk to an LLM inside ComfyUI

By Yahweasel·Created 8 months ago·Updated 7 months ago· 2
HF Transformers create conversation
  • previous
  • image
  • HFT_CONVERSATION
roleuser
text

HuggingFace Transformers doesn't take a raw string and generate - it takes a conversation: a list of messages with roles, the same shape the OpenAI-style chat templates expect. This node is how you build that from inside ComfyUI. It's the input side of the pack's LLM story, and it's the first thing you'll wire up in any prompt-enhancement or image-description workflow built on these nodes.

Three inputs, one output:

  • role - defaults to user; type system or assistant for the others. This is how you slip an instruction preamble in front of your actual request, which is exactly what the pack's "prompt enhancer" example does: a system message teaching the model how to expand prompts, then a user message with your raw one-liner.
  • text - a multiline string, the message content.
  • image (optional) - a PIL_IMAGE. Feed one in and the message becomes a multi-part message with an image, for the vision-language models. Yes, you can chat about an image inside a ComfyUI graph. That's the VLM version of "describe this for me."

There's also an optional previous input: chain the output of one HFTCreateConversation into the previous of the next to build a multi-turn history. First turn, leave it unplugged; the node starts fresh.

The output is a single HFT_CONVERSATION, which feeds directly into HFTRunPipeline's input. That's the whole loop: create a conversation, run the pipeline, get a (possibly extended) conversation back.

Why it exists

This pack's design rule is "generic, not specialized." There's no dedicated "prompt enhancer" node, because prompt enhancement is just a text-to-text pipeline plus a system message. This node is the "plus a system message" - the unopinionated primitive the author expects you to build on. It also means the same node serves totally different purposes: rewriting prompts, describing an image for a img2img loop, classifying, whatever.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/Yahweasel/ComfyUI-HF-Diffusers

or search ComfyUI-HF-Diffusers in ComfyUI Manager, then restart. The pack pins diffusers~=0.36.0 in requirements.txt; for the Transformers side you're relying on the transformers ComfyUI already carries, and the author's GLM example notes some new models want the git versions of both.

Gotchas

  • A conversation isn't text. If you wire HFT_CONVERSATION into a text input, nothing connects. To get the assistant's reply as a string, you need HFTUnpackConversation on the other end.
  • Image in, image type matters. The image input is PIL_IMAGE, so it needs a PIL bridge (Chaoses-Ib's ComfyUI_Ib_CustomNodes) if your image is native ComfyUI.
  • First message isn't necessarily the user's. If you're building a system prompt, create that message first and chain the user message onto it. Get the order backwards and the model reads your instructions as user content - usually still works, but sloppy and sometimes embarrassing.
Categoryhuggingface-transformers

Inputs (4)

NameTypeDefaultDescription
roleSTRINGuser
textSTRING
previousoptHFT_CONVERSATION
imageoptPIL_IMAGE

Outputs (1)

NameTypeDescription
HFT_CONVERSATIONHFT_CONVERSATION