JoyAI_Image_ENCODER

This is the node that makes JoyAI-Image feel different from every checkpoint you've used before. You type a sentence like "Turn the plate blue" in plain English, optionally wire in the image you want edited, and out comes a positive and negative conditioning pair that the pack's own KSampler can consume. No CLIP skip, no ((prompt weighting)), no tag soup - the prompt is a chat message to a vision-language model, and this node is the messenger.

The mechanism matters here, because it explains why your old habits stop working. JoyAI-Image is JD's unified multimodal model: the text encoder isn't CLIP or T5, it's a Qwen3-VL. The ENCODER runs your prompt (and optionally an image) through that VLM using its chat template, grabs the last hidden layer, and slices out the non-masked tokens to build the conditioning. When you attach an image, it injects <image> tokens ahead of your text so the edit instruction has something to look at. The negative side is just the same pass with an empty prompt - for a guidance-scaled flow model, that empty-prompt pass is what the guidance_scale dial pushes against. Expect longer conditioning than you're used to: Qwen3 tokenizes into thousands of tokens, not 77.

The inputs that actually matter

• clip - feed it the output of JoyAI_Image_SM_Clip (the Qwen3-VL loader). Non-negotiable.
• prompt - the instruction. The default, "Turn the plate blue", is the pack's demo prompt and a fine starting point for image editing. Leave image unconnected and this is plain text-to-image conditioning.
• image (optional) - the reference image for editing. Wire it in and the conditioning becomes multimodal.
• infer_device - cuda or cpu. The node pushes the clip to your chosen device for the encode, then shoves it back to CPU and empties the cache. On a 12-16GB card, CPU encoding is the saner choice.
• save_emb - when true, it dumps the positive/negative tensors to your ComfyUI output directory as raw_embeds_JOY_sm.pt / n_raw_embeds_JOY_sm.pt. This is the cache-and-reuse path: JoyAI_Image_SM_KSampler will happily run with no conditioning connected as long as these files exist.

That last one is both a feature and a trap. The cache is real and useful - you can iterate on steps and seeds without re-running the VLM. But if you ever wire a KSampler without conditioning and never ran the ENCODER with save_emb on, the sampler throws "No backup prompt embeddings found. Please run JOY_SM_ENCODER node first." If that error shows up in your console, you didn't break anything; you just skipped the encode step.

Wiring it up

It's the middle of the pack's happy path: JoyAI_Image_SM_Clip → ENCODER → JoyAI_Image_SM_KSampler. Outputs are named positive and negative; wire both into the sampler (the negative is an empty-prompt unconditional, but the sampler expects a pair).

Install

Shared pack install - clone, install requirements, restart:

cd ComfyUI/custom_nodes
git clone https://github.com/smthemex/ComfyUI_JoyAI_Image
cd ComfyUI_JoyAI_Image
pip install -r requirements.txt

ComfyUI Manager can do the same (search "ComfyUI_JoyAI_Image"). The requirements file is the spicy part: flash-attn>=2.8.0 needs a CUDA toolchain to build, and transformers>=4.57.0,<4.58.0 is pinned hard - if another pack of yours needs a different transformers, you'll feel it. You also need the merged text encoder weights (JoyAI-Image-Und-merger_bf16.safetensors, or the Q6_K GGUF) in models/clips or models/gguf respectively, plus the DiT and Wan VAE. The README's GGUF route is the honest one for normal GPUs: the bf16 everything path is a RAM hog.

One warning earned by the model, not the pack: JoyAI-Image is young and its ComfyUI story is community-built (this pack is a third-party port, and an actively maintained one). Treat the first run as a smoke test - if the encode step eats a few gigabytes of RAM, that's expected, not a leak.