AuK Instruction Encode

This is the node that decides everything about what comes out. It takes your instruction text, optionally takes a reference or source clip, and turns the pair into conditioning the sampler can use. Get the instruction right here and the rest of the graph is just knobs.

The name is odd on purpose: this is not CLIP text encoding. AuK's conditioning is a fused stack of hidden states from a Qwen2.5-Omni multimodal encoder, and the fusion is a learned thing - which is why the node asks for the AuK model as well as the encoder.

How it works

The encoder runs your instruction through the Qwen chat template (with an audio placeholder when a clip is attached), reads the hidden states from every layer, and then fuses them: the weights come from the diffusion checkpoint (layer_weights, softmaxed across layers, plus a layer_scale). Base and Flash carry their own learned layer-fusion weights, so the same encoder paired with a different model fuses differently. That is the whole reason model is an input on an encoder node - and the reason you can't shortcut it.

The output conditioning isn't just a tensor. It carries the attention mask and the source audio along with it, so Generate / Edit can later re-encode that reference clip into a latent. Text-only runs append a <no_prompt_audio> marker to the instruction instead.

Encoding happens once, before denoising. It is not re-run per sampling step, so a long reference clip costs you on the encode, not 32 times over.

The inputs that matter

• model - AUK_MODEL from AuK Model Loader. Both outputs of that loader go here and to the sampler.
• encoder - AUK_ENCODER from AuK Encoder Loader.
• instruction - multiline text. There's a default TTS sentence in the box; replace it. The field has examples in its tooltip, and they're worth copying verbatim before you improvise.
• audio (optional) - core AUDIO. Connect it for cloning and every edit task. Leave it empty for text-only speech. This single wire is the difference between "generate a voice" and "work on this recording."

Output: CONDITIONING → AuK Generate / Edit (.conditioning).

Writing instructions that land

The wording is not free-form. AuK tasks are phrased as instructions it was trained on, and paraphrasing costs you:

Say the following with the same voice: "The train leaves at nine."
Replace 'Tuesday' with 'Friday'.
Raise the pitch by 2.0 semitones.
Extract only the singing voice and remove everything else.

Quoted words have to actually exist in the source clip - 'Tuesday' in a recording where nobody says Tuesday is an instruction you can't follow. Same for anchors in insert instructions.

And the honest part: prompt adherence is uneven, and the author says so up front. TTS, cloning, volume, enhancement, singing extraction and speaker separation are the stable ones. Insert/replace, lyric edits and whisper conversion are listed as least stable. When those miss, it's the model, not your wiring, and the useful move is a second seed or a different phrasing - not a reinstall. Keep one word change per lyric edit and pick a replacement that sounds similar to the original, because the model follows prosody better than it follows spelling.

Install

cd ComfyUI/custom_nodes && git clone https://github.com/Saganaki22/ComfyUI-AuK
cd ComfyUI-AuK && python -m pip install -r requirements.txt

No models belong to this node - it takes them from the three loaders. Restart ComfyUI after installing.

Troubleshooting

"Use conditioning from AuK Instruction Encode." The sampler found a conditioning object without AuK's metadata, i.e. you wired something else into it. The core CLIP Text Encode output will not work, ever.

Typing in the box does nothing. If a STRING is connected to instruction, it overrides the widget. Check for a leftover wire from an Instruction Builder or a Prompt Enhance - this is the number-one "the node is ignoring me" report on this pack.

Voice cloning sounds like a different person. The reference clip drives this, so start there: clean, one speaker, no music bed. Then check seconds on the sampler - a too-short duration truncates the clone mid-sentence.

Batch behaviour. One audio item in the batch produces one conditioning entry, and the sampler walks them in order using seed + index. If you're processing ten clips, that's ten conditioning entries, not ten separate runs.