Prompt Dupe Remover V2

PromptDupeRemoverV2 is the prompt hygiene node: it strips repeated words from a prompt so you don't send "beautiful beautiful sunset, golden golden light" to your text encoder. One pass, case handled, and a whitelist so you can protect the words that legitimately repeat. If you've ever built prompts programmatically - stitching wildcards, tags, and batch outputs together - you know the duplicates show up constantly, and this is the mop.

It's a small node in a big pack, but it earns its place in any prompt-assembly workflow. Combine it with PromptCleaner (for removing words) and PromptSwap (for swapping them) and you've got a tidy little text-processing station between your prompt source and your encoder.

How it works

It tokenizes the prompt into words, punctuation, and whitespace, then keeps only the first occurrence of each word, case-insensitively. The controls are where the polish is:

• preserve_case (default true) - keeps your original capitalization for the words it keeps. Turn it off and everything surviving is lowercased, which is handy for normalizing messy tag lists.
• whitelist - a comma-separated list of terms that are never deduplicated. Crucially, it supports multi-word terms in quotes, e.g. "best quality", "no humans", and it recognizes them as a unit. So "no humans no humans" collapses, but "best quality best quality" (if whitelisted) survives.

Whitespace and punctuation are preserved as tokens, so the output reads naturally rather than getting glued together. It logs what it removed to the console, which is nice when you want to check a stubborn case.

One behavior to note: dedupe is across the whole prompt, not per-clause. If you're deliberately using the same word in a positive and negative half, this node sees one string and collapses it - for per-string work you usually want that, just be aware.

The inputs that matter

• text - the prompt (multiline).
• preserve_case - whether survivors keep their original casing.
• whitelist - the "never dedupe these" list; quotes for multi-word terms.

Output: one STRING - the deduplicated prompt, ready for the encoder or the next stage of your prompt pipeline.

Installing it

Part of DJZ-Nodes. ComfyUI Manager → Install Custom Nodes → search DJZ-Nodes → install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/MushroomFleet/DJZ-Nodes
cd DJZ-Nodes
pip install -r requirements.txt

Pure stdlib (re, ast for parsing the whitelist, logging) - zero pack dependencies involved. If DJZ-Nodes is installed, this node is already live.

Common issues

The subtle traps: forgetting to quote multi-word whitelist terms (they'll be treated as single words and the dedupe won't protect the phrase), and expecting it to dedupe across differently-cased words - it does, because matching is case-insensitive, but it keeps the first form you saw, so the casing you keep may not be the casing you wanted. Also, it works on a single string, so if your workflow sends positives and negatives through the same wire, keep that in mind. And while it preserves punctuation, it won't collapse "this, this" into "this" if the duplicate carries trailing punctuation - that's the tokenizer's doing, and it's mostly fine in practice.