Deduplicate Images

If you've ever loaded a folder of images for a training run and realized it's full of near-identical shots - the same character, the same pose, three frames apart - you know the pain. Deduplicate Images removes the near-duplicates so your dataset doesn't train on the same content ten times over. It's a genuinely useful filter, and it's also the node where you should understand exactly how blunt the tool is before you trust its verdict.

How it works

It uses a perceptual hash - a fingerprint of each image that's designed to stay similar when the image looks similar. The implementation is the classic simple version:

1. Resize every image down to 8×8 pixels, grayscale.
2. Compute the average brightness of those 64 pixels.
3. Each pixel becomes a 1 (above average) or 0 (below average) - a 64-bit hash.
4. Compare hashes by Hamming distance (how many of the 64 bits differ), and convert to a similarity score from 0 to 1.
5. If two images score at or above similarity_threshold, the later one is dropped as a duplicate.

It's greedy, which matters: the first image is always kept, and every later image is compared only against images already kept. Order in = order kept, minus the duplicates.

The inputs that matter

• images - the list of IMAGEs to deduplicate. List input; feed from a load-folder or list node.
• similarity_threshold (default 0.95, 0–1) - how similar two images must be to count as duplicates. Higher = more strict about what counts as "the same" = fewer images removed. At 0.95, only near-identical pairs get dropped; crank it down toward 0.8 and the filter gets aggressive and will happily eat distinct-but-similar images.
• Output: the IMAGE list with duplicates removed, first-seen order preserved.

Where the knife gets blunt

The 8×8 average-brightness hash is coarse. Really coarse. It's the kind of hash that's great at catching "same shot re-saved" and genuinely bad at nuanced judgment:

• Composition with similar brightness = collision. Two completely different images can hash the same if their brightness layouts match. A dark scene and another dark scene may look identical to this hash while being totally different photos.
• It's not rotation-, crop-, or exposure-proof. Flip an image, crop it, or change exposure and the hash can change enough to escape the threshold.
• No smart comparison. It doesn't compare actual pixels, just a 64-bit summary. It has no idea that two images show the same person from different angles, and it won't catch that kind of "duplicate."

So treat this as a first-pass dedupe: great for removing literal re-saves and burst-shoot neighbors from a folder before training, not a substitute for actually eyeballing your dataset. For LoRA work, quality and diversity of captions beat raw volume anyway, and this node mostly helps you avoid feeding the same content in ten times.

Common issues

• "It removed things that aren't duplicates." Yes, that's the hash being too coarse. Lower... no, raise the threshold (closer to 1.0) so only very-similar images qualify, and accept that a blunt tool needs conservative settings.
• "It missed obvious duplicates." Usually exposure or crop differences broke the hash. This node isn't the fix for that; a folder-level dedupe tool is.
• Performance. The hash is computed in Python over a resize of every image. On huge folders it's slower than you'd like, but fine for typical dataset sizes.

Ships with ComfyUI core (newer experimental family - update ComfyUI if you can't find it). No models, no install. A useful filter, if you remember it's judging images by their 8×8 silhouette.