Dedup Image Files 🐑

If you build LoRA datasets by pulling frames out of a video, you have a duplicate problem whether you've noticed it or not. Adjacent frames are nearly identical, and a training set stuffed with fifty almost-copies of the same pose teaches the model that pose and not much else. DedupImageFiles scans a folder and deletes the near-duplicates, keeping one representative of each. It's a dataset-hygiene node, and it's more important than it looks.

It's part of ComfyUI-MieNodes (ComfyUI_MieNodes), MieMieeeee's utility pack, filed - a little oddly - under 🐑 Caption Tools alongside the rest of the LoRA-prep kit.

Why you'd reach for it

The "generate a video, use the frames as a dataset" workflow is genuinely popular for character LoRAs, and it's exactly where duplicates breed. The advice you'll hear from people who train a lot is that you want variety - different angles, expressions, lighting - not the same frame forty times, and near-identical images actively hurt: they overweight whatever they have in common and drag the LoRA toward a single look. The KB's LoRA-training doc makes the same point about curation being the work. This node is the cleanup pass that turns a raw frame dump into something worth training on.

Worth knowing: MieMieeeee also ships a separate pack, ComfyUI-CaptionThis, for the captioning half of that same job, and the community does point people to it. DedupImageFiles is the dedup half. Together they're the author's answer to "I have a folder of frames, now what."

How it works

It uses a perceptual hash, not a byte-for-byte comparison. Two files that are visually the same but differ by a pixel, a re-save, or a JPEG artifact have totally different exact hashes - a perceptual hash instead produces a fingerprint that stays close when the images look close. The node computes that fingerprint for every image and measures the Hamming distance (how many bits differ) between them; anything under your threshold counts as a duplicate and gets removed, leaving one behind.

That's why there's a distance knob rather than a simple on/off: "duplicate" is a spectrum, and you decide how aggressive to be.

The inputs and outputs that matter

• directory (STRING) - the folder to clean. Fill in a real absolute path; the default is placeholder text.
• max_distance_threshold (INT, default 10, range 0–64) - the sensitivity dial. 0 means only bit-identical fingerprints are treated as duplicates (very conservative). Higher numbers catch looser matches. The default of 10 is a reasonable middle; if it's leaving obvious dupes, nudge it up, and if it's eating images that are actually different shots, bring it down.
• trigger_signal (optional, any type) - a sequencing input so you can force this to run after whatever produced the images.

Outputs are deleted_file_count (INT) and log (STRING). Read the log - it's how you audit what got removed.

Installing it

ComfyUI Manager → search ComfyUI-MieNodes → install → restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/MieMieeeee/ComfyUI-MieNodes

then restart. Perceptual hashing needs an imaging library under the hood, so if the node errors on load, check that the pack's dependencies installed cleanly (Manager usually handles this; a manual clone may need pip install -r requirements.txt). Nodes appear under 🐑 MieNodes.

Common issues & troubleshooting

This deletes files on disk. There is no undo. Point it at a copy of your dataset the first time, or at least keep the originals somewhere, until you've found a threshold you trust. Read deleted_file_count and the log before you rely on it in an unattended run.

Threshold too high nukes distinct images. Crank max_distance_threshold too far and genuinely different shots that happen to share a background or palette start reading as duplicates. If your set comes back thinner than expected, that's the cause - lower it.

Threshold too low leaves obvious dupes. The flip side. Video frames that look identical to you can still differ enough in fingerprint to survive at a low threshold. Raise it a few points and re-run on a copy.

It's not a substitute for curation. Dedup removes redundancy; it doesn't remove bad images - blurry frames, motion smear, half-blinked eyes. Do a human pass after. The node makes the pile smaller, not automatically better.