Pandas Drop Duplicates

Pandas Drop Duplicates is the "clean up your data" node. Real data is full of repeats - the same row logged twice, an import that appended instead of overwrote, an upsert gone wrong - and duplicate rows will quietly poison every aggregation downstream. A mean() over a table with doubled rows is a mean over the wrong data. This node removes them, and it's one of the genuinely useful cleansers in the pack's data-cleansing category alongside Drop NA.

Mechanism is one call:

return (dataframe.drop_duplicates(),)

Input: dataframe (DATAFRAME). Output: a DATAFRAME with duplicate rows removed. There's exactly one input and one output - no knobs, no options exposed. What you need to know is what the default pandas behavior actually is, because the node doesn't let you change it.

What the default does (and doesn't)

drop_duplicates() with no arguments considers every column when deciding if a row is a duplicate, and keeps the first occurrence of each unique row. Three consequences you'll want to internalize:

• It's a whole-row dedupe. Two rows that match on one column but differ elsewhere are not duplicates and both survive. If you wanted "drop rows where this one column repeats," that's a different operation (drop_duplicates on a subset, which this node doesn't expose) - so don't expect this node to do it.
• Duplicates are judged by values, not by index. Two identical rows with different index labels still count as duplicates and get collapsed.
• "Keep first" means the row that appeared earliest in the table wins. If your data is ordered by time, the earliest record survives - usually what you want, but worth knowing when it isn't.

The output keeps the original index of the kept rows, so after a dedupe the index may have gaps (0, 1, 3, 7…). That's normal pandas behavior, not a bug - reset the index downstream if your workflow assumes contiguous row labels.

Where it fits

This is a data-prep node, so it lives early in the pipeline: load or create your table, run Drop Duplicates (and Drop NA), then feed the clean frame into aggregation, crosstab, or plotting. For the pack's MLB example - computing most hits per year - duplicate rows in the batting log would inflate totals, so a dedupe step is the kind of thing that saves you from a wrong answer you'd never suspect.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/HowToSD/ComfyUI-Data-Analysis.git
mv ComfyUI-Data-Analysis data-analysis
pip install -r requirements.txt

or ComfyUI Manager → search "Data analysis" → install → restart. No GPU, no model downloads - pandas is the whole engine.