Extensions/ComfyUI-huggingface_dataset
ComfyUI Extension

ComfyUI-huggingface_dataset

ComfyUI node to load a Hugging Face dataset

By StableLlamaΒ·Created a day agoΒ·Updated about 24 hours agoΒ· 1
StableLlama/ComfyUI-huggingface_dataset
Nodes17
On cloudLocal install
CategoryHugging Face πŸ€—
Stars1
Updatedabout 24 hours ago

Nodes (17)

πŸ€— Dataset Filter

Filter a Hugging Face dataset by column β€” without writing a single lambda

Hugging Face πŸ€—
πŸ€— Dataset Flatten

Unwrap nested columns β€” turn that JSON-in-a-cell into real columns

Hugging Face πŸ€—
πŸ€— Dataset Map Column

Add a column to every row β€” constants, copies, or a row index, all in the graph

Hugging Face πŸ€—
πŸ€— Dataset Remove Columns

Drop the columns you don't need β€” before the rows ever hit your memory

Hugging Face πŸ€—
πŸ€— Dataset Rename Column

Fix a column name in one box β€” no schema, no migration, no drama

Hugging Face πŸ€—
πŸ€— Dataset Select Rows

Pick exact rows by index β€” comma lists, Python slices, all of it

Hugging Face πŸ€—
πŸ€— Dataset Select Columns

Keep only the columns you need β€” the whitelist version of column trimming

Hugging Face πŸ€—
πŸ€— Dataset Shard

Slice a dataset into N rough pieces and grab one β€” the parallel-work tool

Hugging Face πŸ€—
πŸ€— Dataset Shuffle

Shuffle your dataset with a seed β€” reproducibility that actually reproduces

Hugging Face πŸ€—
πŸ€— Dataset Skip

Skip the first N rows β€” the boring node that keeps showing up in real graphs

Hugging Face πŸ€—
πŸ€— Dataset Sort

Sort by column, then flip with reverse β€” data wrangling's most underrated node

Hugging Face πŸ€—
πŸ€— Dataset Train/Test Split

The honest train/test split β€” randomness you can pin with a seed

Hugging Face πŸ€—
πŸ€— Dataset Take

Keep the first N rows β€” your streaming dataset's best friend

Hugging Face πŸ€—
πŸ€— Dataset To Data List

Materialize a dataset as a Data List β€” the shape the loader's rows output promises

Hugging Face πŸ€—
πŸ€— Dataset To LIST

Turn a dataset into one flat Python LIST that every Basic node can eat

Hugging Face πŸ€—
πŸ€— Dataset Unique

What values actually exist in this column? Ask once, get a Data List

Hugging Face πŸ€—
πŸ€— Dataset Loader

Pull Hugging Face datasets into ComfyUI β€” no API key, no terminal Python

Hugging Face πŸ€—
Readme

Hugging Face Dataset Loader

License: GPL-3.0 Python 3.10+ ComfyUI

A custom ComfyUI node that loads a Hugging Face dataset and makes it available inside ComfyUI for further processing.

Every node ships in-app documentation and parameter tooltips β€” hover a node and open its help (info) panel, or view the node's page in the Node Library β€” and ready-made example workflows are available from the template browser (Workflow β†’ Browse Templates).

It loads a dataset from the Hugging Face Hub or from local/remote files using the datasets library and exposes it in two forms:

  • dataset β€” the raw datasets.Dataset object of the selected split (a lazy datasets.IterableDataset when streaming is on), handed to ComfyUI as an opaque HUGGINGFACE_DATASET value.
  • rows β€” a ComfyUI Data List of row dicts (one dict per row), so every record can be processed further with generic data-handling nodes, for example from the Basic data handling node pack (convert to a ComfyUI LIST or Data List, access fields, filter, ...).

The pack also ships a set of processing nodes that consume the dataset output and expose the data-wrangling methods of the datasets library (shuffling, filtering, column selection, ...) as graph nodes, so a dataset can be shaped before its rows are materialized β€” see Dataset nodes.

Requirements

  • ComfyUI

  • Python package datasets (Hugging Face). It is installed automatically when you install this pack through ComfyUI-Manager, the Comfy Registry or a pip install of the repo β€” there is nothing extra to do. The nodes import it lazily, so ComfyUI still starts even if it is missing; you only get a clear install message when you actually try to load a dataset. When you installed from a bare git clone without an installer, run once:

    pip install -r requirements.txt   # or: pip install datasets
    

[!NOTE] JPEG XL images. If you want to load datasets whose images are stored as JPEG XL (.jxl), make sure the Pillow JPEG XL plugin is installed as well.

pip install pillow-jxl-plugin

The plugin is optional and you must install it yourself: the node starts and runs fine without it, only the JPEG XL images then fail to decode. The ComfyUI startup log tells you whether it is active β€” it prints JPEG XL image support: enabled when the plugin is installed and JPEG XL image support: disabled otherwise.

Quickstart

Recommended Installation (ComfyUI-Manager or the Comfy Registry)

  1. Install ComfyUI-Manager (or use the Registry tab in ComfyUI Desktop / comfy node install).
  2. Look up the "Hugging Face dataset" extension and install it.
  3. Restart ComfyUI.

The datasets dependency is installed for you automatically.

Alternative (Manual Installation)

  1. Install ComfyUI.
  2. Clone this repository under ComfyUI/custom_nodes:
    git clone https://github.com/StableLlama/ComfyUI-huggingface_dataset.git
    
  3. Install the runtime dependencies into ComfyUI's Python environment:
    pip install -r requirements.txt
    
  4. Restart ComfyUI.

Node

Load Hugging Face Dataset

Loads a dataset and outputs the loaded dataset plus a rows Data List.

| Input | Type | Default | Description | | --- | --- | --- | --- | | path | STRING | "" | Hub dataset id (e.g. stanfordnlp/imdb) or a local/remote file or directory (see Sources). | | loader | STRING | auto | auto (infer from file extension), hub, or one of csv, json, parquet, arrow, text. | | split | COMBO | train | Split to load. A dropdown listing the splits of the selected source (see Split selection). | | config | STRING | "" | Config/subset name for Hub datasets that have several configs (e.g. nyu-mll/glue + config mrpc). | | revision | STRING | "" | Optional Hub revision: tag, branch name, or commit hash. | | streaming | BOOLEAN | False | Load the split as a lazy IterableDataset instead of downloading/caching it fully. | | limit | INT | -1 | Maximum number of rows to materialize into rows. -1 = all rows. |

| Output | Type | Description | | --- | --- | --- | | dataset | HUGGINGFACE_DATASET | The raw datasets.Dataset of the selected split (or an IterableDataset with streaming on). | | rows | * (Data List) | List of row dicts (one dict per row), capped by limit. |

Split selection

The split dropdown is filled with the splits the selected dataset actually exposes:

  • it refreshes automatically when you change path, config, revision or loader, and there is also a Refresh splits button to re-query manually;
  • a sensible default is picked (in the order train β†’ validation β†’ test) when the dataset has no train split - e.g. a dataset that only ships a test split selects test automatically;
  • listing the splits requires the datasets package and (for Hub datasets) network access. When the splits cannot be determined (offline, or a multi-config dataset without a config), the dropdown falls back to train/test/validation and the node validates the split when it runs.
  • datasets slicing syntax is still honoured for values that come from an older workflow, e.g. a stored train[:100] or train[:10%] continues to work.

Streaming vs. full load

With streaming off (the default) datasets downloads and caches the whole split before the first limit rows are materialized into the rows Data List - so limit caps the size of the returned rows, but not the download. With streaming on, the node loads the split as a lazy datasets.IterableDataset instead: nothing is downloaded until it is iterated, which makes it a memory/bandwidth-friendly way to feed only the first limit rows of a very large dataset into the graph. Note the dataset output is then an IterableDataset (no len(), single-pass) rather than a datasets.Dataset.

Sources

  • Hugging Face Hub β€” pass the repository id (namespace/name) as path, e.g. stanfordnlp/imdb, nyu-mll/glue (with config), or HuggingFaceFW/fineweb (with a revision). Leave loader at auto.
  • Local/remote files β€” csv, tsv, json, jsonl, parquet, arrow, txt. With loader = auto the format is inferred from the file extension, otherwise pick the matching file builder explicitly. Globs (e.g. data/*.parquet) and remote https:///hf:// URLs work too.
  • Local dataset directory β€” a directory in Hugging Face dataset format.

Example workflows

Load the IMDb reviews stanfordnlp/imdb dataset and count its rows with the Basic data handling pack (Basic β†’ Data List β†’ length):

flowchart LR
    A[Load Hugging Face Dataset<br/>path=stanfordnlp/imdb<br/>split=train] -->|rows| B[Data List β†’ length]

Take the review text of one specific row β€” rows is a Data List of row dicts, so first pull a row, then read its text field:

flowchart LR
    A[Load Hugging Face Dataset] -->|rows| B[Data List β†’ get item<br/>index=0]
    B --> C[DICT β†’ get<br/>key=text]

[!TIP] rows is already a ComfyUI Data List β€” see Working with Basic data handling for many more ways to slice, map and consume it with the Basic data handling pack.

Dataset nodes

The loader's dataset output is an opaque HUGGINGFACE_DATASET value. The processing nodes below take it as input, work on the underlying datasets object and hand the result back as a new HUGGINGFACE_DATASET, so transforms chain together (e.g. Shuffle β†’ Filter β†’ Map Column β†’ Select Columns). Nodes that only exist on a fully-loaded datasets.Dataset β€” marked loaded only below β€” raise a clear error when given a streaming dataset instead of a cryptic traceback: disable streaming on the loader for those.

Transform nodes

| Node | What it does | Streaming? | | --- | --- | --- | | πŸ€— Dataset Shuffle | Randomly reorders the rows (shuffle(seed)). | βœ… | | πŸ€— Dataset Skip | Drops the first n rows (skip(n)). | βœ… | | πŸ€— Dataset Take | Keeps only the first n rows (take(n)). | βœ… | | πŸ€— Dataset Sort | Sorts the rows by a column (sort). | loaded only | | πŸ€— Dataset Shard | Keeps shard index of the dataset split into num_shards (shard). | βœ… | | πŸ€— Dataset Select Rows | Keeps rows by indices β€” comma-separated 0,2,4 and/or slices 0:100, 0:100:2 (select). | loaded only | | πŸ€— Dataset Select Columns | Keeps only the comma-separated columns (select_columns). | βœ… | | πŸ€— Dataset Remove Columns | Removes the comma-separated columns (remove_columns). | βœ… | | πŸ€— Dataset Rename Column | Renames one column (rename_column). | βœ… | | πŸ€— Dataset Flatten | Expands nested columns into top-level ones (flatten). | loaded only | | πŸ€— Dataset Filter | Keeps rows whose column satisfies an operator against value (filter). | βœ… | | πŸ€— Dataset Map Column | Adds/replaces column per row from a constant, a copy of another column, or the row index (map). | βœ… | | πŸ€— Dataset Train/Test Split | Randomly splits into train and test outputs (train_test_split). | loaded only | | πŸ€— Dataset Unique | Returns the unique column values as a Data List (unique). | loaded only |

Filter operators: ==, !=, <, <=, >, >= (the value text is coerced to the column's type, so numeric columns compare with plain numbers), contains / not contains / starts with / ends with, in / not in (comma-separated value list), and is null / is not null.

Conversion nodes (LIST and Data List)

The "Basic data handling" pack distinguishes two list shapes, and either kind of dataset (fully-loaded or streaming) can be turned into both:

| Node | Output | Description | | --- | --- | --- | | πŸ€— Dataset To LIST | LIST | One Python list value of all rows (a LIST as Basic data handling defines it, so it feeds LIST β†’ length, LIST β†’ get item, ...). With a column set, the list holds that column's values instead of row dicts. | | πŸ€— Dataset To Data List | * (Data List) | The same rows exposed as a ComfyUI Data List (like the loader's rows output): Basic Data List nodes receive the whole list in one call, other nodes run once per row. |

Both honour a limit widget (-1 = all rows) β€” handy for pulling only the first rows of a large streaming dataset.

Example workflow

Load IMDb, keep only positive reviews (label == 1), add a row index and feed the result to a Basic-data-handling Data List:

flowchart LR
    A[Load Hugging Face Dataset<br/>path=stanfordnlp/imdb<br/>split=train] -->|dataset| B[πŸ€— Dataset Filter<br/>column=label operator== value=1]
    B -->|dataset| C[πŸ€— Dataset Map Column<br/>column=row_id operation=row index]
    C -->|dataset| D[πŸ€— Dataset To Data List]
    D -->|rows| E[Data List β†’ length]

Working with Basic data handling

The Basic data handling pack provides everyday data nodes β€” lists, dicts, strings, maths, flow control and more. These Hugging Face nodes hand data to it in the two shapes it already understands, so rows can be counted, inspected, mapped and consumed with almost no manual conversion:

  • * Data List β€” one item per row: the loader's rows output, πŸ€— Dataset To Data List, and the values output of πŸ€— Dataset Unique.
  • LIST β€” one Python list value: πŸ€— Dataset To LIST.

Rows are plain dicts keyed by the dataset's column names. Values are converted to plain Python (numpy scalars/arrays are handled for you); objects such as images or audio pass through unchanged. Install the Basic pack from ComfyUI-Manager or the registry β€” its nodes appear under the Basic/… menus (Basic/Data List, Basic/LIST, Basic/DICT, Basic/STRING, Basic/cast, ...).

Whole-list nodes vs. per-row mapping

A Data List can be wired into Basic nodes in two ways:

  • Whole-list nodes (Basic/Data List and Basic/LIST) receive the whole list in one call β€” e.g. length, count, first, get item, convert to LIST, convert to Data List.
  • Per-row mapping β€” connect the Data List into any other Basic node (a Basic/DICT, Basic/STRING or Basic/cast node). ComfyUI then runs that node once for every row, and its output is a new Data List with one result per row. This is the idiomatic way to transform every record in one go.

Recipes

Count the loaded rows, and read the review text of the first row:

flowchart LR
    A[Load Hugging Face Dataset<br/>path=stanfordnlp/imdb] -->|rows| L[Data List β†’ length]
    A -->|rows| G[Data List β†’ get item<br/>index=0]
    G --> T[DICT β†’ get<br/>key=text]

Map every row to one field β€” pull text out of each row with a per-row DICT β†’ get, then join the results into a single string:

flowchart LR
    A[Load Hugging Face Dataset<br/>limit=100] -->|rows| G[DICT β†’ get<br/>key=text]
    G --> J[STRING β†’ join (from data list)<br/>separator= ---]
  • Turn whole rows into text β€” rows β†’ Basic/cast β†’ to STRING, then use any Basic/STRING node.
  • One column of all rows as a LIST β€” πŸ€— Dataset To LIST (column=text) β†’ Basic/LIST β†’ length / get item / first, or Basic/LIST β†’ convert to Data List to switch back to per-row processing.
  • Shape before materialising β€” build the rows you want with the transform nodes first (πŸ€— Dataset Filter, Map Column, Select Columns, ...), then feed πŸ€— Dataset To Data List / To LIST into Basic.
  • Unique values β€” πŸ€— Dataset Unique (column=label) β†’ Basic/Data List β†’ length, or Basic/Data List β†’ convert to LIST.

These patterns work for every Data List the pack produces β€” the loader's rows as well as any πŸ€— Dataset To Data List fed from a transform chain β€” and the limit widget (or a Take / Skip) bounds how many rows are materialised.

Example workflow templates

The repository ships ready-made workflows under example_workflows/. Once the pack is installed they appear in ComfyUI's template browser (Workflow β†’ Browse Templates) under the Hugging Face dataset entry:

  • imdb_filter_to_data_list β€” load IMDb, keep only positive reviews and expose them as a Data List.
  • imdb_map_select_to_list β€” add a row index, keep the text column and export it as a LIST of strings.
  • imdb_streaming_skip_take β€” load IMDb in streaming mode, skip/take rows and materialize a Data List without a full download.

Development

# Lint & format
python -m ruff check .
python -m ruff format .

# Type check (strict)
python -m mypy src tests

# Tests (no network / no datasets required)
python -m pytest tests/

License

GPL-3.0