Nodes/ComfyUI-huggingface_dataset/πŸ€— Dataset Loader
ComfyUI Node

πŸ€— Dataset Loader

Pull Hugging Face datasets into ComfyUI β€” no API key, no terminal Python

By StableLlamaΒ·Created a day agoΒ·Updated a day agoΒ· 1
πŸ€— Dataset Loader
    • dataset
    • rows
    β—„pathβ–Ί
    β—„loaderautoβ–Ί
    β—„splittrainβ–Ί
    β—„configβ–Ί
    β—„revisionβ–Ί
    β—„streamingfalseβ–Ί
    β—„limit-1β–Ί

    Every ComfyUI tutorial you've seen ends the same way: text goes in, image comes out. This node runs in the opposite direction for a change - it drags a table into your graph. Load the stanfordnlp/imdb reviews, a folder of caption CSVs you're assembling for a LoRA run, or a huge corpus from the Hugging Face Hub, and your workflow suddenly knows how to read rows of text and labels, filter them, shuffle them, and hand them to whatever wants them. If your work has any data-prep flavor - building training sets, sampling a corpus, running eval batches - this is the doorway to doing it inside ComfyUI instead of bouncing back and forth to Python.

    What it actually does

    Load Hugging Face Dataset is a thin wrapper around Hugging Face's datasets library (datasets.load_dataset). Nothing about it is an API call: public Hub datasets download straight to a local cache over HTTP, no API key required, and no data leaves your machine afterward. The name "Loader" undersells it - you get two outputs from one node:

    • dataset - the raw datasets.Dataset object of the split you picked, passed through the graph as an opaque HUGGINGFACE_DATASET value. The other nodes in this pack (πŸ€— Dataset Filter, Shuffle, Take, …) consume it, and they chain, so you can shape rows before they ever become Python lists.
    • rows - a ready-made ComfyUI Data List of row dicts (one dict per row, keyed by your column names). This is the "instant gratification" output: wire it straight into generic data nodes from the companion Basic data handling pack to count rows, pull a field, or join values.

    Values are coerced to plain Python (numpy scalars handled for you); only things like image or audio objects pass through untouched.

    Inputs that matter

    Most of the widgets you'll actually touch:

    • path - the Hub repo id (stanfordnlp/imdb) or a local/remote file, glob, or dataset directory. With loader on auto the format is inferred from the file extension, so a .csv, .jsonl, .parquet or .arrow just works.
    • split - a dropdown, and a nice piece of UX: the pack queries the Hub (or your local source) for the dataset's real split names and refreshes the list when you change path/config/revision. If it can't reach the network it falls back to train/test/validation and validates on run. Old slicing syntax like train[:100] in stored workflows still works.
    • config - only for multi-config Hub datasets, where you name the subset (e.g. nyu-mll/glue needs config = mrpc).
    • streaming and limit - see below; these two decide how much of your disk and RAM the node eats.
    • revision - pin a tag, branch, or commit hash if you want reproducibility instead of "whatever's latest."

    Streaming vs. full load - read this before a big dataset

    With streaming off (default), datasets downloads and caches the entire split before the first row is materialized. limit caps how many rows land in rows, but not the download - this is the classic trap. Point it at a 100 GB corpus expecting to peek at ten rows and you'll wait for 100 GB.

    With streaming on, you get a lazy datasets.IterableDataset instead: data files are fetched only as rows are iterated, so limit actually means something. The catch is the dataset output is then single-pass with no len(). The nodes that need a fully materialized dataset (Sort, Select Rows, Flatten, Train/Test Split, Unique) will tell you so with a clear error - just flip streaming off for those.

    Installing it

    ComfyUI-Manager: search "Hugging Face dataset" (repo StableLlama/ComfyUI-huggingface_dataset) and install. Manual route:

    cd ComfyUI/custom_nodes
    git clone https://github.com/StableLlama/ComfyUI-huggingface_dataset
    pip install -r requirements.txt   # the only runtime dep is `datasets`
    

    Then restart ComfyUI. The datasets package is installed automatically by Manager; the pack lazy-imports it anyway, so ComfyUI boots cleanly either way and you only get a "please install datasets" message when you first load something. The only optional extra is pillow-jxl-plugin for image datasets that contain JPEG XL files - the startup log says whether JXL support is on.

    Common issues

    • Whole split downloads before you see rows - you forgot streaming is off. See above.
    • Split dropdown is stuck on defaults - offline, or a multi-config dataset with no config filled in. Fill config or just type the split and let the node validate.
    • The wrong Python - pip install must target ComfyUI's own environment (its venv/portable Python), not your system one. Manager handles this; bare-clone installs are where people trip.
    • rows looks weird when you connect it - it's a Data List, not a LIST. Both are real; the Basic data handling pack distinguishes them. If you want one flat Python list, use the πŸ€— Dataset To LIST node.
    CategoryHugging Face πŸ€—

    Inputs (7)

    NameTypeDefaultDescription
    pathSTRINGHub dataset id (e.g. stanfordnlp/imdb), or a local/remote file, glob or dataset directory.
    loaderCOMBOautoHow to read the source: 'auto' infers from the file extension, 'hub' loads a Hub id, or pick a file builder (csv/json/parquet/arrow/text).
    splitCOMBOtrainSplit to load. The dropdown lists the splits of the selected source; slicing like 'train[:100]' still works.
    configSTRINGConfig/subset name for Hub datasets that have several configs (e.g. glue + mrpc).
    revisionSTRINGOptional Hub revision: tag, branch name, or commit hash.
    streamingBOOLEANfalseLoad the split lazily as an IterableDataset instead of downloading/caching it fully.
    limitINT-1-1–2147483647Maximum number of rows to materialize into the 'rows' Data List; -1 = all.

    Outputs (2)

    NameTypeDescription
    datasetHUGGINGFACE_DATASETRaw datasets.Dataset of the split (a streaming IterableDataset with 'streaming' on); feed it into the πŸ€— dataset nodes.
    rows*ComfyUI Data List of row dicts (one dict per row), capped by 'limit'.