🤗 Dataset Shard

🤗 Dataset Shard is the node you reach for when one graph run over the whole dataset is too slow, too memory-heavy, or just something you want to split across machines. It divides a dataset into num_shards roughly equal pieces and keeps the one you ask for - so the same workflow can run four times, once per shard, and each run handles a quarter of the data.

How it works

It wraps datasets.Dataset.shard (and its streaming counterpart). The inputs are the mechanism:

• num_shards - how many roughly equal pieces to split into (default 2).
• index - which shard to keep, 0 to num_shards - 1 (default 0). One node, one shard: to keep all four you'd have four of these nodes in a row, or one per graph run.
• contiguous - the quiet choice that changes everything. On (default) keeps contiguous blocks: rows 0–24, then 25–49, and so on. Off interleaves: shard 0 gets rows 0, N, 2N…, shard 1 gets 1, N+1, … This is the one to use if the data isn't pre-shuffled and you want each shard to be a fair sample of the whole corpus rather than a contiguous slice.

It works on fully-loaded and streaming datasets, which is notable - most of this pack's loaded-only nodes can't handle lazy streams, but sharding a stream is a genuinely streaming-safe operation (you're partitioning, not reordering globally). That makes Shard the natural way to process a big corpus in parallel batches without ever loading it whole.

Where it earns its keep

Real use cases: splitting a job across multiple ComfyUI instances or GPUs (each picks an index), bounding a single run's memory by processing one shard at a time, or making sure a "sample of the data" actually spans the whole dataset via contiguous = false after a shuffle.

Gotcha territory is small but real: keep index inside 0..num_shards-1, and if you change num_shards between runs, the same index no longer refers to the same rows - shards aren't stable across different shard counts. For most single-run slicing, plain Take/Skip are simpler; Shard is the parallel-work tool.

Installing it

Part of StableLlama/ComfyUI-huggingface_dataset. ComfyUI-Manager → search "Hugging Face dataset", or:

cd ComfyUI/custom_nodes
git clone https://github.com/StableLlama/ComfyUI-huggingface_dataset
pip install -r requirements.txt

Restart ComfyUI. Only runtime dependency is datasets, installed by Manager.