split HF dataset

split_dataset is the bridge between downloading a dataset and training on it. It takes the local_dir path that download_dataset produces, loads every data file it finds there, filters it, splits it into train/validation/test, and saves the result to disk as a Hugging Face DatasetDict. Its output - split_datapaths - is exactly what the pack's CausalLM_trainer wants in its split_datapaths input, so this node is the one that closes the loop on the whole LLM School pipeline.

The mechanics are the important part, because they tell you what data this pipeline is built for. The node walks the directory recursively, loads whatever it finds (csv, json, parquet, txt, xlsx, tsv, xml, and .arrow folders), concatenates everything into one dataset, then runs a filter with a hard assumption: every row must have an answers column with non-empty text. That's SQuAD-style QA data. A generic text corpus will crash on the filter with a KeyError (or silently drop nothing if the column is missing) - this node is a QA-data splitter, not a general-purpose one.

The split itself is a standard train_test_split chain: first carve off 1 - train_ratio, then split that remainder by val_ratio / (val_ratio + test_ratio). The saved output lands in <pack>/datasets/<savepath>_<timestamp>, so every run creates a fresh folder rather than overwriting - harmless, but your datasets directory will accumulate dated copies.

The inputs that matter

• local_dir - the path from download_dataset. Required, no default.
• train_ratio / val_ratio / test_ratio - defaults 0.8 / 0.1 / 0.1. If they don't sum to 1.0, the node silently rescales them and notes it in the log. So don't stress about arithmetic; 0.7/0.2/0.1 and 70/20/10 behave the same.
• savepath - the folder-name prefix; default split_dataset.

The outputs

Two strings. split_datapaths is the path to the saved split - wire it straight into CausalLM_trainer. log is a preview of the head of each split as a pandas table, which is genuinely useful for sanity-checking that your answers actually loaded before you burn GPU hours on a broken dataset.

Install

ComfyUI Manager, search "comfyui_LLM_schools", or:

cd ComfyUI/custom_nodes
git clone https://github.com/heshengtao/comfyui_LLM_schools

then restart. Deps: huggingface_hub, datasets, transformers, peft.

Common issues

• "The node never registers." The pack reads config.ini at import time; missing file → dataset.py fails to load and the loader swallows the error. Copy config.ini.example to config.ini in the pack folder and restart.
• KeyError on answers. Your dataset isn't SQuAD-shaped. Either switch to rajpurkar/squad_v2 (or any QA dataset with question/context/answers), or accept that this pack isn't built for your corpus.
• "No valid datasets found." The loader found files but none with a supported extension, or the directory is empty. The pack's download_dataset writes into <pack>/datasets, so if you changed cache_dir to an absolute path, make sure local_dir matches where the files actually went.

It's a focused, slightly opinionated node - but for QA-style fine-tuning it replaces a script you'd otherwise have to write, and it hands the trainer exactly the input format it expects.