ComfyUI Node

Text Chunker

Breaking a long document into pieces you can actually process

By Excidos·Created 2 years ago·Updated 2 years ago· 67
Text Chunker
    • text_chunks
    text
    chunk_size1000
    chunk_method
    respect_word_boundariestrue

    Document Loader hands you a whole document as one giant string, and that's usually the problem, not the solution. Most text-consuming nodes - and any LLM you're piping that text into - have a limit on how much they can chew on at once. Text Chunker's job is to cut that big string down into a series of smaller, manageable pieces so the rest of your workflow can actually do something with them.

    How it works

    You give it text, a target chunk size, and whether that size is measured in words or characters. It walks through the string and slices it into pieces of roughly that size, with an option to avoid cutting a word in half at the boundary. The output is a batch of chunk strings rather than one - so whatever's downstream needs to expect a list, not a single value.

    The inputs and outputs that matter

    • text - a multiline string input. Normally this comes straight from Document Loader's parsed_text output, but it'll happily take any text your graph produces.
    • chunk_size - how big each piece should be, from 1 to 10,000, default 1000. What "1000" means depends entirely on the next field.
    • chunk_method - words or characters. Pick words and chunk_size counts words per chunk; pick characters and it counts raw characters instead. Words is generally the more predictable choice if you're feeding chunks to something token-sensitive downstream, since word count tracks token count more closely than raw character count does.
    • respect_word_boundaries - a boolean, on by default. When enabled, the chunker won't slice a word in half to hit the exact target size - it'll round to the nearest word boundary instead. Leave this on unless you have a specific reason to want hard character cutoffs; the README frames it as the setting for "more natural" division, and there's little downside to it.

    The output is text_chunks, a list of STRING values - one entry per chunk. It's also marked as an output node, so ComfyUI will run and surface it as a terminal result even without anything wired after it, useful for sanity-checking how your document got split before you build the rest of the pipeline.

    Installing it

    Via ComfyUI Manager: search ComfyUI-Documents, install, restart. Or by hand:

    cd ComfyUI/custom_nodes
    git clone https://github.com/Excidos/ComfyUI-Documents.git
    pip install -r ComfyUI-Documents/requirements.txt
    

    This node itself has no heavy dependency of its own - it's plain string manipulation, no PyMuPDF or python-docx needed unless you're also using the pack's document-loading nodes upstream, which most workflows will be.

    Where people get burned

    • The output is a list, and some nodes don't expect that. If you wire text_chunks into something built for a single string, you'll either get an error or, worse, it'll silently only grab the first chunk. Check that whatever you're connecting to is actually designed to iterate over a batch.
    • chunk_size isn't a hard cap when word boundaries are respected. With respect_word_boundaries on, a chunk can run slightly over your target size to avoid splitting mid-word - that's expected behavior, not a bug, but worth knowing if you're chunking against a strict downstream limit and need headroom.
    • Picking a single chunk out of the list is the next step, and it's the weak point of this pack right now. The obvious next node, Chunk Router, is flagged by the pack's own author as not currently working - see that node's page before you build a workflow around it.
    • This is a small, single-author pack rather than a heavily-used one, so if you hit an edge case with unusual whitespace or non-English text, it's worth testing on a short sample first rather than assuming it'll behave exactly like a more mature chunking library.
    Categorydocument_processing

    Inputs (4)

    NameTypeDefaultDescription
    textSTRING
    chunk_sizeINT10001–10000
    chunk_methodCOMBO2 options: words, characters
    respect_word_boundariesBOOLEANtrue

    Outputs (1)

    NameTypeDescription
    text_chunksSTRING