Text Chunker
Breaking a long document into pieces you can actually process
- text_chunks
Document Loader hands you a whole document as one giant string, and that's usually the problem, not the solution. Most text-consuming nodes - and any LLM you're piping that text into - have a limit on how much they can chew on at once. Text Chunker's job is to cut that big string down into a series of smaller, manageable pieces so the rest of your workflow can actually do something with them.
How it works
You give it text, a target chunk size, and whether that size is measured in words or characters. It walks through the string and slices it into pieces of roughly that size, with an option to avoid cutting a word in half at the boundary. The output is a batch of chunk strings rather than one - so whatever's downstream needs to expect a list, not a single value.
The inputs and outputs that matter
text- a multiline string input. Normally this comes straight from Document Loader'sparsed_textoutput, but it'll happily take any text your graph produces.chunk_size- how big each piece should be, from 1 to 10,000, default 1000. What "1000" means depends entirely on the next field.chunk_method-wordsorcharacters. Pickwordsandchunk_sizecounts words per chunk; pickcharactersand it counts raw characters instead. Words is generally the more predictable choice if you're feeding chunks to something token-sensitive downstream, since word count tracks token count more closely than raw character count does.respect_word_boundaries- a boolean, on by default. When enabled, the chunker won't slice a word in half to hit the exact target size - it'll round to the nearest word boundary instead. Leave this on unless you have a specific reason to want hard character cutoffs; the README frames it as the setting for "more natural" division, and there's little downside to it.
The output is text_chunks, a list of STRING values - one entry per chunk. It's also marked as an output node, so ComfyUI will run and surface it as a terminal result even without anything wired after it, useful for sanity-checking how your document got split before you build the rest of the pipeline.
Installing it
Via ComfyUI Manager: search ComfyUI-Documents, install, restart. Or by hand:
cd ComfyUI/custom_nodes
git clone https://github.com/Excidos/ComfyUI-Documents.git
pip install -r ComfyUI-Documents/requirements.txt
This node itself has no heavy dependency of its own - it's plain string manipulation, no PyMuPDF or python-docx needed unless you're also using the pack's document-loading nodes upstream, which most workflows will be.
Where people get burned
- The output is a list, and some nodes don't expect that. If you wire
text_chunksinto something built for a single string, you'll either get an error or, worse, it'll silently only grab the first chunk. Check that whatever you're connecting to is actually designed to iterate over a batch. chunk_sizeisn't a hard cap when word boundaries are respected. Withrespect_word_boundarieson, a chunk can run slightly over your target size to avoid splitting mid-word - that's expected behavior, not a bug, but worth knowing if you're chunking against a strict downstream limit and need headroom.- Picking a single chunk out of the list is the next step, and it's the weak point of this pack right now. The obvious next node, Chunk Router, is flagged by the pack's own author as not currently working - see that node's page before you build a workflow around it.
- This is a small, single-author pack rather than a heavily-used one, so if you hit an edge case with unusual whitespace or non-English text, it's worth testing on a short sample first rather than assuming it'll behave exactly like a more mature chunking library.
Inputs (4)
| Name | Type | Default | Description |
|---|---|---|---|
| text | STRING | — | |
| chunk_size | INT | 10001–10000 | — |
| chunk_method | COMBO | 2 options: words, characters | |
| respect_word_boundaries | BOOLEAN | true | — |
Outputs (1)
| Name | Type | Description |
|---|---|---|
| text_chunks | STRING | — |