Get Tokens & Segments

CdlGetTokensAndSegments builds the exact input structure BERT expects before it can even look at a sentence: special tokens wrapping the text, and a per-token segment ID saying which sentence each token belongs to. BERT doesn't just read a string - it reads [CLS] + tokens of sentence A + [SEP] + tokens of sentence B + [SEP], plus an ID per position telling it whether that token came from the first or second sentence. This node is the straight port of d2l's get_tokens_and_segments, which is the piece that turns "here's some text" into "here's BERT-ready input."

How it works

You give it the tokens of one sentence (A) and optionally a second (B). It produces:

• tokens - ['<cls>'] + tokens_a + ['<sep>'], and if you supplied B, + tokens_b + ['<sep>']. The <cls> marker is where BERT will accumulate the sentence-level representation (used for classification and next-sentence prediction); <sep> separates sentences.
• segments - a parallel list of IDs: 0 for every position belonging to the first sentence (including the leading <cls>), 1 for the second-sentence positions (including its trailing <sep>). BERT uses this to tell the two sentences apart when doing next-sentence prediction.

Both come back as comma-joined strings, so you can read them directly and see the structure you just built. The default input - "the,quick,brown,fox" for A and "jumps,over" for B - is a worked example you can eyeball before substituting real text.

Inputs and output

• tokens_a - required, comma-separated tokens for sentence A.
• tokens_b - optional, comma-separated tokens for sentence B. Leave it empty and you get single-sentence BERT input (<cls> … <sep> only).

Outputs:

• tokens - the marked-up token list as a comma-joined STRING.
• segments - the segment IDs as a comma-joined STRING.

Where it fits

It's the first step of ComfyDL's NLP pipeline: tokenize → build tokens & segments → encode to vocab indices → feed a BERT-style model. It pairs with CdlTokenize upstream and the vocab-encoding nodes downstream. If you're following along with the BERT chapter, this is the node that makes the mysterious [CLS]/[SEP] ceremony visible instead of buried in a tokenizer class.

Installing it

It ships with ComfyDL, one install for the whole pack:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart ComfyUI, or search "ComfyDL" in ComfyUI Manager.

Gotchas

• Input is comma-separated, not space-separated. The defaults use commas on purpose; paste in space-separated words and the node treats the whole line as one giant token. This is the single most common mistake with this node.
• Empty tokens_b is fine (single-sentence mode), but a blank tokens_a gives you just ['<cls>', '<sep>'] - the markers survive, the content doesn't.
• The output segment IDs are STRINGs, not a tensor. They're meant to be read and then encoded downstream, so don't expect to wire them straight into arithmetic.