Transformer Encoder Block

Every encoder-only transformer you've ever used - BERT, and by inheritance a good chunk of the embedding side of modern image-video pipelines - is a stack of these blocks. CdlTransformerEncoderBlock builds exactly one of them, cleanly and inspectably, which makes it the most useful teaching brick in ComfyDL's transformer family. Build a few and chain them together and you've recreated the encoder stack by hand instead of trusting some library to do it for you.

This is very much a "see the machinery" node, not a "get a better SDXL checkpoint" node. ComfyDL in general wraps the "Dive into Deep Learning" (d2l) textbook code into ComfyUI, and this is the block from the transformer chapter: multi-head self-attention, a residual-add-and-normalize, a position-wise feed-forward net, another add-and-normalize. Two sub-layers, both with the now-ubiquitous residual + LayerNorm pattern that lets gradients flow through a hundred stacked blocks without vanishing.

What the forward pass does

The block takes a sequence of vectors - shaped (batch_size, seq_len, num_hiddens) - and returns the same shape, transformed. First, multi-head self-attention lets every position attend to every other position: the input is projected into queries, keys, and values (use_bias toggles whether those projections have bias terms), split across num_heads heads, attended, and recombined. Then the result is added to the original input and layer-normalized. The same input runs through a two-layer ReLU feed-forward net (ffn_num_hiddens wide in the middle), and again added-and-normalized. Self-attention is how the model learns relationships between distant tokens without the sequential drift that plagues RNNs - every position can look at every other position in one step.

One constraint is enforced in code, and it's the classic gotcha: num_hiddens must be divisible by num_heads. Each head gets num_hiddens / num_heads dimensions, and if the division doesn't come out clean the node raises a ValueError before you even run anything. num_hiddens=8 with num_heads=4 (the defaults) works; num_hiddens=10 with num_heads=4 does not.

The knobs you'll touch

• num_hiddens - the model width. Must stay divisible by num_heads.
• ffn_num_hiddens - how wide the feed-forward middle layer is. Conventionally 4× num_hiddens (the defaults of 8 and 64 follow that).
• num_heads - parallel attention heads; each learns a different "view" of the sequence.
• dropout - regularization inside attention weights and the FFN.
• use_bias - bias on the attention projections. Off by default, matching modern practice; you'll rarely flip it.

The defaults are deliberately tiny (a block that would fit on a postage stamp), because the pack's target is understanding, not scale. Crank them up and it behaves the same, just slower and more parameter-hungry.

How it differs from the full Transformer Encoder

ComfyDL also ships Transformer Encoder, which is the whole stack: token embedding, positional encoding, and num_blks of these blocks already chained. So when do you want the single block instead? When you're doing the d2l exercise of assembling the stack yourself block by block (educational, and genuinely clarifying), or when you want a single block's output wired somewhere unusual. For everything else, the full encoder node is less work. Either way the block's model output is a standard cdlModel you can inspect with Model Info or run through Model Forward to watch shapes flow.

Installing ComfyDL

Same story as every node in this pack:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL ./ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart ComfyUI and find it under ComfyDL/NLP Models. Keep the folder named ComfyDL, and know that ComfyUI Manager may not list this pack in search yet (its registry PublisherId is blank in pyproject.toml), so the clone is the dependable route. Dependencies are only matplotlib, IPython, and matplotlib-inline.

ComfyDL has no real community discussion to lean on yet, so treat the repo's FUNCTIONS.md and the d2l transformer chapter as your documentation. If your block errors with a divisibility complaint, that's not a bug - it's the textbook's one invariant, enforced early, and now you know why it exists.