Position-Wise FFN

The Transformer is famous for attention, but only half of each block is attention. The other half is a plain, boring feed-forward network - two dense layers with a ReLU between them - applied to every token position independently. That's the "Position-Wise FFN," and ComfyDL ships it as a standalone CdlPositionWiseFFN node so you can assemble encoder blocks by hand instead of trusting a black-box encoder.

It's the least glamorous ingredient in a Transformer, and that's sort of the point: it's where the model gets to think in a richer space between attention passes. Attention mixes information across tokens; the FFN transforms each token's representation independently and gives the block most of its actual parameters.

How it works

The structure is exactly what you'd write by hand: dense1 expands the input to a wider hidden space, ReLU chops the negatives, dense2 projects back down to the model width. Two details make it "position-wise" rather than a normal MLP: it's applied identically to each row of the batch (the weights are shared across all sequence positions), and the layers are LazyLinear, meaning the input dimension isn't fixed at build time - it's inferred the first time you run a forward pass, so the node doesn't need to know your embedding width up front.

Inputs and outputs that matter

• ffn_num_hiddens (default 64) - the width of the inner expansion. Conventionally 4× the model width (a width-8 model with ffn_num_hiddens 32 is the textbook ratio), though the default of 64 is loose and fine for demos.
• ffn_num_outputs (default 16) - the width of the output projection. Set it back to your model width (num_hiddens) so the block's output dimension matches what the next layer expects.

The model output (a cdlModel) takes input of shape (batch, seq_len, whatever) and returns (batch, seq_len, ffn_num_outputs). In a hand-built Transformer block, chain: attention → CdlAddNorm → this FFN → another CdlAddNorm. Feed it tensors through a CdlModelForward node to verify shapes before wiring the whole block.

Installing ComfyDL

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL ./ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart ComfyUI and the node appears under NLP Models. Requirements are minimal (matplotlib, IPython, matplotlib-inline), no downloads needed. ComfyUI Manager may or may not list this young educational pack - if search for "ComfyDL" comes up empty, the clone command above is your answer.

Gotchas

The main thing people get wrong is the dimensionality handshake at the block level: if ffn_num_outputs doesn't match your model width, your next CdlAddNorm (which normalizes over that width) will throw. Set ffn_num_outputs to your embedding width and ffn_num_hiddens to roughly 4× it, and the plumbing works. And worth repeating: this node transforms one token at a time - it deliberately has no idea about your sequence structure, so don't expect it to catch ordering mistakes made upstream.