Add & Norm

Open the diagram of any Transformer encoder block and you'll see the same sandwich twice: attention, then a box labeled "Add & Norm", then a feed-forward layer, then another "Add & Norm". That box is doing two of the most quietly important things in deep learning - a residual connection and layer normalization - and ComfyDL's CdlAddNorm is that box as a standalone node.

If you're hand-building Transformer pieces from ComfyDL's Dive into Deep Learning (d2l) building blocks rather than grabbing a whole encoder, this is the node you slot after CdlMultiHeadAttention or CdlPositionWiseFFN.

How it works

The math is deceptively short: LayerNorm(X + dropout(Y)). Two tensors come in - X, the layer's original input, and Y, that same input after it went through attention or a feed-forward net. The node adds them back together (the residual connection, which gives gradients a clean highway back through the network instead of making them fight their way through stacked nonlinearities), drops out a bit of the Y branch, and normalizes the sum with a LayerNorm so activations stay in a healthy range.

That's the whole layer. It's tiny, and its whole job is making the blocks around it trainable.

Inputs and outputs that matter

• norm_shape (default 16) - the feature dimension the LayerNorm normalizes over. This must equal the last dimension of the tensors you feed it, which in practice means your embedding/model width. If you're running width-8 embeddings and leave the default 16, your first forward pass will throw a shape error.
• dropout (default 0, max 0.9) - applied to the Y (residual branch) input before it's added to X, not after normalization.

The model output is a cdlModel whose forward takes (X, Y) and returns a tensor the same shape as X. That two-input signature is the thing to remember: AddNorm doesn't take a single tensor, it takes the pre-layer tensor and the post-layer tensor, because that's what "add the residual" means. Wire the output into a CdlModelForward node to check the shapes, or chain it after an attention layer when assembling a block manually.

Installing ComfyDL

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL ./ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart ComfyUI and look under NLP Models. Dependencies are just matplotlib, IPython, and matplotlib-inline - nothing heavy, no downloads. Manager users can search "ComfyDL", though this young pack may not be in the registry yet; the clone command always works.

Gotchas

norm_shape mismatch is the beginner trap - it has to match your model width exactly, and because the default is 16 you'll hit it the moment you test with width-8 tensors. Also, dropout here sits on the residual branch: with dropout at 0 (the default) the node is literally just LayerNorm(X + Y), which is perfect for inspection. One more: this node is only the Add & Norm sublayer. People sometimes grab it expecting a full block - it's one ingredient in the sandwich, not the sandwich.