Additive Attention

Additive attention (a.k.a. Bahdanau attention) is the older, more flexible cousin of the scaled dot-product kind that every modern transformer runs. Dot-product attention quietly assumes your query and key vectors live in the same dimensional space - it literally multiplies them. Additive attention instead learns a tiny one-layer network that scores how well a query matches a key, which means it works even when the two come from different models with different widths. In ComfyDL this node hands you that layer as a ready-made cdlModel.

If you're here, you're almost certainly following the Dive into Deep Learning (d2l) path through sequence-to-sequence models with attention, where additive attention shows up inside the decoder for machine translation. ComfyDL is that whole textbook ported into nodes, and this is one of its building blocks - you're not going to wire it into a diffusion pipeline, you're going to wire it into an encoder-decoder experiment and actually see the attention weights.

How it works

Under the hood the node builds the classic Bahdanau score function. Queries and keys are each projected through a learned linear layer (W_q, W_k) into the same num_hiddens-wide space, added together with broadcasting, pushed through a tanh, then squeezed down to a single score per pair by a third tiny projection (w_v). Those raw scores go through a masked softmax to become attention weights, which are then used to take a weighted average of the values.

The upshot: instead of "must be the same size to multiply," you get "learn me a compatibility function." That's the whole point of additive attention and the reason it was invented in the first place - the decoder state and the encoder hidden states in Bahdanau et al.'s original translation model didn't have to line up dimensionally.

Inputs and outputs that matter

• num_hiddens (default 8) - the width of the learned score function. In practice set it to the width of your query/embedding space; ComfyDL's default of 8 matches the tiny demo sizes the d2l workflows use.
• dropout (default 0, max 0.9) - applied to the attention weights before they're multiplied with values. Leave it at 0 when you're just inspecting the mechanism; raise it once you're training seriously.

The single output is model (type cdlModel). It expects to be called with (queries, keys, values, valid_lens) tensors shaped (batch_size, seq_len, num_hiddens), and returns the attention-weighted values. Wire the model output into a ComfyDL Model Utils node like CdlModelForward to run a forward pass on hand-made tensors, or into the seq2seq/decoder assemblies if you're building one by hand. Because it's an nn.Module underneath, it also records attention_weights after a forward - that's the part you inspect to "see" what the model is looking at.

Installing ComfyDL

You need ComfyUI first. Then clone into custom_nodes, install the (very light) requirements, and restart:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL ./ComfyDL
pip install -r ./ComfyDL/requirements.txt

That's it - ComfyDL's dependencies are just matplotlib, IPython, and matplotlib-inline; there are no model checkpoints to download for this node. Restart ComfyUI and the NLP Models category will appear. If you use ComfyUI Manager, search "ComfyDL", but note this is a young educational pack that may not be in the Manager registry yet - the clone route above always works.

Gotchas

The classic failure is a shape mismatch at forward time, and it's usually self-inflicted: these attention layers want batch-first tensors, (batch, seq_len, hiddens), while a lot of textbook RNN code is time-major (steps, batch, hiddens). Feed it the wrong layout and you'll get a matmul-size error with a message that doesn't obviously say "transpose me." Also remember num_hiddens here is a score-function width you choose, not something inferred - if it disagrees with your tensor widths, the forward pass will tell you loudly.