Dot-Product Attention

Every modern transformer - every LLM, every diffusion text encoder, the "attention is all you need" line that started it - runs on one piece of math, and this node is that piece exposed as a building block: scaled dot-product attention. If you've ever wanted to actually look at the mechanism instead of trusting the diagram, ComfyDL's CdlDotProductAttention is where you do it. It's the educational unpacking of the thing you've been using blind.

The node belongs to ComfyDL, the pack that ports the Dive into Deep Learning (d2l) textbook into a node graph. It's not for image generation workflows - it's for building and poking at attention layers with your own tensors.

How it works

The recipe is three lines of tensor math. Take your queries and dot them against every key (a batched matrix multiply), divide by the square root of the key dimension so the softmax doesn't saturate, push the result through a softmax to get attention weights, then use those weights to take a weighted average of the values. In code form it's softmax(queries · keysᵀ / √d) · values.

There are two details worth knowing. First, the "scaled" part - the /√d - matters more than it looks: without it, long sequences push the dot products into regions where softmax gradients vanish. Second, the softmax supports a valid_lens mask so you can ignore padding positions. The node keeps the attention_weights it computed after each forward pass, which is gold for a learner: run one forward, then visualize which keys each query actually attends to.

Inputs and outputs that matter

There's exactly one real input, which tells you how clean this little layer is:

• dropout (default 0, max 0.9) - applied to the attention weights before they meet the values. Zero is right for first experiments; raise it when you're training for real and want regularization.

The output is a single model (cdlModel). It expects (queries, keys, values, valid_lens) with queries/keys/values shaped (batch_size, seq_len, num_hiddens) - batch-first, same convention as the rest of ComfyDL's attention nodes. It returns values re-weighted by attention. Wire the model socket into a CdlModelForward node from the Model Utils category to feed it tensors and read the output.

Where it fits

Dot-product attention is also what ComfyDL's CdlMultiHeadAttention uses internally per head, so in practice most people reach for that node instead. This standalone version earns its keep when you want to (a) demonstrate or teach the core mechanism in isolation, or (b) hand-build an attention path where queries, keys, and values genuinely have matching widths and you don't need the multi-head machinery.

Installing ComfyDL

Clone the pack and install its (small) requirements:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL ./ComfyDL
pip install -r ./ComfyDL/requirements.txt

Then restart ComfyUI. Dependencies are just matplotlib, IPython, and matplotlib-inline - no checkpoints, no heavy downloads for this node. Manager users can search "ComfyDL", but the pack is young and may not be listed; cloning is the guaranteed route.

Gotchas

Batch-first shapes again: this node wants (batch, seq, hiddens), so don't feed it time-major (steps, batch, hiddens) data out of habit. And don't skip the dropout thinking it's cosmetic - at 0 it genuinely is for inspection, but the scaling term (√d) is always on, and if your d is huge you may still see flat attention until you check attention_weights.