NNT Define Rotary Positional Embedding

Transformers have no built-in sense of order - feed them a sentence and they can't tell you which word came first. Positional encodings fix that, and rotary positional embeddings (RoPE) are the fix that won. Every modern Llama/GPT-class model uses them. This node is the Neural Network Toolkit's way of adding RoPE to a layer stack.

The inputs

• dim (default 512) - the embedding dimension. It should match your model's embedding width.
• base (default 10000) - the frequency base. This is the famous RoPE knob. Higher base = longer effective context for the same sequence length; models that want to extend context (like scaling tricks) bump it up. 10000 is the original default.
• max_freq (default 10) - caps the highest frequency in the rotation schedule.
• interpolation_factor (default 1) - the position-interpolation dial. Values above 1 "stretch" the rotation schedule across more positions, which is how you extend context length without retraining. Set it to, say, 2 and your model acts like positions are further apart - the same trick the context-extension literature calls PI.

Output: LIST - the growing layer stack.

How it works

RoPE's trick: instead of adding a position vector to the embeddings (the old way), it rotates pairs of embedding dimensions by an angle that depends on position. Each dimension pair gets its own rotation frequency (1 / base^(2i/dim)), and position t rotates the vector by t times that frequency. The rotation carries relative-position information implicitly - attention between tokens naturally ends up depending on how far apart they are, which is what makes RoPE so good at generalization.

The toolkit implements it the textbook way: precompute inverse frequencies, build cos/sin tables for the sequence length, then apply x * cos + rotate_half(x) * sin. The interpolation_factor scales the positions before computing those tables.

The honest caveat

Same one that runs through this pack's transformer nodes: the define nodes record layer dicts onto the stack, and the compile loop currently materializes the classic layer types (Conv, Linear, Flatten, Pool, Norm). A stack entry of type RotaryPositionalEmbedding isn't dispatched into the built model yet in the current source. The implementation code exists (there's a working _create_rotary_embedding with a functional RotaryEmbedding module), but it isn't called by the compile path. So treat this node as a way to learn RoPE's parameterization and the stack format, not as something that will change your compiled model today.

It's a niche node even within a niche pack - 2 impressions in a year. If you're building Llama-style architectures for real, you'd use a proper transformer library; if you're studying, this is a nice way to poke at base and interpolation_factor and see what they mean.

Install

Pack-level:

cd ComfyUI/custom_nodes
git clone https://github.com/inventorado/ComfyUI_NNT.git
cd ComfyUI_NNT
pip install -r requirements.txt

or ComfyUI Manager → "ComfyUI Neural Network Toolkit NNT", then restart. Find it under NNT Neural Network Toolkit/Transformers.