Nodes/ComfyUI-latent-ops/LatentOperationReLU6
ComfyUI Node

LatentOperationReLU6

ReLU with a cap that your latent will almost never hit

By hnmr293·Created about a year ago·Updated about a year ago· 2
LatentOperationReLU6
    • op
    alpha1.0000

    ReLU6 is min(max(x, 0), 6) - ReLU with a ceiling at 6. It was invented for MobileNet-style networks that feed activations into fixed-point quantization, where you want everything bounded so nothing overflows the integer range. Applied to a latent, here's the honest truth: your SD/SDXL latent lives roughly in the ±3 range, so the "6" ceiling is a wall you will essentially never touch. In practice this node behaves almost identically to LatentOperationReLU until you crank alpha high enough to push values past 6. You're mostly choosing the cap for philosophical reasons, not behavioral ones.

    The mechanism

    def relu6(latent, **kwargs):
        return torch.nn.functional.relu6(latent * alpha)
    

    One input: alpha (FLOAT, default 1.0) - the input gain shared by every activation node in this pack. The latent gets scaled by alpha, then rectified with the 0..6 bounds. At alpha = 1 on a realistic latent, the cap never engages and you've built ReLU with extra paperwork. Where ReLU6 differs from ReLU: any value that does cross 6 gets pinned there instead of running free, so if you're amplifying latents hard (large alpha) the cap keeps the positive tail from running away. That's the actual niche - bounded rectification under heavy gain.

    If you're considering this node you've almost certainly also looked at plain ReLU, so the same warnings apply: zeroing the negative half of a roughly zero-mean distribution is a violent edit that usually produces broken or washed decodes. The cap doesn't soften that; it only reins in the top. For gentler nonlinear shaping, the pack's LatentOperationMish or LatentOperationSiLU keep a soft floor instead of a hard zero, and LatentOperationLeakyReLU preserves negatives at low amplitude. ReLU6 is for the specific case where you want a hard floor and a hard ceiling.

    The op output

    As with every node in ComfyUI-latent-ops, the output is op of type LATENT_OPERATION - a deferred closure, not an applied result. The pack builds operations and ships no apply node, so you need a consumer of LATENT_OPERATION (Sonar's SonarApplyLatentOperationCFG is the one that exists in the wild) or your own apply node. Directly wiring op into a VAE Decode gets you a type mismatch - the pack's universal first-encounter trap. Nothing in here applies itself; that's the design.

    Install

    The standard drill. ComfyUI Manager → search ComfyUI-latent-ops, or:

    cd ComfyUI/custom_nodes
    git clone https://github.com/hnmr293/ComfyUI-latent-ops
    

    Restart. No requirements.txt, no model downloads - just PyTorch, which ComfyUI already bundles. The pack is hnmr293's personal latent toolkit (same author as sd-webui-cutoff and llul), barely known to the community, so the source in latent_ops/functions.py is your documentation. Everything registers under hnmr/latent_ops. And if the README's one-line description and the code ever disagree, the code wins.

    Categoryhnmr/latent_ops

    Inputs (1)

    NameTypeDefaultDescription
    alphaFLOAT1.0000-10000–10000

    Outputs (1)

    NameTypeDescription
    opLATENT_OPERATION