Train SAE

Everything in the pack's "Feature Surgery" and "Research" sections - Feature Map, Feature Gate, Feature Probe, Feature Dict, Diff Features, Feature Atlas - needs a trained Sparse Autoencoder to work on. This is the node that trains it. You run it once, save a .pt file, and hand that path to every other Feature node. It's the pack's prerequisite disguised as a node.

How it works

A sparse autoencoder learns to decompose the text encoder's residual-stream activations into a set of sparse, roughly monosemantic features - the "interpretable features" that SAE research (Bricken et al. 2023) popularized. With the default 8× expansion on Qwen's 2560d space that's 20,480 features. The node collects text-encoder activations, then trains the autoencoder to reconstruct them while keeping only a few features active per input (L1 sparsity penalty). The decoder columns it learns are feature directions you can later probe, gate, and label.

Where the training text comes from matters:

• fineweb (recommended) - streams real web text from HuggingFace's FineWeb dataset. 500K+ diverse activation vectors, ~15–30 min on GPU. Real data → well-separated, high-quality features.
• synthetic - generates prompts from templates. Fast (~2–5 min) but only ~15K vectors, which the tooltip warns is underdetermined for 20K features. Fine for experiments, not for anything you rely on.

The inputs that matter

• save_path - where the SAE lands; feed this exact path into Feature Map / Feature Gate.
• data_source - fineweb or synthetic. Choose fineweb for real work.
• layer - 22 default, matching the rest of the pack's convention. Must match what you use everywhere else.
• sae_expansion - 8× = 20,480 features. Higher = finer-grained but slower and bigger.
• n_vectors - fineweb mode only; rule of thumb is 25–50× your feature count, so 500K–1M for 20K features.
• l1_coeff - the sparsity penalty, 0.008 default. Too high → dead features; too low → dense mush. The tooltip's 0.005–0.01 range is the safe zone.
• learning_rate - 3e-4 default, warms up then cosine-decays. Usually leave it.
• hf_dataset / hf_subset - defaults to HuggingFaceFW/fineweb / sample-10BT; any text dataset with a text column works (allenai/c4 is the classic alternative).
• seed - -1 means random features each run. Set a fixed value (42, whatever) to get reproducible feature indices across retrains, and write it down - feature indices are meaningless without knowing which SAE and seed made them.
• cache_activations - on by default; reuses collected activations when you retrain with different hyperparameters. Keep it on.

Output is sae_path, a string you'll paste into every downstream Feature node.

Installing

Same pack install as always - ComfyUI Manager search "Concept Steer", or git clone https://github.com/Nynxz/ComfyUI-ConceptSteer into custom_nodes/ and restart. Beyond transformers and safetensors, the fineweb path needs internet access to stream the dataset, and datasets installed (pip install datasets) so HuggingFace's streaming works. Qwen encoding wants ~10 GB VRAM.

Common gotchas

• "Dead feature" diagnostics. Too-high l1_coeff kills features. The Feature Atlas node exists partly to tell you this - its health diagnostics report dead/alive counts.
• Synthetic mode is a trap for real work. 15K vectors for 20K features is mathematically underdetermined; you'll get a noisy SAE. The tooltip doesn't mince words.
• Feature indices change between runs unless you pin the seed. If you save feature indices for a Gate workflow, note the seed that produced them, or retrain with a fixed seed.
• It's not instant. Fineweb mode is a 15–30 minute GPU job. Cache activations and reuse the file across the pack - you only pay for this once.