Scale-RAE Text to Image

Here's the thing nobody warns you about: the Scale-RAE model is a chat model. Type A field of sunflowers at dusk and there's a decent chance it replies with a description of a field instead of drawing one. Type Can you generate an image of a field of sunflowers at dusk? and it draws. That's not folklore, it's measured: the pack's own testing found 0 of 20 plain descriptions started an image and 60 of 60 request-style prompts did. The node's prompt default - "Can you create an image of a peaceful garden?" - is the author telling you this without telling you this.

Why a chat model is in an image pipeline

Scale-RAE came out of NYU in January 2026, arguing that diffusion transformers should run in the feature space of a frozen vision encoder rather than a VAE's latent. The honest community read was mixed: it trains fast, it reconstructs worse than what you already have, and the resolution is tiny. Interesting architecture, not a Flux killer.

The node runs the paper's inference path in-process:

1. Your prompt goes through Qwen's chat template and a 1.5B Qwen2.5 model reads it.
2. The model greedily decodes until it emits <im_start> - the "okay, here comes the picture" token, or a sentence of text if it decides to be helpful instead.
3. 256 learned latent queries get appended; one forward pass gives their hidden states.
4. A 2.4B rectified-flow DiT samples those into 256 encoder-space tokens over 50 Euler steps.
5. The RAE decoder turns the tokens into a 224×224 image.

No VAE, no CLIP, no separate text encoder - one checkpoint does the lot. That's the interesting part, and it's where the sharp edges are.

The inputs that matter

prompt - phrase it as a request. "Generate an image of…", "Can you create a photo of…". Not guaranteed even then; the author's test saw "Can you generate a close-up photo of a bowl of ramen?" answered with text.

image_start - the field that saves you, and the default (auto) is right. In auto, when the model replies with text, the node starts the image right after the prompt anyway, and the report says so (image_forced: true). model is the official behaviour only: a text-only answer becomes an error. force skips the chat and always starts the image - for request-style prompts it lands very close to the official output.

guidance - leave it at 1.0, the paper setting. Above 1.0 it depends on cfg_mode, and this is the trap: official-cli reproduces the released code, where guidance above 1 does nothing at all because of an upstream bug (ZitengWangNYU/Scale-RAE issue #7). standard does real classifier-free guidance every step, darkening and saturating the image as you push it, and runs the DiT on two rows so it takes about 1.7× longer. Dead-slider complaints are official-cli; deep-fried-output complaints are standard.

steps - 50, the official and paper value. Guidance above 1 doubles the DiT evaluations at every one of them.

seed and batch_size - a batch uses seed, seed + 1, seed + 2… and image k equals a single run at that seed. Same settings, same picture; CPU and GPU draw different noise.

What comes out

Two outputs. image is a 224×224 IMAGE - wire it to Preview/Save, then to an upscaler, because 224 pixels is a thumbnail. Tile-ControlNet passes are what stop an upscale diverging; the KB's upscaling doc covers the options.

report is a JSON string and the most underrated thing in the node: seed, guidance, cfg_mode, steps, image_start, image_forced, the text the model wrote before starting the image, DiT call count, seconds per image, and peak VRAM on CUDA. Read it first when a run misbehaves.

Install

Same pack as the loader node. ComfyUI Manager has it, or:

cd ComfyUI/custom_nodes
git clone https://github.com/hiroki-abe-58/ComfyUI-ScaleRAE
python -m pip install -r ComfyUI-ScaleRAE/requirements.txt
python custom_nodes/ComfyUI-ScaleRAE/tools/download_models.py --models-dir ComfyUI/models/scale_rae

You need a ComfyUI with the V3 node API, and float16 on the loader if you have ~10 GB of VRAM rather than 18 GB.

Where people get burned

"The model answered with text instead of an image." That's image_start = model doing its job. Switch to auto.

An empty prompt, or one containing <image>. The node rejects both outright, and it's the one error that isn't the model's fault.

Everything looks flat and identical across seeds. Check your load dtype. bfloat16 is the one the author measured drifting furthest from the official outputs - it changed images noticeably and sometimes produced the text reply instead of an image. float16 is the better half-precision choice.

Expecting a small Flux. This is a 224×224 research model with a 1.5B language model doing the understanding. Fascinating to have in a graph - text and image in one checkpoint, no VAE loader - and not the node you reach for when you need a poster. Run it, see what a representation-space diffusion model does at 50 steps, then go back to your usual stack for real work.