Audio-Visual Synesthesia (S42)

Audio-Visual Synesthesia is the most ambitious and least predictable node in this pack, and I want to be upfront: the code's own docstring says it "forces the Audio VAE to decode a picture into sound." That's the whole premise - take an image latent, reshape it into the shape an audio decoder expects, and see what the audio VAE makes of it. It is deeply experimental, it will not produce a nice song, and it is genuinely interesting anyway.

Here's the mechanism, which is worth understanding because it demystifies the whole thing. You feed it:

• image_latent - a latent from an image VAE, i.e. ordinary image-encode output.
• target_audio_seconds (1–30, default 5) - how long the resulting audio latent should be.

The node then flattens the spatial dimensions of your image latent into a single time-like axis, pads or trims its channels to 128 (to match the audio-latent format the pack uses everywhere - same 128 channels × 86 steps-per-second convention as S42_AudioLatentEncoder and the AceStep stub), and resamples that flattened stream to a length of seconds × 86. Out pops a LATENT shaped exactly like an audio latent, ready for S42_AudioLatentDecoder.

What that means in practice: the image's spatial structure gets stretched into a timeline and handed to an audio decoder that has never seen an image before. The result is almost certainly going to sound like noise, textured hum, or glitch - whatever the audio VAE hallucinates when given a fundamentally wrong input. Sometimes that's eerily musical, sometimes it's a wall of static. That's the nature of the experiment, and the Experimental category label is the author acknowledging it.

The honest use cases are two, and neither is "make a banger." First: it's a pipeline-integrity test - it proves you have a working audio VAE wired in, because garbage through a working decoder sounds like something, while a wrong or missing VAE yields flat silence. Second: it's a legitimate source of generative foley texture if you like weird, and if one in twenty results is a keeper, you've got a sound no one else has. For serious work, treat it as a curiosity and a debugging tool, not a sampler.

A practical note for the debugging angle: if the decoded output is silent rather than noisy, that's not synesthesia failing - that's your VAE not actually decoding audio latents (see the decoder's silent-fallback behavior). A real audio VAE will produce something audible, even if that something is strange.

Installing it

S42-CutFlow install: ComfyUI Manager → search "S42 CutFlow" → Install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/GeekyGhost/S42-CutFlow.git
pip install -r S42-CutFlow/requirements.txt

One-line requirements (opencv-python-headless); the reshaping is pure torch. Look for the "[S42 CutFlow] Loaded ..." console line.

What to expect

Real audio VAE in the chain → audible weirdness. No audio VAE → silence. It will not replace your music library, but if you've ever wanted to hear what a VAE thinks a picture sounds like, this is the node that finally answers the question.