fear-of-the-art: Audio Analyzer

A ComfyUI custom node project that extracts sonic features from audio and uses a local Ollama LLM to generate painterly, emotionally-driven image-generation prompts. It is an experiment in taking audio seriously as a source of visual instruction — not just as a trigger, but as something that carries meaning worth listening to. Acoustic analysis via librosa describes what the audio does physically; optional CLAP embeddings describe what it feels like semantically; a local language model translates both into image-generation prose. The project is intentionally experimental: results vary, some outputs will surprise you, and that is part of the point.

---

The idea behind it

Most audio-to-image tools treat audio as metadata — a filename, a BPM, a duration. This project treats audio as emotional content. The idea is that if you listen carefully enough to a piece of audio, you can describe what it feels like, and from that description, ask a diffusion model to make something that looks like it feels.

The pipeline works in two phases. First, acoustic features are extracted from the audio — tempo, spectral energy, brightness, dynamic range, textural complexity. These features are passed to a local LLM, which interprets them as mood data and produces structured JSON. Second, that JSON is used as context for a second LLM call that generates image-generation prompts in prose. The two phases use separate temperature settings: analysis calls use low temperature (0.4) for consistency, while prompt generation uses high temperature (0.8) to keep the visual output expressive and non-repetitive.

The CLAP node adds a second kind of listening. Where librosa measures what the audio does physically — "spectral centroid: 2400Hz", "onset rate: 3.2 per second" — CLAP measures what it feels like semantically. It compares the audio's embedding against a configurable set of text anchors ("nocturnal fear", "dark atmospheric tension", "explosive catharsis") and returns the ones that match most closely. The result is not an analysis of sonic properties but a vocabulary of affect.

CLAP and librosa are inputs to the LLM, not outputs of the pipeline. They apply pressure on the LLM's interpretation — they tell it what to pay attention to. The LLM decides what that means visually. This distinction matters: the system is not retrieving images that match an audio embedding; it is asking a language model to imagine a visual world from what the audio communicates.

---

How it works

audio
 ├─ librosa → acoustic features  (tempo, energy, brightness, dynamics…)
 └─ CLAP    → semantic anchors   (nocturnal fear, dark atmospheric tension…)
                     ↓
             Ollama LLM (Qwen / any model)
                     ↓
        mood_json + subject_json
                     ├─ environment_prompt / subject_prompt / merge_prompt
                     └─ Composition Inference → composition_json
                                                  ├─ composition_prompt
                                                  ├─ width  (INT → EmptyLatentImage)
                                                  └─ height (INT → EmptyLatentImage)

CLAP is optional — the pipeline runs without it. When present, its semantic anchors and the librosa features both reach the LLM in a single analysis call, which produces structured mood data used to drive up to three prompt-generation calls: environment, subject, and merged final.

---

Nodes

Audio Mood Analyzer

The base node. Extracts acoustic features from audio and uses a local Ollama LLM to generate up to three image-generation prompts: environment-only, subject-only, and a merged final prompt.

Connection and model

| Name | Type | Description | |------|------|-------------| | audio | AUDIO | Audio input from any ComfyUI audio loader | | ollama_url | STRING | Ollama API endpoint (default: http://localhost:11434/api/generate) | | model | STRING | Ollama model name (default: qwen3:14b) | | analysis_temperature | FLOAT | Temperature for structured JSON analysis calls — lower = more consistent (default: 0.4) | | prompt_temperature | FLOAT | Temperature for creative image-prompt generation — higher = more expressive (default: 0.8) |

Analysis context

These fields are injected into the analysis phase only and have no effect on the visual style of the output prompts.

| Name | Type | Description | |------|------|-------------| | custom_context | STRING | Analytical direction for the mood analysis (e.g. "focus on rhythm over atmosphere") | | lyrics_or_text | STRING | Full lyrics or source text for subject analysis | | focus_fragment | STRING | Key line or phrase used as the primary emotional/visual anchor | | song_title | STRING | Song title used as thematic and symbolic context | | song_description | STRING | General meaning, emotional arc, or artist intent | | song_genre | STRING | Genre or style (e.g. "post-punk, shoegaze") |

Subject analysis only runs when at least one of lyrics_or_text, focus_fragment, song_title, song_description, or song_genre is provided.

Style

These fields control the aesthetic output of the three image-generation prompts.

| Name | Type | Description | |------|------|-------------| | style_preset | COMBO | Visual aesthetic target: painterly, cinematic, raw, abstract, or custom (default: painterly) | | style_notes | STRING | Optional free text appended to the preset, or used alone when preset is custom |

Presets:

| Preset | Target aesthetic | |--------|-----------------| | painterly | Oil painting, raw brushwork, emotionally loaded colour — Bacon, Schiele, Freud | | cinematic | Wide cinematic frame, dramatic lighting, filmic grain — Tarkovsky, Wong Kar-wai, Villeneuve | | raw | Visceral, lo-fi, grainy, desaturated — no production value | | abstract | Non-representational, gestural, colour field — Rothko, Kiefer, Twombly | | custom | Uses style_notes only |

Output toggles

| Name | Type | Description | |------|------|-------------| | generate_environment_prompt | BOOLEAN | Generate an environment-only image prompt (default: true) | | generate_subject_prompt | BOOLEAN | Generate a subject-only image prompt (default: true) | | generate_merge_prompt | BOOLEAN | Generate a merged final image prompt (default: true) |

Outputs

| Name | Type | Description | |------|------|-------------| | mood_json | STRING | Full sonic mood analysis as JSON | | subject_json | STRING | Subject analysis from lyrics/text as JSON | | environment_prompt | STRING | Image prompt focused on environment/atmosphere | | subject_prompt | STRING | Image prompt focused on the human subject | | merge_prompt | STRING | Unified final image-generation prompt | | summary | STRING | Short human-readable mood summary |

---

Audio Mood Analyzer (Advanced)

Identical to the base node (all inputs and outputs from the standard node apply), plus five optional prompt template override fields — one for each internal Ollama call. Leave a field empty to use the built-in template; provide a value to replace it entirely.

| Override field | Replaces | Available template variables | |----------------|----------|------------------------------| | mood_prompt_override | Audio mood analysis prompt | {features}, {custom_context}, {style_block} | | subject_analysis_prompt_override | Subject analysis prompt | {lyrics_or_text}, {focus_fragment}, {song_title}, {song_description}, {song_genre}, {custom_context} | | environment_prompt_override | Environment image-gen prompt | {mood_json}, {subject_json}, {style_block} | | subject_prompt_override | Subject image-gen prompt | {subject_json}, {style_block} | | merge_prompt_override | Merge/final image-gen prompt | {mood_summary}, {environment_prompt}, {subject_prompt}, {style_block} |

Use standard Python {variable_name} placeholders. If a variable name is not recognised, the node logs a warning and falls back to the built-in template for that call.

---

Audio Mood Analyzer (Timeline)

Divides audio into N equal segments and runs the full analysis and generation pipeline on each. Subject analysis runs once from lyrics/text; mood analysis and prompt generation run per segment. Designed for image sequence and video generation workflows.

Additional input

| Name | Type | Description | |------|------|-------------| | n_segments | INT | Number of equal time segments to analyse (default: 8, range: 2–32) |

All other inputs are identical to the standard node.

Outputs

| Name | Type | Description | |------|------|-------------| | prompt_sequence_json | STRING | JSON array — one object per segment containing segment, start_s, end_s, mood_json, environment_prompt, subject_prompt, merge_prompt | | merge_prompts | STRING | Merge prompts only, newline-separated — one per segment | | environment_prompts | STRING | Environment prompts only, newline-separated — one per segment | | subject_prompt | STRING | Single shared subject prompt (computed once from lyrics, same across all segments) |

At n_segments=8 with all toggles on: up to 26 Ollama calls total (2 shared subject calls + 3 per-segment calls × 8 segments). merge_prompts is the most useful output for image batch nodes. prompt_sequence_json feeds the AnimateDiff Schedule Formatter.

---

AnimateDiff Schedule Formatter

Converts AudioMoodAnalyzerTimeline's prompt_sequence_json output into an AnimateDiff Evolved (ADE) prompt travel schedule string. Each audio segment is proportionally mapped to a frame number within the AnimateDiff frame range.

Inputs

| Name | Type | Description | |------|------|-------------| | prompt_sequence_json | STRING | JSON array from Audio Mood Analyzer (Timeline) | | total_frames | INT | Must match the AnimateDiff frame count setting (default: 64, range: 8–256) | | prompt_type | COMBO | Which prompt to use per segment: merge_prompt (default), environment_prompt, or subject_prompt |

Outputs

| Name | Type | Description | |------|------|-------------| | schedule | STRING | ADE prompt travel schedule — wire into an AnimateDiff Evolved prompt scheduling node | | first_frame_prompt | STRING | Prompt for frame 0 — wire into a standard CLIPTextEncode as fallback positive conditioning |

Output format

"0": "dark wasteland, crumbling concrete",
"8": "burning field, ash and smoke",
"16": "flooded ruin, grey water",

Frame numbers are derived proportionally: frame = round(start_s / total_duration × total_frames). Empty prompts (if Ollama failed for a segment) are omitted. When two segments map to the same frame, the later segment's prompt wins.

---

CLAP Audio Analyzer

Extracts a semantic audio embedding using CLAP and ranks a configurable set of text anchors by cosine similarity to the audio. Designed to feed its semantic_summary output into AudioMoodAnalyzer.custom_context, adding semantic pressure to the LLM's interpretation without overriding the acoustic analysis.

Inputs

| Name | Type | Description | |------|------|-------------| | audio | AUDIO | Audio from any ComfyUI audio loader | | clap_model | STRING | HuggingFace model ID (default: laion/clap-htsat-unfused) | | clap_device | COMBO | auto, cpu, cuda, or mps (default: auto) | | clap_text_anchors | STRING (multiline) | One anchor phrase per line — ranked against the audio |

Default anchors (15 phrases): dark atmospheric tension, melancholic isolation, aggressive emotional pressure, fragile human vulnerability, nocturnal fear, ritualistic heaviness, dreamlike surreal space, claustrophobic anxiety, slow emotional collapse, explosive catharsis, cold empty space, distorted memory, spiritual dread, submerged sadness, violent inner pressure

Outputs

| Name | Type | Description | |------|------|-------------| | clap_json | STRING | Full JSON — model name, embedding norm, ranked matches, top-3 inference | | semantic_summary | STRING | "CLAP: anchor1, anchor2, anchor3" — wire into AudioMoodAnalyzer.custom_context |

Wiring semantic_summary into AudioMoodAnalyzer.custom_context injects the top-3 anchor phrases into the LLM's mood analysis prompt as named emotional vocabulary — the same "semantic pressure" mechanism described above, but sourced from the audio embedding rather than typed by hand.

The model is loaded once per ComfyUI session (the Python process lifetime, not per workflow run) and cached by (model_name, device) — changing the device mid-session has no effect until ComfyUI restarts. On error, semantic_summary returns "" and the workflow continues.

---

Composition Inference

Acts as an art director downstream of Audio Mood Analyzer. Takes mood_json and subject_json and runs two Ollama calls to infer optimal visual composition — aspect ratio, orientation, subject placement, camera language — returning a structured JSON and a prose prompt layer alongside parsed width and height integers that wire directly into ComfyUI's EmptyLatentImage.

Inputs

| Name | Type | Description | |------|------|-------------| | mood_json | STRING | JSON from Audio Mood Analyzer.mood_json | | subject_json | STRING | JSON from Audio Mood Analyzer.subject_json | | ollama_url | STRING | Ollama API endpoint (default: http://localhost:11434/api/generate) | | model | STRING | Ollama model name (default: qwen3:14b) | | analysis_temperature | FLOAT | Temperature for structured composition JSON inference (default: 0.4) | | prompt_temperature | FLOAT | Temperature for prose composition prompt generation (default: 0.8) |

Subject analysis is the primary signal — composition is inferred from the semantic and lyrical content of subject_json. If subject_json is empty the node continues, defaulting toward environment-dominant framing.

Outputs

| Name | Type | Description | |------|------|-------------| | composition_json | STRING | Full composition analysis — aspect ratio, subject placement, environment weight, camera | | composition_prompt | STRING | Art direction prose — wire into CLIPTextEncode as a third conditioning layer alongside environment and subject prompts | | width | INT | Inferred canvas width — wire directly into EmptyLatentImage.width | | height | INT | Inferred canvas height — wire directly into EmptyLatentImage.height |

The composition_json schema:

{
  "aspect_ratio": {"orientation": "portrait", "ratio": "4:5", "recommended_resolution": "1024x1280"},
  "subject_placement": {"position": "lower_right", "size_weight": 0.35},
  "environment": {"weight": 0.65, "negative_space": "high"},
  "camera": {"distance": "medium", "framing_style": "environment_dominant", "crop": "waist_up"}
}

On LLM parse failure, falls back to safe defaults (1024×1024, centered subject, balanced framing) and logs a warning. The pipeline continues.

---

Mood JSON Unpacker

Takes the mood_json output from Audio Mood Analyzer and exposes each field as a separate ComfyUI STRING output. Removes the need for a JSON parser node — individual fields wire directly into CLIPTextEncode, string concatenators, or any other downstream node.

Inputs

| Name | Type | Description | |------|------|-------------| | mood_json | STRING | JSON string from Audio Mood Analyzer.mood_json |

Outputs

| Name | Type | Description | |------|------|-------------| | sonic_mood | STRING | Comma-joined mood adjectives (e.g. melancholic, dense, pressured) | | energy_profile | STRING | Prose description of energy level and behaviour | | tension_profile | STRING | Prose description of tension and internal pressure | | color_palette | STRING | Comma-joined color terms — wire into CLIPTextEncode or style notes | | lighting_implications | STRING | Comma-joined lighting descriptors | | texture_implications | STRING | Comma-joined texture descriptors | | composition_suggestions | STRING | Comma-joined composition cues | | avoid | STRING | Comma-joined negative terms — wire into the negative CLIPTextEncode |

On malformed or empty input, all outputs return "". Missing keys return "" individually — the node never raises.

---

Prompt Enricher

Takes any prompt string and deterministically appends selected mood_json fields to it. No LLM call — fast and predictable. The fields_to_inject input controls which fields are appended and in what order.

Inputs

| Name | Type | Description | |------|------|-------------| | prompt | STRING (multiline) | Base prompt to enrich — e.g. merge_prompt or environment_prompt from Audio Mood Analyzer | | mood_json | STRING | JSON string from Audio Mood Analyzer.mood_json | | fields_to_inject | STRING (multiline) | One field name per line — appended to the prompt in listed order (default: color_palette, lighting_implications, texture_implications) |

Outputs

| Name | Type | Description | |------|------|-------------| | enriched_prompt | STRING | Base prompt with mood field values appended, comma-separated |

The avoid field gets a labelled prefix (avoid: ...) rather than a plain comma append — useful if you pipe it into a negative prompt node. Unknown field names and empty values are silently skipped. On malformed mood_json, the base prompt is returned unchanged.

---

Ollama Model Selector

Queries a local Ollama server and returns the list of installed model names. Useful for wiring the correct model name into analysis nodes without hardcoding it.

Inputs

| Name | Type | Description | |------|------|-------------| | ollama_url | STRING | Ollama base URL (default: http://localhost:11434) |

Outputs

| Name | Type | Description | |------|------|-------------| | models_list | STRING | Newline-separated list of installed model names | | first_model | STRING | First model name — wire directly into AudioMoodAnalyzer.model |

---

Example workflows

| File | Demonstrates | |------|-------------| | example_workflow/example.json | Standard dual conditioning — environment + subject as separate CLIPTextEncode inputs, averaged via ConditioningAverage | | example_workflow/example_timeline.json | Full-song timeline analysis, n_segments=8, merge_prompts wired to a single-image sanity check | | example_workflow/example_animatediff.json | Timeline → AnimateDiff formatter → ADE schedule string preview + first_frame_prompt as positive conditioning | | example_workflow/example_clap.json | CLAP semantic embedding — semantic_summary output shown via PreviewAny, ready to wire into AudioMoodAnalyzer.custom_context | | example_workflow/example_composition.json | Composition inference — AudioMoodAnalyzer → CompositionInferenceNode, width/height wired into EmptyLatentImage, composition_prompt shown via PreviewAny | | example_workflow/example_mood_unpack_enrich.json | Unpacking and enrichment — AudioMoodAnalyzer → MoodJsonUnpacker (individual fields shown via PreviewAny) + PromptEnricher (merge_prompt enriched with color, lighting, texture) |

---

Requirements and installation

Requirements:

• Ollama running locally with a model loaded (default: qwen3:14b)
• Python packages: librosa, soundfile, numpy, requests, transformers>=4.35.0

Installation:

cd ComfyUI/custom_nodes
git clone https://github.com/andrea-spoldi/ComfyUI-AudioMoodAnalyzer.git
pip install -r ComfyUI-AudioMoodAnalyzer/requirements.txt

Restart ComfyUI. All nodes appear under audio/analysis.

---

A note on results

Output quality depends heavily on the Ollama model — larger, instruction-tuned models produce more coherent and visually specific prompts, while smaller models tend toward generic language. CLAP anchors apply semantic pressure on the LLM's interpretation, but they are not guarantees: the model may weight them lightly or interpret them in unexpected directions. The prompts this pipeline produces are starting points, not finished images — they will need refinement for production use. Surprises are expected and are often more interesting than what a more deterministic system would produce.

---

License

MIT — see LICENSE.