AnimateDiff Schedule Formatter

The Audio Mood Analyzer (Timeline) splits a song into segments and writes a different prompt for each. That's a beautiful thing to have - and useless, by itself, for animation. AnimateDiff wants its prompts as a schedule: a set of "frame": "prompt" pairs it interpolates between across the clip. AnimateDiff Schedule Formatter is the bridge. It takes the timeline node's prompt_sequence_json and converts it into the exact prompt-travel string AnimateDiff-Evolved (ADE) eats, mapping each audio segment proportionally onto the frame range.

This is a pure string transformer - no Ollama, no models, no analysis. It ships in the same pack so the two nodes plug together, and it exists because hand-writing a 20-line schedule that follows the dynamics of a song is miserable work you should never do twice.

How it works

The mechanism is simple and predictable. It reads the segments from prompt_sequence_json, works out the total duration from the last segment's end_s, and maps each segment to a frame with frame = round(start_s / total_duration × total_frames). Each segment's prompt (whatever prompt_type you pick) becomes one line of the schedule:

"0": "dark wasteland, crumbling concrete",
"8": "burning field, ash and smoke",
"16": "flooded ruin, grey water",

Three behaviors in the source are worth knowing because they're the difference between a clean run and a mysterious blank one:

• Empty prompts are skipped. If Ollama failed on a segment (the timeline node logs and continues), that segment just doesn't appear in the schedule. Your run survives a dead segment instead of producing garbage.
• Frame collisions resolve to the later segment. Two segments mapping to the same frame means the more recent one wins - so a fast-paced song with tight segments won't produce a broken schedule.
• The schedule is clamped to 0..total_frames-1, and if nothing ends up in the map, both outputs come back empty strings.

Inputs and outputs

• prompt_sequence_json - the JSON array from Audio Mood Analyzer (Timeline).
• total_frames - must match your AnimateDiff frame count. Default 64, range 8–256. This is the single most common mistake: set the ADE node to 64 frames and this to 32, and your schedule is misaligned with the actual animation.
• prompt_type - which per-segment prompt to use: merge_prompt (default), environment_prompt, or subject_prompt. Start with merge; switch to environment-only if the subject prompt makes frames too literal.

Outputs are schedule (wire it into an AnimateDiff-Evolved prompt scheduling node) and first_frame_prompt (the prompt for frame 0 - wire it into a standard CLIPTextEncode as the fallback positive conditioning). The first-frame output exists because ADE workflows still need a positive prompt outside the schedule node.

The honest context

Before you go all-in on this pipeline, the KB's position on AnimateDiff deserves a mention: it's a 2023 model, officially frozen since v3, and Wan-based tooling has matched every capability people once kept it around for. What AnimateDiff still does uniquely is the morphing, dreamlike keyframe interpolation - the exact look audio-reactive music videos are built around - and ADE itself is still actively maintained on current ComfyUI. So this node is the right tool for a specific aesthetic: a song's mood arc driving prompt travel in that morph-y, beat-synced style. If you just want video from prompts, you'd be on Wan. If you want this look, the Timeline → formatter → ADE chain is a genuinely nice way to drive it from audio.

Install

Same pack:

cd ComfyUI/custom_nodes
git clone https://github.com/andrea-spoldi/ComfyUI-AudioMoodAnalyzer.git
pip install -r ComfyUI-AudioMoodAnalyzer/requirements.txt

Restart, find it under audio/analysis. You'll also need ComfyUI-AnimateDiff-Evolved installed for the schedule node it targets - that's a separate pack, and the ComfyUI Manager search will sort you out. No Ollama needed for this node itself; it just reshapes text.