🎵 Music2Video

This is the whole Music2Video pack in one node. You drop an audio track in, out comes everything you'd need to shoot a music video for it: shot-by-shot image prompts, MiniMax H3 video prompts in both of its formats, per-shot timings, negatives, and sample-accurate audio slices for lipsync - all on a single pipe. And if you want, it'll do the rendering too: start frames, clips, and the final cut-together film with the music muxed underneath. The name is not a lie about what it does; the surprise is which parts cost money. The analysis is always local and free, prompt writing is local by default, and rendering stays off until you pick a paid provider.

How it works

The pipeline is worth understanding because it's unusually honest about who does what. Whisper large-v3 transcribes the lyrics locally with word-level timestamps. librosa (when installed, otherwise a built-in numpy/scipy fallback) measures the BPM, beat grid, sections and energy. Then a shot planner in plain Python - no LLM touches timing - divides the track into 5–15 second shots snapped to the beats. An LLM writes the creative content: treatment, art direction, a "bible" of recurring subjects, then each shot. Finally, deterministic Python renderers assemble the exact MiniMax H3 prompt skeletons, so the format is correct no matter which model did the writing.

The LLM is never asked to "listen" or count time - it gets measured numbers and fixed shot boundaries and only invents the pictures and words. That's the pack's whole design thesis, and it's a good one: a small local model can write creatively without hallucinating a beat grid.

The inputs that matter

There are a lot of widgets here, but the 30+ advanced ones collapse into the node's native Advanced section (this is a V3-schema node, so that's a ComfyUI feature, not a hack). The ones you'll actually touch:

• audio - the track, from LoadAudio. Its length defines the shot plan.
• instruction - your brief: story, mood, world, constraints. It steers every writing stage.
• llm_provider - lmstudio (local, free, default), or openrouter/openai/anthropic (billed per token).
• clip_seconds, min_shot_seconds, max_shot_seconds - shot length. H3 only accepts 5–15 s, so the defaults respect that.
• creativity, dynamicity, word_influence - three 0–1 sliders that shape how literal, how kinetic, and how lyric-following the writing is.
• image_provider / video_provider - both default to pipe-steps, which renders nothing and bills nothing. Switch to fal or openrouter and it becomes a pay-per-call render farm.

Outputs

Six sockets. pipe (the M2P_PIPE type) carries every prompt, name, timing, the transcript and the analysis JSON - feed it to Music2Video Pipe Expand to get individual outputs. The media stay on their own sockets: images (start frames per shot), subject_images (reference sheets per subject), videos (per-shot clips), final_video (the assembled film), and audio_clips (each shot's exact slice of the track, at original sample rate - straight into a lipsync node).

Install

ComfyUI Manager: search Music2Video, install, restart. Or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/lazniak/comfyui-music2video music2video

It needs ComfyUI ≥ 0.3.48 (the first release with the V3 node schema it's built on). Dependencies are just requests, numpy, scipy, transformers, huggingface_hub - all already in a normal ComfyUI install. First run downloads Whisper large-v3 (~3 GB) to ComfyUI/models/whisper/. For local prompt writing, start LM Studio (Developer → Start Server, default http://127.0.0.1:1234); model dropdowns are fetched live, so if a model doesn't appear, right-click → Refresh model lists.

Where people get burned

VRAM. Whisper large-v3 takes ~2.9 GB of weights and up to ~7 GB peak during word-level timestamp alignment. On an 11 GB card that's the whole GPU. The node handles this aggressively: it unloads ComfyUI's models and the LM Studio model before transcribing, runs Whisper in 30-second windows, and steps down automatically (smaller batch → segment timestamps → CPU) if it still runs out of memory. Set whisper_device to a second card if you have one.

Reasoning models return empty. Gemma 4 and Qwen3 will spend the whole token budget on hidden thinking and hand back nothing. That's why lm_reasoning_effort defaults to none. If your stages come back empty, this is the first thing to check.

The billing surprise. One image per shot, one clip per shot. A 3-minute track cut into 6-second shots is 30 shots - 30 billed images plus 30 billed clips. The live_preview gallery (on by default) is there so you spot a bad prompt at shot 1 instead of paying for all of them, and the node writes a cost report to disk. Refreshingly upfront about the money.

Dead-looking runs. If you wire a media socket on a run where nothing rendered, the node deliberately hands you an execution blocker rather than an empty list - the downstream node is skipped and the log explains why, instead of the whole graph dying with a bare IndexError: list index out of range. Annoying the first time you see it; far better than the alternative.