Nodes/comfyui-music2video/🎡 Music2Video Motion Enhancer
ComfyUI Node

🎡 Music2Video Motion Enhancer

Makes the video prompt describe the frame it's actually starting from

By lazniakΒ·Created 3 days agoΒ·Updated about 12 hours agoΒ· 1
🎡 Music2Video Motion Enhancer
  • pipe
  • images
  • pipe
  • video_prompts_i2va
  • report
β—„llm_providerlmstudioβ–Ί
β—„lm_modelβ–Ύβ–Ί
β—„openrouter_modelβ–Ύβ–Ί
β—„openai_modelβ–Ύβ–Ί
β—„anthropic_modelβ–Ύβ–Ί
β—„lm_urlhttp://127.0.0.1:1234β–Ί
β—„lm_api_keyβ–Ί
β—„openrouter_api_keyβ–Ί
β—„openai_api_keyβ–Ί
β—„anthropic_api_keyβ–Ί
β—„words_per_second8.0β–Ί
β—„rewrite_soundfalseβ–Ί
β—„image_detail768β–Ί
β—„lm_temperature0.40β–Ί
β—„lm_max_tokens2048β–Ί
β—„lm_timeout300β–Ί
β—„lm_retries2β–Ί
β—„verbosefalseβ–Ί

Here's the problem this node exists to fix, and it's a good one. The video prompts from the main Music2Video node are written before a single pixel exists. The LLM describes what it intends the first frame to contain; the image model then draws its own reading of that description. If the image came back different - a different framing, one subject where the prompt said two, a coat that turned out red - the video prompt still describes the intention. And MiniMax H3 resolves that disagreement the only way it can: it starts on the frame it was handed and dissolves into the scene it was told about. Every shot becomes a morph.

The Motion Enhancer closes that loop. Give it the pipe and the rendered start frames, and it shows each frame to a vision-capable LLM together with that shot's own i2va prompt, then rewrites the description so it describes that frame - and so the motion in it fits the shot's real length. The word budget is words_per_second Γ— the shot's duration, clamped to 25–220, which is what stops a six-second shot being described as a minute of events. The H3 skeleton, the <Picture 1> reference and the two sound fields are kept (rewrite_sound opts the sound in), so what comes back is the same prompt with its picture of the world corrected.

The inputs that matter

The pipe and the images are the meat. The rest is mostly LLM plumbing:

  • pipe - the main node's pipe. video_prompts_i2va is what gets rewritten; durations decides how much motion each shot can hold.
  • images - the start frames, one per shot, in shot order. Your sampler's output works as well as the node's own images socket. Fewer frames than shots is fine: a shot without one keeps its prompt.
  • llm_provider - same four providers as the main node (lmstudio, openrouter, openai, anthropic). It must be a vision model - a text-only model "returns a description of nothing" and every shot keeps its original prompt. Load a qwen-vl, llava-class, or gemma vision model in LM Studio.
  • words_per_second - how many words of description one second of clip is worth. 8 is the default; raise it for a model that writes thin, lower it for one that rambles.
  • image_detail - longest side of the frame as sent, in pixels. Bigger sees more and costs more; on a cloud provider this is most of the bill.
  • rewrite_sound - off by default, because a still frame says nothing about sound and the originals were written against the track. Turn it on when the image turned out to be a different place than the prompt assumed.

Outputs: pipe (same pipe with the rewritten video_prompts_i2va), video_prompts_i2va (the rewritten list, usable without an expander), and report - one line per shot saying what disagreed with the frame, or what went wrong. That report is your debugging friend.

Install

Same pack, nothing extra: ComfyUI Manager β†’ search Music2Video β†’ install, restart, or git clone https://github.com/lazniak/comfyui-music2video ComfyUI/custom_nodes/music2video. Needs ComfyUI β‰₯ 0.3.48.

Gotchas

Two real ones. It bills per shot on cloud providers - one LLM call per shot, free on LM Studio, billed per token on the rest, and image_detail is where most of that bill goes. lm_temperature defaults to 0.4 deliberately: this is a correction, not an invention, and a hot model will "fix" the frame into something new. And a failed shot keeps its original prompt rather than failing the run - so if every shot comes back unchanged, check that the model you picked can actually see images before you blame the node.

CategoryMusic2Video

Inputs (20)

NameTypeDefaultDescription
pipeM2P_PIPEThe pipe from the Music2Video node. 'video_prompts_i2va' is what gets rewritten; 'durations' decides how much motion each shot can hold, and 'image_prompts_start' is shown to the model as what the frame was meant to be.
imagesIMAGEThe start frames, one per shot, in shot order - the node's own 'images' socket, or whatever your sampler produced from 'image_prompts_start'. A batch of N frames and a list of N frames are read the same way. Fewer frames than shots is fine: the shots without one keep their prompt untouched.
llm_providerCOMBOlmstudioWhich LLM rewrites the prompts. It must be able to see images: a text-only model returns a description of nothing and the node keeps the original prompt instead. 'lmstudio' is local and free - load a vision model there (gemma, qwen-vl, llava class). The cloud providers bill per token, once per shot.
lm_modelCOMBOModel served by LM Studio. It has to be a vision model.
openrouter_modelCOMBOModel used when 'llm_provider' is 'openrouter'. Pick one that accepts images.
openai_modelCOMBOModel used when 'llm_provider' is 'openai'. Pick one that accepts images.
anthropic_modelCOMBOModel used when 'llm_provider' is 'anthropic'. Pick one that accepts images.
lm_urlSTRINGhttp://127.0.0.1:1234Address of the LM Studio server. Used only for 'lmstudio'.
lm_api_keySTRINGOnly if your LM Studio server asks for one.
openrouter_api_keySTRINGEmpty reads OPENROUTER_API_KEY from the environment.
openai_api_keySTRINGEmpty reads OPENAI_API_KEY from the environment.
anthropic_api_keySTRINGEmpty reads ANTHROPIC_API_KEY from the environment.
words_per_secondFLOAT8.02–25How many words of description one second of clip is worth. The model is given a word budget of duration x this, clamped to 25-220, which is what keeps a six-second shot from being described as a minute of events. Raise it for a model that writes thin, lower it for one that rambles.
rewrite_soundBOOLEANfalseOff: 'overall_soundscape' and 'non_diegetic_music' are carried over untouched, because a still frame says nothing about sound and the originals were written against the track. On: the model rewrites them too, which is worth it when the image turned out to be a different place than the prompt assumed.
image_detailINT768256–1536Longest side of the frame as it is sent, in pixels. Bigger sees more and costs more; on a cloud provider this is most of the bill.
lm_temperatureFLOAT0.400–2Lower than the writing stages on purpose: this is a correction, not an invention.
lm_max_tokensINT2048256–32768Cap on each reply. One shot's description, not the whole film.
lm_timeoutINT30010–3600Seconds to wait for one reply.
lm_retriesINT20–5Retries per shot. A shot that still fails keeps its original prompt.
verboseBOOLEANfalseLog each shot's before and after lengths.

Outputs (3)

NameTypeDescription
pipeM2P_PIPEThe same pipe with the rewritten 'video_prompts_i2va' in it. Everything else is untouched.
video_prompts_i2vaSTRINGThe rewritten prompts, in shot order, so they can be used without an expander.
reportSTRINGOne line per shot: what disagreed with the frame, or what went wrong.