IF VideoDatasetMkr 📚

The most underrated part of any LoRA or video-model fine-tune is the dataset, not the trainer. The KB's LoRA panel hammers this home: dataset curation beats every knob in the trainer, and captions are most of the battle. IF VideoDatasetMkr 📚 is the pack's answer to that chore - a single node that takes a YouTube URL or a local video file and hands you back a clean videos/ + captions/ training set plus a ZIP, ready to dump into ai-toolkit, Kohya, or whatever you actually train with. If you've been piecing together yt-dlp, ffmpeg, and a captioning script in bash, this collapses the whole pipeline into one workflow run.

It's from ImpactFrames (if-ai), the same folks behind the IF_AI LLM-nodes pack, and it's a port of zsxkib's cog-create-video-dataset project. Nothing calls an external API - the caption model runs locally through Hugging Face Transformers.

How it works

Feed it one source (a local video_file path takes precedence if you set both), and the node:

1. Downloads the video with yt-dlp (capped at 1080p) or copies your local file.
2. Runs PySceneDetect's ContentDetector to find real cuts.
3. Cuts each detected scene into 1–5 second clips with ffmpeg (libx264, CRF 23), skipping anything shorter or longer, and drops the last 20 seconds of the video on the assumption it's an outro.
4. Grabs the first frame of each clip, pads/resizes it to a square at your chosen image_size, and captions it with Qwen2.5-VL.
5. Writes captions as {trigger_word}_{number}.txt alongside the clips, names the dataset folder after the video title, and zips the whole thing.

One honest caveat, grounded in the source: each caption describes a single frame, not the motion between frames. The built-in profiles talk about "narrative continuity" and camera movement, but the model only ever sees one still per clip. For a video-generation dataset that's a real limitation worth knowing - you're captioning stills that happen to have been cut from motion.

The inputs that matter

• video_url / video_file - pick one; local file wins if both are set.
• trigger_word - your training trigger, baked into every caption filename. Leave it blank and it auto-generates one from the title.
• autocaption - on by default. Turn it off and give custom_caption a fixed string if you want one caption for every clip.
• model_variant - Qwen2.5-VL 3B/7B/72B, plain or AWQ. The default, 3B-Instruct-AWQ, is the sane choice: a fraction of the VRAM of 7B and the captions are plenty for dataset work.
• profile - the captioning persona pulled from profiles.json: VideoDatasetAnalyzer, HyVideoAnalyzer, or HyVideoDatasetMaker (the training-optimized one).
• model_offload - offloads Qwen to CPU between clips to free VRAM. Leave it on unless you're in a hurry.
• image_size - 448 to 1024; match your training resolution.
• debug_mode - saves ffmpeg commands, requests, and responses to a debug folder when things go sideways.

The single output, dataset_info, is a JSON string (dataset name, paths, clip count, model used). Wire it into any text display node - the example workflow uses the IF_AI pack's IF_LLM_DisplayText - so you can see where everything landed. Paths come back relative to ComfyUI's output folder.

Installing it

Easiest route is ComfyUI Manager - search "IF_DatasetMkr" and install. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/if-ai/ComfyUI-IF_DatasetMkr
pip install -r requirements.txt

Then restart ComfyUI. Two heavy prerequisites: FFmpeg must be installed and in your PATH (system-wide, not just the pip packages), and the first captioning run downloads the Qwen model from Hugging Face - a few GB for the 3B-AWQ default. The AWQ models also want triton and autoawq installed manually if you want the fast path; the README ships those commented out. On macOS skip AWQ entirely and use the plain variants.

Where people get burned

• Missing or stale FFmpeg - clips silently fail validation. ffmpeg -version first.
• YouTube downloads failing - update yt-dlp (pip install -U yt-dlp); the format string is picky about what's still hosted.
• VRAM or gated-model errors - some Qwen repos want an HF token; the hf_token input exists for exactly that.
• It's slow. Captioning every clip is sequential Qwen inference. A 20-clip dataset is minutes, not seconds.

The bigger trap is expectations: this makes an image-frame-plus-caption dataset for training video models. It's genuinely useful, just know what you're getting.