ComfyUI-VideoToText

Video → audio → JSON transcript pipeline for ComfyUI. Pairs with ComfyUI-MiMoASR (Xiaomi MiMo-V2.5-ASR) for the ASR step.

This pack provides the non-ASR pieces of a video-to-text workflow: video loading, audio extraction (ffmpeg), AUDIO dict bridging, JSON packaging, and disk persistence. The ASR itself is delegated to MiMoASR, which gives you strong Mandarin / English code-switching, dialect support (Wu, Cantonese, Hokkien, Sichuanese, Henan, Northeastern Mandarin), and noisy-audio robustness.

| Node | Purpose | | --- | --- | | 🎙️ Load Video (ASR) | Pick a video from ComfyUI/input/ or pass an absolute path. | | 🎙️ Extract Audio (ffmpeg) | Demux audio track to a 16 kHz mono WAV. On-disk cache keyed by mtime + params. | | 🎙️ Audio From Path | Bridge: WAV file → ComfyUI AUDIO dict. Optional resample / mono. Also reports duration. | | 🎙️ Build Transcript JSON | Wrap ASR text + source metadata into a structured v2.0 JSON. | | 🎙️ Save Transcript JSON | Validate + write JSON to ComfyUI/output/ with timestamped filename. |

---

⚠️ Read this before installing

This pack does not ship an ASR model. Three things must be true before you can run a workflow:

1. ComfyUI-MiMoASR installed and working — including its transformers==4.49.0 downgrade and flash-attn build. Verify those nodes register before you even start with this pack.
2. ffmpeg on your server's PATH — $ which ffmpeg must return a path.
3. torchaudio in ComfyUI's venv — installed in Step 2 below.

Quick gut check:

| | Required | How to check | | --- | --- | --- | | ffmpeg | any recent | $ ffmpeg -version | | torchaudio | any | $ python -c "import torchaudio; print(torchaudio.__version__)" | | ComfyUI-MiMoASR | installed | $ curl -s http://SERVER:8188/object_info \| grep -o MiMoASRLoader | | PreviewAny (from rgthree-comfy) | installed | $ curl -s http://SERVER:8188/object_info \| grep -o PreviewAny |

---

Install (the proven order)

Each step has a verification at the end. Don't skip the verifications.

1. Install MiMoASR first

This pack depends on MiMoASRLoader and MiMoASRTranscribe from a separate repo. Follow its full install guide before continuing here. Stop and verify:

$ curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe'):
    print(f'  {n}: {\"OK\" if n in d else \"MISSING\"}')"

Both must print OK. If MISSING, fix MiMoASR before continuing.

2. Clone this node pack

$ cd ComfyUI/custom_nodes
$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git

Or, if installing from a tarball release:

$ cd ComfyUI/custom_nodes
$ tar xzf /path/to/ComfyUI-VideoToText-v0.2.0.tar.gz

Verify:

$ ls ComfyUI/custom_nodes/ComfyUI-VideoToText/__init__.py
# Must exist. If it's nested inside ComfyUI-VideoToText/ComfyUI-VideoToText/, you have a double-wrap — flatten it.

3. Install Python deps (in ComfyUI's venv)

$ cd ComfyUI && source .venv/bin/activate
$ uv pip install torchaudio

Verify:

$ python -c "import torchaudio; print('torchaudio', torchaudio.__version__)"
# Must print a version string. Errors here → re-run install.

4. Install ffmpeg on the server (if missing)

$ which ffmpeg || sudo apt install ffmpeg
$ ffmpeg -version

5. Restart ComfyUI

The same launch incantation MiMoASR needs — both HF_HOME and MIMO_ASR_REPO env vars must be set:

$ pkill -9 -f main.py; sleep 3
$ cd ComfyUI && source .venv/bin/activate
$ export HF_HOME=/big_disk/hf_cache
$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR
$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 &
$ sleep 30
$ grep -iE "IMPORT FAILED|VideoToText|MiMo" ~/comfy_start.log | head -20

6. Verify all 9 required nodes are registered

$ curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath',
          'VTT_BuildTranscriptJSON','VTT_SaveTranscript',
          'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'):
    print(f'  {n}: {\"OK\" if n in d else \"MISSING\"}')"

All 9 should show OK. If PreviewAny is missing, install rgthree-comfy.

---

Usage

Minimal workflow

VTT_LoadVideo  →  VTT_ExtractAudio  →  VTT_AudioFromPath  ─┐
                                                            ├──►  MiMoASRTranscribe  →  text
                                       MiMoASRLoader ───────┘            │
                                                                         ▼
                                                          VTT_BuildTranscriptJSON  →  STRING
                                                                         │
                                                                         ▼
                                                                VTT_SaveTranscript  →  ComfyUI/output/*.json

Required fields:

• MiMoASRLoader (absolute paths!):
  • model_path = /big_disk/models/MiMo-V2.5-ASR
  • tokenizer_path = /big_disk/models/MiMo-Audio-Tokenizer
  • repo_path = leave blank if MIMO_ASR_REPO env var is set
• MiMoASRTranscribe:
  • language = Auto (code-switched), Chinese, or English

The first run loads the 8B MiMo model into VRAM (30–60s). Subsequent runs in the same ComfyUI session are instant — MiMo's loader has a process-wide cache.

An API-format example workflow ships at example_workflows/video_to_json.json. Edit the three MiMo paths to match your server before loading it in the UI.

CLI

A single-file CLI uploads a local video, drives the workflow over ComfyUI's HTTP + WebSocket API, and dumps the JSON transcript locally:

$ uv pip install --system websocket-client requests

$ uv run python cli/video_to_text_cli.py /path/to/lecture.mp4 \
    --ip 10.8.0.12 \
    --model-path /big_disk/models/MiMo-V2.5-ASR \
    --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
    --repo-path /opt/repos/MiMo-V2.5-ASR \
    --language Auto

A wrapper alias to skip retyping paths every time:

# ~/.bashrc
alias v2t='uv run python /path/to/ComfyUI-VideoToText/cli/video_to_text_cli.py \
    --ip 10.8.0.12 \
    --model-path /big_disk/models/MiMo-V2.5-ASR \
    --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
    --repo-path /opt/repos/MiMo-V2.5-ASR'

# Then:
$ v2t my_video.mp4 --language Chinese
$ v2t lecture.mp4 --language Auto --out ./transcripts

See cli/README.md for all flags.

---

JSON schema v2.0

{
  "schema_version": "2.0",
  "source": {
    "audio_path": "/tmp/comfyui_vtt/lecture_abc123.wav",
    "duration_sec": 1834.512,
    "video_path": "/path/to/lecture.mp4"
  },
  "model": {
    "name": "MiMo-V2.5-ASR",
    "tokenizer": "MiMo-Audio-Tokenizer",
    "language": "Auto"
  },
  "text": "你好，今天我们聊一下 ESP32 嵌入式开发……"
}

Trade-off vs faster-whisper

MiMo-V2.5-ASR returns a single plain-text transcript — no word/segment timestamps, no confidence scores. Schema v2.0 reflects this: top-level text only, no segments[]. If you need time-aligned subtitles, run a different ASR backend (or wait for the planned v0.3.0 hybrid mode that pairs MiMo for text quality with faster-whisper for time alignment).

---

Tips

• Pure-language audio: setting language=Chinese or English is slightly more accurate than Auto. Use Auto only for code-switched content.
• GPU pinning: on dual-GPU rigs, $ export CUDA_VISIBLE_DEVICES=1 before launching ComfyUI keeps MiMo from fighting image models on cuda:0.
• Audio cache: VTT_ExtractAudio writes to $TMPDIR/comfyui_vtt/ keyed by source mtime + sample rate + mono flag. Re-runs on the same video skip ffmpeg entirely. Nuke with $ rm -rf /tmp/comfyui_vtt.
• Long audio: MiMo handles multi-hour lectures in a single call. No manual chunking needed at typical podcast / lecture lengths. For 6h+ recordings, consider splitting at silence with ffmpeg.

---

Troubleshooting

| Symptom | Likely cause | Fix | | --- | --- | --- | | VTT_* nodes show MISSING in /object_info | Pack not loaded; ComfyUI didn't restart; or IMPORT FAILED | Check ~/comfy_start.log for IMPORT FAILED ComfyUI-VideoToText traceback | | MiMoASRLoader: MISSING | MiMoASR not installed | See its README | | PreviewAny: MISSING | rgthree-comfy not installed | $ git clone https://github.com/rgthree/rgthree-comfy.git into custom_nodes/ | | RuntimeError: ffmpeg not found on PATH | ffmpeg missing on the server | $ sudo apt install ffmpeg | | RuntimeError: torchaudio is required | torchaudio missing in ComfyUI's venv | $ uv pip install torchaudio in the right venv | | RuntimeError: MiMo ASR repo path is empty | MIMO_ASR_REPO env var not set | Set it before launching ComfyUI, OR pass repo_path in the workflow | | MiMoASRLoader: "model path does not exist" | Relative path in the workflow | Always use absolute paths | | IMPORT FAILED ComfyUI-VideoToText with No module named 'folder_paths' | Pack at wrong location | Must be in ComfyUI/custom_nodes/, not elsewhere | | CUDA OOM on transcribe | GPU 0 held by image models | $ export CUDA_VISIBLE_DEVICES=1 before launching ComfyUI |

See INSTALL_NOTES.md for the full debug log of real-world install gotchas.

---

Compatibility

| | Required | Tested with | | --- | --- | --- | | Python | 3.10+ | 3.13 | | PyTorch | matches ComfyUI's | 2.11.0+cu126 | | torchaudio | any | matches torch | | transformers | 4.49.0 (MiMo pin) | 4.49.0 | | CUDA | ≥12.0 | 12.6 | | GPU VRAM | 24 GB (MiMo needs it) | 2× RTX 3090 | | OS | Linux | Ubuntu 22.04 |

⚠️ The transformers==4.49.0 pin (required by MiMo) will break Sonic, Qwen3TTS, and recent VibeVoice builds — they want >=4.57. If you need both, run MiMo + VideoToText on a dedicated ComfyUI instance.

---

Links

• 🤗 MiMo-V2.5-ASR weights: https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR
• 🤗 MiMo-Audio-Tokenizer: https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer
• MiMo upstream repo: https://github.com/XiaomiMiMo/MiMo-V2.5-ASR
• MiMo ComfyUI nodes: https://github.com/aadebuger/ComfyUI-MiMoASR
• Blog: https://mimo.xiaomi.com/mimo-v2-5-asr

---

License

MIT — see LICENSE. MiMoASR upstream is Apache-2.0; MiMo-V2.5-ASR weights are MIT (Xiaomi).