comfyui-llm-tools

A ComfyUI custom node pack providing local LLM chat, text-to-speech (Kokoro TTS + VibeVoice TTS), and AI voice conversion (RVC).

By JasonHoku·Created 6 months ago·Updated 6 months ago· 3

JasonHoku/comfyui-llm-tools

Nodes—

On cloudLocal install

Stars3

Updated6 months ago

Readme

ComfyUI LLM Tools

A ComfyUI custom node pack providing local LLM chat, text-to-speech (Kokoro TTS + VibeVoice TTS), and AI voice conversion (RVC) — all running locally with no external API keys.

Features

Local LLM Chat — Load any GGUF model and chat with streaming responses via an interactive UI widget
Kokoro TTS — High-quality text-to-speech with 40+ voices across 8 languages (English, Japanese, Chinese, Spanish, French, Hindi, Italian, Portuguese)
VibeVoice TTS — Microsoft's LLM-based TTS (Qwen2.5 backbone + diffusion head); supports 1.5B standard model with voice cloning and 0.5B realtime streaming model
RVC Voice Conversion — Apply trained RVC v1/v2 voice models to TTS output for custom AI voices
Streaming Audio — VibeVoice audio starts playing within seconds via chunked PCM streaming + Web Audio API (no waiting for full generation)
Sentence-by-Sentence TTS — Audio plays as each sentence is generated, with prefetching for smooth playback (Kokoro), or streaming for instant start (VibeVoice)
Python 3.13 Compatible — Vendored RVC package requires no fairseq or librosa

Installation

Clone into your ComfyUI custom nodes directory:

cd ComfyUI/custom_nodes
git clone https://github.com/YOUR_USERNAME/comfyui-llm-tools.git

Install the base dependency:
```
pip install llama-cpp-python
```
(Optional) Install Kokoro TTS:
```
pip install kokoro-onnx onnxruntime
```
The model (~310 MB) and voice pack auto-download on first use.
(Optional) Install VibeVoice TTS:
```
pip install git+https://github.com/FushionHub/VibeVoice.git --no-deps
pip install soundfile diffusers
```
The VibeVoice model (~5.4 GB for 1.5B) auto-downloads from HuggingFace on first use and is cached in ComfyUI/models/TTS/VibeVoice/. A Qwen2.5 tokenizer (~100 MB) is also downloaded alongside.

Note: VibeVoice runs in a subprocess with isolated dependencies (transformers 4.51.3) to avoid conflicts with ComfyUI's transformers 5.x. The vv_deps/ directory is created automatically on first use (or by running install.py). See AINotes.md for architecture details.
(Optional) Install RVC voice conversion:
```
pip install faiss-cpu praat-parselmouth pyworld torchfcpe soundfile transformers safetensors requests
```
The HuBERT encoder (content-vec-best.safetensors) and pitch model (rmvpe.pt) auto-download on first use.

Setup

LLM Models

Place .gguf model files in ComfyUI/models/llm/ (the folder is created automatically).

RVC Voice Models

Place .pth RVC model files (and optional .index files) in ComfyUI/models/TTS/RVC/.

Both RVC v1 (256-dim) and v2 (768-dim) checkpoints are supported, with or without pitch (NSF).

VibeVoice Models

Models are downloaded automatically from HuggingFace and cached in ComfyUI/models/TTS/VibeVoice/. Available models:

| Model | Size | Description | |-------|------|-------------| | vibevoice-1.5B (default) | ~5.4 GB | Multi-speaker with voice cloning support | | vibevoice-realtime-0.5B | ~2.0 GB | Single-speaker, low-latency streaming (experimental) |

Select the model and configure parameters (cfg_scale, seed, inference_steps) via the VV settings button in the chat widget.

Nodes

| Node | Description | |------|-------------| | LLM GGUF | Single-shot text generation from a prompt | | LLM Chat Start | Begin an interactive conversation with a system prompt | | LLM Chat Continue | Continue a multi-turn conversation | | Kokoro TTS | Text-to-speech synthesis (standalone node) |

Interactive Chat Widget

The LLM Chat Start node includes a full interactive chat UI with:

Real-time token streaming via Server-Sent Events
TTS engine selector (Kokoro or VibeVoice)
TTS voice selection (Kokoro voices + RVC models in grouped dropdown)
VibeVoice settings modal (model, cfg_scale, seed, inference_steps)
Three TTS modes: Off, Sentence (plays as sentences complete), Full (plays after entire response)
Streaming playback for VibeVoice — audio starts within ~1-2s via Web Audio API instead of waiting for full generation
Stop generation, clear chat, and per-message speak buttons
Voice and mode settings persist across sessions via localStorage

API Endpoints

The extension adds several HTTP routes to ComfyUI's server:

| Endpoint | Method | Description | |----------|--------|-------------| | /llm_chat/models | GET | List available GGUF models | | /llm_chat/send | POST | Stream chat response (SSE) | | /llm_chat/stop | POST | Stop generation | | /llm_chat/unload | POST | Unload model, free VRAM | | /kokoro_tts/voices | GET | List TTS voices + RVC models | | /kokoro_tts/speak | POST | Generate speech (with optional RVC) | | /kokoro_tts/unload | POST | Unload TTS model | | /vibevoice_tts/voices | GET | List VibeVoice models + voices | | /vibevoice_tts/speak | POST | Generate speech via VibeVoice (full WAV) | | /vibevoice_tts/speak-stream | POST | Stream speech as chunked float32 PCM | | /vibevoice_tts/unload | POST | Unload VibeVoice model | | /rvc/models | GET | List RVC voice models |

Architecture

comfyui-llm-tools/
  __init__.py                # Node registration & web route setup
  llm_gguf_node.py           # Single-shot LLM node
  llm_chat_node.py           # Multi-turn chat nodes
  llm_chat_server.py         # SSE streaming chat API
  llm_utils.py               # GGUF parsing & VRAM estimation
  kokoro_tts_node.py         # Kokoro TTS ComfyUI node
  kokoro_tts_server.py       # TTS + RVC HTTP API
  vibevoice_tts_server.py    # VibeVoice subprocess manager + HTTP proxy
  vibevoice_subprocess.py    # VibeVoice worker (runs with transformers 4.51.3)
  rvc_server.py              # RVC model scanning & conversion API
  install.py                 # Auto-install script (vibevoice + vv_deps)
  vv_deps/                   # Isolated deps for subprocess (auto-created)
  web/js/
    llm_chat_widget.js       # Interactive chat UI widget
  rvc/                       # Vendored RVC inference (no fairseq/librosa)
    config.py                # Device & precision detection
    pipeline.py              # Main voice conversion pipeline
    pitch_extraction.py      # F0 pitch extraction (RMVPE, PM, etc.)
    infer_pack/              # VITS synthesizer, attention, WaveNet, etc.
    lib/rmvpe.py             # RMVPE pitch model (DeepUnet + BiGRU)

Requirements

Python 3.10+ (tested on 3.12 and 3.13)
PyTorch (comes with ComfyUI)
CUDA GPU recommended (automatically falls back to CPU if GPU is incompatible)
llama-cpp-python (required)
kokoro-onnx + onnxruntime (optional, for Kokoro TTS)
vibevoice + diffusers + soundfile (optional, for VibeVoice TTS)
RVC deps (optional, see installation above)

Credits

llama-cpp-python — GGUF model inference
Kokoro — TTS model by Hexgrad
VibeVoice — Microsoft's LLM-based real-time TTS
RVC-Project — Voice conversion architecture
TTS-Audio-Suite — Fairseq-free RVC reference + VibeVoice reference (MIT License)

License

MIT