ComfyUI SmartLML

ComfyUI_SmartLML

Smart Language Model Loader for ComfyUI — two nodes that load and run vision-language models, text-only LLMs, WD14 taggers, and object detection across 8 backends. Pick a model from the unified registry dropdown, choose a task, and generate.

Backward Compatibility

v3.0.0 is a breaking release. All v2 nodes have been removed. Workflows using the old Smart Language Model Loader v2 [SmartLML], Smart Language Model Loader v2 [Eclipse], Smart Language Model Loader v3 [SmartLML], or Pipe Out LM Advanced Options nodes will need to be recreated with the new Smart Model Loader [SML] and Smart Detection [SML] nodes.

Nodes

• Smart Model Loader [SML] — VLM/LLM generation, WD14 tagging, multi-task chaining (up to 4 tasks)
• Smart Detection [SML] — Object detection with Florence-2, Qwen VL, and YOLO. Outputs bounding boxes, masks, and Impact Pack–compatible SEGS

What It Does

• Describe & analyze images — generate captions, detailed descriptions, cinematic scene breakdowns, or structured analyses from any image
• Process text — expand prompts, refine descriptions, rewrite style, convert between tags and natural language, translate, summarize
• Tag images — WD14 tagger with booru-style tag output (5 model variants)
• Chain tasks — run up to 4 tasks sequentially, each feeding its output to the next
• Detect & locate — object detection, phrase grounding, region captioning, instance segmentation (Florence-2, Qwen VL, YOLO)

Supported Models

| Family | Vision | Backends | Examples | |--------|:------:|----------|----------| | Qwen (Qwen2.5-VL, Qwen3-VL) | ✅ | Transformers, GGUF, vLLM, SGLang, Ollama | Qwen2.5-VL-3B/7B, Qwen3-VL-8B | | Mistral (Mistral3, Ministral3) | ✅ | Transformers, vLLM, SGLang, Ollama | Ministral-3-3B/8B-Instruct | | Florence-2 | ✅ | Transformers | base, large, base-ft, large-ft, PromptGen | | LLaVA | ✅ | Transformers, GGUF, Ollama | v1.6-vicuna-7b, mistral-7b | | LLM (text-only) | ❌ | GGUF, Ollama | Lexi-Llama-3-8B, Mistral-7B-Instruct | | YOLO | ✅ | Ultralytics (local) | face_yolov8m, hand_yolov8s, person_yolov8m-seg | | WD14 Tagger | ✅ | ONNX (local) | convnext-v3, eva02-large-v3, swinv2-v3 |

Backends

| Backend | Type | Notes | |---------|------|-------| | Transformers | Local | HuggingFace, full precision or FP8, Flash Attention 2 / SDPA | | GGUF | Local | llama-cpp-python, 10+ quantization levels (Q3–Q8, IQ3–IQ4) | | Ollama | Docker | Recommended for easy setup, auto-pulls models, NVIDIA + AMD/ROCm | | vLLM | Docker | Multi-GPU batched inference, NVIDIA + AMD/ROCm | | SGLang | Docker | Multi-GPU with RadixAttention, NVIDIA + AMD/ROCm | | llama.cpp | Docker | Lightweight GGUF inference, CPU/GPU hybrid | | YOLO | Local | Ultralytics object detection & segmentation, auto-discovers local .pt models | | WD14 | Local | ONNX-based image tagging (SmilingWolf models) |

Tasks

Vision — Simple / Detailed / Ultra Detailed / Cinematic Description, Image Analysis, Detailed Analysis, Tags, Video Summary, OCR Text — Expand Text, Refine & Expand Prompt, Rewrite Style, Tags ↔ Natural Language, Translate to English, Short Story, Summarize Custom — Direct Chat, Question Answering, Custom Instruction Florence-only — PromptGen Analyse / Mixed Caption / Mixed Caption Plus Detection (Smart Detection node) — Caption to Phrase Grounding, Region Caption, Dense Region Caption, Region Proposal, Referring Expression Segmentation, OCR With Region, DocVQA YOLO (Smart Detection node) — all-class detection with optional class filtering, instance segmentation (-seg models)

Key Features

• Registry-based model selection — unified dropdown with models grouped by backend, auto-downloads from HuggingFace on first use with integrity verification
• Multi-task chaining — chain up to 4 tasks sequentially with few-shot training applied per task
• Few-shot training — per-task example pairs (user-editable in config/) for consistent output style
• Editable system prompts — customize per-task instructions in config/system_prompts.json
• Mode bar — toggle chips for Multi-Task, Cleanup, Keep Loaded, Advanced
• Keep model loaded — persist model in VRAM across executions
• Docker lifecycle — auto-start/stop containers, stale image detection, cross-platform (Windows/Linux/macOS)
• Image passthrough — input images flow through to the output for downstream nodes
• Config protection — user settings preserved across updates via .defaults/ extraction

Documentation

• Smart Model Loader Guide — complete guide for the generation/tagging node
• Smart Detection Guide — complete guide for the detection node
• Docker Installation Guide (Windows/WSL2) — WSL2 + Docker + NVIDIA GPU setup
• Docker Installation Guide (Linux) — Docker Engine + NVIDIA Container Toolkit
• Model Repository Reference — HuggingFace URLs for all supported models
• Model Registry Reference (Copy-Paste) — JSON blocks for adding models to registry files

Helper Scripts (Linux)

Located in scripts/ — interactive shell scripts for Docker setup and management:

| Script | Description | |--------|-------------| | install-docker-engine.sh | Install Docker Engine + NVIDIA Container Toolkit + pass (multi-distro) | | remove-docker-nvidia.sh | Safely remove Docker and NVIDIA toolkit, with optional data purge | | manage-docker-images.sh | Pull, update, list, inspect, and clean SmartLML backend images |

Installation

Install via ComfyUI Manager or clone directly:

cd ComfyUI/custom_nodes
git clone https://github.com/r-vage/ComfyUI_SmartLML.git
cd ComfyUI_SmartLML
pip install -r requirements.txt

File Structure

| Path | Purpose | |------|---------| | config.json | Main config: LLM folder path, log level, HF token | | docker_config.json | Docker backend settings: ports, timeouts, images | | registry/ | Model registry JSON files (per-backend + user models + defaults) | | config/ | Few-shot training, system prompts | | .defaults/ | Git-tracked default files (.example suffix); extracted on first run, never overwrites |

License

Apache License 2.0 — see LICENSE for details.