AcademiaSD VLModel (Down)Loader

This is the first half of the pack's dataset-captioning pipeline: a loader that fetches a vision-language model from HuggingFace on first use and loads it into memory, ready for the captioner node. The "Down" in "(Down)Loader" is doing real work - if the model isn't local, it downloads it before loading, no separate trip to the browser required.

How it works

Type a HuggingFace repo ID into repo_id (the default is Qwen/Qwen2-VL-2B-Instruct - a small Qwen vision model), and on first run the node pulls the whole repository into ComfyUI/models/vision/<repo_name> using snapshot_download. It then loads it through the transformers AutoModelForVision2Seq class with fp16 and device_map="auto", which spreads it across whatever GPUs you have. The loaded model and processor are cached in memory, so subsequent runs skip the load entirely.

The low_vram toggle (default "enable") switches on 4-bit quantization via bitsandbytes (load_in_4bit), which is the difference between this running on an 8GB card and not running at all. Leave it enabled unless you've got headroom and want max caption quality.

The output is a single MODEL port of type ACADEMIA_MODEL - a custom type this pack defines. That's not a wire you can connect to a KSampler; it feeds the pack's own captioner node (AcademiaSD LLM Vision). The two nodes are a matched pair.

Install and the real gotcha

Pack install is the usual: ComfyUI Manager → "AcademiaSD", or git clone https://github.com/AcademiaSD/comfyui_AcademiaSD, restart. But here's the thing the README glosses over: the pack ships no requirements.txt, so nothing installs bitsandbytes, accelerate, or transformers for you. ComfyUI's bundled environment usually has transformers and huggingface_hub, but bitsandbytes and accelerate often don't, and the 4-bit path fails without them. If the node throws on load, that's your fix:

pip install bitsandbytes accelerate

The first run also downloads a couple of gigabytes (the 2B Qwen model, plus processor files), so budget a few minutes on a fresh setup. And note the model cache lives outside the usual ComfyUI model folders - if you ever clean house looking for space, it's under models/vision/.

Why you'd bother

If you're building an automated captioning pipeline for LoRA training, this is the zero-touch way to get a local VLM running - no manual HuggingFace downloads, no fiddling with model paths, and the 4-bit mode makes a 2B VLM practical on consumer cards. It's not the best captioner money can buy, but it's the most turnkey one in this pack.