Moondream 3 / 3.1 Loader (Isolated Photon)

The Moondream 3.x models are fast, tiny, and genuinely good at captioning, detection, and pointing - but they ship through Moondream's Photon runtime, which pins an old Pillow that would fight ComfyUI's newer one. Rather than break your install, this pack runs the whole thing in a dedicated sidecar environment. That's why the loader exists and why it's the first Moondream node you need: it starts the isolated Photon worker, downloads the official checkpoint, and hands you a reusable MOONDREAM31_MODEL handle that every other Moondream node consumes.

Think of it like a model cache with attitude. You load once, wire the model output into Caption / Query / Detect / Point / Segment nodes, and they all talk to the same running worker over IPC instead of each loading their own copy. That's a huge practical win for video work - you're not re-spinning a model per node.

Before it'll do anything, flip license_accepted to true - it's the required acknowledgement of the Moondream Model License 1.0. Not optional, and the node raises a clear error if you skip it. device (Auto / NVIDIA CUDA / Apple Silicon MPS) picks where the worker runs, max_batch_size caps concurrent Photon requests (4 is the recommended starting point for video detection), and kv_cache_profile trades VRAM for throughput - Low VRAM (4K pages), Balanced (8K), Maximum throughput (16K), or let Photon decide.

The optional model_or_adapter defaults to moondream3.1-9B-A2B, which is the one to use for query, caption, detection, and pointing. There's a second entry - moondream3-preview - that's for the SVG segment skill only; the final 3.1 model card doesn't ship segment, so don't expect that combo to work. Adapters can use the upstream base/adapter syntax.

Outputs are model (the MOONDREAM31_MODEL handle) and runtime_info - a JSON blob with worker details that's genuinely useful when you're debugging a "why is nothing happening" moment.

Installing the isolated runtime

This is the one node in the pack with a real setup step, and it's deliberate. Read and accept the license, then create the sidecar environment under the registered model folder:

runtime="ComfyUI/models/LLavacheckpoints/moondream31-runtime"
uv venv "$runtime/.venv" --python 3.12
uv pip install --python "$runtime/.venv/bin/python" \
  -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements-moondream31.txt

(Windows PowerShell uses $runtime\.venv\Scripts\python.exe and the .venv under the LLavacheckpoints folder.) The first loader execution downloads the model into that runtime's cache directory. If you use a different environment, set the server-side MOONDREAM_PYTHON variable - and never put the path or any credential in a workflow.

The honest hardware caveat

Photon's local inference supports NVIDIA Ampere-or-newer (Linux/Windows) and Apple Silicon (macOS 13+). There is no local ROCm, Intel GPU, or CPU path - on those platforms this node won't run, and the pack expects you to use its Transformers/GGUF/API nodes instead. On CUDA 12 x86-64 the requirements pin nvidia-cuda-runtime-cu12==12.9.79 inside Photon's private runtime (Kestrel's AOT kernels need an entry point the bundled 12.6 runtime lacks). That's a 5-line footnote in the README and a multi-hour mystery if you skip it - if the worker refuses to start on CUDA, it's the pin doing its job.

unload_after on the task nodes, not the loader, is how you release the worker when you're done; the loader itself caches the model handle for reuse.