Kosmos2 Loader2

The story half of this pack reads your keywords. The vision half reads your image - and KosmosLoader2 is how the weights get into memory for that job. It loads Microsoft's Kosmos-2 (patch14-224), a 2023 multimodal model that does something genuinely useful and still kind of rare: grounded captioning. It doesn't just describe an image, it points at the objects it's talking about - "a man holding a sword" becomes "a man, a sword" plus actual bounding-box coordinates. That grounded output is the whole reason this pack's "look at my image, then write a story about what's in it" loop works.

Under the hood it's stock transformers: AutoModelForVision2Seq.from_pretrained(...) for the model and AutoProcessor for its tokenizer/image processor, then moved to whatever device you pick. It caches the loaded model on the node instance, and only reloads if you change the model name or device - so re-running a workflow doesn't re-download or re-parse the weights.

Where it gets the model

One detail the README skips: the loader first checks a local folder, models/kosmos2/ inside your ComfyUI models directory. If the model name is already there as a subfolder, it uses that and never touches the network. Otherwise it falls back to downloading microsoft/kosmos-2-patch14-224 from HuggingFace - about 1.7 GB on first run. If you have a slow connection or want offline operation, pre-populating that folder with a saved copy of the model is the trick.

Inputs and outputs

• model (enum, one choice) - microsoft/kosmos-2-patch14-224. A single-entry dropdown, like the StoryLoader's. No model picker here.
• device (enum) - cpu or gpu. The GPU option only exists when ComfyUI can actually see CUDA; on a CPU-only build the dropdown is just cpu.

Outputs are model and processor (both CUSTOM), and both wire straight into Kosmos2SamplerSimple2. They're useless anywhere else, which is fine - this node's only job is feeding the sampler.

The gotcha: "cpu" is a trap

The device dropdown is more hopeful than honest. Even if you load the model on CPU, Kosmos2SamplerSimple2 hardcodes .to("cuda") on its inputs - the sampler side refuses to run without a GPU. So picking cpu here doesn't buy you a CPU-only pipeline; it just makes loading slower before you crash in the sampler. Set it to gpu. The example workflow ships with gpu selected for exactly this reason.

Installing it

Pack install, same as the rest:

cd ComfyUI/custom_nodes
git clone https://github.com/oztrkoguz/ComfyUI_StoryCreator

then restart, or use ComfyUI Manager and search "ComfyUI StoryCreator". The real dependency step lives outside the README: the pack ships no requirements.txt but imports transformers and cv2 at module load, so make sure both are in your ComfyUI Python env:

pip install transformers opencv-python

Common issues

• First run hangs on load - that's the 1.7 GB HuggingFace download. It's normal; watch the console for the "loading model" message.
• Nodes missing entirely - the cv2 import is the usual culprit. If opencv-python isn't installed, the whole pack fails to register, including the simple nodes.
• It's a 2023 model. Kosmos-2 is old by AI standards and its captions can miss modern concepts or modern slang. It's good enough to name the objects in your image - which is exactly the job - just don't expect a state-of-the-art captioner.