JoyAI Runtime

This is a supervisor, not a loader

If you arrived from CheckpointLoaderSimple, drop the mental model. JoyAI Runtime has no checkpoint widget, no device picker, no VAE slot. It starts a persistent worker process inside a dedicated WSL2 Ubuntu distribution and asks it to load JD's JoyAI-Video-Edit once - the DiT checkpoint plus a MiMo-VL-7B text encoder - then keeps it warm between queue jobs.

The model lives outside ComfyUI: its own Python 3.10 env, torch 2.9.1+cu128 build and compiled kernels, talking to ComfyUI over wsl.exe pipes and raw frame files. This isn't "load a model on your GPU", it's "raise a service, then hand it work".

That's the right shape for the job. These editing models are heavy by construction - the source clip is part of the input sequence - and JoyAI streams it in 8-frame chunks inside one session, so a job is minutes of video rather than five seconds. A load per queue would make that unusable.

The pack is an unofficial integration and honest about being v0.1.0 pre-release. Exactly one hardware profile was actually run: Windows 11, 64 GB RAM, RTX 5090 32 GB. Everything outside that - Linux hosts, other GPUs, portrait video, other step counts - is explicitly untested. Read it as a careful wrapper around a narrow verified path.

How it works, and where your time goes

Under the hood is a state machine - STOPPED -> STARTING -> LOADING -> READY -> RUNNING -> READY -> STOPPING -> RELEASING -> STOPPED - plus a machine-wide lock file, so only one JoyAI model exists per WSL distro across ComfyUI processes. The lock is an OS-level handle with an owner record, which is why the error can name the PID hogging it.

The node fingerprints itself on MANAGER.generation and the current state, so it re-executes when the worker goes away and is otherwise cached: queue twice and the second run skips the load, Unload and the next queue brings it back. Cold kernel cache, first start took ~14 minutes on the test machine; warm, ~80 seconds.

The two inputs

runtime is a dropdown of runtime ids, and that's all it is. The ids come from joyai.runtimes.json in your ComfyUI user directory (or whatever COMFYUI_JOYAI_CONFIG points at). No config means a single (no runtime configured) entry and an error telling you where the file belongs. Distro, Python path and checkpoints live in that file - a nice security property, irritating when you are the administrator.

idle_unload_minutes (default 30, range 0–240) is the only knob that matters day to day. At 0 the worker stays until you Unload or ComfyUI exits. Given what it holds hostage - ~62 GiB of Windows commit charge at load, ~27.9 GB of a 32 GB card during a job - 30 is sane and I wouldn't raise it casually.

Outputs: runtime is a JOYAI_RUNTIME handle that only JoyAI Video Edit and JoyAI Unload accept. status is JSON with the worker state and a memory snapshot (commit GiB, limit, percent, available RAM) - Preview Any it, or ignore it.

Installing it

The node half is trivial; the runtime half is the project.

cd ComfyUI/custom_nodes
git clone https://github.com/hiroki-abe-58/ComfyUI-JoyAI-Video-Edit

Restart ComfyUI, or search "ComfyUI-JoyAI-Video-Edit" in ComfyUI Manager. There's no requirements.txt - the pyproject declares dependencies = [], and the pack installs and downloads nothing. It's a V3 node pack, so an old ComfyUI just logs [JoyAI-Video-Edit] not loaded: … and shows no nodes.

The real setup is inside a dedicated WSL2 distro: upstream deploy/requirements.txt, CUDA nvcc 12.8, joyomni_ops for sm_120a, SageAttention 2.2.0 with upstream's patch, no FlashAttention 4 in that env, ~50 GB of weights. Then register the runtime:

cp ComfyUI/custom_nodes/ComfyUI-JoyAI-Video-Edit/examples/joyai.runtimes.example.json \
   "<ComfyUI user directory>/joyai.runtimes.json"
python -I -B runtime/joyai_doctor.py <config> --gpu   # optional staged self-check, inside the distro

Where people get burned

The failures are unusually well-signposted, which is the thing I like most about this pack. The memory guard comes first: a load only starts if commit now + 65 GiB fits under 90% of the current commit limit, so a browser plus a game gets you a clear refusal rather than a half-start. The guard can be stricter in config, never looser.

Waiting for WSL memory release is not a bug. After a worker stops, the WSL2 VM keeps 25–30 GiB committed for a while; the loader waits - typically 30–80 seconds - and starts when memory is back. Past 120 seconds you get an error with instructions, not a mystery hang.

attention backend is 'sdpa', expected 'sage' means that env is missing SageAttention, or has FlashAttention 4 installed and silently switched backend. ... is in use by another ComfyUI process means a second ComfyUI holds the lock - Unload there, because this node can't free another process's worker.

Two things to internalize before committing an evening: one worker per distro, so no parallel editing graphs; and on Linux or a non-5090 card you're off the verified path. It may work. Nobody's claiming it does.