Nodes/KJNodes for ComfyUI/Start Recording CUDAMemory History
ComfyUI Node Runs on cloud

Start Recording CUDAMemory History

A proper VRAM allocator trace, not just a printout

By kijai·Created 3 years ago·Updated a day ago· 3,011
Start Recording CUDAMemory History
  • input
  • input
enabledall
contextall
stacksall
max_entries100000

This one wraps PyTorch's own low-level memory-history API almost directly - the parameter names (enabled, context, stacks, max_entries) map straight onto torch.cuda.memory._record_memory_history()'s real arguments. What it starts isn't a friendly summary; it's a raw trace of every CUDA allocation event, optionally with the full call stack behind each one, that you can later dump and load into a proper memory profiler to see exactly what allocated what, when, and from where in the code. This pack also ships a matching EndRecordCUDAMemoryHistory node that stops the recording and saves the snapshot - the two are meant to be used as a pair, start here and end there.

This is a genuinely deep debugging tool, one rung below this pack's ModelMemoryUseReportPatch (which just prints running commentary during sampling). Reach for that one first if you just want a general sense of where memory's going; reach for this one when you need to pin down the exact allocation responsible for an OOM or a leak that a console printout couldn't explain.

The description is blunt about one thing: "THIS NODE ALWAYS RUNS." That's not filler text - it's flagging that this node is deliberately exempt from ComfyUI's normal execution caching. Ordinarily, if none of a node's inputs changed, ComfyUI can skip re-running it on a cached graph. That's exactly wrong for a side-effecting instrumentation node like this one - if it got skipped, you'd think you were recording and silently weren't. So it's marked to always execute, on every single run, whether or not it needs to.

The inputs and outputs that matter

It's a pass-through wrapped around a wildcard, so it can sit anywhere in a graph without breaking type flow:

  • input (any type) in, the same input out - wire it into the middle of whatever pipeline you want to profile.
  • enabled (all / state / None) - what gets recorded.
  • context (all / state / alloc / None) - how much context is captured per event.
  • stacks (python / all) - python captures Python-level call stacks only; all adds native/C++ stacks too, at higher cost.
  • max_entries (default 100,000) - the cap on recorded events before older ones roll off.

How to install it

Standard KJNodes install: ComfyUI Manager → search "KJNodes for ComfyUI" → install, or cd ComfyUI/custom_nodes && git clone https://github.com/kijai/ComfyUI-KJNodes, then pip install -r ComfyUI-KJNodes/requirements.txt, then restart. No extra dependency beyond PyTorch itself, which you already have.

Common issues & troubleshooting

It never stops recording on its own, and it keeps costing you. Because this node always runs, leaving it in a workflow you're not actively debugging means it's silently recording - and using its own extra memory to do so - on every single execution from now on. Pull it back out once you've captured what you needed.

stacks: all is noticeably more expensive than python. If you just need to know roughly where memory is going and don't need native call frames, use python - the difference in overhead is real, especially on a long recording.

Starting the recorder alone saves nothing to disk. You need the paired EndRecordCUDAMemoryHistory node downstream to actually stop and write out a snapshot - this node only opens the recording.

CategoryKJNodes/memory

Inputs (5)

NameTypeDefaultDescription
input*
enabledCOMBOallNone: disable, 'state': keep info for allocated memory, 'all': keep history of all alloc/free calls
contextCOMBOallNone: no tracebacks, 'state': tracebacks for allocated memory, 'alloc': for alloc calls, 'all': for free calls
stacksCOMBOall'python': Python/TorchScript/inductor frames, 'all': also C++ frames
max_entriesINT1000001000–10000000Maximum number of entries to record

Outputs (1)

NameTypeDescription
input*