Nodes/ComfyUI Ino Nodes/Ino S3 Download Folder
ComfyUI Node

Ino S3 Download Folder

Mirror a whole S3 folder locally — concurrent, verifiable, idempotent

By nobandegani·Created about a year ago·Updated 2 months ago· 6
Ino S3 Download Folder
    • success
    • message
    • rel_path
    • abs_path
    • total_files
    • downloaded_successfully
    • skipped_files
    • failed_downloads
    • errors
    enabledtrue
    s3_keyinput/example.png
    parent_folder
    folderinput/
    s3_config{"access_key_id": "", "access_key_secret": "", "bucket_name": "", "endpoint_url": "", "region_name": ""}
    max_concurrent8
    verifyfalse
    overwritefalse

    Some workflows don't need one file from S3 - they need the whole folder: a dataset, a model bundle, a run's output archive. InoS3DownloadFolder is the bulk version of the pack's S3 downloader, and it's built like a proper sync tool rather than a naive loop. It downloads an entire folder from a bucket with concurrent transfers (default 8, up to 16), skips files that already match locally unless you force overwrite, and can verify the result afterward. You get a full report on the other end: totals, successes, skips, failures, and the error strings for anything that didn't make it.

    That reporting is the difference between "it worked, I think" and "123 files synced, 2 skipped, 0 failed." If you're doing the cloud-GPU dance - upload a dataset to S3, spin up a rented box, pull the data down, push results back up - this node is the pull half of that loop.

    How it works

    Required inputs: enabled, s3_key (the folder prefix in the bucket - e.g. input/ or datasets/faces/), parent_folder, and folder (the local destination subfolder).

    Optional inputs worth understanding:

    • max_concurrent - parallelism for downloads, default 8.
    • verify - if True, checks the folder is fully synced after download. The tooltip notes it's skipped if any downloads failed, which is a sensible "don't certify a broken sync" guard.
    • overwrite - default False; when False, each file is skipped if the local copy already matches the S3 object. Flip to True to force everything down fresh.
    • s3_config - credentials JSON, or leave empty for the env vars (S3_ACCESS_KEY, S3_ACCESS_SECRET, S3_BUCKET_NAME, S3_ENDPOINT_URL, S3_REGION_NAME).

    Outputs: success, message, rel_path/abs_path, then the report: total_files, downloaded_successfully, skipped_files, failed_downloads, and errors (the failure strings, so you don't have to guess).

    Installing it

    Part of the 125+-node ComfyUI-InoNodes pack by nobandegani:

    cd ComfyUI/custom_nodes
    git clone https://github.com/nobandegani/comfyui_ino_nodes
    cd comfyui_ino_nodes
    pip install -r requirements.txt
    

    Or search "ComfyUI Ino Nodes" in ComfyUI Manager, install, restart. V3 schema - current ComfyUI required.

    Common issues

    Folder downloads are where people first hit the "is this a key or a folder?" confusion. In S3 there are no real folders - s3_key is a prefix, so input/ with a trailing slash grabs everything under that prefix while input (no slash) might match objects named input-something. Use the trailing slash for the "everything here" semantics you want. Also, the skip logic is per-file: a 500-file folder with 5 changed files only downloads those 5, which is great until you wanted a clean re-download - that's what overwrite: True is for. If failed_downloads is nonzero, read errors; transient network failures on one object don't poison the whole run, and a retry with verify on is a decent reconciliation step.

    CategoryInoS3Helper

    Inputs (8)

    NameTypeDefaultDescription
    enabledBOOLEANtrue
    s3_keySTRINGinput/example.png
    parent_folderCOMBO3 options: input, output, temp
    folderSTRINGinput/
    s3_configoptSTRING{"access_key_id": "", "access_key_secret": "", "bucket_name": "", "endpoint_url": "", "region_name": ""}you can leave it empty and pass it with env vars
    max_concurrentoptINT81–16
    verifyoptBOOLEANfalseIf True, verify folder sync after download (skipped if download had failures).
    overwriteoptBOOLEANfalseIf False (default), per file skip when the local file already matches S3.

    Outputs (9)

    NameTypeDescription
    successBOOLEAN
    messageSTRING
    rel_pathSTRING
    abs_pathSTRING
    total_filesINT
    downloaded_successfullyINT
    skipped_filesINT
    failed_downloadsINT
    errorsSTRING