Nodes/RUI-Nodes/转化为utf-8编码 / Convert to UTF-8
ComfyUI Node

转化为utf-8编码 / Convert to UTF-8

Scrub the broken characters out of your text

By rui40000·Created 3 years ago·Updated 13 days ago· 17
转化为utf-8编码 / Convert to UTF-8
    • filtered_text
    • log
    input_text
    remove_emojitrue
    input_file

    Somewhere between copy-pasting text from random sources and shoving it through a pipeline, encoding can go sideways - lone surrogates, orphaned bytes, characters that are technically there but unreadable. UTF8 Converter is the cleanup node for that mess: it strips out everything that isn't valid UTF-8 and hands you back a clean string, plus a log of exactly what it removed.

    The realistic use case is pre-processing text before it hits something strict - an API call that rejects malformed JSON, a save step that chokes on weird bytes, a subtitle or TTS stage that garbles mojibake. If you've ever had an LLM call fail with a cryptic encoding error and spent an hour hunting a phantom quote character, this is the node that would have found it in one pass. It's the "sanitize before you send" step that nobody builds until they need it once, and then they need it constantly.

    How it works

    Feed it input_text (multi-line) and it filters out invalid sequences, returning filtered_text plus a log string that details what was removed and a summary - useful both for debugging and for verifying the cleanup did what you think it did.

    Two inputs control behavior:

    • remove_emoji - default true, and here's the trap: it force-deletes all 4-byte characters, which includes the vast majority of emoji. If your text contains emoji you want to keep, flip this off.
    • input_file - an optional file path. If you fill it in, the node reads and processes that file instead of the input_text box. Handy for cleaning up a text file on disk without pasting its entire contents into the UI. The node even detects binary files and handles them gracefully rather than producing garbage.

    Outputs: filtered_text (STRING) and log (STRING).

    Installing it

    Part of the RUI-Nodes pack ("Rui-Node🐶"). Install once via ComfyUI Manager (search "RUI-Nodes") or:

    cd ComfyUI/custom_nodes
    git clone https://github.com/rui40000/RUI-Nodes
    cd RUI-Nodes
    pip install -r requirements.txt
    

    Restart after. Pure Python - no extra dependencies.

    Where people get burned

    The emoji default is the one that catches people: remove_emoji being on by default means your "clean" output silently loses all emoji, which is great for API-bound text and awful if your text's meaning lived in those characters. Check the log - it tells you exactly what got dropped. Second, understand what it does not do: it doesn't re-encode mojibake back into proper characters, it deletes what's broken. Text that's been double-encoded will still look wrong after cleaning; the node just guarantees the result is valid UTF-8, not that it's the text you originally wanted. For most pipeline-cleaning jobs that's the right contract.

    CategoryRui-Node🐶/文本处理📝

    Inputs (3)

    NameTypeDefaultDescription
    input_textSTRING待转码的文本。用于把从各处复制来的乱码文本 (GBK/UTF-16 等被误读的内容)修正为正常 UTF-8。 若下方填了文件路径,则以文件内容为准,本框会被忽略。
    remove_emojiBOOLEANtrue是否强制删除所有 4 字节字符(包含绝大多数 Emoji)
    input_fileoptSTRING文本文件路径。填了它就优先读文件,忽略上面的输入框。 会依次尝试 UTF-8 → UTF-16(带BOM) → GB18030 解码, 都失败则忽略非法字节强行读出,尽量不让整个文件报废。

    Outputs (2)

    NameTypeDescription
    filtered_textSTRING
    logSTRING