Y7 T5 Token Counter

T5 V1.1 XXL Token Counter<div style="font-size: 0.8em"><div id="Y7_shortdesc" style="margin-bottom: 15px;">Counts tokens in a text using the T5 XXL tokenizer.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Up to the first 256 or 512 tokens (default) will be displayed, with any overflow tokens shown below.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">The actual limit is 256 or 512 minus one special token reserved for the End-of-Sequence token <code style="border: 1px solid #666; border-radius: 3px; padding: 0px 1px; font-family: monospace; display: inline-block;">&lt;/s&gt;</code></div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Some models such as Flux.1 Schnell impose a 256-token sequence limit.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">The final token in the sequence will be shown, along with a brief context of words leading up to it.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Tokens prefixed with an underscore '_' represent a word boundary (New sentence or a space).</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Tokens without an underscore '_' are usually subword pieces that continue from the previous token.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Inputs:</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- text_in: Any text (string) input.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- show_tokens: Displays tokenized version of text (requires re-run).</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- tokens_per_line: Number of token words per line (requires re-run).</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Output:</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- text_out: A pass-through output for the input string</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Widgets:</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- font_size: Change font size used in the text widget. </div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 20px; font-size: 12px;">- Copy Text: Copy contents of the text widget.</div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;"></div></div><div style="font-size: 0.8em"><div style="margin-bottom: 8px; margin-left: 0px; font-size: 12px;">Note: Longer prompts are supported, but how they are handled depends entirely on the specific implementation of the model and tokenizer. Some implementations may truncate, segment, or otherwise process longer inputs differently.</div></div>