Skip to content

Token & Whitespace Inspector

Loading tokenizer… showing estimate
Characters
54
Tokens (est.)
~14
Whitespace
9
Hidden chars
2

Token heatmap (estimate — tokenizer loading)

Summarize·theNBreportZW·in·3·bullet·points. → Be·concise.

2 hidden / risky characters found

  • nbsppos 13U+00A0No-Break Space
  • zero-widthpos 20U+200BZero Width Space

Bidi controls and zero-width characters can hide instructions or spoof code (the "Trojan Source" class of attacks). Remove them unless you added them intentionally.

What This Tool Does

Token & Whitespace Inspector is built for deterministic developer and agent workflows.

Visualize tokenization boundaries and reveal hidden whitespace and unicode gotchas in a prompt.

Use How to Use for execution steps and FAQ for constraints, policies, and edge cases.

Last updated:

This tool is provided as-is for convenience. Output should be verified before use in any production or critical context.

Agent Invocation

Best Path For Builders

Browser workflow

Runs instantly in the browser with private local processing and copy/export-ready output.

Browser Workflow

This tool is optimized for instant in-browser execution with local data handling. Run it here and copy/export the output directly.

/token-whitespace-inspector/

For automation planning, fetch the canonical contract at /api/tool/token-whitespace-inspector.json.

How to Use Token & Whitespace Inspector

  1. 1

    Paste your prompt

    Drop the prompt or text you want to inspect into the input box. The heatmap and stats update as you type.

  2. 2

    Read the token heatmap

    Each token gets an alternating background so you can see exactly where the tokenizer splits your text, with a live token count.

  3. 3

    Reveal whitespace

    Toggle whitespace markers to turn spaces into middots and surface tabs and newlines, exposing trailing or doubled spacing.

  4. 4

    Reveal hidden characters

    Toggle hidden-character highlighting to catch zero-width, non-breaking, and bidi controls, each labeled with its codepoint and name.

  5. 5

    Copy the report

    Copy a plain-text summary of character, token, whitespace, and hidden-character counts plus the position of each flagged character.

Frequently Asked Questions

What does this tool show?
It renders your prompt as a token heatmap with alternating colors per token, counts characters and tokens, and flags hidden whitespace and risky unicode characters that can change how a model reads your text.
Which tokenizer does it use?
It uses the o200k byte-pair tokenizer (GPT-4o / GPT-5 family) loaded on demand in your browser. Until it loads, token counts fall back to a fast characters-divided-by-four estimate.
What hidden characters does it detect?
Zero-width characters (U+200B-200D, U+FEFF), non-breaking spaces, and bidirectional control characters (U+202A-202E, U+2066-2069) that enable Trojan Source style hidden-instruction attacks.
Why do tokens matter for prompts?
Token count drives cost and context limits, and odd whitespace can split words into extra tokens or shift meaning. Seeing the boundaries helps you trim and clean a prompt.
Is my prompt sent to a server?
No. Tokenization and character analysis run entirely in your browser. Your prompt is never uploaded or stored.