Glossary · Technical
Tokenization
The process of breaking text into tokens before a model can process it, and reassembling tokens back into readable text for the output.
How it works
Before any text reaches the model, a tokenizer splits it into a sequence of tokens based on patterns learned from a large body of text — common words usually stay whole, while rare words, made-up terms, or non-English text often get broken into smaller pieces.
Why it matters for AI visibility
How a brand name tokenizes affects how reliably the model recognizes it as a single, coherent entity. A name that consistently breaks into the same familiar tokens tends to be handled more predictably than one that gets tokenized differently depending on capitalization, spacing, or surrounding punctuation.
See more terms in the full glossary, or read the AI Visibility 101 guide for the full picture.