In plain words
A token is one item in a model’s input or output sequence. For text models, a tokenizer converts text into token IDs according to a vocabulary and set of rules.
A closer look
A token can correspond to a whole word, a word fragment, punctuation, whitespace, or another sequence of characters or bytes. Different tokenizers split the same text differently. The number of tokens therefore depends on the model, language, and content; there is no universal word-to-token conversion.
Token counts matter because context limits, output limits, and many usage charges are expressed in tokens. Models turn token IDs into learned numerical representations before processing them. Multimodal systems also represent non-text input in model-specific units that may enter token accounting in different ways.
In practice
An unfamiliar technical word might be split into several pieces while a common word is represented by one token. A short-looking string can therefore consume more of a context budget than expected.
A useful distinction
One token does not always equal one word or one character. Tokenizing text is also not the same as understanding its meaning; it is an encoding step that prepares the input for later computation.